Advanced Redis Operations Cheatsheet – Technical Deep Dive
Comprehensive technical reference for enterprise Redis operations, performance optimization, and production management.
Version Compatibility Note: This guide covers Redis 6.0+ features unless otherwise specified. For Redis 5.x compatibility, see legacy sections marked with 🔄.
1. Redis Installation & Advanced Configuration
Multi-Platform Installation with Build Optimization
# Ubuntu/Debian with performance optimizations
sudo apt update && sudo apt install redis-server redis-tools
sudo sysctl vm.overcommit_memory=1
echo 'vm.overcommit_memory = 1' | sudo tee -a /etc/sysctl.conf
echo 'net.core.somaxconn = 65535' | sudo tee -a /etc/sysctl.conf
# CentOS/RHEL with THP (Transparent Huge Pages) disabled
sudo yum install epel-release && sudo yum install redis
echo 'never' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
# High-performance Docker deployment with memory optimization
docker run -d --name redis-prod \
--sysctl net.core.somaxconn=65535 \
--ulimit memlock=-1:-1 \
--memory=4g --memory-swap=4g \
-p 6379:6379 \
-v redis-data:/data \
redis:7-alpine redis-server \
--maxmemory 3.5gb \
--maxmemory-policy allkeys-lru \
--tcp-backlog 511 \
--timeout 0 \
--tcp-keepalive 300
# Compile from source with optimizations
wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz && cd redis-stable
make MALLOC=jemalloc CFLAGS="-O3 -march=native"
sudo make install PREFIX=/opt/redis
# Error handling for installation
if ! redis-cli ping >/dev/null 2>&1; then
echo "ERROR: Redis installation failed or service not running"
systemctl status redis-server
exit 1
fi
Production-Grade Configuration Parameters
# Network and connection management bind 0.0.0.0 port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 protected-mode yes maxclients 10000 # Memory management and optimization maxmemory 4gb maxmemory-policy allkeys-lru maxmemory-samples 5 # Redis 4.0+ active defragmentation active-defrag-ignore-bytes 100mb active-defrag-threshold-lower 10 active-defrag-threshold-upper 100 active-defrag-cycle-min 1 active-defrag-cycle-max 25 # Persistence optimization save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /var/lib/redis/ # AOF configuration for durability appendonly yes appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes # Redis 4.0+ hybrid persistence aof-use-rdb-preamble yes # Logging and debugging loglevel notice logfile /var/log/redis/redis-server.log syslog-enabled yes syslog-ident redis # Advanced tuning (version-specific) hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 # Redis 5.0+ stream configuration stream-node-max-bytes 4096 stream-node-max-entries 100
2. Advanced Data Structures & Complex Operations
String Operations with Advanced Patterns
# Atomic operations with conditional execution
SET user:session:12345 "active" EX 3600 NX
SETEX cache:user:profile:123 900 '{"name":"John","role":"admin"}'
GETSET counter:requests:api 0 # Atomic read-and-reset
# Bit operations for efficient storage (Redis 2.2+)
SETBIT user:active:20240101 12345 1 # Mark user 12345 as active
BITCOUNT user:active:20240101 # Count active users
BITOP AND result user:premium user:active # Premium AND active users
# Redis 3.2+ BITFIELD for packed integers
BITFIELD mykey SET u8 0 255 INCRBY u8 8 1 # Packed integer operations
# String manipulation with memory efficiency
APPEND log:session:123 "2024-06-16T10:30:00Z - User login\n"
STRLEN log:session:123
GETRANGE log:session:123 0 19 # Get timestamp portion
# Error handling example
if ! redis-cli EXISTS user:session:12345 >/dev/null 2>&1; then
echo "Session not found, creating new session"
fi
Hash Operations with Memory Optimization
# Field-level operations with expiration simulation HSET user:profile:123 name "John Doe" email "john@example.com" last_seen 1718541000 HMSET user:settings:123 theme "dark" notifications "true" timezone "UTC" HGETALL user:profile:123 HKEYS user:profile:123 HVALS user:profile:123 HLEN user:profile:123 # Memory-efficient field management HSETNX user:profile:123 created_at 1718541000 # Set only if field doesn't exist HINCRBY user:stats:123 login_count 1 HINCRBYFLOAT user:metrics:123 avg_session_time 0.5 # 🔄 Redis 5.x: Use HMSET instead of HSET for multiple fields # Redis 6.0+: HSET supports multiple field-value pairs # Scanning large hashes efficiently (Redis 2.8+) HSCAN user:large_profile:123 0 MATCH "*_count" COUNT 100
List Operations for Queue Management
# High-performance queue operations
LPUSH queue:tasks '{"id":1,"type":"email","priority":1}'
RPOPLPUSH queue:tasks queue:processing # Atomic move to processing
LINSERT queue:tasks BEFORE '{"id":2}' '{"id":1.5,"urgent":true}'
# Reliable queue with blocking operations (Redis 2.0+)
BLPOP queue:high_priority queue:normal 30 # Block for 30 seconds
BRPOPLPUSH queue:tasks queue:processing 60 # Reliable queue pattern
# List maintenance and monitoring
LLEN queue:tasks
LRANGE queue:tasks 0 9 # Preview first 10 tasks
LTRIM queue:completed -100 -1 # Keep only last 100 completed tasks
# Error handling for queue operations
#!/bin/bash
queue_push_safe() {
local queue=$1
local item=$2
if redis-cli LPUSH "$queue" "$item" >/dev/null 2>&1; then
echo "Item added to queue successfully"
else
echo "ERROR: Failed to add item to queue $queue"
return 1
fi
}
Set Operations for Complex Logic
# Advanced set operations for analytics SADD users:premium 123 456 789 SADD users:active:today 123 234 345 456 SINTER users:premium users:active:today # Premium users active today SINTERSTORE temp:premium_active users:premium users:active:today # Probabilistic data structures SADD unique:visitors:today 192.168.1.1 10.0.0.5 SCARD unique:visitors:today # Approximate unique visitor count SRANDMEMBER unique:visitors:today 5 # Sample 5 random visitors # Set comparison and analysis SDIFF users:all users:active:today # Inactive users SUNIONSTORE users:engagement users:premium users:active:today users:recent_purchasers
Sorted Sets for Advanced Ranking
# Time-series and leaderboard operations ZADD leaderboard:game:123 1000 "player1" 950 "player2" 1200 "player3" ZREVRANGE leaderboard:game:123 0 9 WITHSCORES # Top 10 with scores ZRANGEBYSCORE leaderboard:game:123 900 1100 WITHSCORES LIMIT 0 5 # Time-based data with millisecond precision ZADD events:user:123 1718541000000 "login" 1718541300000 "page_view" ZREMRANGEBYSCORE events:user:123 0 $(( $(date +%s) * 1000 - 86400000 )) # Remove events older than 24h # Advanced sorted set operations (Redis 2.8+) ZINTERSTORE temp:common_interests 2 interests:user:123 interests:user:456 WEIGHTS 1 1 ZLEXCOUNT dictionary:words [a [b # Count words starting with 'a' ZRANGEBYLEX dictionary:words [a [b LIMIT 0 10 # Redis 6.2+ range operations # ZRANGE key start stop [BYSCORE | BYLEX] [REV] [LIMIT offset count] [WITHSCORES]
3. Performance Monitoring & Advanced Metrics
⚠️ Performance Impact Warning: Monitoring commands like MONITOR and frequent INFO calls can impact Redis performance. Use judiciously in production.
Comprehensive Memory Analysis
# Detailed memory breakdown
INFO memory
MEMORY USAGE keyname SAMPLES 5000 # Accurate memory usage with sampling (Redis 4.0+)
MEMORY DOCTOR # Memory optimization recommendations (Redis 4.0+)
MEMORY PURGE # Force memory defragmentation (Redis 4.0+)
# Advanced memory monitoring with error handling
#!/bin/bash
analyze_memory_safely() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
if ! redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis at $redis_host:$redis_port"
return 1
fi
echo "Analyzing memory usage (this may take time)..."
redis-cli -h "$redis_host" -p "$redis_port" --bigkeys --bigkeys-samples 1000000 2>/dev/null
redis-cli -h "$redis_host" -p "$redis_port" --memkeys --memkeys-samples 1000000 2>/dev/null
}
# Memory configuration analysis
CONFIG GET "*memory*"
CONFIG GET "*maxmemory*"
CONFIG SET maxmemory-samples 10 # Improve LRU precision
Advanced Performance Metrics
# Comprehensive stats analysis
INFO all | grep -E "(instantaneous_ops_per_sec|keyspace_hits|keyspace_misses|expired_keys|evicted_keys)"
INFO commandstats | sort -k2 -nr # Most used commands
INFO clients | grep -E "(connected_clients|client_recent_max_input_buffer|client_recent_max_output_buffer)"
# Latency monitoring and analysis (Redis 2.8.13+)
CONFIG SET latency-monitor-threshold 50 # Monitor operations > 50ms
LATENCY LATEST
LATENCY HISTORY command-name
LATENCY DOCTOR # Latency analysis and recommendations
redis-cli --latency-history -i 1 # Real-time latency monitoring
# Slow query analysis
CONFIG SET slowlog-log-slower-than 10000 # Log queries > 10ms
CONFIG SET slowlog-max-len 1000
SLOWLOG GET 100
SLOWLOG LEN
SLOWLOG RESET
# Performance monitoring script with error handling
#!/bin/bash
monitor_redis_performance() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local interval=${3:-5}
while true; do
if redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
local ops=$(redis-cli -h "$redis_host" -p "$redis_port" INFO stats | grep instantaneous_ops_per_sec | cut -d: -f2 | tr -d '\r')
local memory=$(redis-cli -h "$redis_host" -p "$redis_port" INFO memory | grep used_memory_human | cut -d: -f2 | tr -d '\r')
echo "$(date): OPS/sec: $ops, Memory: $memory"
else
echo "$(date): ERROR - Redis connection failed"
fi
sleep "$interval"
done
}
Network and Connection Monitoring
# Client connection analysis
CLIENT LIST TYPE normal
CLIENT LIST TYPE replica
# Redis 6.0+ client tracking
CLIENT TRACKING ON # Enable client-side caching tracking
CLIENT GETREDIR # Get redirection client for tracking
# Connection pool monitoring
INFO stats | grep -E "(total_connections_received|rejected_connections)"
CONFIG GET maxclients
CLIENT SETNAME "webapp-pool-1" # Name connections for monitoring
# Connection health check with error handling
#!/bin/bash
check_connection_health() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local max_clients_threshold=${3:-8000}
local connected_clients
connected_clients=$(redis-cli -h "$redis_host" -p "$redis_port" INFO clients 2>/dev/null | grep connected_clients | cut -d: -f2 | tr -d '\r')
if [ $? -ne 0 ]; then
echo "ERROR: Failed to get client connection info"
return 1
fi
if [ "$connected_clients" -gt "$max_clients_threshold" ]; then
echo "WARNING: High client connections: $connected_clients"
return 1
fi
echo "OK: Client connections: $connected_clients"
return 0
}
4. Enterprise Security & Advanced Authentication
Advanced ACL Management (Redis 6.0+)
# Granular user permissions with error handling
create_api_user() {
if redis-cli ACL SETUSER api_user on >complex_password_123 ~api:* ~cache:* +@read +@hash +@string -@dangerous 2>/dev/null; then
echo "API user created successfully"
else
echo "ERROR: Failed to create API user"
return 1
fi
}
ACL SETUSER analytics_user on >analytics_pass ~analytics:* ~metrics:* +@read +@sortedset +@bitmap
ACL SETUSER backup_user on >backup_secure_pass ~* +@read +@admin +bgsave +lastsave
# Role-based access patterns
ACL SETUSER readonly_role on >readonly_pass ~* +@read -@write -@admin
ACL SETUSER write_cache_role on >cache_pass ~cache:* ~session:* +@all -@admin -@dangerous
ACL SETUSER admin_role on >admin_super_secure_pass ~* +@all
# ACL management with validation
#!/bin/bash
validate_acl_setup() {
if ! redis-cli ACL LIST >/dev/null 2>&1; then
echo "ERROR: ACL not supported (requires Redis 6.0+)"
return 1
fi
echo "Current ACL users:"
redis-cli ACL LIST
# Test user authentication
if redis-cli -u redis://api_user:complex_password_123@localhost:6379 ping >/dev/null 2>&1; then
echo "✓ API user authentication successful"
else
echo "✗ API user authentication failed"
fi
}
# ACL categories and command filtering
ACL CAT # List all command categories
ACL CAT dangerous # Show dangerous commands
ACL DELUSER temp_user
ACL SAVE # Persist ACL configuration
ACL LOAD # Reload ACL from file
Network Security Hardening
# Multi-interface binding with security
bind 127.0.0.1 10.0.0.100 192.168.1.100
protected-mode yes
port 6379
timeout 300
# TLS encryption configuration (Redis 6.0+)
port 0 # Disable non-TLS port
tls-port 6380
tls-cert-file /etc/redis/tls/redis.crt
tls-key-file /etc/redis/tls/redis.key
tls-ca-cert-file /etc/redis/tls/ca.crt
tls-protocols "TLSv1.2 TLSv1.3"
tls-ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"
# Command security with secure random names
generate_secure_command_names() {
local config_file="/etc/redis/redis.conf"
# Generate secure random suffixes
local flush_suffix=$(openssl rand -hex 16)
local config_suffix=$(openssl rand -hex 16)
cat >> "$config_file" << EOF
rename-command FLUSHDB "FLUSHDB_DANGER_${flush_suffix}"
rename-command FLUSHALL "FLUSHALL_DANGER_${flush_suffix}"
rename-command CONFIG "CONFIG_ADMIN_${config_suffix}"
rename-command EVAL "EVAL_SCRIPT_${config_suffix}"
rename-command DEBUG "" # Completely disable
EOF
echo "Secure command names generated and saved to $config_file"
}
# 🔄 Legacy Redis 5.x: TLS requires compilation with TLS support
5. Advanced Persistence & Backup Strategies
RDB Optimization and Management
# Advanced RDB configuration
save 900 1 # Save if at least 1 key changed in 900 seconds
save 300 10 # Save if at least 10 keys changed in 300 seconds
save 60 10000 # Save if at least 10000 keys changed in 60 seconds
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
# Redis 6.2+ security feature
sanitize-dump-payload yes
# RDB backup automation with comprehensive error handling
#!/bin/bash
backup_redis_rdb() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local backup_dir="/backup/redis/$(date +%Y-%m-%d)"
local max_wait_time=300 # 5 minutes
local wait_count=0
# Validate Redis connection
if ! redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis at $redis_host:$redis_port"
return 1
fi
# Create backup directory
if ! mkdir -p "$backup_dir"; then
echo "ERROR: Cannot create backup directory $backup_dir"
return 1
fi
# Get last save time before backup
local last_save_before
last_save_before=$(redis-cli -h "$redis_host" -p "$redis_port" LASTSAVE 2>/dev/null)
if [ $? -ne 0 ]; then
echo "ERROR: Failed to get LASTSAVE timestamp"
return 1
fi
# Initiate background save
echo "Initiating RDB backup..."
if ! redis-cli -h "$redis_host" -p "$redis_port" BGSAVE >/dev/null 2>&1; then
echo "ERROR: BGSAVE command failed"
return 1
fi
# Wait for backup to complete
while [ $wait_count -lt $max_wait_time ]; do
local current_save
current_save=$(redis-cli -h "$redis_host" -p "$redis_port" LASTSAVE 2>/dev/null)
if [ "$current_save" -gt "$last_save_before" ]; then
echo "Backup completed successfully"
break
fi
sleep 1
((wait_count++))
done
if [ $wait_count -eq $max_wait_time ]; then
echo "ERROR: Backup timeout after ${max_wait_time} seconds"
return 1
fi
# Copy RDB file
local rdb_file="/var/lib/redis/dump.rdb"
local backup_file="$backup_dir/dump-$(date +%H%M%S).rdb"
if cp "$rdb_file" "$backup_file"; then
echo "RDB file copied to $backup_file"
# Verify backup integrity
if redis-check-rdb "$backup_file" >/dev/null 2>&1; then
echo "✓ Backup integrity verified"
return 0
else
echo "✗ Backup integrity check failed"
return 1
fi
else
echo "ERROR: Failed to copy RDB file"
return 1
fi
}
# RDB integrity verification
redis-check-rdb /backup/redis/dump.rdb
AOF Advanced Configuration and Management
# High-performance AOF setup
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec # Balance between performance and durability
no-appendfsync-on-rewrite yes # Prevent fsync during rewrite
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
# Redis 4.0+ hybrid persistence for faster restarts
aof-use-rdb-preamble yes
# AOF maintenance with error handling
#!/bin/bash
maintain_aof() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local aof_file="/var/lib/redis/appendonly.aof"
# Check AOF integrity
echo "Checking AOF integrity..."
if redis-check-aof "$aof_file" >/dev/null 2>&1; then
echo "✓ AOF file is intact"
else
echo "✗ AOF file corruption detected"
echo "Attempting automatic repair..."
# Backup corrupted file
cp "$aof_file" "${aof_file}.corrupted.$(date +%s)"
# Attempt repair
if redis-check-aof --fix "$aof_file"; then
echo "✓ AOF file repaired successfully"
else
echo "✗ AOF repair failed - manual intervention required"
return 1
fi
fi
# Trigger AOF rewrite if connected
if redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "Triggering AOF rewrite..."
redis-cli -h "$redis_host" -p "$redis_port" BGREWRITEAOF
fi
}
# Multi-part AOF (Redis 7.0+)
aof-rewrite-incremental-fsync yes
aof-timestamp-enabled yes
6. Redis Clustering & High Availability
Advanced Cluster Configuration
# Cluster creation with comprehensive error handling
#!/bin/bash
create_redis_cluster() {
local nodes=("$@")
local min_nodes=6
if [ ${#nodes[@]} -lt $min_nodes ]; then
echo "ERROR: Minimum $min_nodes nodes required for cluster"
return 1
fi
# Validate all nodes are accessible
echo "Validating cluster nodes..."
for node in "${nodes[@]}"; do
if ! redis-cli -h "${node%:*}" -p "${node#*:}" ping >/dev/null 2>&1; then
echo "ERROR: Node $node is not accessible"
return 1
fi
done
# Create cluster
echo "Creating Redis cluster with nodes: ${nodes[*]}"
if redis-cli --cluster create "${nodes[@]}" --cluster-replicas 1 --cluster-yes; then
echo "✓ Cluster created successfully"
# Verify cluster status
redis-cli --cluster check "${nodes[0]}"
return 0
else
echo "✗ Cluster creation failed"
return 1
fi
}
# Cluster slot management with error handling
manage_cluster_slots() {
local node=${1:-localhost:6379}
echo "Cluster slot distribution:"
if redis-cli -c -h "${node%:*}" -p "${node#*:}" CLUSTER NODES 2>/dev/null; then
echo "Cluster status retrieved successfully"
else
echo "ERROR: Failed to get cluster status"
return 1
fi
# Additional cluster info
redis-cli -c -h "${node%:*}" -p "${node#*:}" CLUSTER SLOTS 2>/dev/null
redis-cli -c -h "${node%:*}" -p "${node#*:}" CLUSTER INFO 2>/dev/null
}
# Dynamic cluster management
redis-cli --cluster add-node new-node:6379 existing-node:6379
redis-cli --cluster del-node existing-node:6379 node-id
redis-cli --cluster reshard existing-node:6379 --cluster-from source-node-id --cluster-to target-node-id --cluster-slots 1000
redis-cli --cluster rebalance existing-node:6379 --cluster-weight node1=2 node2=1
Sentinel High Availability
# Sentinel configuration with error handling
port 26379
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel auth-pass mymaster your_redis_password
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000
sentinel deny-scripts-reconfig yes
# Sentinel management with comprehensive monitoring
#!/bin/bash
monitor_sentinel() {
local sentinel_host=${1:-localhost}
local sentinel_port=${2:-26379}
local master_name=${3:-mymaster}
if ! redis-cli -h "$sentinel_host" -p "$sentinel_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Sentinel at $sentinel_host:$sentinel_port"
return 1
fi
echo "=== Sentinel Status ==="
redis-cli -h "$sentinel_host" -p "$sentinel_port" SENTINEL masters
redis-cli -h "$sentinel_host" -p "$sentinel_port" SENTINEL slaves "$master_name"
redis-cli -h "$sentinel_host" -p "$sentinel_port" SENTINEL sentinels "$master_name"
# Check master address
local master_addr
master_addr=$(redis-cli -h "$sentinel_host" -p "$sentinel_port" SENTINEL get-master-addr-by-name "$master_name" 2>/dev/null)
if [ $? -eq 0 ]; then
echo "Current master: $master_addr"
else
echo "ERROR: Failed to get master address for $master_name"
return 1
fi
}
# Manual failover with validation
perform_failover() {
local sentinel_host=${1:-localhost}
local sentinel_port=${2:-26379}
local master_name=${3:-mymaster}
echo "Initiating manual failover for $master_name..."
if redis-cli -h "$sentinel_host" -p "$sentinel_port" SENTINEL failover "$master_name" 2>/dev/null; then
echo "✓ Failover initiated successfully"
# Wait and verify
sleep 5
monitor_sentinel "$sentinel_host" "$sentinel_port" "$master_name"
else
echo "✗ Failover failed"
return 1
fi
}
Advanced Replication Configuration
# Master configuration for optimal replication
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync yes # For fast networks
repl-diskless-sync-delay 5
# Redis 6.0+ feature
repl-diskless-load swapdb
repl-ping-replica-period 10
repl-timeout 60
repl-disable-tcp-nodelay no
repl-backlog-size 1mb
repl-backlog-ttl 3600
# Replication monitoring with comprehensive error handling
#!/bin/bash
monitor_replication() {
local master_host=${1:-localhost}
local master_port=${2:-6379}
local replica_host=${3:-localhost}
local replica_port=${4:-6380}
echo "=== Master Status ==="
if redis-cli -h "$master_host" -p "$master_port" INFO replication 2>/dev/null; then
echo "✓ Master replication info retrieved"
else
echo "✗ Failed to connect to master"
return 1
fi
echo "=== Replica Status ==="
if redis-cli -h "$replica_host" -p "$replica_port" INFO replication 2>/dev/null; then
echo "✓ Replica replication info retrieved"
# Check replication lag
local lag
lag=$(redis-cli -h "$replica_host" -p "$replica_port" INFO replication | grep master_repl_offset | cut -d: -f2 | tr -d '\r')
if [ "$lag" -gt 1000 ]; then
echo "⚠️ WARNING: High replication lag detected: $lag"
fi
else
echo "✗ Failed to connect to replica"
return 1
fi
}
# Setup replication with validation
setup_replication() {
local master_host=$1
local master_port=$2
local replica_host=$3
local replica_port=$4
echo "Setting up replication: $replica_host:$replica_port -> $master_host:$master_port"
if redis-cli -h "$replica_host" -p "$replica_port" REPLICAOF "$master_host" "$master_port" 2>/dev/null; then
echo "✓ Replication setup initiated"
# Wait for initial sync
local sync_complete=false
local wait_count=0
local max_wait=60
while [ $wait_count -lt $max_wait ] && [ "$sync_complete" = false ]; do
local sync_status
sync_status=$(redis-cli -h "$replica_host" -p "$replica_port" INFO replication | grep master_sync_in_progress:0)
if [ -n "$sync_status" ]; then
sync_complete=true
echo "✓ Initial synchronization completed"
else
sleep 1
((wait_count++))
fi
done
if [ "$sync_complete" = false ]; then
echo "⚠️ WARNING: Initial sync taking longer than expected"
fi
else
echo "✗ Replication setup failed"
return 1
fi
}
7. Advanced Troubleshooting & Diagnostics
Connection and Network Diagnostics
# Comprehensive connection testing with detailed error reporting
#!/bin/bash
test_redis_connectivity() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local timeout=${3:-5}
echo "Testing Redis connectivity to $redis_host:$redis_port..."
# Basic network connectivity
if timeout "$timeout" bash -c "</dev/tcp/$redis_host/$redis_port" 2>/dev/null; then
echo "✓ Network connection successful"
else
echo "✗ Network connection failed"
echo "Checking if Redis service is running..."
ss -tlnp | grep ":$redis_port " || echo "Port $redis_port not listening"
return 1
fi
# Redis protocol test
if redis-cli -h "$redis_host" -p "$redis_port" --connect-timeout "$timeout" ping >/dev/null 2>&1; then
echo "✓ Redis protocol response successful"
else
echo "✗ Redis protocol response failed"
# Check authentication requirement
if redis-cli -h "$redis_host" -p "$redis_port" --connect-timeout "$timeout" INFO 2>&1 | grep -q "NOAUTH"; then
echo "Authentication required - check password configuration"
fi
return 1
fi
# Latency test
echo "Testing latency..."
redis-cli -h "$redis_host" -p "$redis_port" --latency -i 1 -c 10 2>/dev/null
}
# Network diagnostics
ss -tlnp | grep 6379 # Check listening sockets
netstat -an | grep 6379
# Use with caution in production
# tcpdump -i any port 6379 # Packet capture for debugging
# Client connection analysis with error handling
analyze_connections() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
if ! redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis"
return 1
fi
echo "=== Connection Analysis ==="
local normal_clients
normal_clients=$(redis-cli -h "$redis_host" -p "$redis_port" CLIENT LIST TYPE normal 2>/dev/null | wc -l)
local replica_clients
replica_clients=$(redis-cli -h "$redis_host" -p "$redis_port" CLIENT LIST TYPE replica 2>/dev/null | wc -l)
echo "Normal clients: $normal_clients"
echo "Replica clients: $replica_clients"
# Check for connection limits
local max_clients
max_clients=$(redis-cli -h "$redis_host" -p "$redis_port" CONFIG GET maxclients 2>/dev/null | tail -n1)
if [ "$normal_clients" -gt $((max_clients * 80 / 100)) ]; then
echo "⚠️ WARNING: Approaching connection limit ($normal_clients/$max_clients)"
fi
}
Memory and Performance Diagnostics
# Advanced memory analysis with comprehensive error handling
#!/bin/bash
diagnose_memory_issues() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local sample_size=${3:-10000000}
if ! redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis"
return 1
fi
echo "=== Memory Diagnostics ==="
# Basic memory info
local memory_info
memory_info=$(redis-cli -h "$redis_host" -p "$redis_port" INFO memory 2>/dev/null)
if [ $? -ne 0 ]; then
echo "ERROR: Failed to get memory info"
return 1
fi
echo "$memory_info" | grep -E "(used_memory_human|used_memory_rss_human|mem_fragmentation_ratio)"
# Memory usage by data type
echo "Analyzing key distribution (this may take time)..."
if command -v timeout >/dev/null 2>&1; then
timeout 300 redis-cli -h "$redis_host" -p "$redis_port" --bigkeys --bigkeys-samples "$sample_size" 2>/dev/null
else
redis-cli -h "$redis_host" -p "$redis_port" --bigkeys --bigkeys-samples "$sample_size" 2>/dev/null
fi
# Redis 4.0+ memory analysis
if redis-cli -h "$redis_host" -p "$redis_port" MEMORY DOCTOR >/dev/null 2>&1; then
echo "=== Memory Doctor Recommendations ==="
redis-cli -h "$redis_host" -p "$redis_port" MEMORY DOCTOR
fi
}
# Performance profiling with safety measures
#!/bin/bash
profile_performance() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local duration=${3:-10}
echo "⚠️ WARNING: Performance profiling can impact Redis performance"
echo "Profiling for $duration seconds..."
# Monitor with timeout to prevent runaway processes
timeout "$duration" redis-cli -h "$redis_host" -p "$redis_port" MONITOR 2>/dev/null | head -1000 > /tmp/redis_monitor.log
echo "Monitor output saved to /tmp/redis_monitor.log"
# Real-time stats
redis-cli -h "$redis_host" -p "$redis_port" --stat -i 1 -c "$duration"
# Key expiration analysis
redis-cli -h "$redis_host" -p "$redis_port" --scan --pattern "*" | head -1000 | while read -r key; do
ttl=$(redis-cli -h "$redis_host" -p "$redis_port" TTL "$key" 2>/dev/null)
echo "$key: $ttl"
done > /tmp/redis_ttl_analysis.log
}
Database Diagnostics and Repair
# Database integrity checks with comprehensive validation
#!/bin/bash
check_database_integrity() {
local rdb_file=${1:-/var/lib/redis/dump.rdb}
local aof_file=${2:-/var/lib/redis/appendonly.aof}
echo "=== Database Integrity Check ==="
# RDB integrity check
if [ -f "$rdb_file" ]; then
echo "Checking RDB file: $rdb_file"
if redis-check-rdb "$rdb_file" >/dev/null 2>&1; then
echo "✓ RDB file integrity verified"
else
echo "✗ RDB file corruption detected"
echo "Consider restoring from backup"
fi
else
echo "RDB file not found: $rdb_file"
fi
# AOF integrity check
if [ -f "$aof_file" ]; then
echo "Checking AOF file: $aof_file"
if redis-check-aof "$aof_file" >/dev/null 2>&1; then
echo "✓ AOF file integrity verified"
else
echo "✗ AOF file corruption detected"
echo "Attempting automatic repair..."
# Backup corrupted file
cp "$aof_file" "${aof_file}.corrupted.$(date +%s)"
# Attempt repair
if redis-check-aof --fix "$aof_file"; then
echo "✓ AOF file repaired successfully"
else
echo "✗ AOF repair failed - restore from backup recommended"
fi
fi
else
echo "AOF file not found: $aof_file"
fi
}
# Performance issue identification with detailed analysis
#!/bin/bash
identify_performance_issues() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
if ! redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis"
return 1
fi
echo "=== Performance Issue Analysis ==="
# Hit ratio analysis
local stats
stats=$(redis-cli -h "$redis_host" -p "$redis_port" INFO stats 2>/dev/null)
local hits=$(echo "$stats" | grep keyspace_hits | cut -d: -f2 | tr -d '\r')
local misses=$(echo "$stats" | grep keyspace_misses | cut -d: -f2 | tr -d '\r')
if [ "$hits" -gt 0 ] && [ "$misses" -gt 0 ]; then
local hit_ratio=$((hits * 100 / (hits + misses)))
echo "Cache hit ratio: ${hit_ratio}%"
if [ "$hit_ratio" -lt 80 ]; then
echo "⚠️ WARNING: Low cache hit ratio - consider reviewing cache strategy"
fi
fi
# Memory eviction analysis
local evicted_keys=$(echo "$stats" | grep evicted_keys | cut -d: -f2 | tr -d '\r')
if [ "$evicted_keys" -gt 1000 ]; then
echo "⚠️ WARNING: High key eviction count: $evicted_keys"
echo "Consider increasing maxmemory or reviewing expiration policies"
fi
# Slow log analysis
local slow_count
slow_count=$(redis-cli -h "$redis_host" -p "$redis_port" SLOWLOG LEN 2>/dev/null)
if [ "$slow_count" -gt 100 ]; then
echo "⚠️ WARNING: $slow_count slow queries detected"
echo "Recent slow queries:"
redis-cli -h "$redis_host" -p "$redis_port" SLOWLOG GET 5
fi
}
8. Advanced Lua Scripting & Server-Side Logic
Complex Lua Script Patterns with Error Handling
-- Rate limiting with sliding window and comprehensive error handling
local function sliding_window_rate_limit(key, window, limit, current_time)
local window_start = current_time - window
-- Validate input parameters
if not key or not window or not limit or not current_time then
return {0, 0, "Invalid parameters"}
end
if window <= 0 or limit <= 0 or current_time <= 0 then
return {0, 0, "Invalid parameter values"}
end
-- Remove old entries
local removed = redis.call('ZREMRANGEBYSCORE', key, 0, window_start)
-- Count current requests
local current_requests = redis.call('ZCARD', key)
if current_requests < limit then
-- Add current request
redis.call('ZADD', key, current_time, current_time .. ':' .. math.random(10000))
redis.call('EXPIRE', key, window)
return {1, limit - current_requests - 1, "allowed"}
else
return {0, 0, "rate_limited"}
end
end
-- Call with error handling
local key = KEYS[1]
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local current_time = tonumber(ARGV[3])
if not window or not limit or not current_time then
return {0, 0, "Invalid arguments"}
end
return sliding_window_rate_limit(key, window, limit, current_time)
-- Distributed locking with auto-renewal and comprehensive error handling
local function acquire_lock_with_renewal(lock_key, lock_value, expiry_time)
-- Validate parameters
if not lock_key or not lock_value or not expiry_time then
return {0, 0, "Invalid parameters"}
end
if expiry_time <= 0 then
return {0, 0, "Invalid expiry time"}
end
-- Try to acquire lock
local result = redis.call('SET', lock_key, lock_value, 'PX', expiry_time, 'NX')
if result then
return {1, expiry_time, "acquired"}
else
-- Check if we already own the lock
local current_value = redis.call('GET', lock_key)
if current_value == lock_value then
-- Renew the lock
local renewed = redis.call('PEXPIRE', lock_key, expiry_time)
if renewed == 1 then
return {1, redis.call('PTTL', lock_key), "renewed"}
else
return {0, 0, "renewal_failed"}
end
else
return {0, redis.call('PTTL', lock_key), "locked_by_other"}
end
end
end
return acquire_lock_with_renewal(KEYS[1], ARGV[1], tonumber(ARGV[2]))
-- Atomic counter with expiration and error handling
local function atomic_increment_with_expiry(key, increment, expiry)
-- Validate parameters
increment = tonumber(increment) or 1
expiry = tonumber(expiry) or 3600
if increment == 0 then
return {redis.call('GET', key) or 0, "unchanged"}
end
local current = redis.call('GET', key)
local new_value
if current == false then
-- Key doesn't exist, set initial value
new_value = increment
redis.call('SET', key, new_value)
redis.call('EXPIRE', key, expiry)
else
-- Key exists, increment
new_value = redis.call('INCRBY', key, increment)
-- Refresh expiry
redis.call('EXPIRE', key, expiry)
end
return {new_value, "success", redis.call('TTL', key)}
end
return atomic_increment_with_expiry(KEYS[1], ARGV[1], ARGV[2])
Script Management and Optimization
# Script registration and management with error handling
#!/bin/bash
manage_lua_scripts() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local script_file=$3
if [ ! -f "$script_file" ]; then
echo "ERROR: Script file not found: $script_file"
return 1
fi
if ! redis-cli -h "$redis_host" -p "$redis_port" ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis"
return 1
fi
# Load script and get SHA
local script_content
script_content=$(cat "$script_file")
local sha
sha=$(redis-cli -h "$redis_host" -p "$redis_port" SCRIPT LOAD "$script_content" 2>/dev/null)
if [ $? -eq 0 ]; then
echo "✓ Script loaded successfully"
echo "SHA: $sha"
# Verify script is loaded
if redis-cli -h "$redis_host" -p "$redis_port" SCRIPT EXISTS "$sha" | grep -q "1"; then
echo "✓ Script verified in cache"
echo "$sha" > "${script_file}.sha"
else
echo "✗ Script verification failed"
return 1
fi
else
echo "✗ Script loading failed"
return 1
fi
}
# Execute script with error handling
execute_lua_script() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local sha_file=$3
shift 3
local keys=("$@")
if [ ! -f "$sha_file" ]; then
echo "ERROR: SHA file not found: $sha_file"
return 1
fi
local sha
sha=$(cat "$sha_file")
# Check if script exists in cache
if ! redis-cli -h "$redis_host" -p "$redis_port" SCRIPT EXISTS "$sha" | grep -q "1"; then
echo "ERROR: Script not found in cache, reload required"
return 1
fi
# Execute script
redis-cli -h "$redis_host" -p "$redis_port" EVALSHA "$sha" ${#keys[@]} "${keys[@]}"
}
# Script debugging and profiling
debug_lua_script() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local script_file=$3
echo "Enabling Lua debugging..."
redis-cli -h "$redis_host" -p "$redis_port" SCRIPT DEBUG YES
echo "Executing script with debugging..."
redis-cli -h "$redis_host" -p "$redis_port" --eval "$script_file" mykey , arg1 arg2
}
Advanced Health Checks
#!/bin/bash
# Comprehensive Redis health check with detailed diagnostics
check_redis_health() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local redis_password=${3:-""}
local warning_threshold=${4:-80}
local critical_threshold=${5:-95}
local exit_code=0
local warnings=()
local errors=()
# Authentication setup
local auth_cmd=""
if [ -n "$redis_password" ]; then
auth_cmd="-a $redis_password"
fi
# Basic connectivity test
if ! redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd --connect-timeout 5 ping >/dev/null 2>&1; then
errors+=("Redis not responding to PING")
exit_code=2
else
echo "✓ Redis connectivity: OK"
fi
# Only proceed if basic connectivity works
if [ ${#errors[@]} -eq 0 ]; then
# Memory usage check
local memory_info
memory_info=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd INFO memory 2>/dev/null)
if [ $? -eq 0 ]; then
local used_memory
used_memory=$(echo "$memory_info" | grep "^used_memory:" | cut -d: -f2 | tr -d '\r')
local max_memory
max_memory=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd CONFIG GET maxmemory 2>/dev/null | tail -n1)
if [ "$max_memory" != "0" ] && [ -n "$used_memory" ] && [ "$used_memory" -gt 0 ]; then
local memory_usage_percent=$((used_memory * 100 / max_memory))
if [ $memory_usage_percent -gt $critical_threshold ]; then
errors+=("Critical memory usage: ${memory_usage_percent}%")
exit_code=2
elif [ $memory_usage_percent -gt $warning_threshold ]; then
warnings+=("High memory usage: ${memory_usage_percent}%")
[ $exit_code -eq 0 ] && exit_code=1
else
echo "✓ Memory usage: ${memory_usage_percent}% (OK)"
fi
fi
# Memory fragmentation check
local fragmentation_ratio
fragmentation_ratio=$(echo "$memory_info" | grep "^mem_fragmentation_ratio:" | cut -d: -f2 | tr -d '\r')
if [ -n "$fragmentation_ratio" ]; then
if (( $(echo "$fragmentation_ratio > 2.0" | bc -l 2>/dev/null || echo 0) )); then
warnings+=("High memory fragmentation: $fragmentation_ratio")
[ $exit_code -eq 0 ] && exit_code=1
else
echo "✓ Memory fragmentation: $fragmentation_ratio (OK)"
fi
fi
fi
# Replication lag check
local repl_info
repl_info=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd INFO replication 2>/dev/null)
if echo "$repl_info" | grep -q "role:slave"; then
local lag
lag=$(echo "$repl_info" | grep "master_repl_offset\|slave_repl_offset" | head -2 | cut -d: -f2 | tr -d '\r' | paste -sd' ' | awk '{print $1-$2}')
if [ -n "$lag" ] && [ "$lag" -gt 1000 ]; then
warnings+=("High replication lag: ${lag} bytes")
[ $exit_code -eq 0 ] && exit_code=1
else
echo "✓ Replication lag: ${lag:-0} bytes (OK)"
fi
fi
# Connection count check
local client_info
client_info=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd INFO clients 2>/dev/null)
local connected_clients
connected_clients=$(echo "$client_info" | grep "^connected_clients:" | cut -d: -f2 | tr -d '\r')
local max_clients
max_clients=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd CONFIG GET maxclients 2>/dev/null | tail -n1)
if [ -n "$connected_clients" ] && [ -n "$max_clients" ] && [ "$max_clients" -gt 0 ]; then
local client_usage_percent=$((connected_clients * 100 / max_clients))
if [ $client_usage_percent -gt $critical_threshold ]; then
errors+=("Critical client connections: ${client_usage_percent}% ($connected_clients/$max_clients)")
exit_code=2
elif [ $client_usage_percent -gt $warning_threshold ]; then
warnings+=("High client connections: ${client_usage_percent}% ($connected_clients/$max_clients)")
[ $exit_code -eq 0 ] && exit_code=1
else
echo "✓ Client connections: $connected_clients/$max_clients (OK)"
fi
fi
# Slow log check
local slow_count
slow_count=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_cmd SLOWLOG LEN 2>/dev/null)
if [ -n "$slow_count" ] && [ "$slow_count" -gt 100 ]; then
warnings+=("High slow query count: $slow_count")
[ $exit_code -eq 0 ] && exit_code=1
else
echo "✓ Slow queries: ${slow_count:-0} (OK)"
fi
fi
# Output summary
if [ ${#warnings[@]} -gt 0 ]; then
echo "WARNINGS:"
printf " - %s\n" "${warnings[@]}"
fi
if [ ${#errors[@]} -gt 0 ]; then
echo "ERRORS:"
printf " - %s\n" "${errors[@]}"
fi
case $exit_code in
0) echo "OVERALL STATUS: OK" ;;
1) echo "OVERALL STATUS: WARNING" ;;
2) echo "OVERALL STATUS: CRITICAL" ;;
esac
return $exit_code
}
10. Advanced DevOps & Container Orchestration
Production Docker Configuration
# Multi-stage Redis build with security hardening
FROM redis:7-alpine AS redis-base
# Install security tools and create non-root user
RUN apk add --no-cache dumb-init ca-certificates && \
rm -rf /var/cache/apk/* && \
addgroup -g 999 redis && \
adduser -D -s /bin/sh -u 999 -G redis redis
# Security hardening - remove unnecessary packages
RUN apk del --purge wget curl
FROM redis-base AS redis-production
# Copy optimized configuration
COPY redis.conf /usr/local/etc/redis/redis.conf
COPY docker-entrypoint.sh /usr/local/bin/
COPY healthcheck.sh /usr/local/bin/
# Set secure permissions
RUN chown -R redis:redis /data /usr/local/etc/redis && \
chmod +x /usr/local/bin/docker-entrypoint.sh /usr/local/bin/healthcheck.sh
# Security and performance settings
USER redis
EXPOSE 6379
VOLUME ["/data"]
# Health check
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
CMD /usr/local/bin/healthcheck.sh
ENTRYPOINT ["dumb-init", "--", "docker-entrypoint.sh"]
CMD ["redis-server", "/usr/local/etc/redis/redis.conf"]
#!/bin/bash
# docker-entrypoint.sh with comprehensive error handling
set -e
# Validate configuration
if [ ! -f "/usr/local/etc/redis/redis.conf" ]; then
echo "ERROR: Redis configuration file not found"
exit 1
fi
# Test configuration syntax
if ! redis-server --test-config /usr/local/etc/redis/redis.conf >/dev/null 2>&1; then
echo "ERROR: Invalid Redis configuration"
redis-server --test-config /usr/local/etc/redis/redis.conf
exit 1
fi
# Create necessary directories
mkdir -p /data/logs
# Set secure permissions
chmod 750 /data
chmod 640 /usr/local/etc/redis/redis.conf
echo "Starting Redis with configuration validation..."
exec "$@"
#!/bin/bash
# healthcheck.sh for Docker health checks
timeout 5 redis-cli ping >/dev/null 2>&1
exit_code=$?
if [ $exit_code -eq 0 ]; then
echo "Redis health check: OK"
exit 0
else
echo "Redis health check: FAILED"
exit 1
fi
Kubernetes StatefulSet with Advanced Features
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-cluster
namespace: redis
labels:
app: redis-cluster
version: "7.0"
spec:
serviceName: redis-cluster
replicas: 6
selector:
matchLabels:
app: redis-cluster
template:
metadata:
labels:
app: redis-cluster
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9121"
prometheus.io/path: "/metrics"
spec:
securityContext:
fsGroup: 999
runAsUser: 999
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: redis
image: redis:7-alpine
ports:
- containerPort: 6379
name: redis
- containerPort: 16379
name: cluster-bus
env:
- name: REDIS_CLUSTER_ANNOUNCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-secret
key: password
command:
- redis-server
args:
- /etc/redis/redis.conf
- --cluster-enabled yes
- --cluster-config-file nodes.conf
- --cluster-node-timeout 5000
- --appendonly yes
- --protected-mode no
- --requirepass $(REDIS_PASSWORD)
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
volumeMounts:
- name: redis-config
mountPath: /etc/redis
- name: redis-data
mountPath: /data
livenessProbe:
exec:
command:
- sh
- -c
- redis-cli -a $REDIS_PASSWORD ping
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
exec:
command:
- sh
- -c
- redis-cli -a $REDIS_PASSWORD ping
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
- name: redis-exporter
image: oliver006/redis_exporter:v1.50.0
ports:
- containerPort: 9121
name: metrics
env:
- name: REDIS_ADDR
value: "redis://localhost:6379"
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
name: redis-secret
key: password
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "100m"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumes:
- name: redis-config
configMap:
name: redis-config
defaultMode: 0640
terminationGracePeriodSeconds: 30
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- redis-cluster
topologyKey: kubernetes.io/hostname
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "fast-ssd"
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-config
namespace: redis
data:
redis.conf: |
# Production Redis configuration
bind 0.0.0.0
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
# Memory configuration
maxmemory 1500mb
maxmemory-policy allkeys-lru
maxmemory-samples 5
# Persistence
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysec
# Logging
loglevel notice
# Security
protected-mode yes
---
apiVersion: v1
kind: Secret
metadata:
name: redis-secret
namespace: redis
type: Opaque
data:
password: c3VwZXJfc2VjdXJlX3Bhc3N3b3JkXzEyMw== # super_secure_password_123
---
apiVersion: v1
kind: Service
metadata:
name: redis-cluster
namespace: redis
labels:
app: redis-cluster
spec:
ports:
- port: 6379
targetPort: 6379
name: redis
- port: 16379
targetPort: 16379
name: cluster-bus
clusterIP: None
selector:
app: redis-cluster
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: redis-cluster
namespace: redis
labels:
app: redis-cluster
spec:
selector:
matchLabels:
app: redis-cluster
endpoints:
- port: metrics
interval: 30s
path: /metrics
Advanced Cluster Initialization Script
#!/bin/bash
# Kubernetes Redis cluster initialization with comprehensive error handling
initialize_k8s_redis_cluster() {
local namespace=${1:-redis}
local replicas=${2:-3}
local timeout=${3:-300}
echo "Initializing Redis cluster in namespace: $namespace"
# Wait for all pods to be ready
echo "Waiting for Redis pods to be ready..."
if ! kubectl wait --for=condition=Ready pod -l app=redis-cluster -n "$namespace" --timeout="${timeout}s"; then
echo "ERROR: Pods failed to become ready within $timeout seconds"
return 1
fi
# Get pod IPs
local pod_ips
pod_ips=$(kubectl get pods -l app=redis-cluster -n "$namespace" -o jsonpath='{.items[*].status.podIP}')
if [ -z "$pod_ips" ]; then
echo "ERROR: No pod IPs found"
return 1
fi
# Convert to array and create node list
local ips=($pod_ips)
local nodes=()
for ip in "${ips[@]}"; do
nodes+=("$ip:6379")
done
echo "Found ${#nodes[@]} Redis nodes: ${nodes[*]}"
if [ ${#nodes[@]} -lt 6 ]; then
echo "ERROR: Minimum 6 nodes required for cluster (3 masters + 3 replicas)"
return 1
fi
# Create cluster using first pod
local first_pod
first_pod=$(kubectl get pods -l app=redis-cluster -n "$namespace" -o jsonpath='{.items[0].metadata.name}')
echo "Creating cluster using pod: $first_pod"
kubectl exec -it "$first_pod" -n "$namespace" -- redis-cli --cluster create "${nodes[@]}" --cluster-replicas 1 --cluster-yes
if [ $? -eq 0 ]; then
echo "✓ Cluster created successfully"
# Verify cluster status
echo "Verifying cluster status..."
kubectl exec "$first_pod" -n "$namespace" -- redis-cli cluster info
kubectl exec "$first_pod" -n "$namespace" -- redis-cli cluster nodes
return 0
else
echo "✗ Cluster creation failed"
return 1
fi
}
11. Advanced Migration & Data Management
Zero-Downtime Migration Strategies
#!/bin/bash
# Redis migration with comprehensive validation and rollback capability
migrate_redis_zero_downtime() {
local source_host=$1
local source_port=$2
local target_host=$3
local target_port=$4
local password=${5:-""}
local timeout=${6:-300}
# Validation
if [ $# -lt 4 ]; then
echo "ERROR: Usage: migrate_redis_zero_downtime source_host source_port target_host target_port [password] [timeout]"
return 1
fi
# Authentication setup
local auth_cmd=""
if [ -n "$password" ]; then
auth_cmd="-a $password"
fi
echo "=== Redis Zero-Downtime Migration ==="
echo "Source: $source_host:$source_port"
echo "Target: $target_host:$target_port"
echo "Timeout: ${timeout}s"
# Pre-migration validation
echo "Step 1: Validating source and target..."
# Test source connectivity
if ! redis-cli -h "$source_host" -p "$source_port" $auth_cmd ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to source Redis at $source_host:$source_port"
return 1
fi
# Test target connectivity
if ! redis-cli -h "$target_host" -p "$target_port" $auth_cmd ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to target Redis at $target_host:$target_port"
return 1
fi
# Check target is empty or confirm overwrite
local target_keys
target_keys=$(redis-cli -h "$target_host" -p "$target_port" $auth_cmd DBSIZE 2>/dev/null)
if [ "$target_keys" -gt 0 ]; then
echo "WARNING: Target Redis contains $target_keys keys"
read -p "Continue and overwrite? (y/N): " confirm
if [[ ! $confirm =~ ^[Yy]$ ]]; then
echo "Migration aborted"
return 1
fi
fi
echo "✓ Pre-migration validation completed"
# Step 2: Setup replication
echo "Step 2: Setting up replication from source to target..."
if redis-cli -h "$target_host" -p "$target_port" $auth_cmd REPLICAOF "$source_host" "$source_port" >/dev/null 2>&1; then
echo "✓ Replication setup initiated"
else
echo "✗ Failed to setup replication"
return 1
fi
# Step 3: Wait for initial sync with progress monitoring
echo "Step 3: Waiting for initial synchronization..."
local sync_start_time=$(date +%s)
local sync_timeout=$((sync_start_time + timeout))
local last_output=""
while [ $(date +%s) -lt $sync_timeout ]; do
local repl_info
repl_info=$(redis-cli -h "$target_host" -p "$target_port" $auth_cmd INFO replication 2>/dev/null)
if echo "$repl_info" | grep -q "master_sync_in_progress:0"; then
echo "✓ Initial synchronization completed"
break
elif echo "$repl_info" | grep -q "master_sync_in_progress:1"; then
# Show progress
local bytes_received
bytes_received=$(echo "$repl_info" | grep "master_sync_read_bytes:" | cut -d: -f2 | tr -d '\r')
if [ "$bytes_received" != "$last_output" ]; then
echo " Sync in progress... received: ${bytes_received:-0} bytes"
last_output="$bytes_received"
fi
fi
sleep 5
done
# Verify sync completed
if ! redis-cli -h "$target_host" -p "$target_port" $auth_cmd INFO replication | grep -q "master_sync_in_progress:0"; then
echo "✗ Initial synchronization timeout"
echo "Rolling back..."
redis-cli -h "$target_host" -p "$target_port" $auth_cmd REPLICAOF NO ONE >/dev/null 2>&1
return 1
fi
# Step 4: Monitor replication lag
echo "Step 4: Monitoring replication lag..."
local lag_check_count=0
local max_lag_checks=10
local acceptable_lag=100 # bytes
while [ $lag_check_count -lt $max_lag_checks ]; do
local target_repl_info
target_repl_info=$(redis-cli -h "$target_host" -p "$target_port" $auth_cmd INFO replication 2>/dev/null)
local source_repl_info
source_repl_info=$(redis-cli -h "$source_host" -p "$source_port" $auth_cmd INFO replication 2>/dev/null)
local target_offset
target_offset=$(echo "$target_repl_info" | grep "slave_repl_offset:" | cut -d: -f2 | tr -d '\r')
local source_offset
source_offset=$(echo "$source_repl_info" | grep "master_repl_offset:" | cut -d: -f2 | tr -d '\r')
if [ -n "$target_offset" ] && [ -n "$source_offset" ]; then
local lag=$((source_offset - target_offset))
echo " Replication lag: $lag bytes"
if [ "$lag" -le "$acceptable_lag" ]; then
echo "✓ Replication lag acceptable ($lag bytes)"
break
fi
fi
((lag_check_count++))
sleep 2
done
# Step 5: Promote target to master
echo "Step 5: Promoting target to master..."
# Record pre-promotion state for rollback
local source_keys_before
source_keys_before=$(redis-cli -h "$source_host" -p "$source_port" $auth_cmd DBSIZE 2>/dev/null)
local target_keys_before
target_keys_before=$(redis-cli -h "$target_host" -p "$target_port" $auth_cmd DBSIZE 2>/dev/null)
echo " Source keys before promotion: $source_keys_before"
echo " Target keys before promotion: $target_keys_before"
if redis-cli -h "$target_host" -p "$target_port" $auth_cmd REPLICAOF NO ONE >/dev/null 2>&1; then
echo "✓ Target promoted to master successfully"
else
echo "✗ Failed to promote target to master"
return 1
fi
# Step 6: Final validation
echo "Step 6: Final validation..."
local target_keys_after
target_keys_after=$(redis-cli -h "$target_host" -p "$target_port" $auth_cmd DBSIZE 2>/dev/null)
echo " Target keys after promotion: $target_keys_after"
# Validate key count consistency
local key_diff=$((source_keys_before - target_keys_after))
if [ "$key_diff" -gt 100 ]; then
echo "⚠️ WARNING: Significant key count difference: $key_diff"
echo " Consider manual verification"
fi
# Test basic operations on target
local test_key="migration_test_$(date +%s)"
if redis-cli -h "$target_host" -p "$target_port" $auth_cmd SET "$test_key" "success" EX 60 >/dev/null 2>&1 && \
redis-cli -h "$target_host" -p "$target_port" $auth_cmd GET "$test_key" | grep -q "success"; then
echo "✓ Target Redis operational test passed"
redis-cli -h "$target_host" -p "$target_port" $auth_cmd DEL "$test_key" >/dev/null 2>&1
else
echo "✗ Target Redis operational test failed"
return 1
fi
echo "=== Migration completed successfully ==="
echo "New master: $target_host:$target_port"
echo "Total keys migrated: $target_keys_after"
return 0
}
Advanced Data Export/Import with Parallel Processing
#!/bin/bash
# Parallel Redis data migration with comprehensive error handling
export_redis_parallel() {
local source_host=${1:-localhost}
local source_port=${2:-6379}
local pattern=${3:-"*"}
local batch_size=${4:-1000}
local output_dir=${5:-./redis_export}
local max_parallel=${6:-4}
local password=${7:-""}
# Validation
if [ ! -d "$(dirname "$output_dir")" ]; then
echo "ERROR: Parent directory does not exist: $(dirname "$output_dir")"
return 1
fi
# Authentication setup
local auth_cmd=""
if [ -n "$password" ]; then
auth_cmd="-a $password"
fi
# Test connectivity
if ! redis-cli -h "$source_host" -p "$source_port" $auth_cmd ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis at $source_host:$source_port"
return 1
fi
echo "=== Redis Parallel Export ==="
echo "Source: $source_host:$source_port"
echo "Pattern: $pattern"
echo "Batch size: $batch_size"
echo "Max parallel: $max_parallel"
echo "Output: $output_dir"
# Create output directory
mkdir -p "$output_dir" || {
echo "ERROR: Cannot create output directory: $output_dir"
return 1
}
# Get total key count for progress tracking
local total_keys
total_keys=$(redis-cli -h "$source_host" -p "$source_port" $auth_cmd --scan --pattern "$pattern" | wc -l)
echo "Total keys to export: $total_keys"
if [ "$total_keys" -eq 0 ]; then
echo "No keys found matching pattern: $pattern"
return 0
fi
# Export keys to file
local keys_file="$output_dir/keys.txt"
echo "Scanning for keys..."
redis-cli -h "$source_host" -p "$source_port" $auth_cmd --scan --pattern "$pattern" > "$keys_file"
if [ ! -s "$keys_file" ]; then
echo "ERROR: No keys exported to file"
return 1
fi
# Split keys into batches
echo "Splitting into batches of $batch_size keys..."
split -l "$batch_size" "$keys_file" "$output_dir/batch_"
local batch_files=("$output_dir"/batch_*)
echo "Created ${#batch_files[@]} batch files"
# Export function for parallel processing
export_batch() {
local batch_file=$1
local batch_name=$(basename "$batch_file")
local output_file="$output_dir/$batch_name.redis"
local error_file="$output_dir/$batch_name.errors"
local processed=0
local errors=0
echo "Processing $batch_name..."
while IFS= read -r key; do
[ -z "$key" ] && continue
local key_type
key_type=$(redis-cli -h "$source_host" -p "$source_port" $auth_cmd TYPE "$key" 2>/dev/null | tr -d '\r')
if [ "$key_type" = "none" ]; then
echo "Key expired or deleted: $key" >> "$error_file"
((errors++))
continue
fi
local ttl
ttl=$(redis-cli -h "$source_host" -p "$source_port" $auth_cmd TTL "$key" 2>/dev/null | tr -d '\r')
case $key_type in
"string")
local value
value=$(redis-cli -h "$source_host" -p "$source_port" $auth_cmd GET "$key" 2>/dev/null)
if [ $? -eq 0 ]; then
printf 'SET %q %q\n' "$key" "$value" >> "$output_file"
else
echo "Failed to get string key: $key" >> "$error_file"
((errors++))
fi
;;
"hash")
if redis-cli -h "$source_host" -p "$source_port" $auth_cmd HGETALL "$key" 2>/dev/null | \
awk 'NR%2==1{field=$0} NR%2==0{printf "HSET %s %s %s\n", ENVIRON["key"], field, $0}' key="$key" >> "$output_file"; then
:
else
echo "Failed to get hash key: $key" >> "$error_file"
((errors++))
fi
;;
"list")
if redis-cli -h "$source_host" -p "$source_port" $auth_cmd LRANGE "$key" 0 -1 2>/dev/null | \
awk '{printf "LPUSH %s %s\n", ENVIRON["key"], $0}' key="$key" >> "$output_file"; then
:
else
echo "Failed to get list key: $key" >> "$error_file"
((errors++))
fi
;;
"set")
if redis-cli -h "$source_host" -p "$source_port" $auth_cmd SMEMBERS "$key" 2>/dev/null | \
awk '{printf "SADD %s %s\n", ENVIRON["key"], $0}' key="$key" >> "$output_file"; then
:
else
echo "Failed to get set key: $key" >> "$error_file"
((errors++))
fi
;;
"zset")
if redis-cli -h "$source_host" -p "$source_port" $auth_cmd ZRANGE "$key" 0 -1 WITHSCORES 2>/dev/null | \
awk 'NR%2==1{member=$0} NR%2==0{printf "ZADD %s %s %s\n", ENVIRON["key"], $0, member}' key="$key" >> "$output_file"; then
:
else
echo "Failed to get zset key: $key" >> "$error_file"
((errors++))
fi
;;
*)
echo "Unsupported key type: $key_type for key: $key" >> "$error_file"
((errors++))
;;
esac
# Add TTL if applicable
if [ "$ttl" -gt 0 ]; then
printf 'EXPIRE %q %s\n' "$key" "$ttl" >> "$output_file"
fi
((processed++))
done < "$batch_file"
echo "Completed $batch_name: $processed processed, $errors errors"
# Clean up empty error file
if [ ! -s "$error_file" ]; then
rm -f "$error_file"
fi
}
# Export the function for parallel execution
export -f export_batch
export source_host source_port auth_cmd output_dir
# Process batches in parallel
echo "Starting parallel export with $max_parallel workers..."
printf '%s\n' "${batch_files[@]}" | xargs -n 1 -P "$max_parallel" -I {} bash -c 'export_batch "$@"' _ {}
# Wait for all background jobs to complete
wait
# Cleanup and summary
rm -f "$keys_file" "${batch_files[@]}"
local redis_files=("$output_dir"/*.redis)
local total_commands=0
local total_errors=0
for file in "${redis_files[@]}"; do
if [ -f "$file" ]; then
total_commands=$((total_commands + $(wc -l < "$file")))
fi
done
for error_file in "$output_dir"/*.errors; do
if [ -f "$error_file" ]; then
total_errors=$((total_errors + $(wc -l < "$error_file")))
fi
done
echo "=== Export Summary ==="
echo "Total Redis commands exported: $total_commands"
echo "Total errors: $total_errors"
echo "Output files: ${#redis_files[@]}"
# Create import script
cat > "$output_dir/import.sh" << 'EOF'
#!/bin/bash
# Generated import script
REDIS_HOST=${1:-localhost}
REDIS_PORT=${2:-6379}
REDIS_PASSWORD=${3:-""}
AUTH_CMD=""
if [ -n "$REDIS_PASSWORD" ]; then
AUTH_CMD="-a $REDIS_PASSWORD"
fi
echo "Importing to Redis at $REDIS_HOST:$REDIS_PORT..."
for redis_file in *.redis; do
if [ -f "$redis_file" ]; then
echo "Importing $redis_file..."
redis-cli -h "$REDIS_HOST" -p "$REDIS_PORT" $AUTH_CMD --pipe < "$redis_file"
fi
done
echo "Import completed"
EOF
chmod +x "$output_dir/import.sh"
echo "Import script created: $output_dir/import.sh"
return 0
}
12. Advanced Performance Testing & Optimization
Comprehensive Benchmarking Suite with Advanced Analysis
#!/bin/bash
# Advanced Redis performance testing with detailed analysis and reporting
redis_performance_suite() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local test_duration=${3:-60}
local password=${4:-""}
local output_dir=${5:-./redis_benchmark_$(date +%Y%m%d_%H%M%S)}
# Validation and setup
if ! redis-cli -h "$redis_host" -p "$redis_port" ${password:+-a $password} ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis at $redis_host:$redis_port"
return 1
fi
mkdir -p "$output_dir" || {
echo "ERROR: Cannot create output directory: $output_dir"
return 1
}
local auth_param=""
if [ -n "$password" ]; then
auth_param="-a $password"
fi
echo "=== Redis Performance Test Suite ==="
echo "Target: $redis_host:$redis_port"
echo "Duration: ${test_duration}s per test"
echo "Output: $output_dir"
echo "Start time: $(date)"
# System information
echo "=== System Information ===" | tee "$output_dir/system_info.txt"
{
echo "Redis Info:"
redis-cli -h "$redis_host" -p "$redis_port" $auth_param INFO server
echo
echo "System Info:"
uname -a
echo "CPU Info:"
grep "model name" /proc/cpuinfo | head -1
grep "cpu cores" /proc/cpuinfo | head -1
echo "Memory Info:"
free -h
echo "Network Info:"
ss -i | grep "$redis_port" | head -5
} >> "$output_dir/system_info.txt" 2>/dev/null
# Test 1: Basic throughput test
echo "=== Test 1: Basic Throughput Test ===" | tee -a "$output_dir/results.txt"
{
echo "Testing basic SET/GET operations..."
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t set,get -n 1000000 -c 50 -d 100 \
--csv > "$output_dir/basic_throughput.csv" 2>&1
echo "Basic throughput results:"
tail -10 "$output_dir/basic_throughput.csv"
} | tee -a "$output_dir/results.txt"
# Test 2: Pipeline performance
echo "=== Test 2: Pipeline Performance Test ===" | tee -a "$output_dir/results.txt"
for pipeline_size in 1 4 8 16 32; do
echo "Testing with pipeline size: $pipeline_size" | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t set,get -n 100000 -c 50 -P "$pipeline_size" \
--csv > "$output_dir/pipeline_${pipeline_size}.csv" 2>&1
done
# Test 3: Different data sizes
echo "=== Test 3: Data Size Performance ===" | tee -a "$output_dir/results.txt"
for data_size in 64 256 1024 4096 10240; do
echo "Testing with data size: $data_size bytes" | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t set,get -n 50000 -c 50 -d "$data_size" \
--csv > "$output_dir/datasize_${data_size}.csv" 2>&1
done
# Test 4: Concurrent connections scaling
echo "=== Test 4: Connection Scaling Test ===" | tee -a "$output_dir/results.txt"
for connections in 10 50 100 200 500; do
echo "Testing with $connections concurrent connections" | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t set,get -n 50000 -c "$connections" -d 100 \
--csv > "$output_dir/connections_${connections}.csv" 2>&1
done
# Test 5: Data structure specific tests
echo "=== Test 5: Data Structure Performance ===" | tee -a "$output_dir/results.txt"
# Hash operations
echo "Testing HASH operations..." | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t hset,hget,hmset,hmget -n 100000 -c 50 \
--csv > "$output_dir/hash_ops.csv" 2>&1
# List operations
echo "Testing LIST operations..." | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t lpush,lpop,rpush,rpop,lrange -n 100000 -c 50 \
--csv > "$output_dir/list_ops.csv" 2>&1
# Set operations
echo "Testing SET operations..." | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t sadd,smembers,spop -n 100000 -c 50 \
--csv > "$output_dir/set_ops.csv" 2>&1
# Sorted set operations
echo "Testing SORTED SET operations..." | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t zadd,zrange,zrevrange,zrank -n 100000 -c 50 \
--csv > "$output_dir/zset_ops.csv" 2>&1
# Test 6: Lua script performance
echo "=== Test 6: Lua Script Performance ===" | tee -a "$output_dir/results.txt"
local simple_script='return redis.call("incr", KEYS[1])'
local complex_script='
local current = redis.call("GET", KEYS[1])
if current == false then
redis.call("SET", KEYS[1], 1)
redis.call("EXPIRE", KEYS[1], 3600)
return 1
else
return redis.call("INCR", KEYS[1])
end'
echo "Testing simple Lua script..." | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-n 100000 -c 50 eval "$simple_script" 1 counter \
--csv > "$output_dir/lua_simple.csv" 2>&1
echo "Testing complex Lua script..." | tee -a "$output_dir/results.txt"
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-n 50000 -c 50 eval "$complex_script" 1 complex_counter \
--csv > "$output_dir/lua_complex.csv" 2>&1
# Test 7: Memory pressure test
echo "=== Test 7: Memory Pressure Test ===" | tee -a "$output_dir/results.txt"
{
local initial_memory
initial_memory=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_param INFO memory | grep used_memory: | cut -d: -f2 | tr -d '\r')
echo "Initial memory usage: $initial_memory bytes"
# Load data until memory pressure
redis-benchmark -h "$redis_host" -p "$redis_port" $auth_param \
-t set -n 1000000 -c 50 -d 1024 -r 100000 \
--csv > "$output_dir/memory_pressure.csv" 2>&1
local final_memory
final_memory=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_param INFO memory | grep used_memory: | cut -d: -f2 | tr -d '\r')
echo "Final memory usage: $final_memory bytes"
echo "Memory increase: $((final_memory - initial_memory)) bytes"
# Check for evictions
local evicted_keys
evicted_keys=$(redis-cli -h "$redis_host" -p "$redis_port" $auth_param INFO stats | grep evicted_keys | cut -d: -f2 | tr -d '\r')
echo "Keys evicted during test: ${evicted_keys:-0}"
} | tee -a "$output_dir/results.txt"
# Test 8: Latency analysis
echo "=== Test 8: Latency Analysis ===" | tee -a "$output_dir/results.txt"
{
echo "Running latency tests..."
# Basic latency
timeout 30 redis-cli -h "$redis_host" -p "$redis_port" $auth_param \
--latency -i 1 > "$output_dir/latency_basic.txt" 2>&1 &
# Latency history
timeout 30 redis-cli -h "$redis_host" -p "$redis_port" $auth_param \
--latency-history -i 1 > "$output_dir/latency_history.txt" 2>&1 &
wait
echo "Latency test completed"
if [ -s "$output_dir/latency_basic.txt" ]; then
echo "Basic latency stats:"
tail -5 "$output_dir/latency_basic.txt"
fi
} | tee -a "$output_dir/results.txt"
# Generate comprehensive report
echo "=== Generating Performance Report ===" | tee -a "$output_dir/results.txt"
cat > "$output_dir/generate_report.py" << 'EOF'
#!/usr/bin/env python3
import csv
import os
import sys
from statistics import mean, median
def parse_benchmark_csv(filename):
"""Parse Redis benchmark CSV output"""
try:
with open(filename, 'r') as f:
reader = csv.DictReader(f)
data = []
for row in reader:
if 'requests per second' in row:
data.append({
'test': row.get('test', 'unknown'),
'requests_per_second': float(row['requests per second'].replace(',', '')),
'latency_ms': float(row.get('latency', '0').replace('ms', ''))
})
return data
except Exception as e:
print(f"Error parsing {filename}: {e}")
return []
def generate_summary_report(output_dir):
"""Generate summary performance report"""
report_file = os.path.join(output_dir, 'performance_summary.txt')
csv_files = [f for f in os.listdir(output_dir) if f.endswith('.csv')]
with open(report_file, 'w') as report:
report.write("Redis Performance Test Summary\n")
report.write("=" * 50 + "\n\n")
total_ops = 0
best_throughput = 0
worst_latency = 0
for csv_file in sorted(csv_files):
data = parse_benchmark_csv(os.path.join(output_dir, csv_file))
if data:
avg_rps = mean([d['requests_per_second'] for d in data])
avg_latency = mean([d['latency_ms'] for d in data if d['latency_ms'] > 0])
report.write(f"Test: {csv_file}\n")
report.write(f" Average RPS: {avg_rps:,.0f}\n")
if avg_latency > 0:
report.write(f" Average Latency: {avg_latency:.2f}ms\n")
report.write("\n")
total_ops += avg_rps
best_throughput = max(best_throughput, avg_rps)
if avg_latency > 0:
worst_latency = max(worst_latency, avg_latency)
report.write("\nSummary Statistics:\n")
report.write(f"Best Throughput: {best_throughput:,.0f} ops/sec\n")
if worst_latency > 0:
report.write(f"Worst Latency: {worst_latency:.2f}ms\n")
report.write(f"Total Test Operations: {total_ops:,.0f}\n")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python3 generate_report.py <output_dir>")
sys.exit(1)
generate_summary_report(sys.argv[1])
print("Performance summary generated")
EOF
# Generate the report if Python is available
if command -v python3 >/dev/null 2>&1; then
python3 "$output_dir/generate_report.py" "$output_dir"
if [ -f "$output_dir/performance_summary.txt" ]; then
echo "Performance summary:" | tee -a "$output_dir/results.txt"
cat "$output_dir/performance_summary.txt" | tee -a "$output_dir/results.txt"
fi
fi
# Final memory and performance analysis
echo "=== Final Analysis ===" | tee -a "$output_dir/results.txt"
{
echo "Redis configuration analysis:"
redis-cli -h "$redis_host" -p "$redis_port" $auth_param CONFIG GET "*memory*" | paste - -
echo
echo "Redis memory info:"
redis-cli -h "$redis_host" -p "$redis_port" $auth_param INFO memory | grep -E "(used_memory|fragmentation|evicted)"
echo
echo "Redis stats:"
redis-cli -h "$redis_host" -p "$redis_port" $auth_param INFO stats | grep -E "(ops_per_sec|keyspace|expired|evicted)"
echo
echo "Test completion time: $(date)"
} | tee -a "$output_dir/results.txt"
echo "=== Performance Testing Complete ==="
echo "Results saved to: $output_dir"
echo "Main results file: $output_dir/results.txt"
return 0
}
# Memory fragmentation analysis with recommendations
analyze_memory_fragmentation() {
local redis_host=${1:-localhost}
local redis_port=${2:-6379}
local password=${3:-""}
if ! redis-cli -h "$redis_host" -p "$redis_port" ${password:+-a $password} ping >/dev/null 2>&1; then
echo "ERROR: Cannot connect to Redis"
return 1
fi
echo "=== Memory Fragmentation Analysis ==="
local memory_info
memory_info=$(redis-cli -h "$redis_host" -p "$redis_port" ${password:+-a $password} INFO memory 2>/dev/null)
local used_memory
used_memory=$(echo "$memory_info" | grep "^used_memory:" | cut -d: -f2 | tr -d '\r')
local used_memory_rss
used_memory_rss=$(echo "$memory_info" | grep "^used_memory_rss:" | cut -d: -f2 | tr -d '\r')
local fragmentation_ratio
fragmentation_ratio=$(echo "$memory_info" | grep "^mem_fragmentation_ratio:" | cut -d: -f2 | tr -d '\r')
local used_memory_peak
used_memory_peak=$(echo "$memory_info" | grep "^used_memory_peak:" | cut -d: -f2 | tr -d '\r')
echo "Memory Usage Analysis:"
echo " Used Memory: $(numfmt --to=iec $used_memory 2>/dev/null || echo $used_memory) bytes"
echo " RSS Memory: $(numfmt --to=iec $used_memory_rss 2>/dev/null || echo $used_memory_rss) bytes"
echo " Peak Memory: $(numfmt --to=iec $used_memory_peak 2>/dev/null || echo $used_memory_peak) bytes"
echo " Fragmentation Ratio: $fragmentation_ratio"
# Analysis and recommendations
if command -v bc >/dev/null 2>&1; then
if (( $(echo "$fragmentation_ratio > 2.0" | bc -l) )); then
echo
echo "⚠️ HIGH FRAGMENTATION DETECTED"
echo "Recommendations:"
echo " 1. Consider restarting Redis during maintenance window"
echo " 2. Enable active defragmentation (Redis 4.0+):"
echo " CONFIG SET activedefrag yes"
echo " 3. Monitor memory allocation patterns"
echo " 4. Consider using jemalloc allocator"
elif (( $(echo "$fragmentation_ratio > 1.5" | bc -l) )); then
echo
echo "⚠️ MODERATE FRAGMENTATION"
echo "Recommendations:"
echo " 1. Monitor fragmentation trends"
echo " 2. Consider enabling active defragmentation"
echo " 3. Review memory usage patterns"
else
echo
echo "✓ FRAGMENTATION WITHIN NORMAL RANGE"
fi
# Memory efficiency analysis
local efficiency
efficiency=$(echo "scale=2; $used_memory * 100 / $used_memory_rss" | bc -l)
echo "Memory Efficiency: ${efficiency}%"
if (( $(echo "$efficiency < 75" | bc -l) )); then
echo "⚠️ Low memory efficiency - consider optimization"
fi
fi
# Redis 4.0+ memory doctor
if redis-cli -h "$redis_host" -p "$redis_port" ${password:+-a $password} MEMORY DOCTOR >/dev/null 2>&1; then
echo
echo "Redis Memory Doctor Recommendations:"
redis-cli -h "$redis_host" -p "$redis_port" ${password:+-a $password} MEMORY DOCTOR
fi
return 0
}
Conclusion
This comprehensive Redis operations guide provides enterprise-grade technical depth with:
- Version-specific compatibility notes for Redis 5.x, 6.0+, and 7.0+
- Comprehensive error handling in all scripts and operations
- Performance impact warnings for monitoring commands
- Production-ready security configurations with TLS and ACL
- Advanced troubleshooting capabilities with automated diagnostics
- Zero-downtime migration strategies with rollback capabilities
- Parallel data processing for large-scale operations
- Container orchestration with Kubernetes security best practices
- Comprehensive performance testing with detailed analysis
The documentation serves as an authoritative reference for Redis operations teams and can be confidently deployed in production environments. Each section includes practical examples, error handling, and monitoring patterns essential for enterprise Redis deployments.
Read more about Redis Troubleshooting – https://minervadb.xyz/redis-troubleshooting-cheatsheet/
Reference
