shahar1 commented on code in PR #55344:
URL: https://github.com/apache/airflow/pull/55344#discussion_r2328822757
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or
\*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
+
+Redis stores task-related data, queue information, and other temporary state.
+
+
+
+Over time, Redis can grow very large if keys are not cleaned up.
+
+This can lead to performance problems or even production outages if the
instance runs out of memory.
+
+
+
+This guide describes how to monitor, clean, and maintain your Redis instance
in an Airflow setup.
+
+
+
+---
+
+
+
+\## Why Maintenance Is Needed
+
+
+
+Redis does not automatically remove all keys created by Celery or Kombu (the
library Airflow uses for queuing).
+
+If keys are left without a \*\*Time-To-Live (TTL)\*\*, they can accumulate
indefinitely.
+
+
+
+\### Common symptoms
+
+
+
+\* Redis memory usage grows continuously.
+
+\* Instance reaches the maximum memory limit and stops accepting writes.
+
+\* Airflow tasks start failing due to Redis errors.
+
+
+
+---
+
+
+
+\## Recommended Cleanup Strategy
+
+
+
+\### 1. Use TTLs on Redis keys
+
+
+
+If you discover keys without TTLs, you can enforce expiry manually.
+
+The example below uses a Lua script to set TTLs for all keys except critical
ones.
+
+
+
+```bash
+
+\#!/bin/bash
+
+
+
+\# Usage: ./cleanup.sh -h <host> -p <port>
+
+
+
+HOST=$1
+
+PORT=$2
+
+
+
+\# Inline Lua script to set TTL on keys
+
+lua\_script='
+
+local keys = redis.call("KEYS", "\*")
+
+for \_, key in ipairs(keys) do
+
+ if string.match(key, "^unacked") or string.match(key,
"^\_kombu%.binding") then
+
+ -- Skip critical keys
+
+ else
+
+ if redis.call("TTL", key) == -1 then
+
+ redis.call("EXPIRE", key, 54000) -- 15 days
+
+ end
+
+ end
+
+end
+
+'
+
+
+
+redis6-cli -h "$HOST" -p "$PORT" --eval <(echo "$lua\_script")
Review Comment:
```suggestion
redis-cli -h "$HOST" -p "$PORT" --eval <(echo "$lua\_script")
```
##########
docs/maintenance/redis.md:
##########
@@ -0,0 +1,234 @@
+
+
+\# Redis Maintenance
+
+
+
+\## Introduction
+
+
+
+When running Airflow with the \*\*CeleryExecutor\*\* or
\*\*CeleryKubernetesExecutor\*\*, Redis is often used as the message broker.
+
+Redis stores task-related data, queue information, and other temporary state.
+
+
+
+Over time, Redis can grow very large if keys are not cleaned up.
Review Comment:
@ashb / @kaxil - isn't there a way today to configure TTL for Celery's
broker's messages? I wonder how it has never been reported as an issue before
#55344.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
