As of last year, the redis Python client was still fully compatible with 
Valkey, has this changed? This would make it even easier. I think they also 
forked redis-py 

> Gesendet: Dienstag, den 15.07.2025 um 21:38 Uhr
> Von: "Jarek Potiuk" <ja...@potiuk.com>
> An: dev@airflow.apache.org
> Cc: gha...@amazon.com.invalid
> Betreff: Re: [DISCUSSION] New Provider: Valkey (Redis OSS Fork)
> 
> > Can you please explain why it should be a provider managed by Airflow
> rather than on your own?
> 
> Just to explain it Elad - and yes, I know I am usually the one that is
> sceptical -> valley is truly community driven replacement for redis
> https://valkey.io/participants/ - you see a lot of well-known community
> recognised entities, and there is no single entity behind it that could
> take "responsibility" - while there are enough stakeholders to make sure
> that it will be maintained here in Airflow community. Also it's a "drop-in"
> replacement for redis generally - and I do not expect a lot of problems
> when we start with both redis provider and redis celery integration.
> 
> I've heard and saw a lot of praise from the community for Valkey being good
> redis replacement, and - even we at the PMC and devlist discussions
> discussed about the dangers Redis licence creates for that - even now we
> limited Redis support in Helm chart of ours to the "really free" version
> and we discussed that Valkey might be a good way forward - especially if
> someone wants to contribute support for it. so if someone wants to
> contribute to Airflow AND make sure that Redis broker support in Celery is
> covered - I am all for it.
> 
> 
> On Tue, Jul 15, 2025 at 9:37 PM Elad Kalif <elad...@gmail.com> wrote:
> 
> > > * Helm Chart support to run valley as Celery Broker instead of Redis
> >
> > Oh yes! didn't consider its usage as a broker but is it related to the
> > provider package?
> > Amazon SQS can be used as broker but it's not related to the SQS
> > integrations in the Airflow amazon provider (hook,operator,sensor)
> > I also don't see ValKey listed as option for brokers in
> >
> > https://docs.celeryq.dev/en/latest/getting-started/backends-and-brokers/index.html#broker-overview
> >
> > On Tue, Jul 15, 2025 at 10:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Yes. Valkey is a good replacement/parallel implementation to Redis. One
> > > thing however is important - I would see this even more useful if it also
> > > allows for the Celery Redis replacement for broker - means that it is
> > fully
> > > supported in our:
> > >
> > > * unit tests (of course)
> > > * integration tests with ("valkey") integration - including running
> > > docker-compose integration to run in CI
> > > * Helm Chart support to run valley as Celery Broker instead of Redis
> > >
> > > I think I'd only consider valley as "successful" provider integration if
> > > all that above is completed.
> > >
> > > J.
> > >
> > > On Tue, Jul 15, 2025 at 9:03 PM Ghaeli, Sean <gha...@amazon.com.invalid>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > >
> > > >
> > > > I’d like to propose the addition of a new provider to support Valkey<
> > > > https://valkey.io/>: a fully open-source, community-governed fork of
> > > > Redis 7.2, now maintained under the Linux Foundation.
> > > >
> > > >
> > > >
> > > > Valkey has been gaining traction recently, in part due to changes by
> > > Redis
> > > > moving their license to one unrecognized by the Open Source Initiative.
> > > > Valkey represents a future-proof alternative. The proposed provider
> > would
> > > > closely mirror the existing Redis provider in functionality and serve
> > as
> > > a
> > > > drop-in, OSS-aligned option for workflows that depend on key-value
> > > storage,
> > > > pub/sub, or caching patterns.
> > > >
> > > >
> > > >
> > > > Tentatively, here are the components I plan on implementing (mirroring
> > > the
> > > > Redis provider):
> > > >
> > > >
> > > >
> > > >   *   ValkeyHook – Manage connections and interactions with the Valkey
> > > > server, enabling reuse across operators and sensors.
> > > >   *   ValkeyPublishOperator – Allows tasks to publish messages to
> > Valkey
> > > > pub/sub channels, useful for event-driven workflows or inter-task
> > > signaling.
> > > >   *   ValkeyKeySensor – Waits for a specific key to exist or reach a
> > > > value, supporting synchronization patterns where one task's output
> > > controls
> > > > downstream logic.
> > > >   *   ValkeyPubSubSensor – Subscribes to a Valkey channel and triggers
> > on
> > > > incoming messages, enabling real-time DAG triggering.
> > > >
> > > >
> > > >
> > > > Here's an example of how we could use the components to publish a
> > message
> > > > and then trigger a DAG:
> > > >
> > > >
> > > >
> > > > `
> > > >
> > > > publish_message = ValkeyPublishOperator(
> > > >
> > > >         task_id="publish_ready_signal",
> > > >
> > > >         channel="processing_status",
> > > >
> > > >         message="READY",
> > > >
> > > >         valkey_conn_id="valkey_default",
> > > >
> > > >     )
> > > >
> > > > `
> > > >
> > > >
> > > >
> > > > This task publishes the message "READY" to the processing_status
> > channel
> > > > on a Valkey server. Other processes listening to this channel will
> > > receive
> > > > the message and proceed with their execution.
> > > >
> > > >
> > > >
> > > > `
> > > >
> > > > wait_for_ready = ValkeyPubSubSensor(
> > > >
> > > >     task_id="wait_for_ready_message",
> > > >
> > > >     channel="processing_status",
> > > >
> > > >     valkey_conn_id="valkey_default",
> > > >
> > > >     timeout=600,
> > > >
> > > > )
> > > >
> > > > `
> > > >
> > > >
> > > >
> > > > Here, a sensor waits for the message on the same channel before
> > > > continuing. Once the message is received, the DAG can continue with
> > > further
> > > > tasks—such as triggering another DAG—in an event-driven manner.
> > > >
> > > >
> > > >
> > > > Given the community's interest in event-driven scheduling (AIP 82<
> > > > https://github.com/apache/airflow/issues/52712>) as well as concerns
> > > > about limited provider optionality (PR #52712<
> > > > https://github.com/apache/airflow/issues/52712>), a Valkey provider
> > > would
> > > > expand user choice.
> > > >
> > > >
> > > >
> > > > I’d be happy to begin working on the initial PR and documentation.
> > > >
> > > >
> > > >
> > > > Looking forward to feedback.
> > > >
> > > > Sean
> > > >
> > > >
> > >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org

Reply via email to