Hey everyone, I created the following (very early) draft PR <https://github.com/apache/airflow/pull/46694> to introduce and socialize the concept of a "Common Message Queue" abstraction, similar to the "Common SQL" and "Common IO" abstractions already present in Airflow.
This "Common Message Queue" abstraction will be a provider package similar to the other "Common" packages and is intended to be a simple abstraction over Apache Kafka, Amazon SQS, Google PubSub, and others messaging frameworks. One of the key initial goals is to provide a simple DAG writing abstraction for integrating Event Driven Scheduling coming with Airflow 3 with streaming systems such as Kafka. This proposal was crafted in collaboration with Vincent Beck, who has been diligently working through AIP-82 as part of Airflow 3. At this stage, this is very much a work in progress draft and is intended to solicit feedback, rather than in any way a completed incarnation. Please feel free to comment on the PR directly or reach out to me over email or slack. Best regards, Vikram -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
