+1 We should also consider extending this abstraction to Airflow publishing (such as notifier use cases). Haven't thought deeply about it, but the current design only includes a consumer hook without a producer hook, suggesting an opportunity to simplify the publishing side as well.
Thanks Shubham On 2025-02-12, 1:30 PM, "Vikram Koka" <vik...@astronomer.io.inva <mailto:vik...@astronomer.io.inva>LID> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Hey everyone, I created the following (very early) draft PR <https://github.com/apache/airflow/pull/46694> <https://github.com/apache/airflow/pull/46694>> to introduce and socialize the concept of a "Common Message Queue" abstraction, similar to the "Common SQL" and "Common IO" abstractions already present in Airflow. This "Common Message Queue" abstraction will be a provider package similar to the other "Common" packages and is intended to be a simple abstraction over Apache Kafka, Amazon SQS, Google PubSub, and others messaging frameworks. One of the key initial goals is to provide a simple DAG writing abstraction for integrating Event Driven Scheduling coming with Airflow 3 with streaming systems such as Kafka. This proposal was crafted in collaboration with Vincent Beck, who has been diligently working through AIP-82 as part of Airflow 3. At this stage, this is very much a work in progress draft and is intended to solicit feedback, rather than in any way a completed incarnation. Please feel free to comment on the PR directly or reach out to me over email or slack. Best regards, Vikram -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <mailto:vik...@astronomer.io> <https://www.astronomer.io/> <https://www.astronomer.io/>>
