Hey everyone,
Thank you for attending the dev call on the 16th. I updated
our meeting notes on the Airflow wiki and the link for those notes is here
<https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304589/2026-07-16+Dev+Call+Minutes>
To everyone who attended the meeting, please check the summary and add
anything I may have missed. For those who could not join, please let us
know if you disagree with anything discussed and agreed upon in
the meeting. Also, please ask questions if something is unclear.
Our next meeting is scheduled for Thursday, the 30th of July at the same
time i.e. for 8 a.m. Pacific Time.
If you would like to add anything to the agenda or if I missed anything
from the last call, please let me know
Best regards,
Vikram
--
Below is the summary from the call:
- Note: Dilnaz Amanzholova was unable to attend. The AIP-85 agenda item
was carried forward again; Dilnaz noted that the AIP has been significantly
updated based on prior feedback and welcomes review and comments.
- *Catch-up on action items from last call:*
- None noted.
- *Airflow 3.3 Development Updates:*
- *3.3.1 Release update (Rahul Vats)*
- Rahul said that RC1 is targeted for August 3rd, with GA wrapping
up around August 10th. This gives about 10-15 days for the
release before
the Airflow Summit.
- Currently 61 open items tagged for this milestone, which need to
be worked on through before the release.
- Elad noted that August will also be slow for mailing list
discussions, code review, and everything else given Summit
prep, and asked
contributors to be less demanding during that period.
- The same applies to providers: critical provider changes should
be submitted before August.
- Elad flagged that he is not currently on the release manager
roster but is available for ad hoc releases or to step in if needed.
- Rahul noted it would be good to align a provider release just
before 3.3.1 so that all constraints are up to date. Elad
confirmed he can
do an ad hoc provider release if needed.
- *Common AI Provider (Vikram Koka / Pavan Kumar)*
- Vikram shared that Kaxil has been driving most of the work on
this and that Pavan is also heavily involved.
- Vikram said that the provider is currently at v0.6.0, with a
target of reaching 1.0 before the Summit.
- The delay to 1.0 is intentional: the team is focused on
stabilizing interfaces across OpenAI, Anthropic, and the
common provider
before locking the API.
- There is significant overlap in users combining the OpenAI
SDK, Anthropic agent SDK, and the common provider, making interface
consistency critical.
- AIP-105 resumable operators are also consuming this work,
with a PR from Amogh recently merged using the provider.
- Vikram also confirmed that this provider was being used in
production already with no major complaints so far, but additional
interaction patterns are still coming in and being incorporated.
- Pavan added that he was looking at adding Strands SDK support as
well.
- Elad shared in chat that the community is already contributing
additional capabilities to the provider, referencing PR #68847
<https://github.com/apache/airflow/pull/68847>.
- Vikram requested the community for feedback on usage patterns
and anything they feel should be addressed before 1.0.
- *Data Quality Provider demo (Pavan Kumar)*
- Pavan demonstrated a new data quality provider built around a
unified rule interface.
- *Core capabilities demonstrated:*
- Define rule sets as objects and attach them to asset
definitions via a helper (asset_quality).
- Execute quality checks at the task level when the asset is
triggered; results show pass/fail per rule.
- Quality scores are carried forward so that consumer DAGs can
gate on a minimum score before proceeding, using a
require_quality helper.
- A plugin showed the full execution history per task. The
backend used was object store with no DB dependency.
- An LLM skill was included in the examples. The rule schema
was generated by an LLM based on data structure.
- Brent confirmed in chat that asset-level plugins will be added
soon, which would allow quality scores to be surfaced in the
asset view as
well.
- *Discussion on scope:*
- Vikram raised a distinction between two categories of quality
checks that have different personas and different
recommended responses:
- Schema and structural validation: the entire data asset is
invalid (e.g. bad vendor data), the pipeline should
fail immediately, and
the data engineer is the relevant persona. This is
similar to the existing
LLM schema check operator in the Common AI provider.
- Business rule validation: individual items fail specific
rules (e.g. a contract value is too high, a date is in
the past), results
should be stored to object store for business owner
review, and this is
genuinely new capability not covered by existing tooling.
- Vikram noted the current provider appeared to cover both and
suggested keeping schema checks in the existing operators
and scoping this
provider to business and content validation.
- Pavan noted the current implementation is SQL-based and that
schema comparison is also possible within the same
framework, but agreed
the distinction is worth discussing further.
- *Action item: Vikram and Pavan to discuss scope offline*,
specifically whether schema/structural checks belong here or in the
existing LLM schema check operator.
- *Discussion Topics:*
- *Kafka provider: Event state listener demo (Christos Bisias)*
- Christos noted that the PR (#68082
<https://github.com/apache/airflow/pull/68082>) had already been
merged before this call, but offered to demo it given it had
been deferred
twice.
- *What the PR adds:*
- A listener under the Kafka provider that publishes a Kafka
message on every DAG and task instance state change.
- Configurable: can be scoped to specific DAGs or tasks via an
allow list; off by default.
- Extension to the Kafka connection config to accept Python
package callbacks resolved by string reference.
- *Demo:* Two DAGs with cross-DAG dependencies implemented via
Kafka messages, using the existing WaitMessageSensor. Messages
showed fields including task instance state, DAG ID, task ID,
and run ID.
- *Discussion: Vikram asked how does this differed from asset
watchers and if Vincent had reviewed this PR*
- Christos clarified that asset watchers sit on the consumer
side; this feature sits on the producer side, emitting
messages on every
state change event within the Airflow cluster.
- Vincent confirmed he had not followed the demo fully and
would review the PR to better understand.
- Ash clarified the key distinction from his understanding of
the demo:
- For DAG authors, asset watchers would continue to be the
preferred mechanism.
- This feature is more suited to system monitoring, for
example, an external observability tool already
listening on Kafka that
wants to react to Airflow state changes, or monitoring
multiple Airflow
instances across deployments.
- Vikram agreed, framing it as analogous to system monitoring
via OTEL: targeted at deployment admins and external
systems rather than
DAG authors.
- Elad raised the core concern: if the 20 people on this call
are confused about when to use this versus asset watchers,
users will be
far more confused. The documentation must make it
immediately clear, and
clarified that it should be in two sentences: what problem
this solves and
how it differs from asset watchers.
- *Terminology concerns:*
- The name "Kafka listener" is confusing because it implies the
feature listens to Kafka, when it actually produces to Kafka.
- "Listener" is an internal Airflow term (same pattern as
OpenLineage) and is not meaningful to most users.
- The config section is also named "Kafka listener," which
compounds the confusion.
- Jake agreed in chat that the "listener" term should not
appear in user-facing docs.
- Ash recommended avoiding the word "listener" entirely in
provider-facing documentation.
- *Action item: Christos to open an issue or PR* to fix the
terminology and documentation before the next provider
release, to avoid
introducing backward compatibility issues.
- Ash suggested creating a PR or issue and flagging it to the
dev list rather than starting a full mailing list thread.
- *Backward compatibility and 2.11:*
- Shahar raised a broader question about the current stance on
Airflow 2.11 backward compatibility for providers. Elad
clarified that 2.11
support has already exceeded the originally promised
deadline and is now
best-effort.
- Shahar noted that Claude had flagged a potential 2.11 issue
in the Kafka provider. A hook that produces the skipped
event does not
exist in 2.11. Elad confirmed that if this causes a
problem, bumping the
Kafka provider minimum version is the right approach.
- Vikram closed by thanking Christos for the demo and apologizing
that the feedback came after the PR was already merged.
- Vincent committed in chat to reviewing the PR the same day or
next.
Vikram Koka
Chief Strategy Officer
Email: [email protected]
<https://www.astronomer.io/>
