Hi Vikram,
Thanks for the feedback. I agree that helpers would be useful for users who
want to run data quality checks from existing tasks or operators, without
always needing to use a dedicated DQ operator.
I updated the PR to include
- importable helpers for running and persisting checks inside tasks or
any existing operators.
For asset integration, this PR does not require Airflow core changes. The
provider adds two asset-oriented helpers:
- asset_quality() to attach quality configuration to an asset. This lets
the asset carry the ruleset that is expected to be executed for it. In the
current implementation this is stored as provider-owned metadata in the
asset extra field.
- require_quality() to let downstream tasks gate on the latest quality
result from the triggering asset event.
Below is an example view of how assets can use data quality with
partition-aware producer and consumer Dags.
```
*Example ruleset:*
orders_quality_rules = RuleSet(
name="orders_quality",
rules=[
DQRule(
id="orders.amount_non_negative",
name="amount_non_negative",
check="min",
column="amount",
condition={"geq_to": 0},
),
],
)
raw_orders = asset_quality(
Asset("daily_raw_orders"),
ruleset=orders_quality_rules,
conn_id="warehouse",
table="raw_orders",
)
*Producer Dag:*
with DAG(
dag_id="daily_raw_orders",
schedule=CronPartitionTimetable("0 2 * * *", timezone="UTC"),
):
@task(outlets=[raw_orders])
def load_and_check():
context = get_current_context()
partition_key = context["partition_key"]
load_raw_orders(partition_key)
result = run_quality_checks(
ruleset=orders_quality_rules,
table="raw_orders",
partition_clause=f"ds = '{partition_key}'",
)
persist_quality_results(
result,
context=context,
outlets=[raw_orders],
)
load_and_check()
*Consumer Dag:*
with DAG(
dag_id="curate_daily_orders",
schedule=PartitionedAssetTimetable(assets=raw_orders),
):
quality_gate = require_quality(raw_orders, min_score=1.0)
@task
def curate():
context = get_current_context()
partition_key = context["partition_key"]
curate_orders(partition_key)
quality_gate >> curate()
```
The flow is:
1. A partitioned producer Dag runs and loads data for the current
partition.
2. The producer runs run_quality_checks() with a partition-scoped filter.
3. persist_quality_results() records the result and adds a DQ summary to
the produced asset event.
4. PartitionedAssetTimetable triggers the consumer Dag for that
partition.
5. require_quality() reads the DQ summary from the triggering asset
event.
6. The consumer processes the partition only if quality passes.
Hope this makes sense?
Regards,
Pavan
On Wed, Jul 15, 2026 at 6:47 PM Vikram Koka via dev <[email protected]>
wrote:
> Hey Pavan,
>
> Thanks for bringing this up and working on it. I strongly support the
> overall concept of the Data Quality provider.
>
> However, I have two concerns / questions:
> 1. I am unsure of the Operator approach shown in the PR.
> I am not certain that these SHOULD be separate Operators which result in
> separate tasks.
> If these are made available as Helper functions, they could be included in
> existing Operators.
> I am not saying they shouldn't be separate Tasks in Dags, but the Helper
> function approach would provide more optionality.
>
> 2. I am wondering how to integrate these with Asset Partitions
> I do believe this can be integrated with the Asset Partitions concept, but
> I am personally unclear on exactly how.
> I think adding an example illustrating this would be very helpful.
>
> Best regards,
> Vikram
>
>
> On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <
> [email protected]>
> wrote:
>
> > Thanks Kaxil..
> >
> > Thanks Everyone, consensus looks positive; I will merge the skeleton PR
> > first.
> >
> > Regards,
> > Pavan
> >
> > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <[email protected]> wrote:
> >
> > > +1 for the high-level need for such data quality checks. Thanks Pavan,
> > will
> > > review the PRs shortly.
> > >
> > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu <
> > [email protected]
> > > >
> > > wrote:
> > >
> > > > Thanks Bugra,
> > > >
> > > > I have updated the naming convention now.
> > > >
> > > > Pavan
> > > >
> > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <[email protected]
> >
> > > > wrote:
> > > >
> > > > > Thanks Pavan for bringing this together and starting the
> discussion!
> > > > > Sounds good! +1 on the idea.
> > > > >
> > > > > Harder than solving problems. Not a strong suggestion, but
> > > > > `common-dataquality` sounds more reasonable to me. It also adds the
> > > value
> > > > > of the `common` part, which provides the separation pattern Jarek
> > > > > mentioned. It gives a better understanding that it is a common
> > > offering.
> > > > >
> > > > > Best regards,
> > > > > Bugra Ozturk
> > > > >
> > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu <
> > > > > [email protected]>
> > > > > wrote:
> > > > >
> > > > > > Thanks Jarek, I agree that the separate provider approach offers
> > much
> > > > > more
> > > > > > flexibility for iterating on features and fixes.
> > > > > >
> > > > > > Naming is always hard :)
> > > > > >
> > > > > > Option 1: apache-airflow-providers-dataquality
> > > > > > Option 2: apache-airflow-providers-common-dataquality (This goes
> > > inside
> > > > > the
> > > > > > common providers folder we already have)
> > > > > >
> > > > > > So, I am up for either option :)
> > > > > >
> > > > > > have removed first short name `apache-airflow-providers-dq`.
> > > > > >
> > > > > > Thanks,
> > > > > > Pavan
> > > > > >
> > > > > >
> > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <[email protected]>
> > > wrote:
> > > > > >
> > > > > > > +1 Good design/ idea. No objections - dataquality is a good
> name
> > -
> > > > but
> > > > > I
> > > > > > > would also consider `common-dataquality" - even if it's longer,
> > it
> > > > > builds
> > > > > > > on the pattern we have already with common-ai. But not a
> blocker.
> > > > > > >
> > > > > > > I also think it's good to have it as a separate provider, even
> if
> > > it
> > > > > > gains
> > > > > > > traction for two reasons:
> > > > > > >
> > > > > > > a) ability to add features or fix issues independently from the
> > > core
> > > > > > > b) an explicit "optional" feature that is easy to promote
> > > > > > >
> > > > > > > I think what we saw with common is that people see airflow
> > already
> > > as
> > > > > too
> > > > > > > heavy - and "too many releases" sometimes, so quite
> > > > > counter-intuitively -
> > > > > > > by having separate providers adding features that "hook in"
> > > existing
> > > > > > > functionalities of core - we do not make airflow "heavier" and
> we
> > > do
> > > > > not
> > > > > > > force people to migrating to future newer versions to use new
> > > > features.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu <
> > > > > > > [email protected]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Amogh,
> > > > > > > >
> > > > > > > > Thanks for the feedback.
> > > > > > > >
> > > > > > > > I am happy to change the provider name to dataquality.
> > > > > > > >
> > > > > > > > Regarding the LLM-assisted features, the current PR does not
> > > > include
> > > > > > any
> > > > > > > > implementation. It only adds the SKILLS [1 ]and the reference
> > > > schema
> > > > > > for
> > > > > > > > the DQ Rule structure. Are you suggesting that I move this
> > SKILL
> > > > > > > > documentation to a separate PR?
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Pavan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai <
> > > [email protected]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Pavan,
> > > > > > > > >
> > > > > > > > > First of all, +1 to this.
> > > > > > > > >
> > > > > > > > > Now, few things:
> > > > > > > > >
> > > > > > > > > * On naming: dataquality over dq for me honestly. Our
> > existing
> > > > > > provider
> > > > > > > > > names spell things out
> > > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq is
> > > > > genuinely
> > > > > > > > > ambiguous outside context.
> > > > > > > > >
> > > > > > > > > * On scope: I also agree with Niko that #69413 is too large
> > for
> > > > one
> > > > > > > pass
> > > > > > > > &
> > > > > > > > > I am glad to see the
> > > > > > > > > backend/UI split already happening in #69575. Would also
> > > suggest
> > > > > > > keeping
> > > > > > > > > the LLM assisted rule
> > > > > > > > > generation pieces (*schema-based
> generate_rules_from_schema*)
> > > out
> > > > > of
> > > > > > > the
> > > > > > > > > initial provider PR entirely
> > > > > > > > > cos as I see it, its a separable capability and bundling it
> > > will
> > > > > slow
> > > > > > > > > review of the core DQRule or
> > > > > > > > > RuleSet or operator surface, which is the part that
> actually
> > > > needs
> > > > > > the
> > > > > > > > most
> > > > > > > > > detailed review.
> > > > > > > > >
> > > > > > > > > In short: go for it!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks & Regards,
> > > > > > > > > Amogh Desai
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu <
> > > > > > > > > [email protected]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > In the meantime, the PR is ready for review. Feel free to
> > > > review
> > > > > > and
> > > > > > > > > > provide any feedback.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Pavan
> > > > > > > > > >
> > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu <
> > > > > > > > > > [email protected]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Sorry, I forgot to add: the draft PR is here
> > > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's
> > still a
> > > > > WIP.
> > > > > > > > > > >
> > > > > > > > > > > some screenshots
> > > > > > > > > > >
> > > > > > >
> > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468
> > > > > > > > > :)
> > > > > > > > > > >
> > > > > > > > > > > Pavan
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu <
> > > > > > > > > > > [email protected]> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi Airflow community,
> > > > > > > > > > >>
> > > > > > > > > > >> I would like to start a discussion regarding a new
> > > provider:
> > > > > > > > > > >> apache-airflow-providers-dq.
> > > > > > > > > > >>
> > > > > > > > > > >> While Airflow already includes SQL check operators
> that
> > > many
> > > > > > users
> > > > > > > > > rely
> > > > > > > > > > >> on for data quality, this new provider builds on that
> > > > > foundation
> > > > > > > by
> > > > > > > > > > >> introducing DQRule and RuleSet objects, stable rule
> > > > identity,
> > > > > > > > > persisted
> > > > > > > > > > >> history, and direct connections to Airflow assets.
> This
> > > > > approach
> > > > > > > > makes
> > > > > > > > > > >> quality results easier to inspect over time, allows
> > > > downstream
> > > > > > > > > > consumers to
> > > > > > > > > > >> gate tasks based on recent quality results, and
> > provides a
> > > > > > unified
> > > > > > > > > > schema
> > > > > > > > > > >> for LLM-assisted workflows. Execution will continue to
> > > > utilize
> > > > > > > > > existing
> > > > > > > > > > >> DbApiHook connections.
> > > > > > > > > > >>
> > > > > > > > > > >> The initial version of the provider is intentionally
> > > > focused:
> > > > > > > > > > >>
> > > > > > > > > > >> - Declarative DQRule and RuleSet objects.
> > > > > > > > > > >> - DQCheckOperator and @task.dq_check.
> > > > > > > > > > >> - DbApiHook-based SQL checks, including built-in
> > checks
> > > > and
> > > > > > > > > > custom_sql.
> > > > > > > > > > >> - Persisted results for tasks, runs, and rules.
> > > > > > > > > > >> - A minimal Airflow UI plugin for viewing results
> and
> > > rule
> > > > > > > > history.
> > > > > > > > > > >> - Experimental asset helpers such as asset_quality()
> > and
> > > > > > > > > > >> require_quality().
> > > > > > > > > > >>
> > > > > > > > > > >> Regarding scope, this first iteration uses object
> > storage
> > > > only
> > > > > > to
> > > > > > > > > > persist
> > > > > > > > > > >> DQ results and history; checks are executed via
> database
> > > > > > > > connections.
> > > > > > > > > > >> Future iterations may include file or object-store
> based
> > > > > checks
> > > > > > > > (e.g.,
> > > > > > > > > > S3,
> > > > > > > > > > >> GCS) where Airflow runs quality rules against data
> > > directly.
> > > > > > > > > > >>
> > > > > > > > > > >> This proposal does not require changes to Airflow
> core.
> > > > Asset
> > > > > > > > support
> > > > > > > > > is
> > > > > > > > > > >> currently provider-owned metadata, with static
> > > configuration
> > > > > > > stored
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > >> asset and runtime summaries stored on asset events. If
> > the
> > > > > > > provider
> > > > > > > > > > gains
> > > > > > > > > > >> traction, we can discuss making Data Quality a
> > first-class
> > > > > > > component
> > > > > > > > > of
> > > > > > > > > > >> Airflow assets.
> > > > > > > > > > >>
> > > > > > > > > > >> This work also serves as a practical follow-up to the
> > data
> > > > > > quality
> > > > > > > > > > >> direction mentioned in AIP-99. Persisted history is
> > > valuable
> > > > > for
> > > > > > > > users
> > > > > > > > > > and
> > > > > > > > > > >> future LLM-assisted workflows, such as those from
> > > Anthropic
> > > > or
> > > > > > > > > > common.ai,
> > > > > > > > > > >> to understand rule performance and generate candidate
> > > rules
> > > > > > based
> > > > > > > on
> > > > > > > > > > schema
> > > > > > > > > > >> context.
> > > > > > > > > > >>
> > > > > > > > > > >> A rough pseudo-flow is provided below:
> > > > > > > > > > >>
> > > > > > > > > > >> seed_rules = RuleSet(
> > > > > > > > > > >> name="orders_quality",
> > > > > > > > > > >> rules=[
> > > > > > > > > > >> DQRule(name="order_id_not_null",
> > > check="null_count",
> > > > > > > > > > >> column="order_id", condition={"equal_to": 0}),
> > > > > > > > > > >> DQRule(name="amount_valid", check="min",
> > > > > > column="amount",
> > > > > > > > > > >> condition={"geq_to": 0}),
> > > > > > > > > > >> ],
> > > > > > > > > > >> )
> > > > > > > > > > >>
> > > > > > > > > > >> orders_asset = asset_quality(
> > > > > > > > > > >> Asset("orders"),
> > > > > > > > > > >> conn_id="warehouse",
> > > > > > > > > > >> table="orders",
> > > > > > > > > > >> ruleset=seed_rules,
> > > > > > > > > > >> )
> > > > > > > > > > >>
> > > > > > > > > > >> # Optional: common.ai / Anthropic provider can
> > generate a
> > > > > > RuleSet
> > > > > > > > > from
> > > > > > > > > > >> schema context.
> > > > > > > > > > >> generated_rules = generate_rules_from_schema(...)
> > > > > > > > > > >>
> > > > > > > > > > >> @task.dq_check(asset=orders_asset)
> > > > > > > > > > >> def check_orders(ruleset):
> > > > > > > > > > >> return ruleset
> > > > > > > > > > >>
> > > > > > > > > > >> checked_orders = check_orders(generated_rules)
> > > > > > > > > > >>
> > > > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset):
> > > > > > > > > > >> require_quality(orders_asset, min_score=0.95) >>
> > > > > > > > consume_orders()
> > > > > > > > > > >>
> > > > > > > > > > >> The UI remains deliberately minimal for this initial
> > > > release,
> > > > > > > > focusing
> > > > > > > > > > on
> > > > > > > > > > >> result and history inspection.
> > > > > > > > > > >>
> > > > > > > > > > >> You can view examples [1] of how it's integrated with
> > > > > > assets/llms.
> > > > > > > > > > >>
> > > > > > > > > > >> currently i named it providers
> > > > `apache-airflow-providers-dq`.
> > > > > if
> > > > > > > any
> > > > > > > > > > >> other preference likely with `dataquality`. Please let
> > me
> > > > know
> > > > > > if
> > > > > > > > you
> > > > > > > > > > have
> > > > > > > > > > >> a preference. naming is hard :)
> > > > > > > > > > >>
> > > > > > > > > > >> [1]:
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> Pavan
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
