Hey Pavan, Thanks for bringing this up and working on it. I strongly support the overall concept of the Data Quality provider.
However, I have two concerns / questions: 1. I am unsure of the Operator approach shown in the PR. I am not certain that these SHOULD be separate Operators which result in separate tasks. If these are made available as Helper functions, they could be included in existing Operators. I am not saying they shouldn't be separate Tasks in Dags, but the Helper function approach would provide more optionality. 2. I am wondering how to integrate these with Asset Partitions I do believe this can be integrated with the Asset Partitions concept, but I am personally unclear on exactly how. I think adding an example illustrating this would be very helpful. Best regards, Vikram On Mon, Jul 13, 2026 at 3:26 PM Pavankumar Gopidesu <[email protected]> wrote: > Thanks Kaxil.. > > Thanks Everyone, consensus looks positive; I will merge the skeleton PR > first. > > Regards, > Pavan > > On Mon, Jul 13, 2026 at 10:05 PM Kaxil Naik <[email protected]> wrote: > > > +1 for the high-level need for such data quality checks. Thanks Pavan, > will > > review the PRs shortly. > > > > On Fri, 10 Jul 2026 at 01:58, Pavankumar Gopidesu < > [email protected] > > > > > wrote: > > > > > Thanks Bugra, > > > > > > I have updated the naming convention now. > > > > > > Pavan > > > > > > On Wed, Jul 8, 2026 at 7:07 PM Buğra Öztürk <[email protected]> > > > wrote: > > > > > > > Thanks Pavan for bringing this together and starting the discussion! > > > > Sounds good! +1 on the idea. > > > > > > > > Harder than solving problems. Not a strong suggestion, but > > > > `common-dataquality` sounds more reasonable to me. It also adds the > > value > > > > of the `common` part, which provides the separation pattern Jarek > > > > mentioned. It gives a better understanding that it is a common > > offering. > > > > > > > > Best regards, > > > > Bugra Ozturk > > > > > > > > On Wed, Jul 8, 2026 at 7:15 PM Pavankumar Gopidesu < > > > > [email protected]> > > > > wrote: > > > > > > > > > Thanks Jarek, I agree that the separate provider approach offers > much > > > > more > > > > > flexibility for iterating on features and fixes. > > > > > > > > > > Naming is always hard :) > > > > > > > > > > Option 1: apache-airflow-providers-dataquality > > > > > Option 2: apache-airflow-providers-common-dataquality (This goes > > inside > > > > the > > > > > common providers folder we already have) > > > > > > > > > > So, I am up for either option :) > > > > > > > > > > have removed first short name `apache-airflow-providers-dq`. > > > > > > > > > > Thanks, > > > > > Pavan > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 12:47 PM Jarek Potiuk <[email protected]> > > wrote: > > > > > > > > > > > +1 Good design/ idea. No objections - dataquality is a good name > - > > > but > > > > I > > > > > > would also consider `common-dataquality" - even if it's longer, > it > > > > builds > > > > > > on the pattern we have already with common-ai. But not a blocker. > > > > > > > > > > > > I also think it's good to have it as a separate provider, even if > > it > > > > > gains > > > > > > traction for two reasons: > > > > > > > > > > > > a) ability to add features or fix issues independently from the > > core > > > > > > b) an explicit "optional" feature that is easy to promote > > > > > > > > > > > > I think what we saw with common is that people see airflow > already > > as > > > > too > > > > > > heavy - and "too many releases" sometimes, so quite > > > > counter-intuitively - > > > > > > by having separate providers adding features that "hook in" > > existing > > > > > > functionalities of core - we do not make airflow "heavier" and we > > do > > > > not > > > > > > force people to migrating to future newer versions to use new > > > features. > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 11:04 AM Pavankumar Gopidesu < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Amogh, > > > > > > > > > > > > > > Thanks for the feedback. > > > > > > > > > > > > > > I am happy to change the provider name to dataquality. > > > > > > > > > > > > > > Regarding the LLM-assisted features, the current PR does not > > > include > > > > > any > > > > > > > implementation. It only adds the SKILLS [1 ]and the reference > > > schema > > > > > for > > > > > > > the DQ Rule structure. Are you suggesting that I move this > SKILL > > > > > > > documentation to a separate PR? > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/gopidesupavan/airflow/blob/9dac869e30d7e1e35aa9297b3098f10667c42aba/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md > > > > > > > > > > > > > > Regards, > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 8, 2026 at 9:48 AM Amogh Desai < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > Hi Pavan, > > > > > > > > > > > > > > > > First of all, +1 to this. > > > > > > > > > > > > > > > > Now, few things: > > > > > > > > > > > > > > > > * On naming: dataquality over dq for me honestly. Our > existing > > > > > provider > > > > > > > > names spell things out > > > > > > > > (common.sql, openlineage, not abbreviated forms) and dq is > > > > genuinely > > > > > > > > ambiguous outside context. > > > > > > > > > > > > > > > > * On scope: I also agree with Niko that #69413 is too large > for > > > one > > > > > > pass > > > > > > > & > > > > > > > > I am glad to see the > > > > > > > > backend/UI split already happening in #69575. Would also > > suggest > > > > > > keeping > > > > > > > > the LLM assisted rule > > > > > > > > generation pieces (*schema-based generate_rules_from_schema*) > > out > > > > of > > > > > > the > > > > > > > > initial provider PR entirely > > > > > > > > cos as I see it, its a separable capability and bundling it > > will > > > > slow > > > > > > > > review of the core DQRule or > > > > > > > > RuleSet or operator surface, which is the part that actually > > > needs > > > > > the > > > > > > > most > > > > > > > > detailed review. > > > > > > > > > > > > > > > > In short: go for it! > > > > > > > > > > > > > > > > > > > > > > > > Thanks & Regards, > > > > > > > > Amogh Desai > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 6, 2026 at 9:35 PM Pavankumar Gopidesu < > > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > In the meantime, the PR is ready for review. Feel free to > > > review > > > > > and > > > > > > > > > provide any feedback. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:20 PM Pavankumar Gopidesu < > > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Sorry, I forgot to add: the draft PR is here > > > > > > > > > > https://github.com/apache/airflow/pull/69413; it's > still a > > > > WIP. > > > > > > > > > > > > > > > > > > > > some screenshots > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/pull/69413#issuecomment-4886311468 > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > Pavan > > > > > > > > > > > > > > > > > > > > On Sun, Jul 5, 2026 at 3:15 PM Pavankumar Gopidesu < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > >> Hi Airflow community, > > > > > > > > > >> > > > > > > > > > >> I would like to start a discussion regarding a new > > provider: > > > > > > > > > >> apache-airflow-providers-dq. > > > > > > > > > >> > > > > > > > > > >> While Airflow already includes SQL check operators that > > many > > > > > users > > > > > > > > rely > > > > > > > > > >> on for data quality, this new provider builds on that > > > > foundation > > > > > > by > > > > > > > > > >> introducing DQRule and RuleSet objects, stable rule > > > identity, > > > > > > > > persisted > > > > > > > > > >> history, and direct connections to Airflow assets. This > > > > approach > > > > > > > makes > > > > > > > > > >> quality results easier to inspect over time, allows > > > downstream > > > > > > > > > consumers to > > > > > > > > > >> gate tasks based on recent quality results, and > provides a > > > > > unified > > > > > > > > > schema > > > > > > > > > >> for LLM-assisted workflows. Execution will continue to > > > utilize > > > > > > > > existing > > > > > > > > > >> DbApiHook connections. > > > > > > > > > >> > > > > > > > > > >> The initial version of the provider is intentionally > > > focused: > > > > > > > > > >> > > > > > > > > > >> - Declarative DQRule and RuleSet objects. > > > > > > > > > >> - DQCheckOperator and @task.dq_check. > > > > > > > > > >> - DbApiHook-based SQL checks, including built-in > checks > > > and > > > > > > > > > custom_sql. > > > > > > > > > >> - Persisted results for tasks, runs, and rules. > > > > > > > > > >> - A minimal Airflow UI plugin for viewing results and > > rule > > > > > > > history. > > > > > > > > > >> - Experimental asset helpers such as asset_quality() > and > > > > > > > > > >> require_quality(). > > > > > > > > > >> > > > > > > > > > >> Regarding scope, this first iteration uses object > storage > > > only > > > > > to > > > > > > > > > persist > > > > > > > > > >> DQ results and history; checks are executed via database > > > > > > > connections. > > > > > > > > > >> Future iterations may include file or object-store based > > > > checks > > > > > > > (e.g., > > > > > > > > > S3, > > > > > > > > > >> GCS) where Airflow runs quality rules against data > > directly. > > > > > > > > > >> > > > > > > > > > >> This proposal does not require changes to Airflow core. > > > Asset > > > > > > > support > > > > > > > > is > > > > > > > > > >> currently provider-owned metadata, with static > > configuration > > > > > > stored > > > > > > > on > > > > > > > > > the > > > > > > > > > >> asset and runtime summaries stored on asset events. If > the > > > > > > provider > > > > > > > > > gains > > > > > > > > > >> traction, we can discuss making Data Quality a > first-class > > > > > > component > > > > > > > > of > > > > > > > > > >> Airflow assets. > > > > > > > > > >> > > > > > > > > > >> This work also serves as a practical follow-up to the > data > > > > > quality > > > > > > > > > >> direction mentioned in AIP-99. Persisted history is > > valuable > > > > for > > > > > > > users > > > > > > > > > and > > > > > > > > > >> future LLM-assisted workflows, such as those from > > Anthropic > > > or > > > > > > > > > common.ai, > > > > > > > > > >> to understand rule performance and generate candidate > > rules > > > > > based > > > > > > on > > > > > > > > > schema > > > > > > > > > >> context. > > > > > > > > > >> > > > > > > > > > >> A rough pseudo-flow is provided below: > > > > > > > > > >> > > > > > > > > > >> seed_rules = RuleSet( > > > > > > > > > >> name="orders_quality", > > > > > > > > > >> rules=[ > > > > > > > > > >> DQRule(name="order_id_not_null", > > check="null_count", > > > > > > > > > >> column="order_id", condition={"equal_to": 0}), > > > > > > > > > >> DQRule(name="amount_valid", check="min", > > > > > column="amount", > > > > > > > > > >> condition={"geq_to": 0}), > > > > > > > > > >> ], > > > > > > > > > >> ) > > > > > > > > > >> > > > > > > > > > >> orders_asset = asset_quality( > > > > > > > > > >> Asset("orders"), > > > > > > > > > >> conn_id="warehouse", > > > > > > > > > >> table="orders", > > > > > > > > > >> ruleset=seed_rules, > > > > > > > > > >> ) > > > > > > > > > >> > > > > > > > > > >> # Optional: common.ai / Anthropic provider can > generate a > > > > > RuleSet > > > > > > > > from > > > > > > > > > >> schema context. > > > > > > > > > >> generated_rules = generate_rules_from_schema(...) > > > > > > > > > >> > > > > > > > > > >> @task.dq_check(asset=orders_asset) > > > > > > > > > >> def check_orders(ruleset): > > > > > > > > > >> return ruleset > > > > > > > > > >> > > > > > > > > > >> checked_orders = check_orders(generated_rules) > > > > > > > > > >> > > > > > > > > > >> with DAG("orders_consumer", schedule=orders_asset): > > > > > > > > > >> require_quality(orders_asset, min_score=0.95) >> > > > > > > > consume_orders() > > > > > > > > > >> > > > > > > > > > >> The UI remains deliberately minimal for this initial > > > release, > > > > > > > focusing > > > > > > > > > on > > > > > > > > > >> result and history inspection. > > > > > > > > > >> > > > > > > > > > >> You can view examples [1] of how it's integrated with > > > > > assets/llms. > > > > > > > > > >> > > > > > > > > > >> currently i named it providers > > > `apache-airflow-providers-dq`. > > > > if > > > > > > any > > > > > > > > > >> other preference likely with `dataquality`. Please let > me > > > know > > > > > if > > > > > > > you > > > > > > > > > have > > > > > > > > > >> a preference. naming is hard :) > > > > > > > > > >> > > > > > > > > > >> [1]: > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/gopidesupavan/airflow/blob/52b447f7acfbae6bd8673e87a2b40098aee3e6fb/providers/dq/src/airflow/providers/dq/example_dags/ > > > > > > > > > >> > > > > > > > > > >> Thanks, > > > > > > > > > >> Pavan > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
