Hi @Jarek Potiuk <[email protected]> and <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=a+global+extension+importer+map+structurally+can+t+give+two+bundles+different+importers+for+the+same+extension+so+as+it+stands+the+foundation+can+t+support+the+resolution+model+the+aip+promises>@Tatiana Al-Chueyr <[email protected]> ,
Thank you so much for the detailed review and for supporting the scoped-down direction. I completely agree with your structural concerns regarding the merged foundation in PR #60127. Reconciling the intended design in the AIP with what currently exists in main is definitely the right move before calling for a VOTE again. Here is how I am addressing your points, many of which align with recent inline discussions on the AIP page: 1. Registry scope: You are right that a global extension -> importer map structurally cannot support the per-bundle resolution we need. Tatiana also brought this up in the inline comments. The intended end-state is indeed per-bundle instantiation with a strict bundle -> global -> default resolution order. I will refactor the DagImporterRegistry out of the singleton pattern to ensure the interface supports this scope. 2. *Archive* handling: I think we are aligned on using a composite ZipImporter. The AIP describes a generic archive importer that delegates to internal-format importers, which provides the flexibility needed for mixed-content archives (e.g., YAML and Python in the same zip). I will ensure the merged PythonDagImporter is updated to hand-over its hard-ownership of .zip files. 3. Cross-file dependency tracking: Tatiana also raised the staleness bug issue for shared configurations like defaults.yaml. <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=without+importer+declared+dependencies+a+change+to+a+shared+file+won+t+re+parse+its+dependents+a+staleness+bug> I have updated the DagImportResult dataclass in the AIP to include a file_dependencies: list[str] field. To clarify the interaction with the DAG Processor: the custom importer will declare these dependencies, and the core DAG Processor will cache them to monitor for modification time changes, triggering a re-parse and a new DAG version when necessary (keeping the importers stateless). 4. Security: This is a good point and something I hadn't explicitly defined yet. Ensuring users don't treat YAML as a "review-free" data payload is critical since resolving operators by classpath has the same power as arbitrary Python. <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=explicitly+that+a+declarative+dag+file+is+fully+code+equivalent+trust+a+yaml+importer+resolving+operators+by+classpath+has+the+same+power+as+arbitrary+python> I would love to take you up on your offer to help fill in these security blanks! I will add a dedicated "Security and Trust Model" section to the AIP for the constraints you mentioned, feel free to modify it. Overall, I would like to learn more about this if you don't mind, as this is my first Airflow community contribution and it would be great to fill in the knowledge blanks. 5. Safe mode: Yes, the "airflow + dag" string check is strictly Python-specific and will fail for YAML. <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=heuristic+file+contains+the+strings+airflow+dag+currently+lives+in+the+abstract+list_dag_files+but+it+s+python+specific+and+will+mis+handle+yaml+it+should+be+delegated+per+importer>The design on the AIP now clarifies that the safe_mode heuristic is delegated to individual importers by overriding the can_handle() method, allowing each format to define its own fast content-level checks. 6. Performance regression: I have added a section to the AIP's "Considerations" to include performance testing. This will include benchmarking basic, large-volume .py workloads to ensure the new registry indirection adds no measurable regression, <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=it+d+be+good+to+confirm+the+registry+indirection+adds+no+measurable+regression+on+the+plain+py+path+some+performance+tests+with+basic+big+numbers+of+dags+scenarios+would+be+great+to+do+as+part+of+the+implementation> alongside comparative benchmarks of native YAML vs. legacy generator patterns (e.g., dag-factory). How should we proceed with the discussion logistics? I can either copy the relevant points from this thread over to the main [DISCUSS] thread, or simply link this thread there as another reference. Once the feedback is incorporated and the AIP is ready to transition back to the voting phase, I can then open a new thread, perhaps titled [FINAL VOTE]. Let me know on your thoughts on this as I don't have strong preference. Kind regards, Dilnaz Amanzholova On Wed, Jul 8, 2026 at 3:42 PM Jarek Potiuk <[email protected]> wrote: > 100% on what Tatiana wrote :D > > On Wed, Jul 8, 2026 at 3:06 PM Tatiana Al-Chueyr Martins < > [email protected]> wrote: > >> Thanks a lot for being so responsive to the feedback on the AIP Confluent >> page, Dilnaz. Since yesterday, you've added file_dependencies to >> DagImportResult, introduced a DagSource class that carries the source >> language for the Code tab, and made the importer precedence order explicit >> (bundle explicit mapping > global config > defaults) - all great >> improvements. >> >> I agree with Jarek on reconciling the merged foundation with the AIP >> (particularly the per-bundle registry) before re-running the vote. And >> yeah, explicitly stating that a declarative DAG file carries the same >> trust >> level as the Python code is important - in our experience, users often >> think YAML is just data, so we should make this explicit. >> >> On the zip question, composite importers feel like the right path forward. >> Once we get non-Python formats treated as first-class citizens, it feels >> like the next natural step is mixed content. Delegating to inner importers >> handles that cleanly, whereas I feel that extending PythonDagImporter >> wouldn't scale as well. >> >> It does feel we should wrap up the bigger questions in [DISCUSS] - this >> will allow maintainers to VOTE on the proposed design with confidence, and >> the AIP will be stronger with it. With the pace I've observed since >> yesterday, it doesn't feel like we're far off - looking forward to seeing >> it back into VOTE. >> >> Kind regards, >> >> Tatiana >> >> On Wed, 8 Jul 2026 at 13:24, Jarek Potiuk <[email protected]> wrote: >> >> > Hi Dilnaz, and indeed - thanks for pushing this forward. >> > >> > The scoped-down "DAG importer" framing is much clearer, and >> > native non-Python DAGs are a genuinely good direction. +1 on the goal. >> > >> > I am happy to make comments and suggestions or even fill-in some blanks >> > (security especially) in the AIP proposal if you are fine with me doing >> it >> > - see >> > some of the comments below. >> > >> > Before I can vote on the design, I'd like to raise one structural >> concern >> > plus a couple of smaller ones. My main point is that the foundation >> already >> > merged in https://github.com/apache/airflow/pull/60127 >> > contradicts the design the AIP describes, and I think we >> > should reconcile the two before the vote closes rather than after. >> > >> > Some of these points are already explicitly stated, either in the AIP or >> > opened >> > questions raised by Tatiana - I just want to reiterate them here and >> see if >> > we can >> > reach consensus on the shape of those changes. >> > >> > Concretely, in airflow-core/src/airflow/dag_processing/importers/: >> > >> > 1. Registry scope — global singleton vs. per-bundle. The AIP describes >> > per-bundle importer mapping with a bundle -> global -> default >> resolution >> > order (and Tatiana's Q4/Q7 ask for the same). But the merged >> > DagImporterRegistry is a process-wide singleton keyed by extension, >> "last >> > registered wins, one importer per extension." A global >> extension->importer >> > map structurally can't give two bundles different importers for the same >> > extension, so as it stands the foundation can't support the resolution >> > model the AIP promises. If per-bundle is the intended end state, I >> think we >> > want that in the interface now, because swapping a singleton out later >> is >> > expensive. >> > >> > 2. Zip handling. The AIP describes a generic ZipImporter that delegates >> to >> > inner-format importers. The merged PythonDagImporter instead hard-owns >> both >> > .py and .zip, with a docstring saying YAML-in-zip would require >> extending >> > it >> > or writing a composite importer — the opposite of the AIP. I think we >> > should >> > decide on one or the other direction, I like the composite importer >> more - >> > even >> > if it is more complex, it allows for greater flexibility if we have >> > composite >> > importers. But I would love to hear what others think. >> > >> > 3. DagImportResult.file_dependencies is in the AIP dataclass but not in >> the >> > merged one — and this is more than cosmetic. Declarative DAGs routinely >> > share a defaults.yaml; the DFP decides what to re-parse from per-file >> > mtime, >> > so without importer-declared dependencies a change to a shared file >> won't >> > re-parse its dependents (a staleness bug). This is Tatiana's Q5, and I'd >> > treat it as a correctness requirement, not a nice-to-have. Worth >> spelling >> > out how it interacts with AIP-66's git-sha bundle versioning too. >> > >> > On security, I'd like the AIP to state explicitly that a declarative DAG >> > file is fully code-equivalent trust: a YAML importer resolving >> operators by >> > classpath has the same power as arbitrary Python. People tend to treat >> YAML >> > as "data" and .py as "code," and I want to make sure nobody builds a >> > "review-free YAML bundle" path on the back of this. I'd also like it >> stated >> > as a hard constraint that importers run only in the DAG processor, >> > in-process, under the same parse timeout, and never touch the metadata >> DB — >> > consistent with the AIP-66/AIP-72 boundary. And a quick confirmation >> that >> > the old "push DAGs over API / ingester" idea stays out of scope, since >> that >> > one genuinely changes the threat model. >> > >> > Again, for security, if you are happy with it, I would love to be able >> to >> > fill >> > all those security blanks - this can be described on high level in the >> AIP, >> > and when implementation progresses, it would turn into security model >> > updates (I'm also happy to help there). >> > >> > Two smaller things: >> > >> > - The safe_mode/might_contain_dag heuristic ("file contains the strings >> > airflow + dag") currently lives in the abstract list_dag_files, but it's >> > Python-specific and will mis-handle YAML — it should be delegated >> > per-importer (Tatiana's Q3). >> > >> > - It'd be good to confirm the registry indirection adds no measurable >> > regression on the plain .py path - some performance tests with basic >> > big-numbers-of-dags scenarios would be great to do as part of the >> > implementation. >> > >> > None of this is a blocker on the idea — I'm supportive. But given the >> open >> > design questions and that the vote has been quiet, It reaffirms we treat >> > this >> > as still in [DISCUSS], reconcile the AIP with the merged foundation >> > (especially point 1), and re-run the vote once the interface reflects >> the >> > intended end state. >> > >> > Happy to help review the per-bundle registry shape. >> > >> > Best, >> > Jarek >> > >> > On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < >> > [email protected]> wrote: >> > >> > > Hi Dilnaz, >> > > >> > > Thanks a lot for pushing this forward, and for the updates after the >> > > earlier feedback - AIP 85 looks much clearer than before. >> > > >> > > I'm one of the maintainers of DAG Factory ( >> > > https://github.com/astronomer/dag-factory), and I like the direction >> the >> > > AIP is taking. I agree with the pain points Igor raised earlier; it >> would >> > > be great to support non-Python DAGs natively in Airflow, rather than >> > > converting them as we currently do in DAG Factory. >> > > >> > > I've reviewed the AIP page and left some inline comments & questions. >> > Since >> > > the authors are already responding on the AIP page, happy to continue >> the >> > > detailed discussion there - sharing the summary here mainly for >> > visibility: >> > > >> > > 1. *Location of schemas and importers*: Where do we see the schemas >> and >> > > actual importers defined - and how do we plan to govern them? It >> would >> > be >> > > important to align, for instance, on how non-primitive operator >> arguments >> > > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) and >> on >> > > the security aspects of loading arbitrary classes from YAML. >> > > >> > > 2. *Validation contract*: If users define unknown or unsupported keys >> and >> > > configurations, should we fail loudly with DagImportErrors referencing >> > > files/lines like the Python DAG import errors? It would be great if we >> > > didn't just skip them. >> > > >> > > 3.* Decision on which files to parse: *As raised by TP on >> > > < >> > > >> > >> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo >> > > > >> > > the >> > > Google >> > > < >> > > >> > >> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo >> > > > >> > > Doc >> > > < >> > > >> > >> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo >> > > > >> > > , I feel that we should have additional criteria to decide which >> files to >> > > treat as a DAG - and not rely only on the file extension. As an >> example, >> > > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, K8s >> > > manifests. Would it make sense to have something similar to the Python >> > > content-level heuristic ("airflow + dag" string check)? >> > > >> > > 4. *Different importers for the same file extension*: Can/should we >> > support >> > > different importers for the same file extension? It would be great if >> the >> > > architecture could support overriding the importer per file. >> > > >> > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely >> > > single-file. They can share default definitions, for example: >> > > >> https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/. >> > > Could importers declare file dependencies in DagImportResult, so the >> > > processor can use them? >> > > >> > > 6. *Code tab UI*: On get_source_code(), could we have a parameter >> related >> > > to the file language, which could be used by the Code tab (it is >> > currently >> > > hard-coded with Python)? How are we planning to resolve ZipImporter >> and >> > > other transformed sources in the UI? >> > > >> > > 7. *Precedence*: What should be the precedence (per-bundle vs global >> vs >> > > default importers)? >> > > >> > > For the performance questions raised earlier in the thread, we could >> use >> > > DAG Factory workloads as a baseline, comparing the >> non-built-in-Airflow >> > > Python layer against a native YAML importer. >> > > >> > > Our team is very keen to contribute to the YAML contract definition. >> > > >> > > Kind regards, >> > > >> > > Tatiana >> > > >> > > >> > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < >> > > [email protected]> wrote: >> > > >> > > > Hi all, I am following up to request your feedback on AIP-85: DAG >> > > importer. >> > > > I attempted to present this at the last Airflow dev call, but the >> > agenda >> > > > was full, and today’s meeting was unfortunately cancelled. As it has >> > been >> > > > some time since this AIP was submitted for review, I would greatly >> > > > appreciate it if you could share your thoughts or any remaining >> > concerns >> > > > here. I understand there are many active discussions, but your input >> > > would >> > > > be invaluable in helping move this proposal forward. Thanks, Dilnaz >> > > > Amanzholova >> > > > >> > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova < >> [email protected] >> > > >> > > > wrote: >> > > > >> > > > > Hey Kaxil, >> > > > > >> > > > > Thanks for the clarification. I have decided to attend, as the >> next >> > > > > meeting is not for another two weeks. I have added the AIP to the >> > > agenda. >> > > > > >> > > > > Best regards, >> > > > > Dilnaz Amanzholova >> > > > > >> > > > > >> > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik <[email protected]> >> > > wrote: >> > > > > >> > > > >> Hey not required at all. >> > > > >> >> > > > >> This is just in case you wanted to solicit feedback live on the >> call >> > > > with >> > > > >> other Airflow devs. Sometimes it is easier that way -- but not >> > > > mandatory. >> > > > >> >> > > > >> More details on the dev calls: >> > > > >> >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call >> > > if >> > > > >> you decide about presenting it next time. >> > > > >> >> > > > >> Regards, >> > > > >> Kaxil >> > > > >> >> > > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova < >> > [email protected]> >> > > > >> wrote: >> > > > >> >> > > > >>> Hi Kaxil, >> > > > >>> >> > > > >>> I was not aware there was a regular call. Is it required to >> present >> > > the >> > > > >>> AIP before voting? >> > > > >>> >> > > > >>> Kind regards, >> > > > >>> Dilnaz Amanzholova >> > > > >>> >> > > > >>> >> > > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik <[email protected] >> > >> > > > wrote: >> > > > >>> >> > > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on the >> next >> > > dev >> > > > >>>> call? >> > > > >>>> >> > > > >>>> >> > > > >>>> >> > > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < >> > > > >>>> [email protected]> wrote: >> > > > >>>> >> > > > >>>>> Hi all, >> > > > >>>>> >> > > > >>>>> I am following up on AIP-85: DAG importer. All previous >> concerns >> > > have >> > > > >>>>> been >> > > > >>>>> addressed, and the content has been updated on the AIP >> Confluence >> > > > page. >> > > > >>>>> >> > > > >>>>> Please review the latest updates at your earliest >> convenience. I >> > > > would >> > > > >>>>> appreciate your feedback and am available to answer any >> further >> > > > >>>>> questions >> > > > >>>>> you may have. >> > > > >>>>> >> > > > >>>>> Best regards, >> > > > >>>>> Dilnaz Amanzholova >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < >> > > > [email protected]> >> > > > >>>>> wrote: >> > > > >>>>> >> > > > >>>>> > Hi all, >> > > > >>>>> > >> > > > >>>>> > I’m calling vote on AIP-85: DAG importer >> > > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg >> > > > >>>>> > >> > > > >>>>> > You can also review the design document here >> > > > >>>>> > < >> > > > >>>>> >> > > > >> > > >> > >> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 >> > > > >>>>> > >> > > > >>>>> > . >> > > > >>>>> > >> > > > >>>>> > Current discussion thread (after scoping down): >> > > > >>>>> > >> > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt >> > > > >>>>> > Original discussion thread (with wider "extendable DAG >> parsing >> > > > >>>>> controls" >> > > > >>>>> > title): >> > > > >>>>> >> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl >> > > > >>>>> > >> > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th >> June >> > > 2026, >> > > > >>>>> at >> > > > >>>>> > 10:00 UTC. >> > > > >>>>> > >> > > > >>>>> > Everyone is encouraged to vote, but only PMC members and >> > > > Committers' >> > > > >>>>> votes >> > > > >>>>> > are considered binding. Please vote accordingly. >> > > > >>>>> > >> > > > >>>>> > [ ] +1 Approve >> > > > >>>>> > [ ] +0 no opinion >> > > > >>>>> > [ ] -1 disapprove with the reason >> > > > >>>>> > >> > > > >>>>> > Kind regards, >> > > > >>>>> > Dilnaz Amanzholova >> > > > >>>>> > >> > > > >>>>> >> > > > >>>> >> > > > >> > > >> > > >> > > -- >> > > Tatiana Al-Chueyr >> > > >> > >> >> >> -- >> Tatiana Al-Chueyr >> >
