Hi Dilnaz, Thanks for your continued work refining the proposal. I'm looking forward to the DagImporter serving as the interface for processing native DAG definitions for Lang-SDK (Java, Go, and the upcoming TypeScript).
I have one final question about the DagImporter interface. I left a comment on the Confluence doc yesterday, but I'm raising it here as well for visibility. Would it make sense to collapse supported_extensions() and can_handle() into list_dag_files()? Extension matching alone seems insufficient: for normal Python DAGs, not all Python files are DAG files, and it's similar for the Java-SDK -- not all JARs are valid Java-SDK entrypoint files. IIUC, list_dag_files() should be the single entrypoint that yields the valid DAG files for further processing. Since the files it yields should be valid anyway, I don't think we'd need can_handle() as a separate public interface. Happy to hear your thoughts on whether this makes sense, or if there's a reason to keep them separate that I'm missing. Thanks! Best regards, Jason On Mon, Jul 13, 2026 at 11:31 PM Dilnaz Amanzholova via dev < [email protected]> wrote: > Hi @Jarek Potiuk <[email protected]> and > < > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=a+global+extension+importer+map+structurally+can+t+give+two+bundles+different+importers+for+the+same+extension+so+as+it+stands+the+foundation+can+t+support+the+resolution+model+the+aip+promises > >@Tatiana > Al-Chueyr <[email protected]> , > > Thank you so much for the detailed review and for supporting the > scoped-down direction. I completely agree with your structural concerns > regarding the merged foundation in PR #60127. Reconciling the intended > design in the AIP with what currently exists in main is definitely the > right move before calling for a VOTE again. > > Here is how I am addressing your points, many of which align with recent > inline discussions on the AIP page: > > 1. Registry scope: You are right that a global extension -> importer map > structurally cannot support the per-bundle resolution we need. Tatiana also > brought this up in the inline comments. The intended end-state is indeed > per-bundle instantiation with a strict bundle -> global -> default > resolution > order. I will refactor the DagImporterRegistry out of the singleton pattern > to ensure the interface supports this scope. > > 2. *Archive* handling: I think we are aligned on using a composite > ZipImporter. The AIP describes a generic archive importer that delegates to > internal-format importers, which provides the flexibility needed for > mixed-content archives (e.g., YAML and Python in the same zip). I will > ensure the merged PythonDagImporter is updated to hand-over its > hard-ownership of .zip files. > > 3. Cross-file dependency tracking: Tatiana also raised the staleness bug > issue for shared configurations like defaults.yaml. > < > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=without+importer+declared+dependencies+a+change+to+a+shared+file+won+t+re+parse+its+dependents+a+staleness+bug > > > I > have updated the DagImportResult dataclass in the AIP to include a > file_dependencies: > list[str] field. To clarify the interaction with the DAG Processor: the > custom importer will declare these dependencies, and the core DAG Processor > will cache them to monitor for modification time changes, triggering a > re-parse and a new DAG version when necessary (keeping the importers > stateless). > > 4. Security: This is a good point and something I hadn't explicitly defined > yet. Ensuring users don't treat YAML as a "review-free" data payload is > critical since resolving operators by classpath has the same power as > arbitrary Python. > < > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=explicitly+that+a+declarative+dag+file+is+fully+code+equivalent+trust+a+yaml+importer+resolving+operators+by+classpath+has+the+same+power+as+arbitrary+python > > > I > would love to take you up on your offer to help fill in these security > blanks! I will add a dedicated "Security and Trust Model" section to the > AIP for the constraints you mentioned, feel free to modify it. Overall, I > would like to learn more about this if you don't mind, as this is my first > Airflow community contribution and it would be great to fill in the > knowledge blanks. > > 5. Safe mode: Yes, the "airflow + dag" string check is strictly > Python-specific and will fail for YAML. > < > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=heuristic+file+contains+the+strings+airflow+dag+currently+lives+in+the+abstract+list_dag_files+but+it+s+python+specific+and+will+mis+handle+yaml+it+should+be+delegated+per+importer > >The > design on the AIP now clarifies that the safe_mode heuristic is delegated > to individual importers by overriding the can_handle() method, allowing > each format to define its own fast content-level checks. > > 6. Performance regression: I have added a section to the AIP's > "Considerations" to include performance testing. This will include > benchmarking basic, large-volume .py workloads to ensure the new registry > indirection adds no measurable regression, > < > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=it+d+be+good+to+confirm+the+registry+indirection+adds+no+measurable+regression+on+the+plain+py+path+some+performance+tests+with+basic+big+numbers+of+dags+scenarios+would+be+great+to+do+as+part+of+the+implementation > > > alongside > comparative benchmarks of native YAML vs. legacy generator patterns (e.g., > dag-factory). > > How should we proceed with the discussion logistics? I can either copy the > relevant points from this thread over to the main [DISCUSS] thread, or > simply link this thread there as another reference. Once the feedback is > incorporated and the AIP is ready to transition back to the voting phase, I > can then open a new thread, perhaps titled [FINAL VOTE]. > > Let me know on your thoughts on this as I don't have strong preference. > > Kind regards, > Dilnaz Amanzholova > > > > On Wed, Jul 8, 2026 at 3:42 PM Jarek Potiuk <[email protected]> wrote: > > > 100% on what Tatiana wrote :D > > > > On Wed, Jul 8, 2026 at 3:06 PM Tatiana Al-Chueyr Martins < > > [email protected]> wrote: > > > >> Thanks a lot for being so responsive to the feedback on the AIP > Confluent > >> page, Dilnaz. Since yesterday, you've added file_dependencies to > >> DagImportResult, introduced a DagSource class that carries the source > >> language for the Code tab, and made the importer precedence order > explicit > >> (bundle explicit mapping > global config > defaults) - all great > >> improvements. > >> > >> I agree with Jarek on reconciling the merged foundation with the AIP > >> (particularly the per-bundle registry) before re-running the vote. And > >> yeah, explicitly stating that a declarative DAG file carries the same > >> trust > >> level as the Python code is important - in our experience, users often > >> think YAML is just data, so we should make this explicit. > >> > >> On the zip question, composite importers feel like the right path > forward. > >> Once we get non-Python formats treated as first-class citizens, it feels > >> like the next natural step is mixed content. Delegating to inner > importers > >> handles that cleanly, whereas I feel that extending PythonDagImporter > >> wouldn't scale as well. > >> > >> It does feel we should wrap up the bigger questions in [DISCUSS] - this > >> will allow maintainers to VOTE on the proposed design with confidence, > and > >> the AIP will be stronger with it. With the pace I've observed since > >> yesterday, it doesn't feel like we're far off - looking forward to > seeing > >> it back into VOTE. > >> > >> Kind regards, > >> > >> Tatiana > >> > >> On Wed, 8 Jul 2026 at 13:24, Jarek Potiuk <[email protected]> wrote: > >> > >> > Hi Dilnaz, and indeed - thanks for pushing this forward. > >> > > >> > The scoped-down "DAG importer" framing is much clearer, and > >> > native non-Python DAGs are a genuinely good direction. +1 on the goal. > >> > > >> > I am happy to make comments and suggestions or even fill-in some > blanks > >> > (security especially) in the AIP proposal if you are fine with me > doing > >> it > >> > - see > >> > some of the comments below. > >> > > >> > Before I can vote on the design, I'd like to raise one structural > >> concern > >> > plus a couple of smaller ones. My main point is that the foundation > >> already > >> > merged in https://github.com/apache/airflow/pull/60127 > >> > contradicts the design the AIP describes, and I think we > >> > should reconcile the two before the vote closes rather than after. > >> > > >> > Some of these points are already explicitly stated, either in the AIP > or > >> > opened > >> > questions raised by Tatiana - I just want to reiterate them here and > >> see if > >> > we can > >> > reach consensus on the shape of those changes. > >> > > >> > Concretely, in airflow-core/src/airflow/dag_processing/importers/: > >> > > >> > 1. Registry scope — global singleton vs. per-bundle. The AIP describes > >> > per-bundle importer mapping with a bundle -> global -> default > >> resolution > >> > order (and Tatiana's Q4/Q7 ask for the same). But the merged > >> > DagImporterRegistry is a process-wide singleton keyed by extension, > >> "last > >> > registered wins, one importer per extension." A global > >> extension->importer > >> > map structurally can't give two bundles different importers for the > same > >> > extension, so as it stands the foundation can't support the resolution > >> > model the AIP promises. If per-bundle is the intended end state, I > >> think we > >> > want that in the interface now, because swapping a singleton out later > >> is > >> > expensive. > >> > > >> > 2. Zip handling. The AIP describes a generic ZipImporter that > delegates > >> to > >> > inner-format importers. The merged PythonDagImporter instead hard-owns > >> both > >> > .py and .zip, with a docstring saying YAML-in-zip would require > >> extending > >> > it > >> > or writing a composite importer — the opposite of the AIP. I think we > >> > should > >> > decide on one or the other direction, I like the composite importer > >> more - > >> > even > >> > if it is more complex, it allows for greater flexibility if we have > >> > composite > >> > importers. But I would love to hear what others think. > >> > > >> > 3. DagImportResult.file_dependencies is in the AIP dataclass but not > in > >> the > >> > merged one — and this is more than cosmetic. Declarative DAGs > routinely > >> > share a defaults.yaml; the DFP decides what to re-parse from per-file > >> > mtime, > >> > so without importer-declared dependencies a change to a shared file > >> won't > >> > re-parse its dependents (a staleness bug). This is Tatiana's Q5, and > I'd > >> > treat it as a correctness requirement, not a nice-to-have. Worth > >> spelling > >> > out how it interacts with AIP-66's git-sha bundle versioning too. > >> > > >> > On security, I'd like the AIP to state explicitly that a declarative > DAG > >> > file is fully code-equivalent trust: a YAML importer resolving > >> operators by > >> > classpath has the same power as arbitrary Python. People tend to treat > >> YAML > >> > as "data" and .py as "code," and I want to make sure nobody builds a > >> > "review-free YAML bundle" path on the back of this. I'd also like it > >> stated > >> > as a hard constraint that importers run only in the DAG processor, > >> > in-process, under the same parse timeout, and never touch the metadata > >> DB — > >> > consistent with the AIP-66/AIP-72 boundary. And a quick confirmation > >> that > >> > the old "push DAGs over API / ingester" idea stays out of scope, since > >> that > >> > one genuinely changes the threat model. > >> > > >> > Again, for security, if you are happy with it, I would love to be able > >> to > >> > fill > >> > all those security blanks - this can be described on high level in the > >> AIP, > >> > and when implementation progresses, it would turn into security model > >> > updates (I'm also happy to help there). > >> > > >> > Two smaller things: > >> > > >> > - The safe_mode/might_contain_dag heuristic ("file contains the > strings > >> > airflow + dag") currently lives in the abstract list_dag_files, but > it's > >> > Python-specific and will mis-handle YAML — it should be delegated > >> > per-importer (Tatiana's Q3). > >> > > >> > - It'd be good to confirm the registry indirection adds no measurable > >> > regression on the plain .py path - some performance tests with basic > >> > big-numbers-of-dags scenarios would be great to do as part of the > >> > implementation. > >> > > >> > None of this is a blocker on the idea — I'm supportive. But given the > >> open > >> > design questions and that the vote has been quiet, It reaffirms we > treat > >> > this > >> > as still in [DISCUSS], reconcile the AIP with the merged foundation > >> > (especially point 1), and re-run the vote once the interface reflects > >> the > >> > intended end state. > >> > > >> > Happy to help review the per-bundle registry shape. > >> > > >> > Best, > >> > Jarek > >> > > >> > On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < > >> > [email protected]> wrote: > >> > > >> > > Hi Dilnaz, > >> > > > >> > > Thanks a lot for pushing this forward, and for the updates after the > >> > > earlier feedback - AIP 85 looks much clearer than before. > >> > > > >> > > I'm one of the maintainers of DAG Factory ( > >> > > https://github.com/astronomer/dag-factory), and I like the > direction > >> the > >> > > AIP is taking. I agree with the pain points Igor raised earlier; it > >> would > >> > > be great to support non-Python DAGs natively in Airflow, rather than > >> > > converting them as we currently do in DAG Factory. > >> > > > >> > > I've reviewed the AIP page and left some inline comments & > questions. > >> > Since > >> > > the authors are already responding on the AIP page, happy to > continue > >> the > >> > > detailed discussion there - sharing the summary here mainly for > >> > visibility: > >> > > > >> > > 1. *Location of schemas and importers*: Where do we see the schemas > >> and > >> > > actual importers defined - and how do we plan to govern them? It > >> would > >> > be > >> > > important to align, for instance, on how non-primitive operator > >> arguments > >> > > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) > and > >> on > >> > > the security aspects of loading arbitrary classes from YAML. > >> > > > >> > > 2. *Validation contract*: If users define unknown or unsupported > keys > >> and > >> > > configurations, should we fail loudly with DagImportErrors > referencing > >> > > files/lines like the Python DAG import errors? It would be great if > we > >> > > didn't just skip them. > >> > > > >> > > 3.* Decision on which files to parse: *As raised by TP on > >> > > < > >> > > > >> > > >> > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > >> > > > > >> > > the > >> > > Google > >> > > < > >> > > > >> > > >> > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > >> > > > > >> > > Doc > >> > > < > >> > > > >> > > >> > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > >> > > > > >> > > , I feel that we should have additional criteria to decide which > >> files to > >> > > treat as a DAG - and not rely only on the file extension. As an > >> example, > >> > > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, > K8s > >> > > manifests. Would it make sense to have something similar to the > Python > >> > > content-level heuristic ("airflow + dag" string check)? > >> > > > >> > > 4. *Different importers for the same file extension*: Can/should we > >> > support > >> > > different importers for the same file extension? It would be great > if > >> the > >> > > architecture could support overriding the importer per file. > >> > > > >> > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely > >> > > single-file. They can share default definitions, for example: > >> > > > >> https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/. > >> > > Could importers declare file dependencies in DagImportResult, so the > >> > > processor can use them? > >> > > > >> > > 6. *Code tab UI*: On get_source_code(), could we have a parameter > >> related > >> > > to the file language, which could be used by the Code tab (it is > >> > currently > >> > > hard-coded with Python)? How are we planning to resolve ZipImporter > >> and > >> > > other transformed sources in the UI? > >> > > > >> > > 7. *Precedence*: What should be the precedence (per-bundle vs global > >> vs > >> > > default importers)? > >> > > > >> > > For the performance questions raised earlier in the thread, we could > >> use > >> > > DAG Factory workloads as a baseline, comparing the > >> non-built-in-Airflow > >> > > Python layer against a native YAML importer. > >> > > > >> > > Our team is very keen to contribute to the YAML contract definition. > >> > > > >> > > Kind regards, > >> > > > >> > > Tatiana > >> > > > >> > > > >> > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < > >> > > [email protected]> wrote: > >> > > > >> > > > Hi all, I am following up to request your feedback on AIP-85: DAG > >> > > importer. > >> > > > I attempted to present this at the last Airflow dev call, but the > >> > agenda > >> > > > was full, and today’s meeting was unfortunately cancelled. As it > has > >> > been > >> > > > some time since this AIP was submitted for review, I would greatly > >> > > > appreciate it if you could share your thoughts or any remaining > >> > concerns > >> > > > here. I understand there are many active discussions, but your > input > >> > > would > >> > > > be invaluable in helping move this proposal forward. Thanks, > Dilnaz > >> > > > Amanzholova > >> > > > > >> > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova < > >> [email protected] > >> > > > >> > > > wrote: > >> > > > > >> > > > > Hey Kaxil, > >> > > > > > >> > > > > Thanks for the clarification. I have decided to attend, as the > >> next > >> > > > > meeting is not for another two weeks. I have added the AIP to > the > >> > > agenda. > >> > > > > > >> > > > > Best regards, > >> > > > > Dilnaz Amanzholova > >> > > > > > >> > > > > > >> > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik <[email protected] > > > >> > > wrote: > >> > > > > > >> > > > >> Hey not required at all. > >> > > > >> > >> > > > >> This is just in case you wanted to solicit feedback live on the > >> call > >> > > > with > >> > > > >> other Airflow devs. Sometimes it is easier that way -- but not > >> > > > mandatory. > >> > > > >> > >> > > > >> More details on the dev calls: > >> > > > >> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call > >> > > if > >> > > > >> you decide about presenting it next time. > >> > > > >> > >> > > > >> Regards, > >> > > > >> Kaxil > >> > > > >> > >> > > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova < > >> > [email protected]> > >> > > > >> wrote: > >> > > > >> > >> > > > >>> Hi Kaxil, > >> > > > >>> > >> > > > >>> I was not aware there was a regular call. Is it required to > >> present > >> > > the > >> > > > >>> AIP before voting? > >> > > > >>> > >> > > > >>> Kind regards, > >> > > > >>> Dilnaz Amanzholova > >> > > > >>> > >> > > > >>> > >> > > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik < > [email protected] > >> > > >> > > > wrote: > >> > > > >>> > >> > > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on the > >> next > >> > > dev > >> > > > >>>> call? > >> > > > >>>> > >> > > > >>>> > >> > > > >>>> > >> > > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < > >> > > > >>>> [email protected]> wrote: > >> > > > >>>> > >> > > > >>>>> Hi all, > >> > > > >>>>> > >> > > > >>>>> I am following up on AIP-85: DAG importer. All previous > >> concerns > >> > > have > >> > > > >>>>> been > >> > > > >>>>> addressed, and the content has been updated on the AIP > >> Confluence > >> > > > page. > >> > > > >>>>> > >> > > > >>>>> Please review the latest updates at your earliest > >> convenience. I > >> > > > would > >> > > > >>>>> appreciate your feedback and am available to answer any > >> further > >> > > > >>>>> questions > >> > > > >>>>> you may have. > >> > > > >>>>> > >> > > > >>>>> Best regards, > >> > > > >>>>> Dilnaz Amanzholova > >> > > > >>>>> > >> > > > >>>>> > >> > > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < > >> > > > [email protected]> > >> > > > >>>>> wrote: > >> > > > >>>>> > >> > > > >>>>> > Hi all, > >> > > > >>>>> > > >> > > > >>>>> > I’m calling vote on AIP-85: DAG importer > >> > > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg > >> > > > >>>>> > > >> > > > >>>>> > You can also review the design document here > >> > > > >>>>> > < > >> > > > >>>>> > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > >> > > > >>>>> > > >> > > > >>>>> > . > >> > > > >>>>> > > >> > > > >>>>> > Current discussion thread (after scoping down): > >> > > > >>>>> > > >> > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > >> > > > >>>>> > Original discussion thread (with wider "extendable DAG > >> parsing > >> > > > >>>>> controls" > >> > > > >>>>> > title): > >> > > > >>>>> > >> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > >> > > > >>>>> > > >> > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th > >> June > >> > > 2026, > >> > > > >>>>> at > >> > > > >>>>> > 10:00 UTC. > >> > > > >>>>> > > >> > > > >>>>> > Everyone is encouraged to vote, but only PMC members and > >> > > > Committers' > >> > > > >>>>> votes > >> > > > >>>>> > are considered binding. Please vote accordingly. > >> > > > >>>>> > > >> > > > >>>>> > [ ] +1 Approve > >> > > > >>>>> > [ ] +0 no opinion > >> > > > >>>>> > [ ] -1 disapprove with the reason > >> > > > >>>>> > > >> > > > >>>>> > Kind regards, > >> > > > >>>>> > Dilnaz Amanzholova > >> > > > >>>>> > > >> > > > >>>>> > >> > > > >>>> > >> > > > > >> > > > >> > > > >> > > -- > >> > > Tatiana Al-Chueyr > >> > > > >> > > >> > >> > >> -- > >> Tatiana Al-Chueyr > >> > > >
