I'd like to suggest revoking this vote thread and start a new discussion thread to open up the discussion around the AIP.
On Fri, Jul 17, 2026 at 2:35 AM Zhe-You(Jason) Liu <[email protected]> wrote: > Hi Dilnaz, > > Thanks for your continued work refining the proposal. I'm looking forward > to the DagImporter serving as the interface for processing native DAG > definitions for Lang-SDK (Java, Go, and the upcoming TypeScript). > > I have one final question about the DagImporter interface. I left a comment > on the Confluence doc yesterday, but I'm raising it here as well for > visibility. > > Would it make sense to collapse supported_extensions() and can_handle() > into list_dag_files()? > > Extension matching alone seems insufficient: for normal Python DAGs, not > all Python files are DAG files, and it's similar for the Java-SDK -- not > all JARs are valid Java-SDK entrypoint files. > IIUC, list_dag_files() should be the single entrypoint that yields the > valid DAG files for further processing. Since the files it yields should be > valid anyway, I don't think we'd need can_handle() as a separate public > interface. > > Happy to hear your thoughts on whether this makes sense, or if there's a > reason to keep them separate that I'm missing. > Thanks! > > Best regards, > Jason > > On Mon, Jul 13, 2026 at 11:31 PM Dilnaz Amanzholova via dev < > [email protected]> wrote: > > > Hi @Jarek Potiuk <[email protected]> and > > < > > > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=a+global+extension+importer+map+structurally+can+t+give+two+bundles+different+importers+for+the+same+extension+so+as+it+stands+the+foundation+can+t+support+the+resolution+model+the+aip+promises > > >@Tatiana > > Al-Chueyr <[email protected]> , > > > > Thank you so much for the detailed review and for supporting the > > scoped-down direction. I completely agree with your structural concerns > > regarding the merged foundation in PR #60127. Reconciling the intended > > design in the AIP with what currently exists in main is definitely the > > right move before calling for a VOTE again. > > > > Here is how I am addressing your points, many of which align with recent > > inline discussions on the AIP page: > > > > 1. Registry scope: You are right that a global extension -> importer map > > structurally cannot support the per-bundle resolution we need. Tatiana > also > > brought this up in the inline comments. The intended end-state is indeed > > per-bundle instantiation with a strict bundle -> global -> default > > resolution > > order. I will refactor the DagImporterRegistry out of the singleton > pattern > > to ensure the interface supports this scope. > > > > 2. *Archive* handling: I think we are aligned on using a composite > > ZipImporter. The AIP describes a generic archive importer that delegates > to > > internal-format importers, which provides the flexibility needed for > > mixed-content archives (e.g., YAML and Python in the same zip). I will > > ensure the merged PythonDagImporter is updated to hand-over its > > hard-ownership of .zip files. > > > > 3. Cross-file dependency tracking: Tatiana also raised the staleness bug > > issue for shared configurations like defaults.yaml. > > < > > > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=without+importer+declared+dependencies+a+change+to+a+shared+file+won+t+re+parse+its+dependents+a+staleness+bug > > > > > I > > have updated the DagImportResult dataclass in the AIP to include a > > file_dependencies: > > list[str] field. To clarify the interaction with the DAG Processor: the > > custom importer will declare these dependencies, and the core DAG > Processor > > will cache them to monitor for modification time changes, triggering a > > re-parse and a new DAG version when necessary (keeping the importers > > stateless). > > > > 4. Security: This is a good point and something I hadn't explicitly > defined > > yet. Ensuring users don't treat YAML as a "review-free" data payload is > > critical since resolving operators by classpath has the same power as > > arbitrary Python. > > < > > > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=explicitly+that+a+declarative+dag+file+is+fully+code+equivalent+trust+a+yaml+importer+resolving+operators+by+classpath+has+the+same+power+as+arbitrary+python > > > > > I > > would love to take you up on your offer to help fill in these security > > blanks! I will add a dedicated "Security and Trust Model" section to the > > AIP for the constraints you mentioned, feel free to modify it. Overall, I > > would like to learn more about this if you don't mind, as this is my > first > > Airflow community contribution and it would be great to fill in the > > knowledge blanks. > > > > 5. Safe mode: Yes, the "airflow + dag" string check is strictly > > Python-specific and will fail for YAML. > > < > > > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=heuristic+file+contains+the+strings+airflow+dag+currently+lives+in+the+abstract+list_dag_files+but+it+s+python+specific+and+will+mis+handle+yaml+it+should+be+delegated+per+importer > > >The > > design on the AIP now clarifies that the safe_mode heuristic is delegated > > to individual importers by overriding the can_handle() method, allowing > > each format to define its own fast content-level checks. > > > > 6. Performance regression: I have added a section to the AIP's > > "Considerations" to include performance testing. This will include > > benchmarking basic, large-volume .py workloads to ensure the new registry > > indirection adds no measurable regression, > > < > > > https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=it+d+be+good+to+confirm+the+registry+indirection+adds+no+measurable+regression+on+the+plain+py+path+some+performance+tests+with+basic+big+numbers+of+dags+scenarios+would+be+great+to+do+as+part+of+the+implementation > > > > > alongside > > comparative benchmarks of native YAML vs. legacy generator patterns > (e.g., > > dag-factory). > > > > How should we proceed with the discussion logistics? I can either copy > the > > relevant points from this thread over to the main [DISCUSS] thread, or > > simply link this thread there as another reference. Once the feedback is > > incorporated and the AIP is ready to transition back to the voting > phase, I > > can then open a new thread, perhaps titled [FINAL VOTE]. > > > > Let me know on your thoughts on this as I don't have strong preference. > > > > Kind regards, > > Dilnaz Amanzholova > > > > > > > > On Wed, Jul 8, 2026 at 3:42 PM Jarek Potiuk <[email protected]> wrote: > > > > > 100% on what Tatiana wrote :D > > > > > > On Wed, Jul 8, 2026 at 3:06 PM Tatiana Al-Chueyr Martins < > > > [email protected]> wrote: > > > > > >> Thanks a lot for being so responsive to the feedback on the AIP > > Confluent > > >> page, Dilnaz. Since yesterday, you've added file_dependencies to > > >> DagImportResult, introduced a DagSource class that carries the source > > >> language for the Code tab, and made the importer precedence order > > explicit > > >> (bundle explicit mapping > global config > defaults) - all great > > >> improvements. > > >> > > >> I agree with Jarek on reconciling the merged foundation with the AIP > > >> (particularly the per-bundle registry) before re-running the vote. And > > >> yeah, explicitly stating that a declarative DAG file carries the same > > >> trust > > >> level as the Python code is important - in our experience, users often > > >> think YAML is just data, so we should make this explicit. > > >> > > >> On the zip question, composite importers feel like the right path > > forward. > > >> Once we get non-Python formats treated as first-class citizens, it > feels > > >> like the next natural step is mixed content. Delegating to inner > > importers > > >> handles that cleanly, whereas I feel that extending PythonDagImporter > > >> wouldn't scale as well. > > >> > > >> It does feel we should wrap up the bigger questions in [DISCUSS] - > this > > >> will allow maintainers to VOTE on the proposed design with confidence, > > and > > >> the AIP will be stronger with it. With the pace I've observed since > > >> yesterday, it doesn't feel like we're far off - looking forward to > > seeing > > >> it back into VOTE. > > >> > > >> Kind regards, > > >> > > >> Tatiana > > >> > > >> On Wed, 8 Jul 2026 at 13:24, Jarek Potiuk <[email protected]> wrote: > > >> > > >> > Hi Dilnaz, and indeed - thanks for pushing this forward. > > >> > > > >> > The scoped-down "DAG importer" framing is much clearer, and > > >> > native non-Python DAGs are a genuinely good direction. +1 on the > goal. > > >> > > > >> > I am happy to make comments and suggestions or even fill-in some > > blanks > > >> > (security especially) in the AIP proposal if you are fine with me > > doing > > >> it > > >> > - see > > >> > some of the comments below. > > >> > > > >> > Before I can vote on the design, I'd like to raise one structural > > >> concern > > >> > plus a couple of smaller ones. My main point is that the foundation > > >> already > > >> > merged in https://github.com/apache/airflow/pull/60127 > > >> > contradicts the design the AIP describes, and I think we > > >> > should reconcile the two before the vote closes rather than after. > > >> > > > >> > Some of these points are already explicitly stated, either in the > AIP > > or > > >> > opened > > >> > questions raised by Tatiana - I just want to reiterate them here and > > >> see if > > >> > we can > > >> > reach consensus on the shape of those changes. > > >> > > > >> > Concretely, in airflow-core/src/airflow/dag_processing/importers/: > > >> > > > >> > 1. Registry scope — global singleton vs. per-bundle. The AIP > describes > > >> > per-bundle importer mapping with a bundle -> global -> default > > >> resolution > > >> > order (and Tatiana's Q4/Q7 ask for the same). But the merged > > >> > DagImporterRegistry is a process-wide singleton keyed by extension, > > >> "last > > >> > registered wins, one importer per extension." A global > > >> extension->importer > > >> > map structurally can't give two bundles different importers for the > > same > > >> > extension, so as it stands the foundation can't support the > resolution > > >> > model the AIP promises. If per-bundle is the intended end state, I > > >> think we > > >> > want that in the interface now, because swapping a singleton out > later > > >> is > > >> > expensive. > > >> > > > >> > 2. Zip handling. The AIP describes a generic ZipImporter that > > delegates > > >> to > > >> > inner-format importers. The merged PythonDagImporter instead > hard-owns > > >> both > > >> > .py and .zip, with a docstring saying YAML-in-zip would require > > >> extending > > >> > it > > >> > or writing a composite importer — the opposite of the AIP. I think > we > > >> > should > > >> > decide on one or the other direction, I like the composite importer > > >> more - > > >> > even > > >> > if it is more complex, it allows for greater flexibility if we have > > >> > composite > > >> > importers. But I would love to hear what others think. > > >> > > > >> > 3. DagImportResult.file_dependencies is in the AIP dataclass but not > > in > > >> the > > >> > merged one — and this is more than cosmetic. Declarative DAGs > > routinely > > >> > share a defaults.yaml; the DFP decides what to re-parse from > per-file > > >> > mtime, > > >> > so without importer-declared dependencies a change to a shared file > > >> won't > > >> > re-parse its dependents (a staleness bug). This is Tatiana's Q5, and > > I'd > > >> > treat it as a correctness requirement, not a nice-to-have. Worth > > >> spelling > > >> > out how it interacts with AIP-66's git-sha bundle versioning too. > > >> > > > >> > On security, I'd like the AIP to state explicitly that a declarative > > DAG > > >> > file is fully code-equivalent trust: a YAML importer resolving > > >> operators by > > >> > classpath has the same power as arbitrary Python. People tend to > treat > > >> YAML > > >> > as "data" and .py as "code," and I want to make sure nobody builds a > > >> > "review-free YAML bundle" path on the back of this. I'd also like it > > >> stated > > >> > as a hard constraint that importers run only in the DAG processor, > > >> > in-process, under the same parse timeout, and never touch the > metadata > > >> DB — > > >> > consistent with the AIP-66/AIP-72 boundary. And a quick confirmation > > >> that > > >> > the old "push DAGs over API / ingester" idea stays out of scope, > since > > >> that > > >> > one genuinely changes the threat model. > > >> > > > >> > Again, for security, if you are happy with it, I would love to be > able > > >> to > > >> > fill > > >> > all those security blanks - this can be described on high level in > the > > >> AIP, > > >> > and when implementation progresses, it would turn into security > model > > >> > updates (I'm also happy to help there). > > >> > > > >> > Two smaller things: > > >> > > > >> > - The safe_mode/might_contain_dag heuristic ("file contains the > > strings > > >> > airflow + dag") currently lives in the abstract list_dag_files, but > > it's > > >> > Python-specific and will mis-handle YAML — it should be delegated > > >> > per-importer (Tatiana's Q3). > > >> > > > >> > - It'd be good to confirm the registry indirection adds no > measurable > > >> > regression on the plain .py path - some performance tests with basic > > >> > big-numbers-of-dags scenarios would be great to do as part of the > > >> > implementation. > > >> > > > >> > None of this is a blocker on the idea — I'm supportive. But given > the > > >> open > > >> > design questions and that the vote has been quiet, It reaffirms we > > treat > > >> > this > > >> > as still in [DISCUSS], reconcile the AIP with the merged foundation > > >> > (especially point 1), and re-run the vote once the interface > reflects > > >> the > > >> > intended end state. > > >> > > > >> > Happy to help review the per-bundle registry shape. > > >> > > > >> > Best, > > >> > Jarek > > >> > > > >> > On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < > > >> > [email protected]> wrote: > > >> > > > >> > > Hi Dilnaz, > > >> > > > > >> > > Thanks a lot for pushing this forward, and for the updates after > the > > >> > > earlier feedback - AIP 85 looks much clearer than before. > > >> > > > > >> > > I'm one of the maintainers of DAG Factory ( > > >> > > https://github.com/astronomer/dag-factory), and I like the > > direction > > >> the > > >> > > AIP is taking. I agree with the pain points Igor raised earlier; > it > > >> would > > >> > > be great to support non-Python DAGs natively in Airflow, rather > than > > >> > > converting them as we currently do in DAG Factory. > > >> > > > > >> > > I've reviewed the AIP page and left some inline comments & > > questions. > > >> > Since > > >> > > the authors are already responding on the AIP page, happy to > > continue > > >> the > > >> > > detailed discussion there - sharing the summary here mainly for > > >> > visibility: > > >> > > > > >> > > 1. *Location of schemas and importers*: Where do we see the > schemas > > >> and > > >> > > actual importers defined - and how do we plan to govern them? It > > >> would > > >> > be > > >> > > important to align, for instance, on how non-primitive operator > > >> arguments > > >> > > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) > > and > > >> on > > >> > > the security aspects of loading arbitrary classes from YAML. > > >> > > > > >> > > 2. *Validation contract*: If users define unknown or unsupported > > keys > > >> and > > >> > > configurations, should we fail loudly with DagImportErrors > > referencing > > >> > > files/lines like the Python DAG import errors? It would be great > if > > we > > >> > > didn't just skip them. > > >> > > > > >> > > 3.* Decision on which files to parse: *As raised by TP on > > >> > > < > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > >> > > > > > >> > > the > > >> > > Google > > >> > > < > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > >> > > > > > >> > > Doc > > >> > > < > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo > > >> > > > > > >> > > , I feel that we should have additional criteria to decide which > > >> files to > > >> > > treat as a DAG - and not rely only on the file extension. As an > > >> example, > > >> > > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, > > K8s > > >> > > manifests. Would it make sense to have something similar to the > > Python > > >> > > content-level heuristic ("airflow + dag" string check)? > > >> > > > > >> > > 4. *Different importers for the same file extension*: Can/should > we > > >> > support > > >> > > different importers for the same file extension? It would be great > > if > > >> the > > >> > > architecture could support overriding the importer per file. > > >> > > > > >> > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely > > >> > > single-file. They can share default definitions, for example: > > >> > > > > >> > https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/. > > >> > > Could importers declare file dependencies in DagImportResult, so > the > > >> > > processor can use them? > > >> > > > > >> > > 6. *Code tab UI*: On get_source_code(), could we have a parameter > > >> related > > >> > > to the file language, which could be used by the Code tab (it is > > >> > currently > > >> > > hard-coded with Python)? How are we planning to resolve > ZipImporter > > >> and > > >> > > other transformed sources in the UI? > > >> > > > > >> > > 7. *Precedence*: What should be the precedence (per-bundle vs > global > > >> vs > > >> > > default importers)? > > >> > > > > >> > > For the performance questions raised earlier in the thread, we > could > > >> use > > >> > > DAG Factory workloads as a baseline, comparing the > > >> non-built-in-Airflow > > >> > > Python layer against a native YAML importer. > > >> > > > > >> > > Our team is very keen to contribute to the YAML contract > definition. > > >> > > > > >> > > Kind regards, > > >> > > > > >> > > Tatiana > > >> > > > > >> > > > > >> > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < > > >> > > [email protected]> wrote: > > >> > > > > >> > > > Hi all, I am following up to request your feedback on AIP-85: > DAG > > >> > > importer. > > >> > > > I attempted to present this at the last Airflow dev call, but > the > > >> > agenda > > >> > > > was full, and today’s meeting was unfortunately cancelled. As it > > has > > >> > been > > >> > > > some time since this AIP was submitted for review, I would > greatly > > >> > > > appreciate it if you could share your thoughts or any remaining > > >> > concerns > > >> > > > here. I understand there are many active discussions, but your > > input > > >> > > would > > >> > > > be invaluable in helping move this proposal forward. Thanks, > > Dilnaz > > >> > > > Amanzholova > > >> > > > > > >> > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova < > > >> [email protected] > > >> > > > > >> > > > wrote: > > >> > > > > > >> > > > > Hey Kaxil, > > >> > > > > > > >> > > > > Thanks for the clarification. I have decided to attend, as the > > >> next > > >> > > > > meeting is not for another two weeks. I have added the AIP to > > the > > >> > > agenda. > > >> > > > > > > >> > > > > Best regards, > > >> > > > > Dilnaz Amanzholova > > >> > > > > > > >> > > > > > > >> > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik < > [email protected] > > > > > >> > > wrote: > > >> > > > > > > >> > > > >> Hey not required at all. > > >> > > > >> > > >> > > > >> This is just in case you wanted to solicit feedback live on > the > > >> call > > >> > > > with > > >> > > > >> other Airflow devs. Sometimes it is easier that way -- but > not > > >> > > > mandatory. > > >> > > > >> > > >> > > > >> More details on the dev calls: > > >> > > > >> > > >> > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call > > >> > > if > > >> > > > >> you decide about presenting it next time. > > >> > > > >> > > >> > > > >> Regards, > > >> > > > >> Kaxil > > >> > > > >> > > >> > > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova < > > >> > [email protected]> > > >> > > > >> wrote: > > >> > > > >> > > >> > > > >>> Hi Kaxil, > > >> > > > >>> > > >> > > > >>> I was not aware there was a regular call. Is it required to > > >> present > > >> > > the > > >> > > > >>> AIP before voting? > > >> > > > >>> > > >> > > > >>> Kind regards, > > >> > > > >>> Dilnaz Amanzholova > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik < > > [email protected] > > >> > > > >> > > > wrote: > > >> > > > >>> > > >> > > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on > the > > >> next > > >> > > dev > > >> > > > >>>> call? > > >> > > > >>>> > > >> > > > >>>> > > >> > > > >>>> > > >> > > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < > > >> > > > >>>> [email protected]> wrote: > > >> > > > >>>> > > >> > > > >>>>> Hi all, > > >> > > > >>>>> > > >> > > > >>>>> I am following up on AIP-85: DAG importer. All previous > > >> concerns > > >> > > have > > >> > > > >>>>> been > > >> > > > >>>>> addressed, and the content has been updated on the AIP > > >> Confluence > > >> > > > page. > > >> > > > >>>>> > > >> > > > >>>>> Please review the latest updates at your earliest > > >> convenience. I > > >> > > > would > > >> > > > >>>>> appreciate your feedback and am available to answer any > > >> further > > >> > > > >>>>> questions > > >> > > > >>>>> you may have. > > >> > > > >>>>> > > >> > > > >>>>> Best regards, > > >> > > > >>>>> Dilnaz Amanzholova > > >> > > > >>>>> > > >> > > > >>>>> > > >> > > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < > > >> > > > [email protected]> > > >> > > > >>>>> wrote: > > >> > > > >>>>> > > >> > > > >>>>> > Hi all, > > >> > > > >>>>> > > > >> > > > >>>>> > I’m calling vote on AIP-85: DAG importer > > >> > > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg > > >> > > > >>>>> > > > >> > > > >>>>> > You can also review the design document here > > >> > > > >>>>> > < > > >> > > > >>>>> > > >> > > > > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 > > >> > > > >>>>> > > > >> > > > >>>>> > . > > >> > > > >>>>> > > > >> > > > >>>>> > Current discussion thread (after scoping down): > > >> > > > >>>>> > > > >> > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt > > >> > > > >>>>> > Original discussion thread (with wider "extendable DAG > > >> parsing > > >> > > > >>>>> controls" > > >> > > > >>>>> > title): > > >> > > > >>>>> > > >> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl > > >> > > > >>>>> > > > >> > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th > > >> June > > >> > > 2026, > > >> > > > >>>>> at > > >> > > > >>>>> > 10:00 UTC. > > >> > > > >>>>> > > > >> > > > >>>>> > Everyone is encouraged to vote, but only PMC members and > > >> > > > Committers' > > >> > > > >>>>> votes > > >> > > > >>>>> > are considered binding. Please vote accordingly. > > >> > > > >>>>> > > > >> > > > >>>>> > [ ] +1 Approve > > >> > > > >>>>> > [ ] +0 no opinion > > >> > > > >>>>> > [ ] -1 disapprove with the reason > > >> > > > >>>>> > > > >> > > > >>>>> > Kind regards, > > >> > > > >>>>> > Dilnaz Amanzholova > > >> > > > >>>>> > > > >> > > > >>>>> > > >> > > > >>>> > > >> > > > > > >> > > > > >> > > > > >> > > -- > > >> > > Tatiana Al-Chueyr > > >> > > > > >> > > > >> > > >> > > >> -- > > >> Tatiana Al-Chueyr > > >> > > > > > >
