(please ignore the broken links in the previous message, apologize for any confusion)
On Mon, Jul 13, 2026 at 5:30 PM Dilnaz Amanzholova <[email protected]> wrote: > Hi @Jarek Potiuk <[email protected]> and > <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=a+global+extension+importer+map+structurally+can+t+give+two+bundles+different+importers+for+the+same+extension+so+as+it+stands+the+foundation+can+t+support+the+resolution+model+the+aip+promises>@Tatiana > Al-Chueyr <[email protected]> , > > Thank you so much for the detailed review and for supporting the > scoped-down direction. I completely agree with your structural concerns > regarding the merged foundation in PR #60127. Reconciling the intended > design in the AIP with what currently exists in main is definitely the > right move before calling for a VOTE again. > > Here is how I am addressing your points, many of which align with recent > inline discussions on the AIP page: > > 1. Registry scope: You are right that a global extension -> importer map > structurally cannot support the per-bundle resolution we need. Tatiana also > brought this up in the inline comments. The intended end-state is indeed > per-bundle instantiation with a strict bundle -> global -> default resolution > order. I will refactor the DagImporterRegistry out of the singleton > pattern to ensure the interface supports this scope. > > 2. *Archive* handling: I think we are aligned on using a composite > ZipImporter. The AIP describes a generic archive importer that delegates > to internal-format importers, which provides the flexibility needed for > mixed-content archives (e.g., YAML and Python in the same zip). I will > ensure the merged PythonDagImporter is updated to hand-over its > hard-ownership of .zip files. > > 3. Cross-file dependency tracking: Tatiana also raised the staleness bug > issue for shared configurations like defaults.yaml. > <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=without+importer+declared+dependencies+a+change+to+a+shared+file+won+t+re+parse+its+dependents+a+staleness+bug> > I > have updated the DagImportResult dataclass in the AIP to include a > file_dependencies: > list[str] field. To clarify the interaction with the DAG Processor: the > custom importer will declare these dependencies, and the core DAG Processor > will cache them to monitor for modification time changes, triggering a > re-parse and a new DAG version when necessary (keeping the importers > stateless). > > 4. Security: This is a good point and something I hadn't explicitly > defined yet. Ensuring users don't treat YAML as a "review-free" data > payload is critical since resolving operators by classpath has the same > power as arbitrary Python. > <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=explicitly+that+a+declarative+dag+file+is+fully+code+equivalent+trust+a+yaml+importer+resolving+operators+by+classpath+has+the+same+power+as+arbitrary+python> > I > would love to take you up on your offer to help fill in these security > blanks! I will add a dedicated "Security and Trust Model" section to the > AIP for the constraints you mentioned, feel free to modify it. Overall, I > would like to learn more about this if you don't mind, as this is my first > Airflow community contribution and it would be great to fill in the > knowledge blanks. > > 5. Safe mode: Yes, the "airflow + dag" string check is strictly > Python-specific and will fail for YAML. > <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=heuristic+file+contains+the+strings+airflow+dag+currently+lives+in+the+abstract+list_dag_files+but+it+s+python+specific+and+will+mis+handle+yaml+it+should+be+delegated+per+importer>The > design on the AIP now clarifies that the safe_mode heuristic is delegated > to individual importers by overriding the can_handle() method, allowing > each format to define its own fast content-level checks. > > 6. Performance regression: I have added a section to the AIP's > "Considerations" to include performance testing. This will include > benchmarking basic, large-volume .py workloads to ensure the new registry > indirection adds no measurable regression, > <https://lists.apache.org/thread/df3f0z7l29nr9t5g2pvgxzpqvq74bv2w?content_ref=it+d+be+good+to+confirm+the+registry+indirection+adds+no+measurable+regression+on+the+plain+py+path+some+performance+tests+with+basic+big+numbers+of+dags+scenarios+would+be+great+to+do+as+part+of+the+implementation> > alongside > comparative benchmarks of native YAML vs. legacy generator patterns (e.g., > dag-factory). > > How should we proceed with the discussion logistics? I can either copy the > relevant points from this thread over to the main [DISCUSS] thread, or > simply link this thread there as another reference. Once the feedback is > incorporated and the AIP is ready to transition back to the voting phase, I > can then open a new thread, perhaps titled [FINAL VOTE]. > > Let me know on your thoughts on this as I don't have strong preference. > > Kind regards, > Dilnaz Amanzholova > > > > On Wed, Jul 8, 2026 at 3:42 PM Jarek Potiuk <[email protected]> wrote: > >> 100% on what Tatiana wrote :D >> >> On Wed, Jul 8, 2026 at 3:06 PM Tatiana Al-Chueyr Martins < >> [email protected]> wrote: >> >>> Thanks a lot for being so responsive to the feedback on the AIP Confluent >>> page, Dilnaz. Since yesterday, you've added file_dependencies to >>> DagImportResult, introduced a DagSource class that carries the source >>> language for the Code tab, and made the importer precedence order >>> explicit >>> (bundle explicit mapping > global config > defaults) - all great >>> improvements. >>> >>> I agree with Jarek on reconciling the merged foundation with the AIP >>> (particularly the per-bundle registry) before re-running the vote. And >>> yeah, explicitly stating that a declarative DAG file carries the same >>> trust >>> level as the Python code is important - in our experience, users often >>> think YAML is just data, so we should make this explicit. >>> >>> On the zip question, composite importers feel like the right path >>> forward. >>> Once we get non-Python formats treated as first-class citizens, it feels >>> like the next natural step is mixed content. Delegating to inner >>> importers >>> handles that cleanly, whereas I feel that extending PythonDagImporter >>> wouldn't scale as well. >>> >>> It does feel we should wrap up the bigger questions in [DISCUSS] - this >>> will allow maintainers to VOTE on the proposed design with confidence, >>> and >>> the AIP will be stronger with it. With the pace I've observed since >>> yesterday, it doesn't feel like we're far off - looking forward to seeing >>> it back into VOTE. >>> >>> Kind regards, >>> >>> Tatiana >>> >>> On Wed, 8 Jul 2026 at 13:24, Jarek Potiuk <[email protected]> wrote: >>> >>> > Hi Dilnaz, and indeed - thanks for pushing this forward. >>> > >>> > The scoped-down "DAG importer" framing is much clearer, and >>> > native non-Python DAGs are a genuinely good direction. +1 on the goal. >>> > >>> > I am happy to make comments and suggestions or even fill-in some blanks >>> > (security especially) in the AIP proposal if you are fine with me >>> doing it >>> > - see >>> > some of the comments below. >>> > >>> > Before I can vote on the design, I'd like to raise one structural >>> concern >>> > plus a couple of smaller ones. My main point is that the foundation >>> already >>> > merged in https://github.com/apache/airflow/pull/60127 >>> > contradicts the design the AIP describes, and I think we >>> > should reconcile the two before the vote closes rather than after. >>> > >>> > Some of these points are already explicitly stated, either in the AIP >>> or >>> > opened >>> > questions raised by Tatiana - I just want to reiterate them here and >>> see if >>> > we can >>> > reach consensus on the shape of those changes. >>> > >>> > Concretely, in airflow-core/src/airflow/dag_processing/importers/: >>> > >>> > 1. Registry scope — global singleton vs. per-bundle. The AIP describes >>> > per-bundle importer mapping with a bundle -> global -> default >>> resolution >>> > order (and Tatiana's Q4/Q7 ask for the same). But the merged >>> > DagImporterRegistry is a process-wide singleton keyed by extension, >>> "last >>> > registered wins, one importer per extension." A global >>> extension->importer >>> > map structurally can't give two bundles different importers for the >>> same >>> > extension, so as it stands the foundation can't support the resolution >>> > model the AIP promises. If per-bundle is the intended end state, I >>> think we >>> > want that in the interface now, because swapping a singleton out later >>> is >>> > expensive. >>> > >>> > 2. Zip handling. The AIP describes a generic ZipImporter that >>> delegates to >>> > inner-format importers. The merged PythonDagImporter instead hard-owns >>> both >>> > .py and .zip, with a docstring saying YAML-in-zip would require >>> extending >>> > it >>> > or writing a composite importer — the opposite of the AIP. I think we >>> > should >>> > decide on one or the other direction, I like the composite importer >>> more - >>> > even >>> > if it is more complex, it allows for greater flexibility if we have >>> > composite >>> > importers. But I would love to hear what others think. >>> > >>> > 3. DagImportResult.file_dependencies is in the AIP dataclass but not >>> in the >>> > merged one — and this is more than cosmetic. Declarative DAGs routinely >>> > share a defaults.yaml; the DFP decides what to re-parse from per-file >>> > mtime, >>> > so without importer-declared dependencies a change to a shared file >>> won't >>> > re-parse its dependents (a staleness bug). This is Tatiana's Q5, and >>> I'd >>> > treat it as a correctness requirement, not a nice-to-have. Worth >>> spelling >>> > out how it interacts with AIP-66's git-sha bundle versioning too. >>> > >>> > On security, I'd like the AIP to state explicitly that a declarative >>> DAG >>> > file is fully code-equivalent trust: a YAML importer resolving >>> operators by >>> > classpath has the same power as arbitrary Python. People tend to treat >>> YAML >>> > as "data" and .py as "code," and I want to make sure nobody builds a >>> > "review-free YAML bundle" path on the back of this. I'd also like it >>> stated >>> > as a hard constraint that importers run only in the DAG processor, >>> > in-process, under the same parse timeout, and never touch the metadata >>> DB — >>> > consistent with the AIP-66/AIP-72 boundary. And a quick confirmation >>> that >>> > the old "push DAGs over API / ingester" idea stays out of scope, since >>> that >>> > one genuinely changes the threat model. >>> > >>> > Again, for security, if you are happy with it, I would love to be able >>> to >>> > fill >>> > all those security blanks - this can be described on high level in the >>> AIP, >>> > and when implementation progresses, it would turn into security model >>> > updates (I'm also happy to help there). >>> > >>> > Two smaller things: >>> > >>> > - The safe_mode/might_contain_dag heuristic ("file contains the strings >>> > airflow + dag") currently lives in the abstract list_dag_files, but >>> it's >>> > Python-specific and will mis-handle YAML — it should be delegated >>> > per-importer (Tatiana's Q3). >>> > >>> > - It'd be good to confirm the registry indirection adds no measurable >>> > regression on the plain .py path - some performance tests with basic >>> > big-numbers-of-dags scenarios would be great to do as part of the >>> > implementation. >>> > >>> > None of this is a blocker on the idea — I'm supportive. But given the >>> open >>> > design questions and that the vote has been quiet, It reaffirms we >>> treat >>> > this >>> > as still in [DISCUSS], reconcile the AIP with the merged foundation >>> > (especially point 1), and re-run the vote once the interface reflects >>> the >>> > intended end state. >>> > >>> > Happy to help review the per-bundle registry shape. >>> > >>> > Best, >>> > Jarek >>> > >>> > On Tue, Jul 7, 2026 at 4:45 PM Tatiana Al-Chueyr Martins < >>> > [email protected]> wrote: >>> > >>> > > Hi Dilnaz, >>> > > >>> > > Thanks a lot for pushing this forward, and for the updates after the >>> > > earlier feedback - AIP 85 looks much clearer than before. >>> > > >>> > > I'm one of the maintainers of DAG Factory ( >>> > > https://github.com/astronomer/dag-factory), and I like the >>> direction the >>> > > AIP is taking. I agree with the pain points Igor raised earlier; it >>> would >>> > > be great to support non-Python DAGs natively in Airflow, rather than >>> > > converting them as we currently do in DAG Factory. >>> > > >>> > > I've reviewed the AIP page and left some inline comments & questions. >>> > Since >>> > > the authors are already responding on the AIP page, happy to >>> continue the >>> > > detailed discussion there - sharing the summary here mainly for >>> > visibility: >>> > > >>> > > 1. *Location of schemas and importers*: Where do we see the schemas >>> and >>> > > actual importers defined - and how do we plan to govern them? It >>> would >>> > be >>> > > important to align, for instance, on how non-primitive operator >>> arguments >>> > > should be defined in YAML (e.g., Asset, V1Pod, arbitrary classes) >>> and on >>> > > the security aspects of loading arbitrary classes from YAML. >>> > > >>> > > 2. *Validation contract*: If users define unknown or unsupported >>> keys and >>> > > configurations, should we fail loudly with DagImportErrors >>> referencing >>> > > files/lines like the Python DAG import errors? It would be great if >>> we >>> > > didn't just skip them. >>> > > >>> > > 3.* Decision on which files to parse: *As raised by TP on >>> > > < >>> > > >>> > >>> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo >>> > > > >>> > > the >>> > > Google >>> > > < >>> > > >>> > >>> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo >>> > > > >>> > > Doc >>> > > < >>> > > >>> > >>> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?disco=AAAB8ofIMRo >>> > > > >>> > > , I feel that we should have additional criteria to decide which >>> files to >>> > > treat as a DAG - and not rely only on the file extension. As an >>> example, >>> > > bundles normally have non-DAG YAML - dbt_project.yml, CI configs, K8s >>> > > manifests. Would it make sense to have something similar to the >>> Python >>> > > content-level heuristic ("airflow + dag" string check)? >>> > > >>> > > 4. *Different importers for the same file extension*: Can/should we >>> > support >>> > > different importers for the same file extension? It would be great >>> if the >>> > > architecture could support overriding the importer per file. >>> > > >>> > > 5.* Cross-file dependency tracking:* Declarative DAGs are rarely >>> > > single-file. They can share default definitions, for example: >>> > > >>> https://astronomer.github.io/dag-factory/1.1.0/configuration/defaults/. >>> > > Could importers declare file dependencies in DagImportResult, so the >>> > > processor can use them? >>> > > >>> > > 6. *Code tab UI*: On get_source_code(), could we have a parameter >>> related >>> > > to the file language, which could be used by the Code tab (it is >>> > currently >>> > > hard-coded with Python)? How are we planning to resolve ZipImporter >>> and >>> > > other transformed sources in the UI? >>> > > >>> > > 7. *Precedence*: What should be the precedence (per-bundle vs global >>> vs >>> > > default importers)? >>> > > >>> > > For the performance questions raised earlier in the thread, we could >>> use >>> > > DAG Factory workloads as a baseline, comparing the >>> non-built-in-Airflow >>> > > Python layer against a native YAML importer. >>> > > >>> > > Our team is very keen to contribute to the YAML contract definition. >>> > > >>> > > Kind regards, >>> > > >>> > > Tatiana >>> > > >>> > > >>> > > On Tue, 7 Jul 2026 at 13:48, Dilnaz Amanzholova via dev < >>> > > [email protected]> wrote: >>> > > >>> > > > Hi all, I am following up to request your feedback on AIP-85: DAG >>> > > importer. >>> > > > I attempted to present this at the last Airflow dev call, but the >>> > agenda >>> > > > was full, and today’s meeting was unfortunately cancelled. As it >>> has >>> > been >>> > > > some time since this AIP was submitted for review, I would greatly >>> > > > appreciate it if you could share your thoughts or any remaining >>> > concerns >>> > > > here. I understand there are many active discussions, but your >>> input >>> > > would >>> > > > be invaluable in helping move this proposal forward. Thanks, Dilnaz >>> > > > Amanzholova >>> > > > >>> > > > On Thu, Jun 18, 2026 at 2:08 PM Dilnaz Amanzholova < >>> [email protected] >>> > > >>> > > > wrote: >>> > > > >>> > > > > Hey Kaxil, >>> > > > > >>> > > > > Thanks for the clarification. I have decided to attend, as the >>> next >>> > > > > meeting is not for another two weeks. I have added the AIP to the >>> > > agenda. >>> > > > > >>> > > > > Best regards, >>> > > > > Dilnaz Amanzholova >>> > > > > >>> > > > > >>> > > > > On Wed, Jun 17, 2026 at 6:37 PM Kaxil Naik <[email protected]> >>> > > wrote: >>> > > > > >>> > > > >> Hey not required at all. >>> > > > >> >>> > > > >> This is just in case you wanted to solicit feedback live on the >>> call >>> > > > with >>> > > > >> other Airflow devs. Sometimes it is easier that way -- but not >>> > > > mandatory. >>> > > > >> >>> > > > >> More details on the dev calls: >>> > > > >> >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Dev+Call >>> > > if >>> > > > >> you decide about presenting it next time. >>> > > > >> >>> > > > >> Regards, >>> > > > >> Kaxil >>> > > > >> >>> > > > >> On Wed, 17 Jun 2026 at 17:32, Dilnaz Amanzholova < >>> > [email protected]> >>> > > > >> wrote: >>> > > > >> >>> > > > >>> Hi Kaxil, >>> > > > >>> >>> > > > >>> I was not aware there was a regular call. Is it required to >>> present >>> > > the >>> > > > >>> AIP before voting? >>> > > > >>> >>> > > > >>> Kind regards, >>> > > > >>> Dilnaz Amanzholova >>> > > > >>> >>> > > > >>> >>> > > > >>> On Wed, Jun 17, 2026 at 6:26 PM Kaxil Naik < >>> [email protected]> >>> > > > wrote: >>> > > > >>> >>> > > > >>>> Hi @dilnaz -- Are you up for presenting it tomorrow or on the >>> next >>> > > dev >>> > > > >>>> call? >>> > > > >>>> >>> > > > >>>> >>> > > > >>>> >>> > > > >>>> On Tue, 16 Jun 2026 at 15:15, Dilnaz Amanzholova via dev < >>> > > > >>>> [email protected]> wrote: >>> > > > >>>> >>> > > > >>>>> Hi all, >>> > > > >>>>> >>> > > > >>>>> I am following up on AIP-85: DAG importer. All previous >>> concerns >>> > > have >>> > > > >>>>> been >>> > > > >>>>> addressed, and the content has been updated on the AIP >>> Confluence >>> > > > page. >>> > > > >>>>> >>> > > > >>>>> Please review the latest updates at your earliest >>> convenience. I >>> > > > would >>> > > > >>>>> appreciate your feedback and am available to answer any >>> further >>> > > > >>>>> questions >>> > > > >>>>> you may have. >>> > > > >>>>> >>> > > > >>>>> Best regards, >>> > > > >>>>> Dilnaz Amanzholova >>> > > > >>>>> >>> > > > >>>>> >>> > > > >>>>> On Fri, Jun 5, 2026 at 1:23 PM Dilnaz Amanzholova < >>> > > > [email protected]> >>> > > > >>>>> wrote: >>> > > > >>>>> >>> > > > >>>>> > Hi all, >>> > > > >>>>> > >>> > > > >>>>> > I’m calling vote on AIP-85: DAG importer >>> > > > >>>>> > https://cwiki.apache.org/confluence/x/_Q7OEg >>> > > > >>>>> > >>> > > > >>>>> > You can also review the design document here >>> > > > >>>>> > < >>> > > > >>>>> >>> > > > >>> > > >>> > >>> https://docs.google.com/document/d/1K6-4cGoZItXGQHZjOydNbc7rGtOp_XKfFurMnFptKe0/edit?tab=t.0 >>> > > > >>>>> > >>> > > > >>>>> > . >>> > > > >>>>> > >>> > > > >>>>> > Current discussion thread (after scoping down): >>> > > > >>>>> > >>> > https://lists.apache.org/thread/wtfog0qjrf3oh7355db0x6mqk3o7l2dt >>> > > > >>>>> > Original discussion thread (with wider "extendable DAG >>> parsing >>> > > > >>>>> controls" >>> > > > >>>>> > title): >>> > > > >>>>> >>> https://lists.apache.org/thread/bn0oo47j48xh8r335gd2jrrjz0o7vnjl >>> > > > >>>>> > >>> > > > >>>>> > The vote will run for 5 days, closing on Wednesday, 10th >>> June >>> > > 2026, >>> > > > >>>>> at >>> > > > >>>>> > 10:00 UTC. >>> > > > >>>>> > >>> > > > >>>>> > Everyone is encouraged to vote, but only PMC members and >>> > > > Committers' >>> > > > >>>>> votes >>> > > > >>>>> > are considered binding. Please vote accordingly. >>> > > > >>>>> > >>> > > > >>>>> > [ ] +1 Approve >>> > > > >>>>> > [ ] +0 no opinion >>> > > > >>>>> > [ ] -1 disapprove with the reason >>> > > > >>>>> > >>> > > > >>>>> > Kind regards, >>> > > > >>>>> > Dilnaz Amanzholova >>> > > > >>>>> > >>> > > > >>>>> >>> > > > >>>> >>> > > > >>> > > >>> > > >>> > > -- >>> > > Tatiana Al-Chueyr >>> > > >>> > >>> >>> >>> -- >>> Tatiana Al-Chueyr >>> >>
