I think I agree with option1, if the user chose to show example_dags then they should get what they asked for. Option 2 feels like a cleaner option, essentially curating the examples for new users, but it's opening a can of worms debating which are "important" and where we draw the line, etc. So I think I'd go with 1.
- ferruzzi ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Wednesday, April 30, 2025 10:53 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSS] Example dags CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. The PR is finally green, sources updated, example dags moved to "standard provider", yet in breeze they are visible. I run all the local testing in Breeze for the moved dags and they work nicely. I think we might continue discussing what should be the "final" approach - but for now - merging that one seems like a good idea. Unless someone objects ? J. On Wed, Apr 30, 2025 at 5:16 PM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: Yep - all the dags will be visible in `breeze` in my PR https://github.com/apache/airflow/pull/49978 - just pushed a change for it: [image.png] On Wed, Apr 30, 2025 at 4:31 PM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: Answering Constance (somewhat it went to another thread:) > I know we rely on the example dags for dev purposes, especially for UI development. As part of moving them out of core, can we update breeze to automatically load them from their new home? Yes. Very good point. I strongly think we need to do it. And it's part of the issue :). I already - in my PR - made it so in `breeze` and local venv where `tests` are part of the system we have now two "example" bundles - the second one coming from the standard provider. That was one of the reasons my PR was not green initially - because also some of our tests in core assume that "standard" provider examples are present. I adjusted some of them to expect "simplest" dag - where it made no difference, but I also created an extra dag bundle to add the standard provider in case the test system is importable locally in tests - so far so good. But I think in order to make things **really** work for breeze installation (and in local venv) - we would have to make it so the example dags are: a) available in the provider packages b) airflow should discover them from there I can (quickly) make a hack for Breeze (that's the question of adding standard provider's "tests" to PYTHONPATH) - in this PR - so It will cover that one and we will be able to regenerate documentation with links to sources. But I think in this discussion (and follow up PR) is to answer the question of what users should see when they enable it in "installed" airflow. * should users see the standard provider package dags when they install airflow locally from packages and say "load_example_dags". if we think "yes", then we generally need to include example_dags in provider packages (they are excluded now) - and we can do it, no problem. Now - I think we should also decide what users should "see" in this case - my first reaction would be "standard" + "simplest" + "maybe a few others? (which ones) ?" J. On Wed, Apr 30, 2025 at 1:12 PM Kaxil Naik <kaxiln...@gmail.com<mailto:kaxiln...@gmail.com>> wrote: just an idea: With "git bundles", maybe example DAGs from standard provider can be dynamically imported? But from your options: (1) seems reasonable On Wed, 30 Apr 2025 at 15:07, Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: > Hello here. > > While fixing a bug in the doc generation to bring "source" link to example > dags https://github.com/apache/airflow/pull/49978 I realized that we have > not moved a number of python/bash etc. example dags to standard provider - > where they now belong. I do it as part of my PR - but as result the number > of "example_dags" shown in the UI when "load example dags" is true is going > to decrease (and also they are not necessarily the dags you would really > want to see). > > I thought maybe about a follow up PR where we will do a few things: > > 1) allow to show example dags also from providers (or maybe just standard > provider) > 2) limit a number of dags shown in airflow when "load example dags" config > is set - for example by tags - to only see those that are really useful (I > found the bash/python/comlex and few others really useful for quick > demo/testing - maybe there are others). > 3) We could optionally have "show all examples" setting for the "really > power user case" and make it easy to test things in airfow UI > out-of-the-box - or even limit it per provider. > > WDYT ? > > J >
