The PR is finally green, sources updated, example dags moved to "standard provider", yet in breeze they are visible. I run all the local testing in Breeze for the moved dags and they work nicely.
I think we might continue discussing what should be the "final" approach - but for now - merging that one seems like a good idea. Unless someone objects ? J. On Wed, Apr 30, 2025 at 5:16 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Yep - all the dags will be visible in `breeze` in my PR > https://github.com/apache/airflow/pull/49978 - just pushed a change for > it: > > [image: image.png] > > On Wed, Apr 30, 2025 at 4:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Answering Constance (somewhat it went to another thread:) >> >> > I know we rely on the example dags for dev purposes, especially for UI >> development. As part of moving them out of core, can we update breeze to >> automatically load them from their new home? >> >> Yes. Very good point. I strongly think we need to do it. And it's part >> of the issue :). >> >> I already - in my PR - made it so in `breeze` and local venv where >> `tests` are part of the system we have now two "example" bundles - the >> second one coming from the standard provider. That was one of the reasons >> my PR was not green initially - because also some of our tests in core >> assume that "standard" provider examples are present. I adjusted some of >> them to expect "simplest" dag - where it made no difference, but I also >> created an extra dag bundle to add the standard provider in case the test >> system is importable locally in tests - so far so good. >> >> But I think in order to make things **really** work for breeze >> installation (and in local venv) - we would have to make it so the example >> dags are: >> >> a) available in the provider packages >> b) airflow should discover them from there >> >> I can (quickly) make a hack for Breeze (that's the question of adding >> standard provider's "tests" to PYTHONPATH) - in this PR - so It will cover >> that one and we will be able to regenerate documentation with links to >> sources. >> >> But I think in this discussion (and follow up PR) is to answer the >> question of what users should see when they enable it in "installed" >> airflow. >> >> * should users see the standard provider package dags when they install >> airflow locally from packages and say "load_example_dags". >> >> if we think "yes", then we generally need to include example_dags in >> provider packages (they are excluded now) - and we can do it, no problem. >> Now - I think we should also decide what users should "see" in this case >> - my first reaction would be "standard" + "simplest" + "maybe a few others? >> (which ones) ?" >> >> J. >> >> >> On Wed, Apr 30, 2025 at 1:12 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >> >>> just an idea: With "git bundles", maybe example DAGs from standard >>> provider >>> can be dynamically imported? >>> >>> But from your options: (1) seems reasonable >>> >>> On Wed, 30 Apr 2025 at 15:07, Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>> > Hello here. >>> > >>> > While fixing a bug in the doc generation to bring "source" link to >>> example >>> > dags https://github.com/apache/airflow/pull/49978 I realized that we >>> have >>> > not moved a number of python/bash etc. example dags to standard >>> provider - >>> > where they now belong. I do it as part of my PR - but as result the >>> number >>> > of "example_dags" shown in the UI when "load example dags" is true is >>> going >>> > to decrease (and also they are not necessarily the dags you would >>> really >>> > want to see). >>> > >>> > I thought maybe about a follow up PR where we will do a few things: >>> > >>> > 1) allow to show example dags also from providers (or maybe just >>> standard >>> > provider) >>> > 2) limit a number of dags shown in airflow when "load example dags" >>> config >>> > is set - for example by tags - to only see those that are really >>> useful (I >>> > found the bash/python/comlex and few others really useful for quick >>> > demo/testing - maybe there are others). >>> > 3) We could optionally have "show all examples" setting for the "really >>> > power user case" and make it easy to test things in airfow UI >>> > out-of-the-box - or even limit it per provider. >>> > >>> > WDYT ? >>> > >>> > J >>> > >>> >>
