Very nice summary Jens and I 100% agree with the proposal. And BTW, this is not as difficult to implement as we might think. With bundle implementation, making providers contribute their own "example_bundle" seems like a very natural thing to do - this is actually what I implemented in order to show the "standard" example dags in Breeze. And extending provider discovery to also provide list of examples embedded in the provider should be super-easy as well, I am about to create a PR that moves the new queues to be "provided" by providers rather than embedded into "common.messaging", so I am happy to also do similar thing for examples and happy to join the 2-5 people team to work out the story told by examples. That would be really nice thing for the users/
J On Wed, Apr 30, 2025 at 11:06 PM Ferruzzi, Dennis <ferru...@amazon.com.invalid> wrote: > I retract my previous answer and agree with Jens. Well said! That's a > bigger lift to get it done, so maybe not part of the immediate migration, > but it would be a nice overall improvement project. > > > - ferruzzi > > > ________________________________ > From: Jens Scheffler <j_scheff...@gmx.de.INVALID> > Sent: Wednesday, April 30, 2025 1:56 PM > To: dev@airflow.apache.org > Subject: RE: [EXT] [DISCUSS] Example dags > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > Thanks for opening the discussion. > > I have since a longer time the idea that we need to clean and re-work > the examples. I like how examples are used in some places in docs and > that examples. But at least 50% of them do not add real value to be > loaded. I think these are too many. > > I'd propose to take some effort and consolidate such that (1) a reduced > but meaningful number of examples is available which (2) are all used > and referenced for details and features in docs [=means the examples > should cover all core features of the product with examples] and (3) > maybe even tell a storyline. I remember from my former times all the > microsoft trainings I received with "northwind" database as example. It > would be cool having examples also around some artificial but realistic > use case where we can "eat your own dog food" implement some best > practices with proper documentation. (4) bonus really could be that if > you add a provider that each provider might bring another piece of the > puzzle to the overall use case [=e.g. once you add kafka provider one > example polls for customer survey events] > > I assume this would be a bit tooo much to have a creative discussion on > the devlist... but if there are 2-5 people who like to join, would there > be somebody else having a creative mind such that we can think about a > storyline and rework examples to this? > > (And the technical layer I also agree to option 1 of having bundles per > provider added also would be cool. Then also taggin all examples > consistently and adding tags per provider for easier filtering etc... > some kind of housekeeping and beautifying would be cool (I also once > proposed to add proper docs and display names to all examples to have > them pretty) > > Options 2+3 would not be needed if we make some cleanup and > consolidation and get the number down to a meaningful number > > On 30.04.25 22:06, Ferruzzi, Dennis wrote: > > I think I agree with option1, if the user chose to show example_dags > then they should get what they asked for. Option 2 feels like a cleaner > option, essentially curating the examples for new users, but it's opening a > can of worms debating which are "important" and where we draw the line, > etc. So I think I'd go with 1. > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Wednesday, April 30, 2025 10:53 AM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSS] Example dags > > > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > The PR is finally green, sources updated, example dags moved to > "standard provider", yet in breeze they are visible. I run all the local > testing in Breeze for the moved dags and they work nicely. > > > > I think we might continue discussing what should be the "final" approach > - but for now - merging that one seems like a good idea. Unless someone > objects ? > > > > J. > > > > > > On Wed, Apr 30, 2025 at 5:16 PM Jarek Potiuk <ja...@potiuk.com<mailto: > ja...@potiuk.com>> wrote: > > Yep - all the dags will be visible in `breeze` in my PR > https://github.com/apache/airflow/pull/49978 - just pushed a change for > it: > > > > [image.png] > > > > On Wed, Apr 30, 2025 at 4:31 PM Jarek Potiuk <ja...@potiuk.com<mailto: > ja...@potiuk.com>> wrote: > > Answering Constance (somewhat it went to another thread:) > > > >> I know we rely on the example dags for dev purposes, especially for UI > > development. As part of moving them out of core, can we update breeze to > > automatically load them from their new home? > > > > Yes. Very good point. I strongly think we need to do it. And it's part > of the issue :). > > > > I already - in my PR - made it so in `breeze` and local venv where > `tests` are part of the system we have now two "example" bundles - the > second one coming from the standard provider. That was one of the reasons > my PR was not green initially - because also some of our tests in core > assume that "standard" provider examples are present. I adjusted some of > them to expect "simplest" dag - where it made no difference, but I also > created an extra dag bundle to add the standard provider in case the test > system is importable locally in tests - so far so good. > > > > But I think in order to make things **really** work for breeze > installation (and in local venv) - we would have to make it so the example > dags are: > > > > a) available in the provider packages > > b) airflow should discover them from there > > > > I can (quickly) make a hack for Breeze (that's the question of adding > standard provider's "tests" to PYTHONPATH) - in this PR - so It will cover > that one and we will be able to regenerate documentation with links to > sources. > > > > But I think in this discussion (and follow up PR) is to answer the > question of what users should see when they enable it in "installed" > airflow. > > > > * should users see the standard provider package dags when they install > airflow locally from packages and say "load_example_dags". > > > > if we think "yes", then we generally need to include example_dags in > provider packages (they are excluded now) - and we can do it, no problem. > > Now - I think we should also decide what users should "see" in this case > - my first reaction would be "standard" + "simplest" + "maybe a few others? > (which ones) ?" > > > > J. > > > > > > On Wed, Apr 30, 2025 at 1:12 PM Kaxil Naik <kaxiln...@gmail.com<mailto: > kaxiln...@gmail.com>> wrote: > > just an idea: With "git bundles", maybe example DAGs from standard > provider > > can be dynamically imported? > > > > But from your options: (1) seems reasonable > > > > On Wed, 30 Apr 2025 at 15:07, Jarek Potiuk <ja...@potiuk.com<mailto: > ja...@potiuk.com>> wrote: > > > >> Hello here. > >> > >> While fixing a bug in the doc generation to bring "source" link to > example > >> dags https://github.com/apache/airflow/pull/49978 I realized that we > have > >> not moved a number of python/bash etc. example dags to standard > provider - > >> where they now belong. I do it as part of my PR - but as result the > number > >> of "example_dags" shown in the UI when "load example dags" is true is > going > >> to decrease (and also they are not necessarily the dags you would really > >> want to see). > >> > >> I thought maybe about a follow up PR where we will do a few things: > >> > >> 1) allow to show example dags also from providers (or maybe just > standard > >> provider) > >> 2) limit a number of dags shown in airflow when "load example dags" > config > >> is set - for example by tags - to only see those that are really useful > (I > >> found the bash/python/comlex and few others really useful for quick > >> demo/testing - maybe there are others). > >> 3) We could optionally have "show all examples" setting for the "really > >> power user case" and make it easy to test things in airfow UI > >> out-of-the-box - or even limit it per provider. > >> > >> WDYT ? > >> > >> J > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
