I'd be happy to help with the proposal as well Jens, you can count me in. -- Regards, Aritra Basu
On Thu, 1 May 2025, 2:43 am Jarek Potiuk, <ja...@potiuk.com> wrote: > Very nice summary Jens and I 100% agree with the proposal. > > And BTW, this is not as difficult to implement as we might think. With > bundle implementation, making providers contribute their own > "example_bundle" seems like a very natural thing to do - this is actually > what I implemented in order to show the "standard" example dags in Breeze. > And extending provider discovery to also provide list of examples embedded > in the provider should be super-easy as well, I am about to create a PR > that moves the new queues to be "provided" by providers rather than > embedded into "common.messaging", so I am happy to also do similar thing > for examples and happy to join the 2-5 people team to work out the story > told by examples. That would be really nice thing for the users/ > > J > > > On Wed, Apr 30, 2025 at 11:06 PM Ferruzzi, Dennis > <ferru...@amazon.com.invalid> wrote: > > > I retract my previous answer and agree with Jens. Well said! That's a > > bigger lift to get it done, so maybe not part of the immediate migration, > > but it would be a nice overall improvement project. > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Jens Scheffler <j_scheff...@gmx.de.INVALID> > > Sent: Wednesday, April 30, 2025 1:56 PM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSS] Example dags > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > Thanks for opening the discussion. > > > > I have since a longer time the idea that we need to clean and re-work > > the examples. I like how examples are used in some places in docs and > > that examples. But at least 50% of them do not add real value to be > > loaded. I think these are too many. > > > > I'd propose to take some effort and consolidate such that (1) a reduced > > but meaningful number of examples is available which (2) are all used > > and referenced for details and features in docs [=means the examples > > should cover all core features of the product with examples] and (3) > > maybe even tell a storyline. I remember from my former times all the > > microsoft trainings I received with "northwind" database as example. It > > would be cool having examples also around some artificial but realistic > > use case where we can "eat your own dog food" implement some best > > practices with proper documentation. (4) bonus really could be that if > > you add a provider that each provider might bring another piece of the > > puzzle to the overall use case [=e.g. once you add kafka provider one > > example polls for customer survey events] > > > > I assume this would be a bit tooo much to have a creative discussion on > > the devlist... but if there are 2-5 people who like to join, would there > > be somebody else having a creative mind such that we can think about a > > storyline and rework examples to this? > > > > (And the technical layer I also agree to option 1 of having bundles per > > provider added also would be cool. Then also taggin all examples > > consistently and adding tags per provider for easier filtering etc... > > some kind of housekeeping and beautifying would be cool (I also once > > proposed to add proper docs and display names to all examples to have > > them pretty) > > > > Options 2+3 would not be needed if we make some cleanup and > > consolidation and get the number down to a meaningful number > > > > On 30.04.25 22:06, Ferruzzi, Dennis wrote: > > > I think I agree with option1, if the user chose to show example_dags > > then they should get what they asked for. Option 2 feels like a cleaner > > option, essentially curating the examples for new users, but it's > opening a > > can of worms debating which are "important" and where we draw the line, > > etc. So I think I'd go with 1. > > > > > > > > > - ferruzzi > > > > > > > > > ________________________________ > > > From: Jarek Potiuk <ja...@potiuk.com> > > > Sent: Wednesday, April 30, 2025 10:53 AM > > > To: dev@airflow.apache.org > > > Subject: RE: [EXT] [DISCUSS] Example dags > > > > > > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain que le contenu ne présente aucun risque. > > > > > > > > > The PR is finally green, sources updated, example dags moved to > > "standard provider", yet in breeze they are visible. I run all the local > > testing in Breeze for the moved dags and they work nicely. > > > > > > I think we might continue discussing what should be the "final" > approach > > - but for now - merging that one seems like a good idea. Unless someone > > objects ? > > > > > > J. > > > > > > > > > On Wed, Apr 30, 2025 at 5:16 PM Jarek Potiuk <ja...@potiuk.com<mailto: > > ja...@potiuk.com>> wrote: > > > Yep - all the dags will be visible in `breeze` in my PR > > https://github.com/apache/airflow/pull/49978 - just pushed a change for > > it: > > > > > > [image.png] > > > > > > On Wed, Apr 30, 2025 at 4:31 PM Jarek Potiuk <ja...@potiuk.com<mailto: > > ja...@potiuk.com>> wrote: > > > Answering Constance (somewhat it went to another thread:) > > > > > >> I know we rely on the example dags for dev purposes, especially for UI > > > development. As part of moving them out of core, can we update breeze > to > > > automatically load them from their new home? > > > > > > Yes. Very good point. I strongly think we need to do it. And it's > part > > of the issue :). > > > > > > I already - in my PR - made it so in `breeze` and local venv where > > `tests` are part of the system we have now two "example" bundles - the > > second one coming from the standard provider. That was one of the reasons > > my PR was not green initially - because also some of our tests in core > > assume that "standard" provider examples are present. I adjusted some of > > them to expect "simplest" dag - where it made no difference, but I also > > created an extra dag bundle to add the standard provider in case the test > > system is importable locally in tests - so far so good. > > > > > > But I think in order to make things **really** work for breeze > > installation (and in local venv) - we would have to make it so the > example > > dags are: > > > > > > a) available in the provider packages > > > b) airflow should discover them from there > > > > > > I can (quickly) make a hack for Breeze (that's the question of adding > > standard provider's "tests" to PYTHONPATH) - in this PR - so It will > cover > > that one and we will be able to regenerate documentation with links to > > sources. > > > > > > But I think in this discussion (and follow up PR) is to answer the > > question of what users should see when they enable it in "installed" > > airflow. > > > > > > * should users see the standard provider package dags when they > install > > airflow locally from packages and say "load_example_dags". > > > > > > if we think "yes", then we generally need to include example_dags in > > provider packages (they are excluded now) - and we can do it, no problem. > > > Now - I think we should also decide what users should "see" in this > case > > - my first reaction would be "standard" + "simplest" + "maybe a few > others? > > (which ones) ?" > > > > > > J. > > > > > > > > > On Wed, Apr 30, 2025 at 1:12 PM Kaxil Naik <kaxiln...@gmail.com > <mailto: > > kaxiln...@gmail.com>> wrote: > > > just an idea: With "git bundles", maybe example DAGs from standard > > provider > > > can be dynamically imported? > > > > > > But from your options: (1) seems reasonable > > > > > > On Wed, 30 Apr 2025 at 15:07, Jarek Potiuk <ja...@potiuk.com<mailto: > > ja...@potiuk.com>> wrote: > > > > > >> Hello here. > > >> > > >> While fixing a bug in the doc generation to bring "source" link to > > example > > >> dags https://github.com/apache/airflow/pull/49978 I realized that we > > have > > >> not moved a number of python/bash etc. example dags to standard > > provider - > > >> where they now belong. I do it as part of my PR - but as result the > > number > > >> of "example_dags" shown in the UI when "load example dags" is true is > > going > > >> to decrease (and also they are not necessarily the dags you would > really > > >> want to see). > > >> > > >> I thought maybe about a follow up PR where we will do a few things: > > >> > > >> 1) allow to show example dags also from providers (or maybe just > > standard > > >> provider) > > >> 2) limit a number of dags shown in airflow when "load example dags" > > config > > >> is set - for example by tags - to only see those that are really > useful > > (I > > >> found the bash/python/comlex and few others really useful for quick > > >> demo/testing - maybe there are others). > > >> 3) We could optionally have "show all examples" setting for the > "really > > >> power user case" and make it easy to test things in airfow UI > > >> out-of-the-box - or even limit it per provider. > > >> > > >> WDYT ? > > >> > > >> J > > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
