Thanks for opening the discussion.
I have since a longer time the idea that we need to clean and re-work
the examples. I like how examples are used in some places in docs and
that examples. But at least 50% of them do not add real value to be
loaded. I think these are too many.
I'd propose to take some effort and consolidate such that (1) a reduced
but meaningful number of examples is available which (2) are all used
and referenced for details and features in docs [=means the examples
should cover all core features of the product with examples] and (3)
maybe even tell a storyline. I remember from my former times all the
microsoft trainings I received with "northwind" database as example. It
would be cool having examples also around some artificial but realistic
use case where we can "eat your own dog food" implement some best
practices with proper documentation. (4) bonus really could be that if
you add a provider that each provider might bring another piece of the
puzzle to the overall use case [=e.g. once you add kafka provider one
example polls for customer survey events]
I assume this would be a bit tooo much to have a creative discussion on
the devlist... but if there are 2-5 people who like to join, would there
be somebody else having a creative mind such that we can think about a
storyline and rework examples to this?
(And the technical layer I also agree to option 1 of having bundles per
provider added also would be cool. Then also taggin all examples
consistently and adding tags per provider for easier filtering etc...
some kind of housekeeping and beautifying would be cool (I also once
proposed to add proper docs and display names to all examples to have
them pretty)
Options 2+3 would not be needed if we make some cleanup and
consolidation and get the number down to a meaningful number
On 30.04.25 22:06, Ferruzzi, Dennis wrote:
I think I agree with option1, if the user chose to show example_dags then they should get
what they asked for. Option 2 feels like a cleaner option, essentially curating the
examples for new users, but it's opening a can of worms debating which are
"important" and where we draw the line, etc. So I think I'd go with 1.
- ferruzzi
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Wednesday, April 30, 2025 10:53 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Example dags
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le
contenu ne présente aucun risque.
The PR is finally green, sources updated, example dags moved to "standard
provider", yet in breeze they are visible. I run all the local testing in Breeze for
the moved dags and they work nicely.
I think we might continue discussing what should be the "final" approach - but
for now - merging that one seems like a good idea. Unless someone objects ?
J.
On Wed, Apr 30, 2025 at 5:16 PM Jarek Potiuk
<ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote:
Yep - all the dags will be visible in `breeze` in my PR
https://github.com/apache/airflow/pull/49978 - just pushed a change for it:
[image.png]
On Wed, Apr 30, 2025 at 4:31 PM Jarek Potiuk
<ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote:
Answering Constance (somewhat it went to another thread:)
I know we rely on the example dags for dev purposes, especially for UI
development. As part of moving them out of core, can we update breeze to
automatically load them from their new home?
Yes. Very good point. I strongly think we need to do it. And it's part of the
issue :).
I already - in my PR - made it so in `breeze` and local venv where `tests` are part of the system we have now
two "example" bundles - the second one coming from the standard provider. That was one of the
reasons my PR was not green initially - because also some of our tests in core assume that
"standard" provider examples are present. I adjusted some of them to expect "simplest"
dag - where it made no difference, but I also created an extra dag bundle to add the standard provider in
case the test system is importable locally in tests - so far so good.
But I think in order to make things **really** work for breeze installation
(and in local venv) - we would have to make it so the example dags are:
a) available in the provider packages
b) airflow should discover them from there
I can (quickly) make a hack for Breeze (that's the question of adding standard provider's
"tests" to PYTHONPATH) - in this PR - so It will cover that one and we will be
able to regenerate documentation with links to sources.
But I think in this discussion (and follow up PR) is to answer the question of what users
should see when they enable it in "installed" airflow.
* should users see the standard provider package dags when they install airflow locally
from packages and say "load_example_dags".
if we think "yes", then we generally need to include example_dags in provider
packages (they are excluded now) - and we can do it, no problem.
Now - I think we should also decide what users should "see" in this case - my first reaction would be
"standard" + "simplest" + "maybe a few others? (which ones) ?"
J.
On Wed, Apr 30, 2025 at 1:12 PM Kaxil Naik
<kaxiln...@gmail.com<mailto:kaxiln...@gmail.com>> wrote:
just an idea: With "git bundles", maybe example DAGs from standard provider
can be dynamically imported?
But from your options: (1) seems reasonable
On Wed, 30 Apr 2025 at 15:07, Jarek Potiuk
<ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote:
Hello here.
While fixing a bug in the doc generation to bring "source" link to example
dags https://github.com/apache/airflow/pull/49978 I realized that we have
not moved a number of python/bash etc. example dags to standard provider -
where they now belong. I do it as part of my PR - but as result the number
of "example_dags" shown in the UI when "load example dags" is true is going
to decrease (and also they are not necessarily the dags you would really
want to see).
I thought maybe about a follow up PR where we will do a few things:
1) allow to show example dags also from providers (or maybe just standard
provider)
2) limit a number of dags shown in airflow when "load example dags" config
is set - for example by tags - to only see those that are really useful (I
found the bash/python/comlex and few others really useful for quick
demo/testing - maybe there are others).
3) We could optionally have "show all examples" setting for the "really
power user case" and make it easy to test things in airfow UI
out-of-the-box - or even limit it per provider.
WDYT ?
J
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
