Hello All, I would like to start the lazy consensus based on the above - I think it is pretty natural next steps of internal isolation between the distributions we have in the repo and it will be a nice step towards more reuse between them :)
On Tue, Nov 18, 2025 at 7:20 AM Amogh Desai <[email protected]> wrote: > Thanks for sharing this. > > Some / Most providers already follow a similar structure: > https://github.com/apache/airflow/tree/main/providers/amazon/tests > and having related things closer will provide better maintainability as > well as discoverability. Any effort in that area is highly > appreciated. > > That said, for k8s_tests, I think the best course of action would be to > separate it out and place it where it belongs. What I mean > by that is, the k8s_tests/ has tests varying from testing running dags > using the KubernetesExecutor to testing KPO and also > testing any executor / general execution with K8s as the environment. > > A lot of this can be cleaned up, move KE / KPO tests to cncf provider and > general execution to helm_tests as e2e tests or the > likes of it. Although this is a separate discussion that can be taken up > later. > > Thanks & Regards, > Amogh Desai > > > On Tue, Nov 18, 2025 at 3:47 AM Jarek Potiuk <[email protected]> wrote: > > > > I'd put them rather in core as an integration for this... but as said, > > there might be more options and much more opinions in the room. But > > capability is more important so "praise" to all who have contributed to > > this! (Hope Edge will be added there soon as well...) > > > > We can also put them to core, yes, but my reasoning why `chart` is > > that they **actually** use the chart and they **mainly** test if > > Airflow installed with the chart "works". So they are (at least the > > main ones that work with all executor+k8s matrix) are pretty muc > > testing the chart e2e. > > > > But there are indeed some tests that are testing k8s pod operators, > > and maybe a good compromise will be to split those and put the "chart" > > tests in chart, but the "k8s" Pod operator ones in "cncf.kubernetes" > > provider. Those tests are anyhow quite a bit different and we won't > > touch them for a while - so we can decide later. Also I think the k8s > > POD operators might be anyhow eventually separated and done > > differently - Kalyan currently works on making the `breeze --executor > > Kubernetes` to work and once it does, we might want to separate the > > K8SPod Operators anyway to use this rather than setting up airflow > > with the chart - as it's not needed there at all. > > > > I would just defer the decision on k8s_tests for now. > > > > > > J. > > > > On Mon, Nov 17, 2025 at 11:03 PM Jens Scheffler <[email protected]> > > wrote: > > > > > > Very cool! > > > > > > Not sure whether they are really placed well in "chart" because we do > > > not want to use K8s tests to check the chart but the full integration > of > > > Airflow with Kubernetes. I tincludes core, K8s provider, Task SDK... so > > > for K8s tests we need at least one exception to the rule. > > > > > > I'd put them rather in core as an integration for this... but as said, > > > there might be more options and much more opinions in the room. But > > > capability is more important so "praise" to all who have contributed to > > > this! (Hope Edge will be added there soon as well...) > > > > > > Jens > > > > > > On 11/17/25 19:41, Jarek Potiuk wrote: > > > >> I have one question about the k8s tests in the future: should they > > live > > > > under airflow-core, under task-sdk, or in a separate dedicated > > location? > > > > > > > > This is an excellent question. And as surprising as it might be I > think > > > > there should be in .... chart > > > > > > > > chart > > > > tests > > > > unit <- current "helm_tests" > > > > e2e <- current "k8s_tests" > > > > > > > > > > > > > > > > On Mon, Nov 17, 2025 at 7:09 PM Zhe-You Liu <[email protected]> > > wrote: > > > > > > > >> Hi Jarek, > > > >> > > > >> Thanks for proposing the new e2e test structure, the task-sdk e2e > > setup > > > >> looks great to me! > > > >> I have one question about the k8s tests in the future: should they > > live > > > >> under airflow-core, under task-sdk, or in a separate dedicated > > location? > > > >> > > > >> Best regards, > > > >> Jason > > > >> > > > >> > > > >> On Mon, Nov 17, 2025 at 11:56 PM Jarek Potiuk <[email protected]> > > wrote: > > > >> > > > >>> Hello here, > > > >>> > > > >>> *TL;DR; We want to have better organized, and much improved new > > `e2e` > > > >> test > > > >>> type, a common thing in Airflow - one that others will be able to > add > > > >> tests > > > >>> to and add their own similar tests following the blueprint.* > > > >>> > > > >>> Over the last few weeks quite a few of us were working on something > > we > > > >> have > > > >>> not fully realise we will end up and it was mostly bottom -up grass > > root > > > >>> ideas that simply "clicked" together and we would like to propose > > adding > > > >>> (or more formalising) a new type of tests in airflow - e2e tests. > The > > > >>> people who worked on it were mostly Amogh, Zhe You lIu, Bugra, > > myself. > > > >>> > > > >>> Wa already have a few attempts to do it (in various stages - as > > top-level > > > >>> folders: *docker-tests, k8s-tests, airflow-e2e-tests, > > airflow-ctl-tests, > > > >>> task-sdk-integration-tests* - yes the lack of consistency in names > is > > > >> very > > > >>> confusing) but with recent improvements in > > task-sdk-integration-tests - > > > >> we > > > >>> think we have a good proposal on how we can implement those tests > and > > > >>> standardise it across airflow distributions. > > > >>> > > > >>> Current (best) approach is > > > >>> > > https://github.com/apache/airflow/tree/main/task-sdk-integration-tests > > > >>> > > > >>> And it has the following properties: > > > >>> > > > >>> 1) uses uv pytest (standard pytest tests), docker-compose and > > > >>> python-on-whales together, also has nice breeze wrapper for CI > > > >>> > > > >>> 2) automatically sets-up (and keeps running during iteration) > > > >>> docker-compose based airflow deployment - with components (any - > > > >>> api-server, dag processor, scheduler, triggerer) necessary during > the > > > >> tests > > > >>> and allows to interact with it, including triggering local dags and > > the > > > >>> like > > > >>> > > > >>> 3) pytest tests are run in a local venv controlled by uv sync > > > >>> > > > >>> 4) it allows for extremely fast (almost like pytest-native in venv) > > > >>> iteration speed with tests - it uses Zhe's hot-reload functionality > > > >> added a > > > >>> week ago in order to reload components inside the > docker-containers, > > and > > > >>> local sources are mounted to those - which means that just saving a > > file > > > >> in > > > >>> your IDE or local env will make automated (sub-second, really) > > reload of > > > >>> the components you interact with. Basically at the time you press > > > >> Shift-F10 > > > >>> (or whatever shortcut you have) to re-run your tests, your just > > > >> auto-saved > > > >>> modified Airflow code already runs in those docker-compose > > components (!) > > > >>> - *this is the most important feature of it.* > > > >>> > > > >>> 5) All those tests are automatically run in CI - in relevant PRs > > (driven > > > >> by > > > >>> selective checks) - but also they are very easily runnable locally > in > > > >> local > > > >>> venv (no breeze CI image needed). Basically: > > > >>> > > > >>> cd task-sdk-integration-tests > > > >>> uv run pytest > > > >>> > > > >>> 6) Last minute addition - we've been literally asked by Kacper an > > hour > > > >> ago > > > >>> about having "something like that" for open lineage - every current > > > >>> distribution is able to have their own set of tests like that if > the > > > >>> stewards of that part want it > > > >>> > > > >>> Enough bragging.... > > > >>> > > > >>> *Our proposal:* > > > >>> > > > >>> * we consistently rename those tests as `*e2e*` tests > > > >>> > > > >>> * we adapt all of them (including *k8s* tests eventually - where we > > have > > > >> to > > > >>> hack kind a bit) - to follow the same patterns and principles > > > >>> > > > >>> * we extract common code for that to `*devel-common*` and reuse > > across > > > >>> those tests > > > >>> > > > >>> * we get rid of all the top-level `*SMTH-test*` folders we have now > > and > > > >>> move those different kinds of tests to `*tests/e2e*` (next to > `unit`, > > > >>> `system`, `integration` we already have) of distributions that the > > tests > > > >>> are relevant to > > > >>> > > > >>> For example - instead of: > > > >>> > > > >>> task-sdk > > > >>> src > > > >>> tests > > > >>> task_sdk > > > >>> some_unit_test.py > > > >>> task-sdk-integration-tests > > > >>> dags <- here e2e test dags are stored > > > >>> logs <- here logs from all components are kept (.gitignored) > > > >>> tests > > > >>> task_sdk_tests > > > >>> some_end_2_end_test.py > > > >>> > > > >>> We propose: > > > >>> > > > >>> > > > >>> task-sdk > > > >>> src > > > >>> tests > > > >>> unit > > > >>> task_sdk > > > >>> some_unit_test.py > > > >>> e2e > > > >>> dags > > > >>> logs > > > >>> task_sdk > > > >>> some_end_2_end_test.py > > > >>> > > > >>> Of course - this is a proposal - and if there are other ideas on > how > > to > > > >>> restructure the test folders, that might be a good idea to do it > now. > > > >> *Only > > > >>> serious and complete offers are going to be considered :)*, so I > > propose > > > >>> constructive, complete proposals rather than criticising the > current > > one. > > > >>> > > > >>> *Let us know what you think,* > > > >>> > > > >>> Jarek, Zhe, Bugra, Amogh > > > >>> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
