Hi all, I have a few announcements regarding AIP-47 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests> .
First of all, after the AIP has been approved on 24th February, we were working hard on preparing the first PR that will include a first batch with migrated system tests and all related changes compliant with the AIP-47. We have prepared code changes with related documentation, new pre-commit hooks, deleted old tests, updated paths and so on... We tried to keep the PR as small as possible (it only covers Google BigQuery operators) but also to have a complete solution so that you can see how it will all work in practice. The PR is available here: https://github.com/apache/airflow/pull/22311 Please take a look and share your comments in the issue. Secondly, we did some changes to the AIP-47 document and I wanted to call for a lazy consensus since these changes were strongly useful for the design - some simplifications and improvements were done. The voting will be running until the first PR is merged to the Airflow main branch. Until then, we can discuss, do other improvements and updates. Some more important changes in the AIP: - DebugExecutor is now contained in conftest.py inside tests/system, not a part of each system test, - if __name__ == "__main__" condition is no longer enclosed in each system test (we decided that running these tests through pytest is enough and we don't need to support the execution via python command), - updated "Process of migration in details" and "Complete test example" with more detailed information on the steps, to make it more clear how to do the migration, - updated "Which users are affected by the change" and added there a number of example DAGs that may need to be migrated to the new design. Also, updated the numbers as of 14th March 2022, - updated "Test structure" with information about modified pytest.ini, which now also recognize file starting with example_* as test files. The diff reflecting changes in the AIP from the moment when it was accepted till today can be found here ( https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=199532973&selectedPageVersions=7&selectedPageVersions=11) for your review. The linked PR will hang there as long as there are no concerns and all required approvals are provided. Reading the PR would be probably the best way to familiarize with the changes and the new design. You are also more than welcome to share the news about this AIP and hopefully the rest of the community can join us with migrating system tests to this new design. We have more Google-related operators already covered by new system tests, but we will hold with creating PRs with those changes till the moment where this first PR is merged. Last, but not least, Google has deposited 10,000 $ in credits into the GCP project dedicated for the exclusive use of running CI in Airflow and it is now maintained by the community. It is under preparation to make the new system tests run automatically. More info and the progress of configuring it for the community needs will be announced later. When the mentioned PR <https://github.com/apache/airflow/pull/22311> will be merged, all providers are welcome to start migrating their system tests and decide whether they want to prepare their example DAGs to be also run in the new design (even if they were not run before). Kind regards, Mateusz Nojek
