Hello there! Since the main PR (#22311 <https://github.com/apache/airflow/pull/22311>) with the implementation of AIP-47 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests> has been merged, I am now closing the lazy consensus voting. The discussion was conducted in the comments section to the PR. All suggestions were addressed or implemented. The PR is in the Top #60 of most commented issues in the whole history of Airflow on GitHub! :)
The AIP is now also completed, and the next phase is to migrate old system tests to the new design in compliance with the AIP-47. It was a busy and exciting journey - it took exactly a month for the AIP to be accepted and another straight month to be implemented! Thank you all for help, comments, ideas and support! Kind regards, Mateusz Nojek pon., 21 mar 2022 o 15:09 Mateusz Nojek <matno...@gmail.com> napisał(a): > Hello, > > After the discussion in the comments section of PR #22311 > <https://github.com/apache/airflow/pull/22311>, there were several > updates and changes in the code. The overall hasn't changed significantly, > but few enhancements were added. Along with the changes in the code, I > updated the AIP-47 > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests> > document > and wanted to list here the modified sections: > > - Updated point 7. with watcher import moved closer to its usage in > "Process of migration in details" > - Updated point 9. with updated pytest function in "Process of > migration in details" > - Updated "Complete test example" with latest changes to the design > - Updated "Design details"/"Test structure" with new content of > conftest.py and utils file for system tests > > I would also like to thank you for your great suggestions and encourage > others to familiarize with this design. It is a huge improvement that will > hopefully increase the quality of testing and drop the number of new bugs > in the next releases. We are really getting this whole initiative closer to > perfection. > > Kind regards, > Mateusz Nojek > > > śr., 16 mar 2022 o 15:21 Mateusz Nojek <matno...@gmail.com> napisał(a): > >> Hi all, >> >> I have a few announcements regarding AIP-47 >> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests> >> . >> >> First of all, after the AIP has been approved on 24th February, we were >> working hard on preparing the first PR that will include a first batch with >> migrated system tests and all related changes compliant with the AIP-47. We >> have prepared code changes with related documentation, new pre-commit >> hooks, deleted old tests, updated paths and so on... We tried to keep the >> PR as small as possible (it only covers Google BigQuery operators) but also >> to have a complete solution so that you can see how it will all work in >> practice. >> The PR is available here: https://github.com/apache/airflow/pull/22311 >> Please take a look and share your comments in the issue. >> >> Secondly, we did some changes to the AIP-47 document and I wanted to call >> for a lazy consensus since these changes were strongly useful for the >> design - some simplifications and improvements were done. The voting will >> be running until the first PR is merged to the Airflow main branch. Until >> then, we can discuss, do other improvements and updates. >> >> Some more important changes in the AIP: >> - DebugExecutor is now contained in conftest.py inside tests/system, not >> a part of each system test, >> - if __name__ == "__main__" condition is no longer enclosed in each >> system test (we decided that running these tests through pytest is enough >> and we don't need to support the execution via python command), >> - updated "Process of migration in details" and "Complete test example" >> with more detailed information on the steps, to make it more clear how to >> do the migration, >> - updated "Which users are affected by the change" and added there a >> number of example DAGs that may need to be migrated to the new design. >> Also, updated the numbers as of 14th March 2022, >> - updated "Test structure" with information about modified pytest.ini, >> which now also recognize file starting with example_* as test files. >> >> The diff reflecting changes in the AIP from the moment when it was >> accepted till today can be found here ( >> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=199532973&selectedPageVersions=7&selectedPageVersions=11) >> for your review. >> >> The linked PR will hang there as long as there are no concerns and all >> required approvals are provided. Reading the PR would be probably the best >> way to familiarize with the changes and the new design. You are also more >> than welcome to share the news about this AIP and hopefully the rest of the >> community can join us with migrating system tests to this new design. >> >> We have more Google-related operators already covered by new system >> tests, but we will hold with creating PRs with those changes till the >> moment where this first PR is merged. >> >> Last, but not least, Google has deposited 10,000 $ in credits into the >> GCP project dedicated for the exclusive use of running CI in Airflow and it >> is now maintained by the community. It is under preparation to make the new >> system tests run automatically. More info and the progress of configuring >> it for the community needs will be announced later. >> >> When the mentioned PR <https://github.com/apache/airflow/pull/22311> >> will be merged, all providers are welcome to start migrating their system >> tests and decide whether they want to prepare their example DAGs to be also >> run in the new design (even if they were not run before). >> >> Kind regards, >> Mateusz Nojek >> >
