IMHO, if they were deprecated that long ago then adding a ruff rule is just enabling the users to ignore the deprecation. They should /eventually/ have to clear some of their tech debt and actually update their code to modern standards, no? They've had four and half YEARS of warning at this point. But to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. I've lost track of who is working on what at this point, but I know I heard talk about that almost since the beginning, no? That seems like as good of a place as any to handle this redirect.
- ferruzzi ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Tuesday, April 1, 2025 9:32 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Ah.. I missed that we have those _operators ... Those indeed should have been fixed a long time ago and automated conversion rules from ruff should be fixing them. On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Those were deprecated 4.5+ years ago: > > https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py > > On Tue, 1 Apr 2025 at 21:45, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Instead handle it via ruff rules AIR2 something > > > > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > >> ` - ModuleNotFoundError: No module named > >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x > old > >> > >> We had already stripped `_operator` from the module names in Airflow > >> 2.0.0 -- so IMO there is no need to keep back-compatibility for > something > >> that was working 2 major versions ago > >> > >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >>> An example where deprection_tools are still used > >>> > >>> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py > >>> > >>> It's rather straightforward = needs a package with __init__.py - only > >>> where > >>> you list all the classes and provide redirections. It will > >>> automatically raise deprecation warnings: > >>> > >>> from airflow.utils.deprecation_tools import add_deprecated_classes > >>> > >>> __deprecated_classes = { > >>> "cloudwatch_task_handler": { > >>> "CloudwatchTaskHandler": ( > >>> > >>> > >>> > "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler" > >>> ), > >>> }, > >>> "es_task_handler": { > >>> "ElasticsearchTaskHandler": ( > >>> > >>> > >>> > "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler" > >>> ), > >>> }, > >>> "gcs_task_handler": { > >>> "GCSTaskHandler": > >>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler", > >>> }, > >>> "s3_task_handler": { > >>> "S3TaskHandler": > >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler", > >>> }, > >>> "stackdriver_task_handler": { > >>> "StackdriverTaskHandler": ( > >>> > >>> "airflow.providers.google > >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler" > >>> ), > >>> }, > >>> "wasb_task_handler": { > >>> "WasbTaskHandler": > >>> > >>> > "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler", > >>> }, > >>> } > >>> > >>> add_deprecated_classes(__deprecated_classes, __name__) > >>> > >>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >>> > >>> > We were going to have compatibility shims to redirect the imports - > >>> with - > >>> > there are few ways to do it - Ash had a little POC with module > loader, > >>> but > >>> > I think it has some potential side effect and I think Ash abandoned > >>> > the idea and I would personally prefer to use our old PEP-563 > mechanism > >>> > using airflow-core/src/airflow/utils/deprecation_tools.py, > >>> > > >>> > Very nice and small PR to implement if you want to contribute - and > >>> since > >>> > you are testing it now with some existing DAGS it might also be a > good > >>> test > >>> > if no redirect has been forgotten > >>> > > >>> > J. > >>> > > >>> > > >>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev <eu...@kosteev.com> > >>> wrote: > >>> > > >>> >> Hi everyone. > >>> >> > >>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and > would > >>> >> like > >>> >> to discuss this topic. > >>> >> > >>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed > >>> them > >>> >> to > >>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors: > >>> >> - ModuleNotFoundError: No module named > >>> >> 'airflow.operators.python_operator' > >>> >> - ModuleNotFoundError: No module named > >>> 'airflow.operators.bash_operator' > >>> >> - ImportError: cannot import name 'email_operator' from > >>> >> 'airflow.operators' > >>> >> - ModuleNotFoundError: No module named > >>> >> 'airflow.operators.dummy_operator' > >>> >> > >>> >> I know that users are supposed to migrate from using > >>> "airflow.operators" > >>> >> to > >>> >> standard/stmp/.. provider packages before upgrading to Airflow 3. > >>> >> > >>> >> But I also remember some discussions around keeping old imports > work, > >>> by > >>> >> rerouting them to the correct module (similarly as we do in case of > >>> >> deprecated classes, etc.). > >>> >> So, it will be very smooth for users to migrate to Airflow 3. > >>> >> > >>> >> What is our stand on this? Do we abandon "airflow.operators" usage > in > >>> DAGs > >>> >> in Airflow 3 completely? > >>> >> Or this is something that needs to be done in Airflow 3, but not > yet. > >>> >> > >>> >> -- > >>> >> Eugene > >>> >> > >>> > > >>> > >> >
