Re: New committer: Rahul Vats
Congratulations Rahul 🎊 On Tue, 1 Apr, 2025, 08:56 Aritra Basu, wrote: > Congrats Rahul, great job! > -- > Regards, > Aritra Basu > > On Tue, 1 Apr 2025, 8:42 am kalyan reddy, wrote: > > > Congratulations Rahul! Thanks for your amazing work. Well deserved !! > > > > From: Hemkumar Chheda > > Sent: 01 April 2025 07:12 > > To: dev@airflow.apache.org > > Subject: Re: New committer: Rahul Vats > > > > Congratulations Rahul! Well Deserved 🥳🎉 > > > > Best Regards, > > Hemkumar > > > > > On 1 Apr 2025, at 6:37 AM, Wei Lee wrote: > > > > > > Congratulations! > > > > > > Best, > > > Wei > > > > > >> On Apr 1, 2025, at 8:36 AM, Zhe You Liu wrote: > > >> > > >> Congrats Rahul, well deserved! > > >> > > >> On Tue, Apr 1, 2025 at 9:28 AM Ephraim Anierobi < > > ephraimanier...@gmail.com> > > >> wrote: > > >> > > >>> Congratulations Rahul! Well deserved > > >>> > > >>> On Tue, 1 Apr 2025 at 01:27, Pierre Jeambrun > > >>> wrote: > > >>> > > Amazing work Rahul, congratulations. > > > > On Mon 31 Mar 2025 at 23:14, Pavankumar Gopidesu < > > >>> gopidesupa...@gmail.com> > > wrote: > > > > > Woohoo, Congratulations Rahul 🎉🎉 > > > > > > Pavan. > > > > > > On Mon, Mar 31, 2025 at 9:17 PM Pankaj Koti > > > wrote: > > > > > >> Many congratulations Rahul! Very happy for you and well deserved > > 🎉🎉 > > >> > > >> On Tue, 1 Apr, 2025, 01:17 Buğra Öztürk, > > > > wrote: > > >> > > >>> Amazing news! Congratulations Rahul! Well deserved! > > >>> > > >>> On Mon, Mar 31, 2025 at 9:14 PM Vincent Beck < > vincb...@apache.org> > > >> wrote: > > >>> > > Congrats Rahul! Very well deserved! > > > > On 2025/03/31 19:08:09 Shubham Raj wrote: > > > Congrats Rahul, well deserved!! > > > > > > On Tue, 1 Apr 2025 at 00:33, Vishnu Chilukoori < > > vish.chiluko...@gmail.com> > > > wrote: > > > > > >> Congrats Rahul...well deserved!! > > >> > > >> > > >> -- > > >> Regards, > > >> Vishnu Chilukoori > > >> > > >> On Mon, Mar 31, 2025 at 11:53 AM Idris Adebisi > > >> wrote: > > >> > > >>> Congratulations, Rahul! > > >>> > > >>> On Monday, March 31, 2025 at 07:45:27 PM GMT+1, Ambika > > > Garg < > > >>> ambikagarg1...@gmail.com> wrote: > > >>> > > >>> Congratulations Rahul!! > > >>> On Mon, Mar 31, 2025 at 2:43 PM Ankit Chaurasia < > > >>> sunank...@gmail.com > > > > > >>> wrote: > > >>> > > Congratulations, Rahul! Well deserved. > > > > > > > > > > > > > > > > On Tue, Apr 1, 2025 at 12:16 AM Constance Martineau > > wrote: > > > > > Congrats Rahul! Well deserved :) > > > > > > On Mon, Mar 31, 2025 at 2:28 PM Oliveira, Niko > > > >> > > > wrote: > > > > > >> Congrats Rahul, great work! > > >> > > >> > > >> From: Jed Cunningham > > >> Sent: Monday, March 31, 2025 11:19:53 AM > > >> To: dev@airflow.apache.org > > >> Subject: [EXT] New committer: Rahul Vats > > >> > > >> CAUTION: This email originated from outside of the > > organization. Do > > >>> not > > >> click links or open attachments unless you can > > >>> confirm > > > the > > sender > > >> and > > > know > > >> the content is safe. > > >> > > >> > > >> > > >> AVERTISSEMENT: Ce courrier électronique provient d’un > > expéditeur > > externe. > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > jointe > > >> si > > vous > > >> ne > > > pouvez > > >> pas confirmer l’identité de l’expéditeur et si vous > > > n’êtes > > >>> pas > > >>> certain > > > que > > >> le contenu ne présente aucun risque. > > >> > > >> > > >> > > >> The Project Management Committee (PMC) for Apache > > Airflow > > >> has invited Rahul Vats to become a committer and we > > >>> are > > >>> pleased > > >> to announce that they have accepted. > > >> > > >> Please join us in welcoming Rahul to their new role > > >>> and > > >> responsibility in our project community. > > >> > > >> Rahul has had a tremendous impact on Airflow 3, > > primarily > > around > > testing. > > >>>
[ANNOUNCE] SequentialExecutor & DebugExecutor removed
Hi All, I’m excited to announce that the *Sequential Executor* and *Debug Executor* have been removed from the in-tree executors in our repository, thanks to the efforts of the community! (Discussed in https://lists.apache.org/thread/65jw3o3mn7qlc56f9bt2146fmoy0z5t6). These executors were primarily used for local testing and debugging but we have the LocalExecutor( https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html) which can achieve better results since it supports parallelism too. The LocalExecutor has also been ported over to use the new Airflow 3 way of running tasks using task SDK and task execution interface + API server. One of the reasons the Sequential Executor was kept around for so long was due to some "single" process restrictions while using Sqlite. With the newer versions of Sqlite supporting WAL mode, LocalExecutor now works well with Sqlite, making the SequentialExecutor redundant. [3] helped in accomplishing that. This was a long shot for Airflow 3 but we have managed to do this even before the release candidate, thanks to everyone who helped in reviewing the code and helping me get this one through. *What does this mean for you?* If you were using these executors, you should switch to alternatives. *LocalExecutor* is the easiest and the most natural replacement to any of those, but *CeleryExecutor* is a good option too if you do not mind an extra worker being spun up. I do not anticipate any major issues from this change, but do shout out on Slack, on #airflow-3-dev channel for help if you encounter something. Thanks & Best Regards, Amogh [1] - https://github.com/apache/airflow/issues/48522 [2] - https://github.com/apache/airflow/issues/48523 [3] - https://github.com/apache/airflow/pull/44839
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
We were going to have compatibility shims to redirect the imports - with - there are few ways to do it - Ash had a little POC with module loader, but I think it has some potential side effect and I think Ash abandoned the idea and I would personally prefer to use our old PEP-563 mechanism using airflow-core/src/airflow/utils/deprecation_tools.py, Very nice and small PR to implement if you want to contribute - and since you are testing it now with some existing DAGS it might also be a good test if no redirect has been forgotten J. On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote: > Hi everyone. > > I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would like > to discuss this topic. > > I took bunch of DAGs from existing Airflow 2 instances and deployed them to > instance with Airflow 3 (3.0.0b4) and have bunch of import errors: > - ModuleNotFoundError: No module named > 'airflow.operators.python_operator' > - ModuleNotFoundError: No module named 'airflow.operators.bash_operator' > - ImportError: cannot import name 'email_operator' from > 'airflow.operators' > - ModuleNotFoundError: No module named 'airflow.operators.dummy_operator' > > I know that users are supposed to migrate from using "airflow.operators" to > standard/stmp/.. provider packages before upgrading to Airflow 3. > > But I also remember some discussions around keeping old imports work, by > rerouting them to the correct module (similarly as we do in case of > deprecated classes, etc.). > So, it will be very smooth for users to migrate to Airflow 3. > > What is our stand on this? Do we abandon "airflow.operators" usage in DAGs > in Airflow 3 completely? > Or this is something that needs to be done in Airflow 3, but not yet. > > -- > Eugene >
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
An example where deprection_tools are still used
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
It's rather straightforward = needs a package with __init__.py - only where
you list all the classes and provide redirections. It will
automatically raise deprecation warnings:
from airflow.utils.deprecation_tools import add_deprecated_classes
__deprecated_classes = {
"cloudwatch_task_handler": {
"CloudwatchTaskHandler": (
"airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
),
},
"es_task_handler": {
"ElasticsearchTaskHandler": (
"airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
),
},
"gcs_task_handler": {
"GCSTaskHandler":
"airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
},
"s3_task_handler": {
"S3TaskHandler":
"airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
},
"stackdriver_task_handler": {
"StackdriverTaskHandler": (
"airflow.providers.google.cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
),
},
"wasb_task_handler": {
"WasbTaskHandler":
"airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
},
}
add_deprecated_classes(__deprecated_classes, __name__)
On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
> We were going to have compatibility shims to redirect the imports - with -
> there are few ways to do it - Ash had a little POC with module loader, but
> I think it has some potential side effect and I think Ash abandoned
> the idea and I would personally prefer to use our old PEP-563 mechanism
> using airflow-core/src/airflow/utils/deprecation_tools.py,
>
> Very nice and small PR to implement if you want to contribute - and since
> you are testing it now with some existing DAGS it might also be a good test
> if no redirect has been forgotten
>
> J.
>
>
> On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote:
>
>> Hi everyone.
>>
>> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
>> like
>> to discuss this topic.
>>
>> I took bunch of DAGs from existing Airflow 2 instances and deployed them
>> to
>> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
>> - ModuleNotFoundError: No module named
>> 'airflow.operators.python_operator'
>> - ModuleNotFoundError: No module named 'airflow.operators.bash_operator'
>> - ImportError: cannot import name 'email_operator' from
>> 'airflow.operators'
>> - ModuleNotFoundError: No module named
>> 'airflow.operators.dummy_operator'
>>
>> I know that users are supposed to migrate from using "airflow.operators"
>> to
>> standard/stmp/.. provider packages before upgrading to Airflow 3.
>>
>> But I also remember some discussions around keeping old imports work, by
>> rerouting them to the correct module (similarly as we do in case of
>> deprecated classes, etc.).
>> So, it will be very smooth for users to migrate to Airflow 3.
>>
>> What is our stand on this? Do we abandon "airflow.operators" usage in DAGs
>> in Airflow 3 completely?
>> Or this is something that needs to be done in Airflow 3, but not yet.
>>
>> --
>> Eugene
>>
>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Those were deprecated 4.5+ years ago:
https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
> Instead handle it via ruff rules AIR2 something
>
> On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
>
>> ` - ModuleNotFoundError: No module named
>> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x old
>>
>> We had already stripped `_operator` from the module names in Airflow
>> 2.0.0 -- so IMO there is no need to keep back-compatibility for something
>> that was working 2 major versions ago
>>
>> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
>>
>>> An example where deprection_tools are still used
>>>
>>> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
>>>
>>> It's rather straightforward = needs a package with __init__.py - only
>>> where
>>> you list all the classes and provide redirections. It will
>>> automatically raise deprecation warnings:
>>>
>>> from airflow.utils.deprecation_tools import add_deprecated_classes
>>>
>>> __deprecated_classes = {
>>> "cloudwatch_task_handler": {
>>> "CloudwatchTaskHandler": (
>>>
>>>
>>> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
>>> ),
>>> },
>>> "es_task_handler": {
>>> "ElasticsearchTaskHandler": (
>>>
>>>
>>> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
>>> ),
>>> },
>>> "gcs_task_handler": {
>>> "GCSTaskHandler":
>>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
>>> },
>>> "s3_task_handler": {
>>> "S3TaskHandler":
>>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
>>> },
>>> "stackdriver_task_handler": {
>>> "StackdriverTaskHandler": (
>>>
>>> "airflow.providers.google
>>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
>>> ),
>>> },
>>> "wasb_task_handler": {
>>> "WasbTaskHandler":
>>>
>>> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
>>> },
>>> }
>>>
>>> add_deprecated_classes(__deprecated_classes, __name__)
>>>
>>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
>>>
>>> > We were going to have compatibility shims to redirect the imports -
>>> with -
>>> > there are few ways to do it - Ash had a little POC with module loader,
>>> but
>>> > I think it has some potential side effect and I think Ash abandoned
>>> > the idea and I would personally prefer to use our old PEP-563 mechanism
>>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
>>> >
>>> > Very nice and small PR to implement if you want to contribute - and
>>> since
>>> > you are testing it now with some existing DAGS it might also be a good
>>> test
>>> > if no redirect has been forgotten
>>> >
>>> > J.
>>> >
>>> >
>>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev
>>> wrote:
>>> >
>>> >> Hi everyone.
>>> >>
>>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
>>> >> like
>>> >> to discuss this topic.
>>> >>
>>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
>>> them
>>> >> to
>>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
>>> >> - ModuleNotFoundError: No module named
>>> >> 'airflow.operators.python_operator'
>>> >> - ModuleNotFoundError: No module named
>>> 'airflow.operators.bash_operator'
>>> >> - ImportError: cannot import name 'email_operator' from
>>> >> 'airflow.operators'
>>> >> - ModuleNotFoundError: No module named
>>> >> 'airflow.operators.dummy_operator'
>>> >>
>>> >> I know that users are supposed to migrate from using
>>> "airflow.operators"
>>> >> to
>>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
>>> >>
>>> >> But I also remember some discussions around keeping old imports work,
>>> by
>>> >> rerouting them to the correct module (similarly as we do in case of
>>> >> deprecated classes, etc.).
>>> >> So, it will be very smooth for users to migrate to Airflow 3.
>>> >>
>>> >> What is our stand on this? Do we abandon "airflow.operators" usage in
>>> DAGs
>>> >> in Airflow 3 completely?
>>> >> Or this is something that needs to be done in Airflow 3, but not yet.
>>> >>
>>> >> --
>>> >> Eugene
>>> >>
>>> >
>>>
>>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Ah.. I missed that we have those _operators ... Those indeed should have
been fixed a long time ago and automated conversion rules from ruff
should be fixing them.
On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote:
> Those were deprecated 4.5+ years ago:
>
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
>
> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
>
> > Instead handle it via ruff rules AIR2 something
> >
> > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> >
> >> ` - ModuleNotFoundError: No module named
> >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> old
> >>
> >> We had already stripped `_operator` from the module names in Airflow
> >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> something
> >> that was working 2 major versions ago
> >>
> >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> >>
> >>> An example where deprection_tools are still used
> >>>
> >>>
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> >>>
> >>> It's rather straightforward = needs a package with __init__.py - only
> >>> where
> >>> you list all the classes and provide redirections. It will
> >>> automatically raise deprecation warnings:
> >>>
> >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> >>>
> >>> __deprecated_classes = {
> >>> "cloudwatch_task_handler": {
> >>> "CloudwatchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> >>> ),
> >>> },
> >>> "es_task_handler": {
> >>> "ElasticsearchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> >>> ),
> >>> },
> >>> "gcs_task_handler": {
> >>> "GCSTaskHandler":
> >>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
> >>> },
> >>> "s3_task_handler": {
> >>> "S3TaskHandler":
> >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> >>> },
> >>> "stackdriver_task_handler": {
> >>> "StackdriverTaskHandler": (
> >>>
> >>> "airflow.providers.google
> >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> >>> ),
> >>> },
> >>> "wasb_task_handler": {
> >>> "WasbTaskHandler":
> >>>
> >>>
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> >>> },
> >>> }
> >>>
> >>> add_deprecated_classes(__deprecated_classes, __name__)
> >>>
> >>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
> >>>
> >>> > We were going to have compatibility shims to redirect the imports -
> >>> with -
> >>> > there are few ways to do it - Ash had a little POC with module
> loader,
> >>> but
> >>> > I think it has some potential side effect and I think Ash abandoned
> >>> > the idea and I would personally prefer to use our old PEP-563
> mechanism
> >>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> >>> >
> >>> > Very nice and small PR to implement if you want to contribute - and
> >>> since
> >>> > you are testing it now with some existing DAGS it might also be a
> good
> >>> test
> >>> > if no redirect has been forgotten
> >>> >
> >>> > J.
> >>> >
> >>> >
> >>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev
> >>> wrote:
> >>> >
> >>> >> Hi everyone.
> >>> >>
> >>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and
> would
> >>> >> like
> >>> >> to discuss this topic.
> >>> >>
> >>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
> >>> them
> >>> >> to
> >>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> >>> >> - ModuleNotFoundError: No module named
> >>> >> 'airflow.operators.python_operator'
> >>> >> - ModuleNotFoundError: No module named
> >>> 'airflow.operators.bash_operator'
> >>> >> - ImportError: cannot import name 'email_operator' from
> >>> >> 'airflow.operators'
> >>> >> - ModuleNotFoundError: No module named
> >>> >> 'airflow.operators.dummy_operator'
> >>> >>
> >>> >> I know that users are supposed to migrate from using
> >>> "airflow.operators"
> >>> >> to
> >>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
> >>> >>
> >>> >> But I also remember some discussions around keeping old imports
> work,
> >>> by
> >>> >> rerouting them to the correct module (similarly as we do in case of
> >>> >> deprecated classes, etc.).
> >>> >> So, it will be very smooth for users to migrate to Airflow 3.
> >>> >>
> >>> >> What is our stand on this? Do we abandon "airflow.operators" usage
> in
> >>> DAGs
> >>> >> in Airflow 3 completely?
> >>> >> Or this is something that needs to be done in Airflow 3, but not
> yet.
> >>> >>
> >>> >> --
> >>> >> Eugene
> >>> >>
> >>> >
> >>>
> >>
>
Re: [VOTE] Release Apache Airflow Helm Chart 1.16.0 based on 1.16.0rc1
+1 binding. Checked signatures, checksums, licences, reproducibility - all good. All my changes are in (updated images mostly), installed and run airflow and 2.10.5 was up-and-running flawlessly. All good to go. On Tue, Apr 1, 2025 at 7:41 AM Amogh Desai wrote: > +1 non binding. > > I didn't have any particular change in there but I deployed the new chart > and tested it with various dags > and also clicked around the UI a bit here and there. Seems to work fine! > > Thanks & Regards, > Amogh Desai > > > On Sun, Mar 30, 2025 at 10:15 AM Vishnu Chilukoori < > vish.chiluko...@gmail.com> wrote: > > > +1 non-binding > > > > > > -- > > Regards, > > Vishnu Chilukoori > > > > On Sat, Mar 29, 2025 at 12:15 PM Jens Scheffler > > > > > wrote: > > > > > +1 (binding) - Checked SVN, Reproducible package build, Licenses, > > > Signatures > > > > > > On 29.03.25 15:20, Jed Cunningham wrote: > > > > Hello Apache Airflow Community, > > > > > > > > This is a call for the vote to release Helm Chart version 1.16.0. > > > > > > > > The release candidate is available at: > > > > https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.16.0rc1/ > > > > > > > > airflow-chart-1.16.0-source.tar.gz - is the "main source release" > that > > > > comes with INSTALL instructions. > > > > airflow-1.16.0.tgz - is the binary Helm Chart release. > > > > > > > > Public keys are available at: > https://www.apache.org/dist/airflow/KEYS > > > > > > > > For convenience "index.yaml" has been uploaded (though excluded from > > > > voting), so you can also run the below commands. > > > > > > > > helm repo add apache-airflow-dev > > > > https://dist.apache.org/repos/dist/dev/airflow/helm-chart/1.16.0rc1/ > > > > helm repo update > > > > helm install airflow apache-airflow-dev/airflow > > > > > > > > airflow-1.16.0.tgz.prov - is also uploaded for verifying Chart > > Integrity, > > > > though not strictly required for releasing the artifact based on ASF > > > > Guidelines. > > > > > > > > $ helm gpg verify airflow-1.16.0.tgz > > > > gpg: Signature made Sat Mar 29 07:59:32 2025 MDT > > > > gpg:using RSA key > > > E1A1E984F55B8F280BD9CBA20BB7163892A2E48E > > > > gpg:issuer "jedcunning...@apache.org" > > > > gpg: Good signature from "Jed Cunningham " > > > > [ultimate] > > > > plugin: Chart SHA verified. > > > > > sha256:45f7bc33fffb3f80113ea41051c49956cddf67b6b0dd47e97219459fed0a83f1 > > > > > > > > The vote will be open for at least 72 hours (2025-04-01 14:24 UTC) or > > > until > > > > the necessary number of votes is reached. > > > > > > > > > > > > > > https://www.timeanddate.com/countdown/to?iso=20250401T1424&p0=136&font=cursive > > > > > > > > Please vote accordingly: > > > > > > > > [ ] +1 approve > > > > [ ] +0 no opinion > > > > [ ] -1 disapprove with the reason > > > > > > > > Only votes from PMC members are binding, but members of the community > > are > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > Consider this my (binding) +1. > > > > > > > > For license checks, the .rat-excludes files is included, so you can > run > > > the > > > > following to verify licenses (just update your path to rat): > > > > > > > > tar -xvf airflow-chart-1.16.0-source.tar.gz > > > > cd airflow-chart-1.16.0 > > > > java -jar apache-rat-0.13.jar chart -E .rat-excludes > > > > > > > > Please note that the version number excludes the `rcX` string, so > it's > > > now > > > > simply 1.16.0. This will allow us to rename the artifact without > > > modifying > > > > the artifact checksums when we actually release it. > > > > > > > > The status of testing the Helm Chart by the community is kept here: > > > > https://github.com/apache/airflow/issues/48531 > > > > > > > > Thanks, > > > > Jed > > > > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
[RESULT][VOTE] Release Apache Airflow Helm Chart 1.16.0 based on 1.16.0rc1
Hello all, The vote to release Apache Airflow Helm Chart version 1.0.1 based on 1.0.1rc1 is now closed. The vote PASSED with 3 binding "+1", 2 non-binding "+1" and 0 "-1" votes: "+1" Binding votes: - Jed Cunningham - Jens Scheffler - Jarek Potiuk "+1" Non-Binding votes: - Vishnu Chilukoori - Amogh Desai Vote thread: https://lists.apache.org/thread/8css7sh2bj2ny8d86cd2g4v55b3cob8b I'll continue with the release process and the release announcement will follow shortly. Thanks, Jed
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Hi Kaxil, > Instead handle it via ruff rules AIR2 something Does that mean we need to add an `AIR2` for `_operator`? In the current implementation, they are in `AIR303` (moved to provider). It also raises a new thing I'd like to confirm. We're to mark the suggested change for `airflow.operators.python` as they are supposed to work ok on Airflow 3, but it would be better to change them to a standard provider. For `airflow.operators.python_operator`, it sounds like something is no longer working in Airflow 3. If this is the case, we probably need to mark them as required changes in Airflow 3 and suggested changes in AIR2? --- ferruzzi, > But to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. > I've lost track of who is working on what at this point I'm working on the ruff thing. AIR3 things only for now, though. --- Tamara, > but might change to "AIR30" for crucial changes and "AIR31" for suggested changes We already reached a consensus with the ruff team but will need some time to implement it. https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 --- Best, Wei > On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > wrote: > > Hi Eugen > > As others have said >from airflow.operators.python_operator import PythonOperator > has been deprecated a long time ago and wont work anymore in Airflow 3. > >from airflow.operators.python import PythonOperator > which was the preferred import for recent years is going to get deprecated > for Airflow 3 but will still work in 3.0 (likely will be removed eventually) > > and the new preferred import will be from standard providers: >from airflow.providers.standard.operators.python import PythonOperator > > There is a ruff rule in the works ( > https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" but > might change to "AIR30" for crucial changes and "AIR31" for suggested > changes and it is still being worked on (currently it would catch the > removed import from python_operator but recommend the deprecated statement, > not the one from standard providers yet). > >ruff check --preview --select AIR3 > is the command right now if you want to test, but I'd wait until Airflow 3 > release so it is all final. There will also be upgrading guides. :) > > > > On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > wrote: > >> IMHO, if they were deprecated that long ago then adding a ruff rule is >> just enabling the users to ignore the deprecation. They should >> /eventually/ have to clear some of their tech debt and actually update >> their code to modern standards, no? They've had four and half YEARS of >> warning at this point. But to the original topic, an AF2-AF3 ruff rule >> was "always" the plan AFAIK. I've lost track of who is working on what at >> this point, but I know I heard talk about that almost since the beginning, >> no? That seems like as good of a place as any to handle this redirect. >> >> >> - ferruzzi >> >> >> >> From: Jarek Potiuk >> Sent: Tuesday, April 1, 2025 9:32 AM >> To: dev@airflow.apache.org >> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que >> le contenu ne présente aucun risque. >> >> >> >> Ah.. I missed that we have those _operators ... Those indeed should have >> been fixed a long time ago and automated conversion rules from ruff >> should be fixing them. >> >> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote: >> >>> Those were deprecated 4.5+ years ago: >>> >>> >> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py >>> >>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote: >>> Instead handle it via ruff rules AIR2 something On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote: > ` - ModuleNotFoundError: No module named > 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x >>> old > > We had already stripped `_operator` from the module names in Airflow > 2.0.0 -- so IMO there is no need to keep back-compatibility for >>> something > that was working 2 major versions ago > > On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote: > >> An example where deprection_tools are still used >> >> >>> >> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py >> >> It's rather straightforward = needs a package with __init__.py - only >> where >> you list all the classes and provide redirections. It will >
[ANNOUNCE] Apache Airflow Helm Chart version 1.16.0 Released
Dear Airflow community, I am pleased to announce that we have released Apache Airflow Helm chart 1.16.0 🎉 🎊 The source release, as well as the "binary" Helm Chart release, are available: 📦 Official Sources: https://airflow.apache.org/docs/helm-chart/1.16.0/installing-helm-chart-from-sources.html 📦 ArtifactHub: https://artifacthub.io/packages/helm/apache-airflow/airflow 📚 Docs: https://airflow.apache.org/docs/helm-chart/1.16.0/ 🚀 Quick Start Installation Guide: https://airflow.apache.org/docs/helm-chart/1.16.0/quick-start.html 🛠️ Release Notes: https://airflow.apache.org/docs/helm-chart/1.16.0/release_notes.html Thanks to all the contributors who made this possible. Thanks, Jed
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
IMHO, if they were deprecated that long ago then adding a ruff rule is just
enabling the users to ignore the deprecation. They should /eventually/ have to
clear some of their tech debt and actually update their code to modern
standards, no? They've had four and half YEARS of warning at this point. But
to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. I've
lost track of who is working on what at this point, but I know I heard talk
about that almost since the beginning, no? That seems like as good of a place
as any to handle this redirect.
- ferruzzi
From: Jarek Potiuk
Sent: Tuesday, April 1, 2025 9:32 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le
contenu ne présente aucun risque.
Ah.. I missed that we have those _operators ... Those indeed should have
been fixed a long time ago and automated conversion rules from ruff
should be fixing them.
On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote:
> Those were deprecated 4.5+ years ago:
>
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
>
> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
>
> > Instead handle it via ruff rules AIR2 something
> >
> > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> >
> >> ` - ModuleNotFoundError: No module named
> >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> old
> >>
> >> We had already stripped `_operator` from the module names in Airflow
> >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> something
> >> that was working 2 major versions ago
> >>
> >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> >>
> >>> An example where deprection_tools are still used
> >>>
> >>>
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> >>>
> >>> It's rather straightforward = needs a package with __init__.py - only
> >>> where
> >>> you list all the classes and provide redirections. It will
> >>> automatically raise deprecation warnings:
> >>>
> >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> >>>
> >>> __deprecated_classes = {
> >>> "cloudwatch_task_handler": {
> >>> "CloudwatchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> >>> ),
> >>> },
> >>> "es_task_handler": {
> >>> "ElasticsearchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> >>> ),
> >>> },
> >>> "gcs_task_handler": {
> >>> "GCSTaskHandler":
> >>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
> >>> },
> >>> "s3_task_handler": {
> >>> "S3TaskHandler":
> >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> >>> },
> >>> "stackdriver_task_handler": {
> >>> "StackdriverTaskHandler": (
> >>>
> >>> "airflow.providers.google
> >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> >>> ),
> >>> },
> >>> "wasb_task_handler": {
> >>> "WasbTaskHandler":
> >>>
> >>>
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> >>> },
> >>> }
> >>>
> >>> add_deprecated_classes(__deprecated_classes, __name__)
> >>>
> >>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
> >>>
> >>> > We were going to have compatibility shims to redirect the imports -
> >>> with -
> >>> > there are few ways to do it - Ash had a little POC with module
> loader,
> >>> but
> >>> > I think it has some potential side effect and I think Ash abandoned
> >>> > the idea and I would personally prefer to use our old PEP-563
> mechanism
> >>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> >>> >
> >>> > Very nice and small PR to implement if you want to contribute - and
> >>> since
> >>> > you are testing it now with some existing DAGS it might also be a
> good
> >>> test
> >>> > if no redirect has been forgotten
> >>> >
> >>> > J.
> >>> >
> >>> >
> >>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev
> >>> wrote:
> >>> >
> >>> >> Hi everyone.
> >>> >>
> >>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and
> would
> >>> >> like
> >>> >> to discuss this topic.
> >>> >>
> >>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
> >>> them
> >>> >> to
> >>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> >>> >> - ModuleNotFoundError: No module named
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Instead handle it via ruff rules AIR2 something
On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> ` - ModuleNotFoundError: No module named
> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x old
>
> We had already stripped `_operator` from the module names in Airflow 2.0.0
> -- so IMO there is no need to keep back-compatibility for something that
> was working 2 major versions ago
>
> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
>
>> An example where deprection_tools are still used
>>
>> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
>>
>> It's rather straightforward = needs a package with __init__.py - only
>> where
>> you list all the classes and provide redirections. It will
>> automatically raise deprecation warnings:
>>
>> from airflow.utils.deprecation_tools import add_deprecated_classes
>>
>> __deprecated_classes = {
>> "cloudwatch_task_handler": {
>> "CloudwatchTaskHandler": (
>>
>>
>> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
>> ),
>> },
>> "es_task_handler": {
>> "ElasticsearchTaskHandler": (
>>
>>
>> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
>> ),
>> },
>> "gcs_task_handler": {
>> "GCSTaskHandler":
>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
>> },
>> "s3_task_handler": {
>> "S3TaskHandler":
>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
>> },
>> "stackdriver_task_handler": {
>> "StackdriverTaskHandler": (
>>
>> "airflow.providers.google
>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
>> ),
>> },
>> "wasb_task_handler": {
>> "WasbTaskHandler":
>> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
>> },
>> }
>>
>> add_deprecated_classes(__deprecated_classes, __name__)
>>
>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
>>
>> > We were going to have compatibility shims to redirect the imports -
>> with -
>> > there are few ways to do it - Ash had a little POC with module loader,
>> but
>> > I think it has some potential side effect and I think Ash abandoned
>> > the idea and I would personally prefer to use our old PEP-563 mechanism
>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
>> >
>> > Very nice and small PR to implement if you want to contribute - and
>> since
>> > you are testing it now with some existing DAGS it might also be a good
>> test
>> > if no redirect has been forgotten
>> >
>> > J.
>> >
>> >
>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote:
>> >
>> >> Hi everyone.
>> >>
>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
>> >> like
>> >> to discuss this topic.
>> >>
>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
>> them
>> >> to
>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
>> >> - ModuleNotFoundError: No module named
>> >> 'airflow.operators.python_operator'
>> >> - ModuleNotFoundError: No module named
>> 'airflow.operators.bash_operator'
>> >> - ImportError: cannot import name 'email_operator' from
>> >> 'airflow.operators'
>> >> - ModuleNotFoundError: No module named
>> >> 'airflow.operators.dummy_operator'
>> >>
>> >> I know that users are supposed to migrate from using
>> "airflow.operators"
>> >> to
>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
>> >>
>> >> But I also remember some discussions around keeping old imports work,
>> by
>> >> rerouting them to the correct module (similarly as we do in case of
>> >> deprecated classes, etc.).
>> >> So, it will be very smooth for users to migrate to Airflow 3.
>> >>
>> >> What is our stand on this? Do we abandon "airflow.operators" usage in
>> DAGs
>> >> in Airflow 3 completely?
>> >> Or this is something that needs to be done in Airflow 3, but not yet.
>> >>
>> >> --
>> >> Eugene
>> >>
>> >
>>
>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
` - ModuleNotFoundError: No module named
'airflow.operators.python_operator'` <-- those paths are Airflow 1.x old
We had already stripped `_operator` from the module names in Airflow 2.0.0
-- so IMO there is no need to keep back-compatibility for something that
was working 2 major versions ago
On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> An example where deprection_tools are still used
>
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
>
> It's rather straightforward = needs a package with __init__.py - only where
> you list all the classes and provide redirections. It will
> automatically raise deprecation warnings:
>
> from airflow.utils.deprecation_tools import add_deprecated_classes
>
> __deprecated_classes = {
> "cloudwatch_task_handler": {
> "CloudwatchTaskHandler": (
>
>
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> ),
> },
> "es_task_handler": {
> "ElasticsearchTaskHandler": (
>
>
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> ),
> },
> "gcs_task_handler": {
> "GCSTaskHandler":
> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
> },
> "s3_task_handler": {
> "S3TaskHandler":
> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> },
> "stackdriver_task_handler": {
> "StackdriverTaskHandler": (
>
> "airflow.providers.google
> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> ),
> },
> "wasb_task_handler": {
> "WasbTaskHandler":
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> },
> }
>
> add_deprecated_classes(__deprecated_classes, __name__)
>
> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
>
> > We were going to have compatibility shims to redirect the imports - with
> -
> > there are few ways to do it - Ash had a little POC with module loader,
> but
> > I think it has some potential side effect and I think Ash abandoned
> > the idea and I would personally prefer to use our old PEP-563 mechanism
> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> >
> > Very nice and small PR to implement if you want to contribute - and
> since
> > you are testing it now with some existing DAGS it might also be a good
> test
> > if no redirect has been forgotten
> >
> > J.
> >
> >
> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote:
> >
> >> Hi everyone.
> >>
> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
> >> like
> >> to discuss this topic.
> >>
> >> I took bunch of DAGs from existing Airflow 2 instances and deployed them
> >> to
> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> >> - ModuleNotFoundError: No module named
> >> 'airflow.operators.python_operator'
> >> - ModuleNotFoundError: No module named
> 'airflow.operators.bash_operator'
> >> - ImportError: cannot import name 'email_operator' from
> >> 'airflow.operators'
> >> - ModuleNotFoundError: No module named
> >> 'airflow.operators.dummy_operator'
> >>
> >> I know that users are supposed to migrate from using "airflow.operators"
> >> to
> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
> >>
> >> But I also remember some discussions around keeping old imports work, by
> >> rerouting them to the correct module (similarly as we do in case of
> >> deprecated classes, etc.).
> >> So, it will be very smooth for users to migrate to Airflow 3.
> >>
> >> What is our stand on this? Do we abandon "airflow.operators" usage in
> DAGs
> >> in Airflow 3 completely?
> >> Or this is something that needs to be done in Airflow 3, but not yet.
> >>
> >> --
> >> Eugene
> >>
> >
>
Re: New committer: Rahul Vats
Congratulations Rahul, very well deserved! On Tue, Apr 1, 2025 at 10:35 AM Karthikeyan wrote: > Congratulations Rahul. Thanks for all your efforts towards the Airflow 3 > release including detailed bug reports that improved release stability. > > On Tue, Apr 1, 2025 at 12:53 PM Kunal Bhattacharya > wrote: > > > Congratulations Rahul, very well deserved :) > > > > Regards, > > Kunal Bhattacharya > > > > On Tue, Apr 1, 2025 at 11:04 AM Amogh Desai > > wrote: > > > > > Congratulations Rahul! > > > > > > I have first hand seen the impact you create in Airflow and the way you > > > drive > > > testing efforts. Kudos to you. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Tue, Apr 1, 2025 at 9:24 AM Phani Kumar > > > wrote: > > > > > > > Congratulations Rahul 🎊 > > > > > > > > On Tue, 1 Apr, 2025, 08:56 Aritra Basu, > > > wrote: > > > > > > > > > Congrats Rahul, great job! > > > > > -- > > > > > Regards, > > > > > Aritra Basu > > > > > > > > > > On Tue, 1 Apr 2025, 8:42 am kalyan reddy, > > > wrote: > > > > > > > > > > > Congratulations Rahul! Thanks for your amazing work. Well > deserved > > !! > > > > > > > > > > > > From: Hemkumar Chheda > > > > > > Sent: 01 April 2025 07:12 > > > > > > To: dev@airflow.apache.org > > > > > > Subject: Re: New committer: Rahul Vats > > > > > > > > > > > > Congratulations Rahul! Well Deserved 🥳🎉 > > > > > > > > > > > > Best Regards, > > > > > > Hemkumar > > > > > > > > > > > > > On 1 Apr 2025, at 6:37 AM, Wei Lee > wrote: > > > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > Best, > > > > > > > Wei > > > > > > > > > > > > > >> On Apr 1, 2025, at 8:36 AM, Zhe You Liu > > > > > wrote: > > > > > > >> > > > > > > >> Congrats Rahul, well deserved! > > > > > > >> > > > > > > >> On Tue, Apr 1, 2025 at 9:28 AM Ephraim Anierobi < > > > > > > ephraimanier...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Congratulations Rahul! Well deserved > > > > > > >>> > > > > > > >>> On Tue, 1 Apr 2025 at 01:27, Pierre Jeambrun < > > > > pierrejb...@gmail.com> > > > > > > >>> wrote: > > > > > > >>> > > > > > > Amazing work Rahul, congratulations. > > > > > > > > > > > > On Mon 31 Mar 2025 at 23:14, Pavankumar Gopidesu < > > > > > > >>> gopidesupa...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Woohoo, Congratulations Rahul 🎉🎉 > > > > > > > > > > > > > > Pavan. > > > > > > > > > > > > > > On Mon, Mar 31, 2025 at 9:17 PM Pankaj Koti > > > > > > > wrote: > > > > > > > > > > > > > >> Many congratulations Rahul! Very happy for you and well > > > deserved > > > > > > 🎉🎉 > > > > > > >> > > > > > > >> On Tue, 1 Apr, 2025, 01:17 Buğra Öztürk, < > > > > ozturkbugr...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > >> > > > > > > >>> Amazing news! Congratulations Rahul! Well deserved! > > > > > > >>> > > > > > > >>> On Mon, Mar 31, 2025 at 9:14 PM Vincent Beck < > > > > > vincb...@apache.org> > > > > > > >> wrote: > > > > > > >>> > > > > > > Congrats Rahul! Very well deserved! > > > > > > > > > > > > On 2025/03/31 19:08:09 Shubham Raj wrote: > > > > > > > Congrats Rahul, well deserved!! > > > > > > > > > > > > > > On Tue, 1 Apr 2025 at 00:33, Vishnu Chilukoori < > > > > > > vish.chiluko...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > >> Congrats Rahul...well deserved!! > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> Regards, > > > > > > >> Vishnu Chilukoori > > > > > > >> > > > > > > >> On Mon, Mar 31, 2025 at 11:53 AM Idris Adebisi > > > > > > >> wrote: > > > > > > >> > > > > > > >>> Congratulations, Rahul! > > > > > > >>> > > > > > > >>> On Monday, March 31, 2025 at 07:45:27 PM GMT+1, > > Ambika > > > > > > > Garg < > > > > > > >>> ambikagarg1...@gmail.com> wrote: > > > > > > >>> > > > > > > >>> Congratulations Rahul!! > > > > > > >>> On Mon, Mar 31, 2025 at 2:43 PM Ankit Chaurasia < > > > > > > >>> sunank...@gmail.com > > > > > > > > > > > > > >>> wrote: > > > > > > >>> > > > > > > Congratulations, Rahul! Well deserved. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 1, 2025 at 12:16 AM Constance Martineau > > > > > > wrote: > > > > > > > > > > > > > Congrats Rahul! Well deserved :) > > > > > > > > > > > > > > On Mon, Mar 31, 2025 at 2:28 PM Oliveira, Niko > > > > > > > > > > > >> > > > > > > > wrote: > > > > > > > > > > > > > >> Co
Re: [ANNOUNCE] SequentialExecutor & DebugExecutor removed
Yay! Moar deletions! Just one comment to that - we **know** that forking + threading in MacOS causes SEGV (Segmentation Violation) issues when you are using some additional libraries that can get initialized by **just** importing them and they start threads, when it happens before we attempt to make the first fork with Local Executor. This is a known issue in MacOS + Python, it's racy (i.e. it might or might not happen at any given execution) and there is not much we can do - with exception of **not using** those libraries or using some workarounds. An example of that - that might be happening when logging configuration uses some libraries and initialises loggers (like AWS with boto). In this case setting `NO_PROXY=true" env variable helps for example. But this is case-by-case and it's hard to both - know what the root cause is for SIGSEGV and finding remediations is more of a "don't use this or that". Another option for MacOS users is to use containerized / dockerized airflow rather than in their local environment. We should be aware of that - because that's what was the feature of the Sequential Executor - it did not do forking when executing tasks. We might add better protection or detection over time, but for now we should be wary of running airflow locally with more "production-like" settings and we should likely gather more information about those cases to find a better remedy. That's calculated risk, really - we know it's an issue, and we know some users will have problems with running airflow locally with Local Executor, but we think it's better to remove those old executors anyway. Let's monitor and try to improve that part over time. J. On Tue, Apr 1, 2025 at 11:52 AM Amogh Desai wrote: > Hi All, > > I’m excited to announce that the *Sequential Executor* and *Debug Executor* > have been removed from the in-tree executors in our repository, thanks to > the efforts of the community! (Discussed in > https://lists.apache.org/thread/65jw3o3mn7qlc56f9bt2146fmoy0z5t6). > > These executors were primarily used for local testing and debugging but we > have the LocalExecutor( > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html > ) > which can achieve better results since it supports parallelism too. The > LocalExecutor has also been ported over to use the new Airflow 3 way of > running tasks using task SDK and task execution interface + API server. > > One of the reasons the Sequential Executor was kept around for so long was > due to some "single" process restrictions while using Sqlite. With the > newer versions of Sqlite supporting WAL mode, LocalExecutor now works well > with Sqlite, making the SequentialExecutor redundant. [3] helped in > accomplishing that. > > This was a long shot for Airflow 3 but we have managed to do this even > before the release candidate, thanks to everyone who helped in reviewing > the code and helping me get this one through. > > *What does this mean for you?* > If you were using these executors, you should switch to alternatives. > *LocalExecutor* is the easiest and the most natural replacement to any of > those, but *CeleryExecutor* is a good option too if you do not mind an > extra worker being spun up. > > I do not anticipate any major issues from this change, but do shout out on > Slack, on #airflow-3-dev channel for help if you encounter something. > > Thanks & Best Regards, > Amogh > > [1] - https://github.com/apache/airflow/issues/48522 > [2] - https://github.com/apache/airflow/issues/48523 > [3] - https://github.com/apache/airflow/pull/44839 >
[DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Hi everyone. I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would like to discuss this topic. I took bunch of DAGs from existing Airflow 2 instances and deployed them to instance with Airflow 3 (3.0.0b4) and have bunch of import errors: - ModuleNotFoundError: No module named 'airflow.operators.python_operator' - ModuleNotFoundError: No module named 'airflow.operators.bash_operator' - ImportError: cannot import name 'email_operator' from 'airflow.operators' - ModuleNotFoundError: No module named 'airflow.operators.dummy_operator' I know that users are supposed to migrate from using "airflow.operators" to standard/stmp/.. provider packages before upgrading to Airflow 3. But I also remember some discussions around keeping old imports work, by rerouting them to the correct module (similarly as we do in case of deprecated classes, etc.). So, it will be very smooth for users to migrate to Airflow 3. What is our stand on this? Do we abandon "airflow.operators" usage in DAGs in Airflow 3 completely? Or this is something that needs to be done in Airflow 3, but not yet. -- Eugene
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Hi Eugen
As others have said
from airflow.operators.python_operator import PythonOperator
has been deprecated a long time ago and wont work anymore in Airflow 3.
from airflow.operators.python import PythonOperator
which was the preferred import for recent years is going to get deprecated
for Airflow 3 but will still work in 3.0 (likely will be removed eventually)
and the new preferred import will be from standard providers:
from airflow.providers.standard.operators.python import PythonOperator
There is a ruff rule in the works (
https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" but
might change to "AIR30" for crucial changes and "AIR31" for suggested
changes and it is still being worked on (currently it would catch the
removed import from python_operator but recommend the deprecated statement,
not the one from standard providers yet).
ruff check --preview --select AIR3
is the command right now if you want to test, but I'd wait until Airflow 3
release so it is all final. There will also be upgrading guides. :)
On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis
wrote:
> IMHO, if they were deprecated that long ago then adding a ruff rule is
> just enabling the users to ignore the deprecation. They should
> /eventually/ have to clear some of their tech debt and actually update
> their code to modern standards, no? They've had four and half YEARS of
> warning at this point. But to the original topic, an AF2-AF3 ruff rule
> was "always" the plan AFAIK. I've lost track of who is working on what at
> this point, but I know I heard talk about that almost since the beginning,
> no? That seems like as good of a place as any to handle this redirect.
>
>
> - ferruzzi
>
>
>
> From: Jarek Potiuk
> Sent: Tuesday, April 1, 2025 9:32 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> Ah.. I missed that we have those _operators ... Those indeed should have
> been fixed a long time ago and automated conversion rules from ruff
> should be fixing them.
>
> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote:
>
> > Those were deprecated 4.5+ years ago:
> >
> >
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
> >
> > On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
> >
> > > Instead handle it via ruff rules AIR2 something
> > >
> > > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> > >
> > >> ` - ModuleNotFoundError: No module named
> > >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> > old
> > >>
> > >> We had already stripped `_operator` from the module names in Airflow
> > >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> > something
> > >> that was working 2 major versions ago
> > >>
> > >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> > >>
> > >>> An example where deprection_tools are still used
> > >>>
> > >>>
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> > >>>
> > >>> It's rather straightforward = needs a package with __init__.py - only
> > >>> where
> > >>> you list all the classes and provide redirections. It will
> > >>> automatically raise deprecation warnings:
> > >>>
> > >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> > >>>
> > >>> __deprecated_classes = {
> > >>> "cloudwatch_task_handler": {
> > >>> "CloudwatchTaskHandler": (
> > >>>
> > >>>
> > >>>
> >
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> > >>> ),
> > >>> },
> > >>> "es_task_handler": {
> > >>> "ElasticsearchTaskHandler": (
> > >>>
> > >>>
> > >>>
> >
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> > >>> ),
> > >>> },
> > >>> "gcs_task_handler": {
> > >>> "GCSTaskHandler":
> > >>> "airflow.providers.google
> .cloud.log.gcs_task_handler.GCSTaskHandler",
> > >>> },
> > >>> "s3_task_handler": {
> > >>> "S3TaskHandler":
> > >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> > >>> },
> > >>> "stackdriver_task_handler": {
> > >>> "StackdriverTaskHandler": (
> > >>>
> > >>> "airflow.providers.google
> > >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> > >>> ),
> > >>> },
> > >>> "wasb_task_handler": {
> > >>> "WasbTaskHandler":
> > >>>
> > >>>
> >
> "airflow.providers.microsoft.azure.log.was
