Hi everyone,
As Airflow already supports lineage functionality through pluggable lineage
backends, I think OpenLineage and other lineage systems integration should
follow this path. I think more 'native' integration with OpenLineage (or
any other lineage system) in Airflow while maintaining the ge
-1 (non-binding)
Dask is more niche than Celery or K8s - let's err on the side of making
Airflow core dependencies more light-weight and not preinstall it.
On Fri, Jul 21, 2023 at 10:57 AM Pankaj Koti
wrote:
> 0 (non-binding)
>
> I agree with most of what has been discussed in the thread
> http
In general I support this suggestion for more clarity and separation, but
on the assumption we at least protect the core sections as in point 2).
With a scale large enough, someone, consciously or not, will break the
gentleman's agreement.
On Fri, Jul 21, 2023 at 7:13 AM Jarek Potiuk wrote:
> He
+1 - let's avoid forcing users to modify their DAGs
On Sun, Aug 13, 2023 at 7:19 PM Vikram Koka
wrote:
> +1 (with emphasis)
>
>
> On Sun, Aug 13, 2023 at 5:23 AM Ephraim Anierobi <
> ephraimanier...@gmail.com>
> wrote:
>
> > +1 binding
> >
> > On Sun, Aug 13, 2023 at 6:57 AM Elad Kalif wrote:
>
+1 - I strongly agree with the direction too.
On Fri, Aug 18, 2023 at 2:31 PM Jarek Potiuk wrote:
> I am strongly for it. I wanted to raise the same proposal after we see the
> results of the survey (which I hope we will run after the Summit) - in
> order to get more data points on how many user
sers, developers and passionates. The meetup will be hybrid - we'll share
the link with the participants.
Best,
Michał Modras
First, I'd like to say that I support Eugen's proposal and I agree with the
enhancements suggested by Jarek.
I'm a bit confused about Elad's point 3 - are you suggesting having a
global policy for all providers, or that we should not codify our approach
at all since different providers have differ
parameter/operator will be removed
> > >
> > > Everything else, such as decommission/removal only once bumping major
> > > version, cadence of releases, etc. is not something that I propose to
> > > change.
> > >
> > > The idea in general is to mak
My 2 cents: it must be possible to opt-out, preferably it should be
possible to deploy Airflow instances without bundling the telemetry library
dependencies. Other than that I don't mind it being e.g. optional provider.
śr., 3 kwi 2024, 22:42 użytkownik Hussein Awala napisał:
> > I'd like to pro
rs who
> don't want to send metrics will disable it.
>
>
> On Fri, Apr 5, 2024 at 6:19 PM Michał Modras
> wrote:
>
> > My 2 cents: it must be possible to opt-out, preferably it should be
> > possible to deploy Airflow instances without bundling the telemetry
> l
+1 to Jens's & Bolke's points here and in the doc
I agree we should work on clarifying the directions we would like Airflow
to go. Introducing a new major Airflow version is a massive overhead for
users, who would need to plan for migrations, onboarding the new Airflow
(with a slightly different a
> > > makes
> > > >> no
> > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is
> > coming
> > > -
> > > >> > that generally triples effort needed to get them out. We should
> drop
> > > new
&
and perhaps move
> from a model where TaskInstance listeners are executed on the worker to one
> where they simply
> depend on that metadata rather than the TaskInstance object. They could be
> executed elsewhere - on
> some specialized component working asynchronously, like the trigger
+1 for moving the meeting by a week - 30th is a bank holiday in Poland too.
On Tue, May 28, 2024 at 6:40 AM Amogh Desai
wrote:
> Lovely!
>
> Looking forward to it. Btw, as Jarek mentioned, many folks would be
> travelling for Community Over Code, next week
> and we also have some public holidays
Great work Freddy!
On Thu, Jun 13, 2024 at 4:08 AM Wei Lee wrote:
> This is great!
>
> Best,
> Wei
>
> > On Jun 12, 2024, at 9:47 PM, Bishundeo, Rajeshwar
> wrote:
> >
> > Fantastic job from the Google team. Love it!!
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > On 2024-06-12, 9:20 AM, "Panka
Hi,
+1 to this idea. I think standardizing the format of the presented test run
results makes sense. I also agree that we don't necessarily need to enforce
it in any hard way. However, given that we have dashboards of these three
major providers, we could consider enforcing the presence of *some*
+1 from my side as well, as mentioned before there's no clear downside to
it. Good stuff
czw., 27 cze 2024, 06:34 użytkownik Amogh Desai
napisał:
> Excellent proposal! I see no down-side to the proposal
>
> Good investigation on the higher level implementation part as well.
>
> Thanks & Regards,
Thanks for work on this Ash, great to see how the proposal is getting more
material. Overall I'm positive about the general direction, I left a few
comments (and questions) on some of the sections.
On Mon, Jul 8, 2024 at 6:11 PM Jarek Potiuk wrote:
> > I’d love to support websockets/async but ha
Good discussion here, few points from our perspective:
- +1 for separating DAG submission into an API endpoint or SDK method. Not
only it creates a clean interface boundary for a parsed DAG, but also
enables other modes of using that endpoint, e.g. event-driven, which could
be a prelude for on-dem
Huge +1 - I think this would be a super useful process, in these critical
situations taking the burden off the release managers, and empowering
parties that are actually pressured to act (say owners of the service the
provider has integrations with). It also means the fixes for the overlooked
issue
+1 non-binding
In general I think the AIP makes sense, and there are of course details to
clarify and iron out - but this can happen on the way.
On Fri, Jul 12, 2024 at 11:20 AM Ephraim Anierobi
wrote:
> +1 binding
>
> On Fri, 12 Jul 2024 at 06:41, Scheffler Jens (XC-AS/EAE-ADA-T)
> wrote:
>
>
I think it makes sense to orchestrate backfills in a more managed way than
through a CLI command, especially that execution of tasks would happen
through regular executors configured in the Airflow deployment. Once
concern I have, also called out in the AIP, is the increased load on the
scheduler.
I think we should finish the AIP within Airflow 2 - it will take time until
Airflow 3 is out, and I believe some learnings from finishing and running
this AIP might be useful for Airflow 3. We plan to contribute to finishing
this AIP.
On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T
-1 (non-binding)
While the cleaner approach to templates is appealing, the blast radius of
this change in its current shape is enormous. I am worried that it would
strongly impede migration of users from Airflow 2 to Airflow 3, especially
that not all Airflow users are proficient in Airflow, and f
+1 for both, non-binding
On Thu, Aug 1, 2024 at 9:34 PM Igor Kholopov
wrote:
> +1 (non-binding, both)
>
> On Thu, Aug 1, 2024 at 9:04 PM Vikram Koka
> wrote:
>
> > +1 binding on both AIP-79 and AIP-84.
> >
> > Vikram
> >
> >
> > On Thu, Aug 1, 2024 at 11:35 AM Vincent Beck
> wrote:
> >
> > > +
I understand that the compatibility layer mentioned by TP would allow
providers to work with both Airflow 2.x and 3.x. I could see it working in
the following ways:
1) You can use DAGs with Airflow 3.x syntax in Airflow 2.x, so you can
migrate your DAGs gradually to 3.x syntax before upgrading to i
Congratulations Jens, well deserved!
On Tue, Aug 6, 2024 at 9:51 AM Jarek Potiuk wrote:
> The Project Management Committee (PMC) for Apache [PROJECT]
> has invited Jens to become a PMC member and we are pleased
> to announce that they have accepted.
>
> Jens has contributed for a long time alrea
al compatibility layer). But it will not be actively maintained.
>
> TP
>
>
> > On 6 Aug 2024, at 04:55, Michał Modras
> wrote:
> >
> > I understand that the compatibility layer mentioned by TP would allow
> > providers to work with both Airflow 2.x and 3.x. I
aintainers :)), it actually will discentivise some users from moving to
Airflow 3 entirely. If we care for Airflow 3 adoption and not putting
blockers for users for it, we should go with 2). I believe it should be at
least a provider and maintained.
On Wed, Aug 7, 2024 at 10:18 AM Michał Modras
w
ementation easier too.
>
> TP
>
> > On 7 Aug 2024, at 16:19, Michał Modras
> wrote:
> >
> > I don't think (obviously) is so obvious. While 2) might be seen as not
> > incentivising to change anything (btw software forcing people to change
> > something tha
Yes, there are two options. One - forward compatibility layer, and two -
backwards compatibility layer.
I strongly believe that if we care for Airflow 3 adoption, providing
forward compatibility layers only is not enough, and lack of backwards
compatibility layer in case of changes that bring mostl
+1
On Mon, Aug 19, 2024 at 4:21 PM Wei Lee wrote:
> +1
>
> Best,
> Wei
>
> > On Aug 19, 2024, at 9:59 PM, Pierre Jeambrun
> wrote:
> >
> > +1
> >
> > On Mon, Aug 19, 2024 at 3:29 PM Jed Cunningham >
> > wrote:
> >
> >> +1
> >>
>
>
> -
Congratulations Vikram!
On Mon, Oct 21, 2024 at 6:55 AM Vikram Koka wrote:
> Thank you Kaxil, Tomek, Pankaj Singh, Shubham, Wei, Bugra, Pankaj Koti,
> Amogh, Jarek, Rom, Elad, Rahul, Vishnu, Phani and the Airflow PMC. I very
> much appreciate your kind works.
>
> I am honored to join the Apache
Hi Kaxil,
Thanks for the reminder. Unfortunately I won't make it today, but on moving
the mtg to the 1st of November - it is a bank holiday in Poland, so on
behalf of Poland-based community members, could we please pick some other
date? Thanks!
Best,
Michal
On Thu, Sep 19, 2024 at 4:56 PM Kaxil
I've also witnessed the zipped DAGs feature to be used quite a bit - in
scenarios similar to what Jarek & Constance described, and also to e.g.
avoid downloading a multitude of files from blob storage (less effective
cost & performance wise).
On Wed, Feb 5, 2025 at 6:08 PM Constance Martineau
wro
>This change would bring parity between the `output` property and the
classic `xcom_pull()` method. The obvious drawback is this would be a
slight authoring change for existing DAGs that use the `output` property.
Perhaps if the change could be automated in migration tooling the behavior
change wou
+1 - separating these workloads makes sense to me - we remove
unnecessary coupling and make them more single-responsibility, which eases
reasoning about the system and any potential debugging
On Fri, Jan 10, 2025 at 9:15 AM Kaxil Naik wrote:
> Yeah, purely from operational perspective, debuggi
Email notifications are a *massively* used Airflow feature. I suggest that
any change does not break the current syntax of BaseOperator, avoiding
forcing users to modify their DAGs code. Changes on the config side and
provider separation should be fine though (seems separating provider while
keepin
default_to_email if we want to mitigate that
> risk.
>
> On Tue, Jan 28, 2025 at 12:19 AM Hussein Awala wrote:
>
> > > avoiding forcing users to modify their DAGs code
> >
> > Actually, modifying DAGs code is the easiest part of the migration
> > for users, becau
them. We’re not asking users to update all their
> > code at once - any option we choose will be backported to 2.11, allowing
> > users to clean up their code and resolve incompatibilities smoothly
> before
> > upgrading to Airflow 3.0.
> >
> > On Mon, Jan
ere are no judgment
> calls.
> I do not follow on why this is risky nor why it is expensive.
> Is the issue you are worried about that this change can not be automated by
> the migration tool we plan to have?
>
> Am I missing something?
>
>
> On Tue, Jan 28, 2025 at 3:48 PM
+1, I fully agree with the mechanism - it is open, transparent, gives
everyone an opportunity to participate, and keeps track of how the
decisions are made.
On Sat, Mar 22, 2025 at 5:34 PM Shahar Epstein wrote:
> Overall I agree that decisions should be made in the devlist.
> One improvement tha
; > >> >>> Just one thing - the pre / post mechanisms are executed in-process
> > of
> > >> the
> > >> >>> task rather than the DAG. So they are not equal to setup/teardown.
> > Are
> > >> >> you
> >
This is huge, congratulations to everyone involved!
On Wed, Apr 23, 2025 at 7:54 AM Amogh Desai
wrote:
> Congratulations all! Airflow 2 is literally nostalgia now!
>
> Looking forward to initial set of bugs to keep our release vehicle moving
> and feedback from early adopters.
>
>
> Thanks & Reg
Would removing it imply that there's not going to be Airflow 2.12 for sure?
Do we want to limit ourselves this way?
On Tue, Feb 18, 2025 at 12:35 PM Abhishek Bhakat
wrote:
> Where do I find the docs for plugins with the new web UI alternative for <
>
> https://airflow.apache.org/docs/apache-airf
Thanks Ash. Then it makes sense to me.
wt., 18 lut 2025, 13:51 użytkownik Ash Berlin-Taylor
napisał:
> This is only talking about the main branch and Airflow 3.0, Airflow 2 is
> already branched off and that UI won’t be removed in 2.x at all.
>
> > On 18 Feb 2025, at 11:54
I strongly disagree with the proposal of changing the default for all DAGs.
This requires every user that does not specify catchup to modify their
DAGs. As pointed out in other similar threads about changes requiring DAG
code changes:
>I am concerned simply because it is a physical code change, an
can even suggest it as part of
> migration process) - and the compatibility is set.
>
> And if my understanding is right, I am quite in favour of this proposal.
>
> J.
>
>
>
>
> On Wed, Mar 5, 2025 at 7:54 PM Michał Modras
> wrote:
>
> > I strongly disagr
I'd prefer a world without separate pre_execute and post_execute functions
- as pointed out in the PR, they make reasoning about DAGs more complex,
and can be error prone.
Having said that, I know there are multiple users relying on these
functionalities, so I'll bring up my usual - another breaki
49 matches
Mail list logo
