The example I give was a simplified version, and also a continuation of another
DAG process.
The issue I tried to solve in Airflow here for this case (we have also other
use cases where we ran into the same issue) was reading n number of users from
MSGraph, which where updated and had to be syn
Curious why you want to model this as many tasks, e.g. one page == one task.
Another option would be to handle many pages in one task. And I'm curious
what were the factors that led you to split it out more granularly.
Hello,
At our company we have DAG's which have to process lots of paged results (e.g.
XComs results from REST endpoints or multiple files from FTP downloads).
The Xcom isn't the issue, as we use a custom provider which allows us to store
large data on a Persistent Volume Claim without overloadi
+1 (non-binding)
--
Regards
Vishnu Chilukoori
On Wed, Sep 18, 2024, 15:03 Wei Lee wrote:
> Tested my changes and a few example dags. Work fine.
>
> +1 (non-binding)
>
> Best,
> Wei
>
> > On Sep 18, 2024, at 4:17 AM, Jens Scheffler
> wrote:
> >
> > Hi,
> >
> > I tested reporoducibility, SVN, l
I like what Daniel suggested. But I guess we’ll probably need to split a few
executors from existing providers for this? If that’s the case, are we going to
have those executors exist in both side for some time? or is there a better for
it?
Best,
Wei
> On Sep 17, 2024, at 1:16 AM, Jens Scheffl
Tested my changes and a few example dags. Work fine.
+1 (non-binding)
Best,
Wei
> On Sep 18, 2024, at 4:17 AM, Jens Scheffler
> wrote:
>
> Hi,
>
> I tested reporoducibility, SVN, licenses and signatures. All OK.
>
> Also I tested the fixed I applied/contributes/merged in the patch
> release
Thanks for the meeting Jarek, it was truly valuable.
בתאריך יום ב׳, 16 בספט׳ 2024 ב-8:48 מאת Elad Kalif <elad...@apache.org
>:
> It was a great meeting.
> Thabk you for setting it Jarek
>
> בתאריך שבת, 14 בספט׳ 2024, 01:34, מאת Jarek Potiuk :
>
> > Hey here,
> >
> > Following a discussio
