Small clarification: the DAG processor manager still has access to the db -
but the DagFileProcessorProcess (eventually) doesn't, and uses the SDK
machinery to communicate with the DagFileProcessorManager. The manager acts
as its own "api server" in a way.
Either way though, in the SQLite context,
Thanks everyone, lazy consensus here:
https://lists.apache.org/thread/cx1mg5g1pw4k1km56hzr06d0o39kdk8d
On Fri, Jan 10, 2025 at 2:08 PM Jed Cunningham wrote:
> Small clarification: the DAG processor manager still has access to the db
> - but the DagFileProcessorProcess (eventually) doesn't, and u
Hello everyone,
I'm calling a lazy consensus on dropping support for the DAG processor
embedded in the scheduler. All of the responses in the discussion thread
are positive.
For more details on "why", go check out the discussion thread:
https://lists.apache.org/thread/g0vm5sh37ww2c36mz2mss311gxp6
+1 - as a very common problem in #user-troubleshooting is that DAGs are
vanishing because of poor DB performance or (mostly) poorly written
DAGs. If Performance problems appear in DAG parsing, separating the
parser is the first hand fix. So we should have this per default.
(Besides the case I hope
Dag processor does not access the DB at all - it communicates with Task SDK
to write the serialized DAGS.
On Fri, Jan 10, 2025 at 4:51 PM Igor Kholopov
wrote:
> The only thing that we need to clear out before this is sealed - what are
> we going to do with SQLite? SQLite supports concurrent conn
With Task API :)
On Fri, Jan 10, 2025 at 5:54 PM Jarek Potiuk wrote:
> Dag processor does not access the DB at all - it communicates with Task
> SDK to write the serialized DAGS.
>
> On Fri, Jan 10, 2025 at 4:51 PM Igor Kholopov
> wrote:
>
>> The only thing that we need to clear out before this
+1
On 2025/01/10 15:50:55 Igor Kholopov wrote:
> The only thing that we need to clear out before this is sealed - what are
> we going to do with SQLite? SQLite supports concurrent connections if the
> processes are on the same host and we already have WAL enabled by Ash in
> https://github.com/apa
Same, I am not strongly opinionated, just a preference :)
On 2025/01/10 15:36:05 Vikram Koka wrote:
> I agree this makes sense.
>
> I was originally concerned that this would make it more difficult to ensure
> compatibility across providers for capabilities such as common.sql,
> objectstore, and
Vikram,
I was not able to attend yesterday's call - much appreciate the quick update on
the notes. Out of curiosity, was there any discussion yesterday with regards to
an alpha/beta build for Airflow 3.0? A few months back, we were talking about
trying to have something available for Jan - not
The only thing that we need to clear out before this is sealed - what are
we going to do with SQLite? SQLite supports concurrent connections if the
processes are on the same host and we already have WAL enabled by Ash in
https://github.com/apache/airflow/pull/44839/files. But I think we still
need
+1
Regards,
Pavan
On Fri, Jan 10, 2025, 15:34 Vikram Koka
wrote:
> +1 on this
>
> For many reasons, which have already been brought up in the thread.
>
>
> On Fri, Jan 10, 2025 at 7:30 AM Igor Kholopov >
> wrote:
>
> > +1, there are a lot of old code paths that exist only because of the
> > e
I agree this makes sense.
I was originally concerned that this would make it more difficult to ensure
compatibility across providers for capabilities such as common.sql,
objectstore, and so on.
However, seeing that the "common" pattern would remain the same and it's
only the code layout that is ch
+1 on this
For many reasons, which have already been brought up in the thread.
On Fri, Jan 10, 2025 at 7:30 AM Igor Kholopov
wrote:
> +1, there are a lot of old code paths that exist only because of the
> embedding in the scheduler support. Focusing on a single supported mode of
> operation wi
+1, there are a lot of old code paths that exist only because of the
embedding in the scheduler support. Focusing on a single supported mode of
operation will allow us to significantly reduce the size (and complexity)
of the DAG processing code.
On Fri, Jan 10, 2025 at 11:49 AM Pierre Jeambrun
wr
So I propose letting the "doer" make the decision if we are split.
On Fri, Jan 10, 2025 at 11:52 AM Ash Berlin-Taylor wrote:
> Not strong at all, preference Is all. It sounds like Vincent and I are in
> the hyphen camp and you and Maciej are in the slash camp.
>
> +1 on the “I don’t care what co
Not strong at all, preference Is all. It sounds like Vincent and I are in the
hyphen camp and you and Maciej are in the slash camp.
+1 on the “I don’t care what code style is used as long as it is
programmatically enforced”.
-a
> On 10 Jan 2025, at 09:41, Jarek Potiuk wrote:
>
> Is there any
+1
On Fri, Jan 10, 2025 at 11:01 AM Michał Modras
wrote:
> +1 - separating these workloads makes sense to me - we remove
> unnecessary coupling and make them more single-responsibility, which eases
> reasoning about the system and any potential debugging
>
>
>
> On Fri, Jan 10, 2025 at 9:15 AM K
+1 - separating these workloads makes sense to me - we remove
unnecessary coupling and make them more single-responsibility, which eases
reasoning about the system and any potential debugging
On Fri, Jan 10, 2025 at 9:15 AM Kaxil Naik wrote:
> Yeah, purely from operational perspective, debuggi
Is there anything else that "tastes" Ash ? A concrete reason that makes you
think the "-" prefix in this case is better than the "/" folder? How
strong is your "taste" preference and do you think it will have some
lasting effect if we choose to flatten the folder structure?
I might make a small v
Also just for the record (while having everyone attention to actions
security) how important this whole subject of actions security is:
* yesterday GitHub released (and we are going to enable it very soon)
update to Code QL to check security of Github Actions
https://github.blog/security/applicati
Yeah, purely from operational perspective, debugging issues become lots
simpler if they are separated as one is CPU hungry while the other is
memory hungry!
On Fri, 10 Jan 2025 at 13:09, Jarek Potiuk wrote:
> +1 (or rather +10). There are two additional things - both related to
> security and
21 matches
Mail list logo
