Hi all,
thanks for picking-up the discussion. So following the email chain a bit
I would recommend to spin an AIP for the implementation. There might be
one or multiple cases where this is a cool feature. Still it will add
complexity and needs a closer discussion. The best discussion might be
on the AIP itself and then once all questions and details are described
we still can VOTE on it.
@David, can you follow as described in
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals
?
(I also have another use case in mind and am courious if the propsal
would also support this)
Jens
On 15.10.24 18:24, Daniel Standish wrote:
RE SLAs there was actually a lot of people who chimed in and expressed
concerns with the approach, but no one took the step of actually down
voting it. It's hard to down vote and say no this does not seem right.
And sometimes these things gain a momentum and you don't want to be a stick
in the mud, particularly if you don't have a better solution and someone
has spent a lot of time on it. But yeah we should not be so timid about
saying no that we never do it.
I think I did not really engage with it until substantially later in the
process, wish I could have engaged earlier.
On the topic of streaming, yeah, I'm trying to do my part to engage in this
thread. I don't yet see and understand the value so that's why I suggested
fleshing out the proposal in a doc. I'm not ready to give any thumbs up yet
cus I'd don't see it. That doesn't mean the value isn't there, just I
don't see it / understand it yet.
And yeah we're only two people here engaging with this one so, it's good if
others could consider the proposal also. But people only have so much
time. And anyway, I think the proposal needs more clarity to be
efficiently and accurately evaluated -- so formalizing it a bit, even if
not precisely an AIP, would help in others to chime in. Really get into
what problem it's solving and why and how.
On Tue, Oct 15, 2024 at 9:05 AM Jarek Potiuk <ja...@potiuk.com> wrote:
So I think what David really needs (from you Daniel and others) if is the
idaa sounds right, if it does and we agree it is something that should be
clarified in detail and there are no major blockers to move in this
direction - this can be turned into detailed proposal with the syntax,
I think we had a long story of some cases (like SLA) where we asked for
detailed AIPs and then after it has been delivered it turned out that the
idea from the very beginning was not right, but this feedback has been
missing. SLA feature sufferred from late feedback that "the whole idea
seems wrong".
I think we should avoid such an approach. If we see that the general idea
is wrong we should give early feedback - and then engage in detailed
discussion - but without the "I have not paid attention before but the
whole thing is wrong".
I think David is looking for this kind of confirmation, so that he does not
spend days and weeks on detailing a proposal then was strangled to death
because we did not like the idea in the first place. That's very
discouraging.
J,
On Tue, Oct 15, 2024 at 6:00 PM Jarek Potiuk <ja...@potiuk.com> wrote:
It's about the same David's proposal is about stream syntax to run the
operators in the task. So those are not two things - this is the "idea"
(run operators in a loop in a task) and implementation detail (stream
syntax).
I think at this stage I distilled the idea from the syntax proposal, and
what we could do in the future is to make sure that syntax is good.
J.
On Tue, Oct 15, 2024 at 4:11 PM Daniel Standish
<daniel.stand...@astronomer.io.invalid> wrote:
I'm still a bit fuzzy on the proposal. It also seems at times like you
two
(David and Jarek) are sorta talking about two different things. David:
"stream" syntax. Jarek: run operator in a task.
I would suggest @David maybe just produce a sort of draft AIP maybe in
google docs or something and share and interested parties can review and
understand better and possibly help shape the direction.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
