I'd also be happy to help out with this effort.
We've been interested in improving the quality of signals produced by
Airflow for a while - they are critical for making task failure easier and
more accessible.
Let me know once the plan is updated, I will be happy to pick up some work!
On Sat,
I know Howard (AIP creator) was looking for developers to assist on this.
Howard - Can you share status/progress?
On Sat, Jan 28, 2023 at 12:50 AM Ferruzzi, Dennis
wrote:
> Third time’s the charm?
>
> Hi folks, looks like AIP-49 [1] hasn’t been touched in a while and I am
> interested in getti
Third time’s the charm?
Hi folks, looks like AIP-49 [1] hasn’t been touched in a while and I am
interested in getting it implemented. If nobody has any objections, I’m going
to start coming up with a plan to get this one done in the next little while.
The last PoC for it was almost a year ago
Thinking about this some more.
As an airflow developer, a lot of our backcompat concerns (which takes up a
substantial portion of our energy), is about the concern that we might
break something for someone who has extended either built-in classes or
provider classes. Maybe we are to permissive in
okie doke, took a crack at it https://github.com/apache/airflow/pull/29200
Strong +1 for the BaseExecutor.
This relates to a discussion we had with Niko about AIP-51 where we talked
about having at some point all our Executors to extend this common base
class. At the moment, most of our mixed executors do not. As a consequence
we had a few surprises while implementing A
Yes. BaseExecutor is the way to go.
On Fri, Jan 27, 2023 at 5:30 PM Daniel Standish
wrote:
> Question though... I'm not sure we need to mark anything in the subclasses
> (K8sExecutor, LocalExecutor etc) as public... how would we choose what to
> make public? Why would we do so? Why not just ke
Question though... I'm not sure we need to mark anything in the subclasses
(K8sExecutor, LocalExecutor etc) as public... how would we choose what to
make public? Why would we do so? Why not just keep it simple and say that
BaseExecutor is the public interface and that's what's public and the
indi
> - We may not want to make a strong statement like "all executors are public".
> It's just
> impossible IMHO.
> - Let's just mark a limited number of key methods/interfaces in each executor
> as public,
> and we ensure a strong backcompat for them. For any methods/interfaces not
> marked, no
Yep. I see your point Daniel. And yes. I think you are right it might be
too much to add back-compat guarantees to all executors.
After a bit of thinking on it, I am perfectly fine with clarification that
"Executor interface" is public but "K8S executor" internal details might be
changed any time
General comment from my side: I think Open Lineage is (and should be
even more) a feature of Airflow that expands Airflow's capabilities
greatly and opens up the direction we've been all working on - Airflow
as a Platform.
I think closely integrating it with Open-Lineage goes the same
direction (a
11 matches
Mail list logo
