Re: AIP-49 OpenTelemetry Support - Take Three

Hi everybody. Howard Yoo here. I authored the AIP-49, and just to give you all some updates, I haven't been successful in making much progress on this proposal, due to the fact that my original intention was to publish AIP-49 after my POC on open telemetry, and wanted to hand it over to who

Re: [AIP-49 OTel] [Discussion] Metrics Provider packages

I never conceived of the idea of reverting everything in such a radical fashion, because I never thought those actions would even be possible! So having the OTEL as the backbone of Airflow's metrics/logs/traces is actually a great idea, in my opinion. With the usage of OTEL collector that can then

[DISCUSSION] Proposal to Add OTEL Provider to Apache Airflow

ffort to ensure a smooth and effective implementation. Please let me know if there are any specific guidelines or processes that I should follow to initiate this proposal. Thanks and regards, Howard Yoo

Re: [VOTE] AIP-65: Improve DAG history in UI

+1 On Fri, Jul 19, 2024 at 3:50 PM Jed Cunningham wrote: > I’m calling for a vote on AIP 65: > https://cwiki.apache.org/confluence/x/T4qSEQ > > Discussion thread: > https://lists.apache.org/thread/vvm43tfchyo92hmf40fqvmq0f5845bjr > > The vote will run for 5 days and last till next Wednesday, 202

Re: [DISCUSSION] Proposal to Add OTEL Provider to Apache Airflow

ly especially with a lot of auto > > instrumentation enabled might be fantastic way to help our users - but in > > order to do that - we need to give them some easy-to-follow-instructions. > > > > If that would be part of the work then I am even +10 on that. > >

Re: [DISCUSSION] Proposal to Add OTEL Provider to Apache Airflow

spect your system - so I can't tell you what is wrong" - > > having > > > such a snapshot that you can load locally especially with a lot of auto > > > instrumentation enabled might be fantastic way to help our users - but > in > > > order to do that

[VOTE] OTEL Provider to Apache Airflow

Fellow Airflowers, I would like to start the voting process for OTEL providers for airflow mentioned in previous discussion in this dev mailing list here: https://lists.apache.org/thread/l0m06llz171m5yhqd016pv58sh5wnqmr The voting will last at least for 72 hours, until Friday 10:00 am CST time.

Re: [VOTE] OTEL Provider to Apache Airflow

is very new concept for us. > > On Tue, Aug 13, 2024 at 5:49 PM Ash Berlin-Taylor wrote: > > > Hi Howard, > > > > It's not clear to me what we are actually voting on here. Could you > > elaborate please? > > > > > > > > On 12 August 2024

Re: [VOTE] OTEL Provider to Apache Airflow

y the community. Please let me know if there's any more questions or comments. Howard On Tue, Aug 13, 2024 at 10:17 PM Howard Yoo wrote: > Ah, got it. > I'll put some more details. Thanks! > > On Tue, Aug 13, 2024 at 6:46 PM Jarek Potiuk wrote: > >> Yeah. I think -

Re: [VOTE] OTEL Provider to Apache Airflow

riday 10:00 am CST time. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason Only votes from PMC members are binding, but all members of the community are encouraged to vote with "(non-binding)". On Thu, Aug 15, 2024 at 3:33 PM Howard Yoo w

[RESULT][VOTE] OTEL Provider to Apache Airflow

Hi all, The vote for OTEL Provider to Apache Airflow has not received any votes, therefore the proposed implementation is not going to proceed. Thanks to everybody for their interest in this matter. Discussion thread https://lists.apache.org/thread/9234s5c33k3hrwdp1jq5mhhcyl8ntc0w Vote thread: h

[DISCUSS] airflow telemetry : improve with open telemetry

Hi all, I’m a staff product manager in Astronomer, and wanted to post this email according to the guide from https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals . Currently, th

RE: Re: [DISCUSS] airflow telemetry : improve with open telemetry

then pipe to many log > > >> forwarders of choice. imo this is the easiest way to set it up and a > > >> default that should work in the vast majority of airflow use cases. > > >> > > >> the PR looks like a great start! what can I do to help? > > &

RE: Re: [DISCUSS] airflow telemetry : improve with open telemetry

? > > > On Sat, Jan 8, 2022 at 12:19 AM Howard Yoo > wrote: > > > Hi all, > > > > I’m a staff product manager in Astronomer, and wanted to post this email > > according to the guide from > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflo

RE: Re: Re: [DISCUSS] airflow telemetry : improve with open telemetry

I am very much interested in how we can improve Not only the instrumentation by using OpenTelemetry, but also Think about how we can make the existing metrics list better. For example, perhaps in the future, maybe we can provide things like how much CPU, memory, and disk I/O a task instance is u

RE: AIP 39: schedule_date renamed to logical_date

Hi TP, I do have another opinion about this change. I agree that ’schedule_date’ is not the right word to describe it, but I also believe that `logical date` is also a bit confusing. My argument is that there is not that much ‘logical’ thing about the period of which the DAG run will base its da

RE: RE: Re: Re: [DISCUSS] airflow telemetry : improve with open telemetry

with OpenTelemetry. Howard On 2022/01/12 03:21:27 Howard Yoo wrote: > I am very much interested in how we can improve > Not only the instrumentation by using OpenTelemetry, but also > Think about how we can make the existing metrics list better. > > For example, perhaps in the f

Re: [DISCUSS] how to name classes with abbreviations?

Just my two cents : Have preference for not capitalizing all the characters in abbrevications, so BranchSqlOperator is totally fine with me. It's also easier to type (less pressing on Caps key!), and also Might be easier to read and check. +1 on that. - Howard On Fri, Feb 4, 2022 at 1:20 PM Ferru

Re: AIP 39: schedule_date renamed to logical_date

Now I can understand why the data_date may not be a perfect fit to describe the term. This is not to be against the logical_date, but what about ‘interval_date?’ We have the schedule interval, which defines the duration of the interval (e.g. 1day), so wouldn’t interval start and end date be a b

Re: AIP 39: schedule_date renamed to logical_date

e?" - is > the question that I anticipate happening a lot if we mix those. > > J. > > > >> On Sun, Feb 6, 2022 at 6:04 PM Howard Yoo wrote: >> Now I can understand why the data_date may not be a perfect fit to describe >> the term. >> >> Th

Re: AIP 39: schedule_date renamed to logical_date

ot;end". It does not > matter. but if they process a data interval, they have very clear boundaries > of ("start" <-> "end") range that they can use without even thinking on. how > "logical_date" maps to it. > > For me - those are c

Re: AIP 39: schedule_date renamed to logical_date

; Yeah. That discussion actually made me think that probably we need to > explain it better :) > > On Sun, Feb 6, 2022 at 11:10 PM Howard Yoo wrote: > >> As we discuss this topic, the more and more I get to understand the >> reasons behind all those philosophies behind, so I

Re: Improving contributor experience for "trusted" users -- faster CI by using self-hosted runners

+1 Sent from my iPhone > On Feb 18, 2022, at 9:18 AM, Kaxil Naik wrote: > >  > +1 > >> On Fri, 18 Feb 2022 at 15:07, Ash Berlin-Taylor wrote: >> Hi all, >> >> I'd like to propose we start allowing more users to use the self-hosted >> runners -- they are much much quicker to run test workfl

Request for access to creating a new AIP proposal at cwiki.apache.org.

Hi there, I'd like to request for edit access to the Confluence page of apache airflow so that I can proceed with creating an AIP (airflow improvement proposal). My cwiki id is howardyoo Please let me know if you need anything from my side. Thank you, Howard

[DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

Hi all, I am pleased to announce the start of the discussion for the new AIP draft that was recently been published: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow Jarek Potuik and I have been discussing about this proposal since early this ye

Re: [LAZY CONSENSUS] Changes to AIP-47 and the first PR with a complete solution!

wasn't involved in this AIP at all, but just wanted to say Congratulations for getting through this! Howard On Mon, Mar 28, 2022 at 3:49 AM Mateusz Nojek wrote: > Hello there! > > Since the main PR (#22311 ) > with the implementation of AIP-47 >

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

ue" side. I looked at it from the technical integration > > POV - and Melodie helped to validate some of the assumptions and > > expose some of the technical challenges. > > The composite result is good, but we are looking with Howard on some > > insightful co

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

o the AIP a scope paragraph listing what > will be included in the first phase and what is left for other phases. > > On Fri, Apr 1, 2022 at 4:26 PM Jarek Potiuk wrote: > >> Or maybe that people are so stunned by the beauty and usefulness of it >> that they cannot even say a w

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

t;> >> >> 2. regarding adding metrics. >> >> Do we intend to let users define their own KPIs/metrics to be measured >> or it will be a closed list set by Airflow? >> >> >> >> 3. The POC specifies it uses a feature of open-telemetry (add met

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

Hi Elad, I have updated the AIP-49 with the appropriate changes to contain what you requested in your previous comments. I think the AIP is fairly comprehensive and ready to be voted, unless there is any objections. Sincerely, Howard On Mon, Apr 25, 2022 at 11:03 PM Howard Yoo wrote: > Hi E

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

do - only the > critical ones where it couples with airflow core. > You have changes for dag_proccessing, executors, scheduler_job and even > migration to add columns. > > https://github.com/howardyoo/airflow/commit/cc83c2b377ac22f0e7ef82e7f59784df972037fd > These are important &

Re: [DISCUSS] Support "slim" PROD image(s) for Airflow

I also like the idea of SLIM images - always helpful. Howard On Wed, May 4, 2022 at 4:53 PM Ping Zhang wrote: > Hi Jarek, > > I really like the idea of having a slim airflow docker image. 500MB > uncompressed is tiny 👍 > > > Thanks, > > Ping > > > On Sun, May 1, 2022 at 8:41 AM Jarek Potiuk w

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

Hope this helps! Howard On Fri, Apr 29, 2022 at 11:49 AM Howard Yoo wrote: > Got it. Will describe the changes that might be subjected to be done - > however, please note that the changes were the things that I did on my own > during the POC, so the final changes may obviously be differen

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

initial implementation of metrics and traces. Howard On Wed, May 18, 2022 at 5:57 AM Malthe wrote: > On Fri, 25 Mar 2022 at 20:40, Howard Yoo wrote: > > I am pleased to announce the start of the discussion for the new AIP > draft that was recently been published: > https://cwiki.apac

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

y 21, 2022 at 2:22 AM Malthe wrote: > On Wed, 18 May 2022 at 16:44, Howard Yoo wrote: > > 2. So, the reason why I ended up implementing span_json was that between > the scheduler who submits the tasks to be processed, and the worker that > needs to pick them up from the queue (whic

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

be ready for a Voting attempt actually :). I'd > wait however with it till after the Summit (like with few other > discussions we are having now). > > J > > On Sun, May 22, 2022 at 4:53 PM Howard Yoo wrote: > > > > But isn't the span uniquely ide

Re: [DISCUSSION] AIP-49 OpenTelemetry Support for Apache Airflow

Yes, I would agree. As for the deferred tasks, I believe the Otel should create 'spans' when the task instances are actually 'running' during their deferred tasks were running - so even though the task id would be the same, we would have multiple 'spans' on the course of its lifetime. On Fri, May

[VOTE] AIP-49 OpenTelemetry Support for Apache Airflow

Now that Summit is over (well done all the speakers! The talks I've caught so far have been great) I'm ready to push forward with OpenTelemetry Support for Airflow, and I would like to call for a vote on https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Air

Re: Airflow Survey Writeup Draft

Just want to say amazing work, and great content! Very helpful in understanding where Airflow is, and what could be in its future! Howard On Wed, Jun 15, 2022 at 10:55 AM Jarek Potiuk wrote: > Thanks John, Ewa, > > From what I saw it's an amazing job and (and I really like what I saw in > the s

[RESULT][VOTE] AIP-49 OpenTelemetry Support for Apache Airflow

Dear Airflow Community, The voting for AIP-49 ( https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow) was completed on June 22, 2022, and I am happy announce the following voting result: *Binding (+5) Votes* Jarek Potuik Tomasz Urbaszek Elad Kalif Jo

[DISCUSSION] Opentelemetry support for LOGS in Apache Airflow

Hi Airflow community members, Apache Airflow has been moving steadily from conventional statsd based telemetry into OpenTelemetry (opentelemetry.io), which has gained popularity and support for the last few years. There has been AIP proposals which had a 'phased' approach to incorporate 1. Metric

Re: [DISCUSSION] Opentelemetry support for LOGS in Apache Airflow

> > Your timing is…. Interesting given we are focusing on getting 3.0 > > shipped > > > > and are in the final throws of that. > > > > > > > > Let's revisit this in May or June once the dust has settled from 3.0, > > any > > > > im