Re: [DISCUSS] Donating Airflow k8s Operator

+1 here! On Thu, Sep 12, 2019 at 10:20 AM Jarek Potiuk wrote: > The name is indeed misleading. My proposal would be > k8s-operator-for-airflow - this might make it unambiguous . > > czw., 12 wrz 2019, 16:17 użytkownik Jarek Potiuk > > napisał: > > > Fantastic! Strong +1 from me. > > > > To alle

Re: [PROPOSAL] Migrate to Pytest

Huge +1 from me on this one! On Mon, Oct 14, 2019 at 10:46 AM Bas Harenslak < basharens...@godatadriven.com> wrote: > Big +1 for pytest > > Bas > > > On 14 Oct 2019, at 15:29, Jarek Potiuk wrote: > > > > Fantastic! Pytest is sooo much better than nosetests! > > > > On Mon, Oct 14, 2019 at 1:44 P

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

Hi All! Good to hear from you @gsemet. > - do you want to take full "ownership" of it, if so, how to proceed ? How the governance of the chart should be managed? Yes I think that adding me and Jarek would be a good starting point with the helm chart. We can figure out long-term governance later.

Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

Hello fellow Airflowers! As some of you have heard, recently we’ve been looking into knative as a new executor that will offer both the flexibility of the KubernetesExecutor and the speed of the CeleryExecutor. The Initial tests have been successful (with pretty awesome speed/scaling results),

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

PM Aizhamal Nurmamat kyzy wrote: > This is awesome! Thank you for setting this up, Daniel. > > Also adding few Cloud Composer folks who are super interested in > contributing to this effort. > > On Wed, Oct 16, 2019, 7:34 PM Daniel Imberman > wrote: > > > Hello fellow Air

Re: [DISCUSS] Smarter test execution for CI (trivial changes without full tests)

I’m still somewhat hesitant on this as it could allow regressions to peak through, though as long as we’re still doing the daily build, and with our soon-to-be-created prerelease load testing I think we should be okay. On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk wrote: Seems that our tests g

Re: [DISCUSS] Smarter test execution for CI (trivial changes without full tests)

master full suite of tests will be run - which means that we can immediately see which commit caused the problem. It's only at the PR level the tests will be "smart". On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman wrote: > I’m still somewhat hesitant on this as it could allow re

Re: New PMC Member: Jarek Potiuk

Congratulations Jarek! You absolutely deserve this! On Fri, Oct 18, 2019 at 6:12pm, Kevin Yang < yrql...@gmail.com [yrql...@gmail.com] > wrote: Congrats! Well deserved. Ash Berlin-Taylor 于2019年10月18日 周五下午2:08写道： > And we'll deserved too! Congrats. > > -a > > On 18 October 2019 22:05:33 BST, Ka

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

uchemin < > > maximebeauche...@gmail.com> wrote: > > > > > Nice! > > > > > > On Wed, Oct 16, 2019 at 1:19 PM Ash Berlin-Taylor > > > wrote: > > > > > > > We meant October 30th (the date in the calendar is correct) > &

Tomorrow is our first SIG-Knative meeting

Hello fellow Airflowers! Wanted to send a quick reminder that our first airflow SIG-knative meeting will take place tomorrow morning at 11AM EST! We'll be going to go over the proposed architecture, current progress, and potential opportunities for collaboration on the upcoming KnativeExecutor

Re: Closing JIRA Issue for Merged PRs

+1 for probot, the k8s community has a pretty slick GitHub automation system as well we should look into. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Tue, Nov 19, 2019 at 2:09 PM, Maxime Beauchemin wrote: Quick note as I'm playing w

Re: [VOTE] Accept donation of Airflow k8s Operator to Apache Airflow

+1 (non-binding) Yay! I’m excited to get to work on this once it’s donated :D via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Thu, Nov 21, 2019 at 9:39 AM, Jarek Potiuk wrote: +1 (binding) On Thu, Nov 21, 2019 at 9:27 AM Kaxil Naik wro

Re: [VOTE] Accept donation of Airflow k8s Operator to Apache Airflow

via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Thu, Nov 21, 2019 at 1:00 PM, Kaxil Naik wrote: Daniel's vote is binding ;) Committers vote are binding excepts on releases AFAIK. On Thu, Nov 21, 2019 at 5:55 PM Danie

Re: Using implicit namespace packages in Airflow (removing __init__.py where empty)

Fine by me if it doesn’t break anything. I’m always on the side of less code :). via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Fri, Nov 22, 2019 at 8:12 AM, Ash Berlin-Taylor wrote: Probably worth it, but things to check: - if nose or

Re: New PMC members: Aizhamal and Kevin

Congrats!! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Fri, Nov 22, 2019 at 12:17 PM, Pablo Estrada wrote: Yayyy congrats! : ) On Fri, Nov 22, 2019 at 3:06 AM Jarek Potiuk wrote: > Super happy about both. And looking forward to co

Knative Executor Meeting Delay

Hello everyone! Due to the fact that many people in the US will be on vacation this week, let's have our Knative monthly meeting next Wednesday on December 4th. I look forward to seeing you all then! Also we have posted the video for the first meeting last month https://youtu.be/f0y80U2-l0M See

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

+1 on my end as well. @jarek does this affect anything involving breeze? Do GitHub actions have an easy docker/docker-compose based workflow? via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Tue, Dec 10, 2019 at 5:28 AM, Ash Berlin-Taylor

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

subsequent steps. But we can do it as the next step - optimising the > experience. > > J. > > On Tue, Dec 10, 2019 at 11:34 PM Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > +1 on my end as well. > > > > @jarek does this affect anythin

RE: Pros and cons of Oracle db as Airflow SqlAlchemy backend

Mehmet, for what it’s worth. I don’t know anyone who uses oracledb for an airflow backend, and I’m pretty certain it’s not high on the committer/PMC radar so you might be on your own for any in-depth debugging. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&sour

Re: Call to organize more First time Airflow Contributor's Workshops

Hi Jarek, I’d be glad to help at PyCon US as well, perhaps we could collaborate on something or submit two talks in hopes of getting twice the airflow coverage? :). On Sun, Dec 15, 2019 at 8:04 AM, Jarek Potiuk wrote: Now hat you wrote it PyCon US looks great. And It is still 5 days

Re: [VOTE] Add Probot Integrations to Airflow Github Repo

+1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 +1 … Yes please. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Thu, Dec 19, 2019 at 5:51 PM, Kaxil Naik wrote: Hello all, I would like to propose that w

Re: New Committer: Tomasz Urbaszek

Congratulations Tomasz! It’s been awesome seeing your contributions to the community and I look forward to what comes next. You deserve this :). via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Fri, Dec 20, 2019 at 7:51 AM, Michał Słowik

Re: Automated insertion of JIRA issue links in PR description

Thank you Jarek! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Mon, Jan 6, 2020 at 7:02 AM, Kaxil Naik wrote: Just love this 💟 On Mon, Jan 6, 2020 at 2:41 PM Jarek Potiuk wrote: > On top of the auto-links (which are for commits) we a

SIG-Autoscaling meetup tomorrow

Hello all! We will be meeting tomorrow for #sig-autoscaling monthly meetup tomorrow. In this chat we’ll be discussing progress in using Knative, Kombu, and Kubernetes task-pooling to improve the story of airflow autoscaling/efficiency! We will be meeting at 8:00AM PST Zoom link here https://a

Re: Stateful Tasks (was Poke Reschedule)

@jarek reading through the history of this and will try to come back with thoughts. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2] On Mon, Jan 13, 2020 at 3:47 PM, Jarek Potiuk wrote: I think the main subject has been hijacked (sorry Ja

Re: Remove Dask Executor in Airflow 2.0 ?

I think we need to ask “who is going to champion this executor.” I see that it is being used (a bit), but am concerned if no one with knowledge of this executor is willing to maintain it. I’ve personally never used Dask and the DaskExecutor isn’t super high on my priority list compared to thing

Re: Remove Dask Executor in Airflow 2.0 ?

t (no matter committer or contributor), I would be for dropping it. J. On Fri, Jan 17, 2020 at 6:07 PM Daniel Imberman wrote: > I think we need to ask “who is going to champion this executor.” I see > that it is being used (a bit), but am concerned if no one with knowledge of > this e

SIG-Autoscaling meeting

Hello all! The SIG autoscaling meeting is happening now, please feel free to join if you are interested. The link can be found here https://astronomer.zoom.us/j/115707599 [https://astronomer.zoom.us/j/115707599] Cheers! Daniel via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv

Re: [DISCUSS] Airflow functional DAGs

I like this idea a lot. We could create something similar to an “executor_config” so people can pre-populate most of the parameters necessary for a python_operator and pass it in e.g. Config = AirflowPythonConfig(…) @airflow.make_python_operator(config) def my_func(): On Mon, Feb 3, 2020 a

Re: [DISCUSS] Airflow functional DAGs

I think if we’re not breaking any other operators (which I doubt we are) it’s a great 2.0 feature. It would also look great in a “What’s New in Airflow 2.0” announcement ;). Docs are always a challenge, but we could set up a google doc and hack it out in a day or two. +1 via Newton Mail [htt

Re: [VOTE] Release Apache Airflow 1.10.8 based on 1.10.8rc1

+1 (binding). Tested with the kubernetes executor On Thu, Feb 6, 2020, 7:17 AM Ash Berlin-Taylor wrote: > +1 (binding) > > I tested LocalExecutor, CeleryExecutor (with redis and postgres as > brokers), and Kube executor. All looking good! > On Feb 5 2020, at 11:26 pm, Jarek Potiuk wrote: > > +1

Re: Shorter release vote or 1.10.9 to pin Werkzeug dep

+1 binding via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Feb 7, 2020 at 6:22 AM, Bolke de Bruin wrote: +1 binding Sent from my iPhone > On 7 Feb 2020, at 14:08, Jarek Potiuk wrote: > > Agree. > >> On Fri, Feb 7, 2020 at 2:05 PM

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

+1 on my end! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Mon, Feb 17, 2020 at 12:30 AM, Driesprong, Fokko wrote: I like this as well. It will hopefully also reduce the memory footprint of Airflow. The only thing I can think of is

Airflow and Machine Learning

Hello everyone! I’m working on a few proposals to make Apache Airflow more friendly for ML/Data science use-cases, and I wanted to reach out in hopes of hearing from people that are using/wish to use Airflow for ML. If you have any opinions on the subject, I’d love to hear what you’re all worki

Re: Airflow and Machine Learning

r and > > ML/Data science engineer can reuse a template and choose which type of > > machine they needs (highcpu, highmem, GPU or not..etc) > > > > We have a process in place describe in the second part of this article > > (Industrializing machine learning pipeline) : >

Proposal: SIG-Kubernetes

Hello Airflowers! Recently we've had some pretty great success with the SIG-Autoscaling and SIG-DAG-Serialization groups. In this same vein, I'd like to propose a SIG-Kubernetes group where we can discuss all efforts to improve the airflow Kubernetes story. We would meet once a month and discus

Re: Proposal: SIG-Kubernetes

n in on these > > >>> > > >>> On Wed, Feb 26, 2020 at 11:52 AM Tomasz Urbaszek < > > >>> tomasz.urbas...@polidea.com> wrote: > > >>> > > >>>> +1 for the idea. Should Airflow k8s operator be included in those > > >

Re: [PROPOSAL][AIP-32] Airflow REST API

I think it’s totally reasonable to support multiple versions but offer a deprecation system. We can look at how kubernetes handles deprecation in its API for a template. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Mar 6, 2020 at

Re: Proposal: SIG-Kubernetes

le on Kubernetes slack were wondering if the project is dead or not :) T. On Sat, Feb 29, 2020 at 11:54 AM Roi Teveth wrote: > > +1 would love to be a part of this > > > On Wed, Feb 26, 2020 at 10:49 PM Kaxil Naik wrote: > > > +1 > > > > On Wed, Feb 26, 2020

Re: [VOTE] Backport package release process

+1 (binding) On Mar 8, 2020, 7:46 AM -0400, Tomasz Urbaszek , wrote: > +1 (binding) > > On Sat, Mar 7, 2020 at 10:40 PM Kamil Breguła > wrote: > > > +1 (binding) > > > > On Sat, Mar 7, 2020 at 10:29 PM Kaxil Naik wrote: > > > > > > +1 (binding) > > > > > > On Sat, Mar 7, 2020 at 2:36 AM Jarek Po

Re: Let's agree to guidelines for AIP (Airflow Improvement Proposal)

I agree with Jarek. All -1’s should be considered bur need to be qualified. Daniel On Mar 16, 2020, 4:35 AM -0700, Kaxil Naik , wrote: > Yes, I like the procedural issues one (that includes lazy consensus) too. > > > > On Mon, Mar 16, 2020, 11:31 Jarek Potiuk wrote: > > > (and BTW when we vote on

Re: [VOTE] Switch from using Jira to Github Issues

+1 Binding No point in having a whole extra site if we’re not using any of the JIRA specific features Daniel On Mar 16, 2020, 9:58 PM -0700, Sumit Maheshwari , wrote: > +1 binding > > On Mon, Mar 16, 2020 at 11:44 PM Kevin Yang wrote: > > > +1 binding > > > > On Mon, Mar 16, 2020 at 9:34 AM Ta

Re: Let's agree to guidelines for AIP (Airflow Improvement Proposal)

ons? > > On Tue, Mar 17, 2020 at 1:37 PM Daniel Imberman > wrote: > > > I agree with Jarek. All -1’s should be considered bur need to be qualified. > > > > Daniel > > On Mar 16, 2020, 4:35 AM -0700, Kaxil Naik , wrote: > > > Yes, I like the procedur

SIG-Kubernetes meeting

Hello everyone! We are starting our fist SIG-Kubernetes meeting on zoom right now. If you are around please join :) https://astronomer.zoom.us/j/796880800 Daniel

Let's talk Airflow 2.0

Hello, fellow Airflowers! I wanted to start this discussion in hopes of "kicking the tires" and agreeing on a timeline for the Airflow 2.0 release. It seems to me that our 1.10 branch is straying farther and farther from master, and with that, there are two significant consequences. Releases beco

Re: Let's talk Airflow 2.0

Hi Jarek, thank you for your support on this effort! For sure. One comment here. One of the ways we discussed with Kamil > about the approach we want to take is to engage more people in > implementing it - so Kamil would create a basic framework and we might > have many people implementing and te

Re: Let's talk Airflow 2.0

Great! Hope to get a few more folx to give +1's but I think we have a good path forward here :) On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk wrote: > > > > > > I agree especially for larger-scale users migrations are a difficult > > process. Perhaps we can adopt something similar to a blockchai

Re: Helm Chart for Airflow

@jarek I agree completely. I think that pairing an official helm chart with the official image would make for a REALLY powerful “up and running with airflow” story :). Tomek and I have also been looking into operator-sdk which has the ability to create custom controllers from helm charts. We mig

Re: Let's talk Airflow 2.0

need to be feature complete or just enough > > functionality to warrant removing the existing experimental one. > > > > Like you said, less sooner is better. > > > > R > > > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, > > wrote: > > > > >

Re: [DISCUSS] Back to (some) dependency pinning

Agreed. Kind of a “best we can do” considering the current nature of python. On Mar 24, 2020, 2:45 PM -0700, Driesprong, Fokko , wrote: > Yes, I'd be in favor of not having two packages, and just pinning the > versions then. In this case, all the versions will be pinned, so if a user > wants to ins

SIG-Autoscaling meeting

Hello all! Since the KEDA autoscaler has been released, for the moment there’s not too much going on with SIG-Autoscaling. For this reason I think we should suspend the monthly meetings and take them on a more ad-hoc basis (e.g. if there is a call for feedback on the FargateExecutor). If new pr

Re: Requesting to give details on FacebookAdsHook project

Hi Chetan, Please see our documentation here https://airflow.apache.org/. Please feel free to reach out if you have further questions. On Mar 25, 2020, 10:29 AM -0700, Chetan Agarwal , wrote: > Dear Sir, > > I am Chetan Agarwal, third-year Computer Science and Engineering > Student at National In

Re: Helm Chart for Airflow

; > > > > Dockerfiles > > > > > > pull this package in. These scripts are used to coordinate running > > > > > > migrations and cleaning up failed pods. > > > > > > > > > > > > > > > > I see two scripts: >

Re: [VOTE] AIP-31: Airflow functional DAG definition

+1 binding. I volunteer to help implement as well :D On Mar 26, 2020, 7:15 AM -0700, Jarek Potiuk , wrote: > +1 binding > > On Thu, Mar 26, 2020 at 8:59 AM Sumit Maheshwari > wrote: > > > +1 binding > > > > On Thu, Mar 26, 2020 at 9:50 AM Chao-Han Tsai > > wrote: > > > > > +1 binding > > > > > >

Re: Helm Chart for Airflow

ail :) > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman > > wrote: > > > We’re not using CRDs for the tests at the moment. We just have deployment > > > files. If anything having the helm chart as a part of the airflow repo > > > could mean that the helm

Re: Helm Chart for Airflow

CI working to test it. Where > do we want to run it? Do you all just use Travis only? > > Steven > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman > wrote: > > > @Jarek I think with the helm chart + prod image we can go even further > > than that :). We can test C

Re: Helm Chart for Airflow

ove forward all of that and have a PR that we > can start running tests on and replacing the resources with helm > chart. > > J. > > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman > wrote: > > > > @Steven glad to help you out on that. > > > > For the moment

Re: How about switching from Celery to RQ?

I mean… we’re not planning it (kind of an “if it’s not broke don’t fix it” situation), but I don’t think we’re super set on Celery. Would you be interested in making an AIP to discuss potential benefits? On Mar 26, 2020, 10:14 AM -0700, Alexandre Vermeerbergen , wrote: > Hi there, > > Looks like

Re: New committer: Jiajie Zhong

Woo!! Congratulations Jiajie! Well deserved :) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Tue, Mar 31, 2020 at 10:29 AM, Jarek Potiuk wrote: The Project Management Committee (PMC) for Apache Airflow has invited Jiajie Zhong to be

Re: Let's talk Airflow 2.0

Core changes without rewriting the PRs. Regards, Kaxil On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman wrote: > Hi Robin, > > > > I feel some of the stuff for instance Schedular HA could wait for a > point > > > release of version 2 (although maybe this a lot

Re: Let's talk Airflow 2.0

new and old REST APIs simultaneously. It is not easy to upgrade two complex systems at the same time. However, if we do this, some users will have to do it. Older versions can be hidden behind the feature gate. We can also add deprecation warnings. > > R > > On Fri, 20 Mar 2020, 20:29 Daniel Imbe

Re: Let's talk Airflow 2.0

nium. What ever > we choose we need to pay careful attention to avoid slow or flakey UI tests. > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've been taking time to learn cypress recently, and once the airflow helm chart is merged I think merging a set of inte

Re: Let's talk Airflow 2.0

m in/out in the UI, we can add a column to the task_instance table as a marker to group these tasks together when rendered in the UI. Let me know how you guys think. Thanks Bin On Thu, Apr 2, 2020 at 9:55 AM Daniel Imberman wrote: > Hello all, > > I've been reviewing This wiki page > &

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

@jarek once we merge the helm chart we can add a breeze command to do the same with the k8sexecutor on a kubernetes cluster :) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Apr 3, 2020 at 6:41 AM, Jarek Potiuk wrote: Yep. And t

Re: Helm Chart for Airflow

ished for 1.10.10 and beyond. Happy to hear your review comments and hope to merge it very soon. J. On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk < jarek.pot...@polidea.com [jarek.pot...@polidea.com] > wrote: Absolutely! Please do :). The more eyes on this the better! On Thu, Mar 26,

Re: [VOTE] AIP-32: Airflow REST API

Looking forward to it! +1! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Apr 3, 2020 at 4:06 AM, Kaxil Naik wrote: I had the same concern with the Auth model (Had commented on the AIP on confluence: https://cwiki.apache.org/conflu

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc1

@jarek when you’re running on breeze how do you actually run airflow s.t. I can see the UI/run DAGs? It seems like it downloads everything, but it doesn’t seem like a full airflow deployment. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]

Re: [VOTE] Release Apache Airflow 1.10.10 based on 1.10.10rc2

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Fri, Apr 3, 2020 at 11:10 AM, Kaxil Naik wrote: +1 (binding) On Fri, Apr 3, 2020 at 6:55 PM Kaxil Naik wrote: > Hello Airflow Community, > > This is a call for the vote to

Re: [DISCUSS] DockerHub vs. Github Image registry for "master"/"v1-10-test" builds

Yeah, I'm not worried about DDOS as long as the URL is stored in a secret/doesn't show up in the github action UI. On Tue, Apr 21, 2020 at 6:29 AM Ash Berlin-Taylor wrote: > I'm still not quite sure what problem are we solving here either...? > What is broken with the current/already merged sol

Re: Proposal: Change TI from having execution date to dag_run_id (in API at least)

+1 for sure! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Thu, May 14, 2020 at 1:18 PM, Dan Davydov wrote: +1 but in the future I think better would be /dags/{dag_id}/dagRuns/{execution_date}/{run_number}. That would give an automatic

Re: Our CI architecture as a model for GA <> DockerHub integration for ASF projects

Cool!! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Thu, May 28, 2020 at 12:20 AM, Jarek Potiuk wrote: Hello everyone, I just wanted you to know that our way of integration GitHub Actions with the DockerHub has become one of the mod

Re: Unsubscribe from dev-list

Not a worry Fokko! Mental health is crucial right now. We look forward to seeing you in these conversations again soon :) On Mon, Jun 1, 2020 at 9:13 AM, Driesprong, Fokko wrote: Hi all, I've decided to take a break from the dev-list. There is a lot of action on the list, which is great. Awe

Re: [AIP-34] Rewrite SubDagOperator

I hadn’t thought about using the `>>` operator to tie dags together but I think that sounds pretty great! I wonder if we could essentially write in the ability to set dependencies to all starter-tasks for that DAG. I’m personally ok with SubDag being a mostly UI concept. It doesn’t need to exec

Re: [AIP-34] Rewrite SubDagOperator

ransfer from one to another. > > > I wonder if we could essentially write in the ability to set > dependencies to all starter-tasks for that DAG. > Can you elaborate on *the ability to set dependencies to all starter-tasks > for that DAG*? > > Bin > > On Fri, Jun 12, 2020

Re: [AIP-34] Rewrite SubDagOperator

> execution > > > > > > > and scheduling. It will be the same as not using SubDag. > > > > > > > 2. Still have the benefits of modularized and reusable dag code > > and > > > > > > > declare dependencies between them. And w

Re: [UPDATE] AIP-31 .output update

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Wed, Jun 17, 2020 at 12:13 AM, Tomasz Urbaszek wrote: +1 (binding) On Wed, Jun 17, 2020 at 3:39 AM 蒋晓峰 wrote: > +1(not binding) > > On Wed, Jun 17, 2020 at 3:03 AM Gerard Cas

Re: [DISCUSS] Enable 'Black' for Auto Code Formatting

I think that once we add integration tests (which I’ll do now that we have helm for k8s) and we no longer have to back port this is a great idea. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Wed, Jul 1, 2020 at 4:17 PM, Kaxil Naik wro

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

I’m fine with keeping it as three separate repos but merging testing somehow (e.g. the source code chart would pull the helm/docker chart into .build) but we need to do it in a way that doesn’t make testing too difficult. So for example: How do I test/integration test a change that involves a ch

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

and helm. Tomek On Thu, Jul 2, 2020 at 9:26 AM Jarek Potiuk wrote: > > On Thu, Jul 2, 2020 at 3:16 AM Daniel Imberman > wrote: > > I’m fine with keeping it as three separate repos but merging testing > > somehow (e.g. the source code chart would pull the helm/docker chart into

Re: [VOTE] Release Apache Airflow 1.10.11 based on 1.10.11rc1

@kaxil could we post release candidates to our docker hub? Would make it easier to test via the helm chart. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Thu, Jul 2, 2020 at 7:59 AM, Kaxil Naik wrote: Hello Airflow Community, This i

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

ntities" > > >> >> at the > > >> >> same time. Even just explaining how to work with Pull Requests and > in > > >> what > > >> >> sequence those PRs would have to be opened and merged in case of > > >> changes > >

Re: Separate Repo vs MonoRepo for Dockerfile & Helm Chart

/git-cherry-pick-or-merge-specific-directory-from-another-branch via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Sun, Jul 5, 2020 at 9:57 AM, Daniel Imberman wrote: I can’t agree with this enough :). I think writing a few bots

Re: new committer: QP Hou

Welcome aboard!!! On Wed, Jul 15, 2020 at 10:30 AM, Tao Feng wrote: Welcome QP! On Tue, Jul 14, 2020 at 1:53 PM Jarek Potiuk wrote: > The Project Management Committee (PMC) for Apache Airflow has invited > QP Hou to become a committer and we are pleased to announce that he > has accepted. > >

Re: New Committers (Leah Cole & Ry Walker)

Congratulations both of you! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Mon, Jul 20, 2020 at 7:26 AM, Tomasz Urbaszek wrote: Big congrats to you Leah and Ry! T. On Mon, Jul 20, 2020 at 4:18 PM Kaxil Naik wrote: > Hello Airflow

Simplifying the KubernetesExecutor

Hello, fellow Airflowers! I hope you are all well in these trying times. With the recent launch of Airflow 2.0 preparation, it now seems like a good time to review the project's state and where we can fit in some breaking changes that will improve the project for the future. When we first cre

Re: Simplifying the KubernetesExecutor

.1 to me. J. On Wed, Aug 12, 2020 at 4:24 PM Daniel Imberman wrote: > Hello, fellow Airflowers! I hope you are all well in these trying times. > > > With the recent launch of Airflow 2.0 preparation, it now seems like a > good time to review the project's state and where

Re: Simplifying the KubernetesExecutor

ail_footer_2] On Wed, Aug 12, 2020 at 8:09 AM, Daniel Imberman wrote: 100% agreed on timing. I think 2.0 should be for the breaking aspect (losing the configurations) and then 2.1/2.2 we can start adding on the new features. I also like the idea of a migration tool. We can make a script tha

Re: Simplifying the KubernetesExecutor

ote: > +1 for the idea, thanks Daniel! I agree that multi-executor and pod > templates should be 2.1. > > T. > > On Wed, Aug 12, 2020 at 5:13 PM Daniel Imberman > wrote: > >> (Also funny enough we could use a lot of the existing infrastructure int >> 1.10 to create

Re: [VOTE] Release Apache Airflow 1.10.12 based on 1.10.12rc1

+1! On Sat, Aug 15, 2020, 1:38 PM Jarek Potiuk wrote: > Cool! Will take a look tomorrow ! > > sob., 15 sie 2020, 19:05 użytkownik Kaxil Naik > napisał: > > > Hello Apache Airflow Community, > > > > This is a call for the vote to release Apache Airflow version 1.10.12. > > > > The release candid

Re: Simplifying the KubernetesExecutor

k. > > > > Best wishes > > > > Ping Zhang > > > > > > On Thu, Aug 13, 2020 at 12:58 PM Jarek Potiuk > > wrote: > > > >> +1 for pod mutation in config. It's not a "yamly" thing - it's a python > >> > >

Re: Optimize KuberneteExecutor pod labels to task instance key

Hi Ping, I think that’s a great idea! Would be glad to help merge this. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2] On Sun, Aug 23, 2020 at 11:33 PM, Ping Zhang wrote: Hi everyone, I was evaluating using *KubernetesExcutor* and found t

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 binding via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2] On Tue, Aug 25, 2020 at 8:45 AM, Gerard Casas Saez wrote: + 1 (non-binding) Gerard Casas Saez Twitter | Cortex | @casassaez On Tue, Aug 25, 2020

Re: Generic Transfer Operator

Interesting! Beam also could potentially allow transfers within Dask/any other system with a java/python SDK? I think @jarek and Polidea do a lot of work with Beam as well so I’d love their thoughts if this a good use-case. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv

Re: Generic Transfer Operator

out of the box and allows us to simplify > > > >> > read/write from external sources. Thus, it requires no external > > > >> > dependency other than the library to communicate with the API. In > the > > > >> > case of Beam we need more than that I thin

Re: Generic Transfer Operator

Beam based approach: > >> > > >> >1. Dependency management. If I do `pip install apache-airflow[gcp]` > >> >will it install `apache-beam[gcp]`? What if there's a version clash > >> >between dependencies? > >> > > >> >2. Th

Re: Generic Transfer Operator

irect/ Tomek On Sun, Sep 6, 2020 at 9:45 PM Daniel Imberman wrote: > > Oof ok yeah. I hadn't realized that beam had a hard JVM requirement. I > think that initially offering a local or block storage based solution with > easy extensions for users is totally in line with airf

Re: Import style in Airflow codebase

+1 on my end! via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2] On Wed, Sep 9, 2020 at 7:38 PM, Maxime Beauchemin wrote: +1 On Wed, Sep 9, 2020 at 10:36 AM Vikram Koka wrote: > +1 on absolute import. Honestly, a huge fan of doing it as a

Re: Docker/Helm releases

Thank you Niclas, this will help us a lot in figuring out our helm situation via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2] On Wed, Sep 9, 2020 at 5:36 AM, Kaxil Naik wrote: Credits to Jarek on that one, he is the one who is actually dra

Re: [VOTE] AIP-8 Split Providers into Separate Packages for Airflow 2.0

+1 (binding). via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2] On Sun, Sep 13, 2020 at 1:57 PM, Kevin Yang wrote: +1 (binding) On Sun, Sep 13, 2020 at 1:29 PM Jarek Potiuk wrote: > Hello Everyone, > > Last week, at the Airflow 2.0 meetin

Re: Discuss: should we allow HTML emails on dev@ list

+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.6&source=email_footer_2] On Mon, Sep 14, 2020 at 9:03 AM, Jarek Potiuk wrote: Oh noo .. I just started to use Plain Text Mode in my Gmail more often than not. Seriously though: +1 Binding.

  1   2   >