Re: "Breeze" dev environment

e's "docs" directory and you should see the generated files there. I think that should help a lot when dealing with documentation errors :). It helped me a lot at least. > > On Thu, Aug 29, 2019 at 4:29 AM Jarek Potiuk > wrote: > > > Fantastic. Thanks. Do y

Re: Redeploying the website?

ote: > Hi everyone, > > Can anyone help me figure out what is the process of deploying the > https://airflow.apache.org/ website? > > I have noticed that few changes that we pushed recently haven't appeared > yet. > > Thanks, > Aizhamal > -- Jarek Potiuk Pol

Re: Codecov Integration

Not yet. Will do soon :). J. On Wed, Sep 4, 2019 at 3:51 PM Driesprong, Fokko wrote: > Did you find any time to fix this Jarek? Would be great to get this up and > running again. > > Cheers, Fokko > > Op wo 28 aug. 2019 om 14:28 schreef Jarek Potiuk >: > > >

Re: Proposal: Automatically mark stale PRs in github

ink the stalebot is a great addition. Even if an > issue gets closed unresolved, it is an indication to me that the issue > might not be relevant. In the end you can always reopen issues again. > > Cheers, Fokko > > Op di 2 jul. 2019 om 07:41 schreef Jarek Potiuk > > >

Re: "Breeze" dev environment

, 2019 at 6:06 PM Jarek Potiuk wrote: > Hey Kevin, > > Thanks for those - really helpful explanations! Sorry for spamming > everyone but I post answers here as maybe some people might find it useful > for their own usage of Breeze. I am also adding Elena - who is the > technical wri

Re: Proposal: Automatically mark stale PRs in github

for serialization implementation). J. On Wed, Sep 4, 2019 at 4:43 PM Jarek Potiuk wrote: > Hello everyone, > > ACTION NEEDED BY ALL CONTRIBUTORS: Please add comment to the PRs that you > are still working and you got notified that they are marked as "stale". If > you do not comme

Re: Airflow @ApacheCon?

is coming up next week. > > Looking at the schedule, I don't see anything on Airflow... Am I not > finding, or is nobody talking/going from Airflow? > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+4866079

Cloud Native Warsaw - talking about Breeze

If you happen to be in Poland - tomorrow I speak at Cloud Native Warsaw conference about Breeze: http://cloudnativewarsaw.com/talksspeakers/#rec119200062 J.` -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [im

Re: Codecov Integration

Dockerfile # J. On Wed, Sep 4, 2019 at 3:52 PM Jarek Potiuk wrote: > Not yet. Will do soon :). > > J. > > > On Wed, Sep 4, 2019 at 3:51 PM Driesprong, Fokko > wrote: > >>

Help needed from the community - use cases for the new landing page

case study are CTOs) 1. What was the problem? (about 100 - 200 words) 2. How did Apache Airflow help solve this problem? (about 100 - 200 words) 3. What are the results? (about 50 words) 4. Why would you recommend Apache Airflow? (1-2 sentences) Please respond directly to Marta ! J. -- Jarek P

Re: [ANNOUNCE] Please welcome new Airflow committer Felix Uellendall

Congrats Felix! pon., 9 wrz 2019, 12:58 użytkownik Ash Berlin-Taylor napisał: > All, > > I'm pleased to announce that the Airflow PMC has voted in Felix Uellendall > to be > a committer. Felix has been actively involved in the project for some time > now, contributing patches and testing the rel

Re: Airflow Contributors Workshop at Cumbre de Contribuidores de Open Source Software (CCOSS)

27;ll remove > the tag and they'll be fair game. > > Thanks, > Jakob > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Proposal: Automatically mark stale PRs in github

committers that the PR is still "alive". I truly hope this process will be much better than 100s of abandoned issues. J. On Fri, Sep 6, 2019 at 1:38 PM Jarek Potiuk wrote: > Hello All contributors, > > ACTION MIGHT BE NEEDED FROM YOUR SIDE :). Please review your prs marked as

Re: Outstanding review requests

t; > https://reviews.apache.org/r/71282/ > > * Set log level to warn for relationships without relationship > definition : > > https://reviews.apache.org/r/71278/. (Under review) > > > > Thanks! > > Bolke > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [DISCUSS] Donating Airflow k8s Operator

Fantastic! Strong +1 from me. To alleviate some of the concerns - we @Polidea are just starting a project for big and serious customer where we will most likely use and extend Kubernetes Operator. We have 3 people that could become contributors and commiters day one, so we can already help buildi

Re: [DISCUSS] Donating Airflow k8s Operator

The name is indeed misleading. My proposal would be k8s-operator-for-airflow - this might make it unambiguous . czw., 12 wrz 2019, 16:17 użytkownik Jarek Potiuk napisał: > Fantastic! Strong +1 from me. > > To alleviate some of the concerns - we @Polidea are just starting a > project

Re: [DISCUSS] Donating Airflow k8s Operator

r better because > > *airflow* is the first word rather than the last. > > > > -Ry > > > > czw., 12 wrz 2019, 16:17 użytkownik Jarek Potiuk < Jarek. Potiuk@ > > polidea. com ( jarek.pot...@polidea.com ) > napisał: > > > > > > > > &g

Failing master of airflow

-progress PRs in case you see 16 unrelated errors in your build (related to impersonation etc.).. J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Failing master of airflow

It's fixed now - master is GREEN again. Feel free to rebase your changes on top of it if they failed before with impersonation-related errors. On Mon, Sep 16, 2019 at 11:54 AM Jarek Potiuk wrote: > Hello everyone, > > It seems that we have failing master now due to some ha

Re: Airflow 1.10.5 Python 2.7 compatability

> > > > File > > > > > "/usr/local/lib/python3.5/dist-packages/marshmallow_sqlalchemy/__init__.py", > > line 1, in > > > > from .schema import TableSchemaOpts, ModelSchemaOpts, TableSchema, > > ModelSchema > > > >

AIP-7 completed :)

aking after that (kind for Kubernetes for one and production-ready Airflow Image). And more :). Cannot wait to be more engaged in different parts of Airflow (though current customer engagement might add some delays there). J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Soft

AIP-21 (Move operators to Core) - "cross_transfer" packages

ly against this? Or maybe we can follow lazy consensus rule for this? Or maybe someone can come up with a better name :) ? J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Warsaw Apache Airflow Meetup

.com/Warsaw-Airflow-Meetup/events/264867971/ > > Please send your talk proposal if you wish to share a talk or your use > case at this event. > > Looking forward to seeing you all! > > Thanks, > Aizhamal > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Pri

Re: Issues for Hacktoberfest 2019

This is a great idea (both - Hacktoberfest and Pycon DE). I actually got my Hacktoberfest T-shirt last year by contributing to Airflow :). We will need to enable Github Issues in the project if we should follow that (this is not technically needed but then people won't find the issues through the

Re: [PLEASE PARTICIPATE][AIP-11] UX phase of Airflow website

gt; >> > You can click through the prototype but not all elements are > > > > interactive. > > > > > >> > Here is the link: https://invis.io/YSTUZ7S98ZN > > > > > >> > > > > > > >> > For further information please read AIP: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-11+Create+a+Landing+Page+for+Apache+Airflow > > > > > >> > > > > > > >> > If you have any comments regarding this prototype, please add > > them > > > > by > > > > > >> > replying to this email. Preferably till 21th September 11:00 > am > > > > > >> > (UTC+01:00). > > > > > >> > > > > > > https://www.timeanddate.com/worldclock/fixedtime.html?iso=20190921T11 > > > > > >> > > > > > > >> > Thanks! > > > > > >> > Kamil Breguła > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > === > > > > > >> > > > > > > >> > Please access the attached hyperlink for an important > electronic > > > > > >> > communications disclaimer: > > > > > >> > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > > > > >> > > > > > > >> > > > > > > > > > > === > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> Tomasz Urbaszek > > > > > >> Polidea <https://www.polidea.com/> | Junior Software Engineer > > > > > >> > > > > > >> M: [+48 505 628 493 <+48%20505%20628%20493>](tel:+48505628493 > > <+48%20505%20628%20493>) > > > > <+48%[20505](tel:20505)%[20628](tel:20628)%[20493](tel:20493)> > > > > <[+48505628493 <+48%20505%20628%20493>](tel:+48505628493) > > > > > >> <+48%[20505](tel:20505)%[20628](tel:20628)%[20493](tel:20493)>> > > > > > >> E: tomasz.urbas...@polidea.com > > > > > >> > > > > > >> Unique Tech > > > > > >> Check out our projects! <https://www.polidea.com/our-work> > > > > > >> > > > > > > > > > > > > -- > > > > > > > > > > > > Leah Cole | Developer Programs Engineer | colel...@google.com | > > > > [(925](tel:925)) > > > > > > [257-2112](tel:2572112) > > > > > > > > > > > > > -- > > > > > > Tomasz Urbaszek > > > Polidea <https://www.polidea.com/> | Junior Software Engineer > > > > > > M: +48 505 628 493 <+48%20505%20628%20493> <+48505628493 > > <+48%20505%20628%20493>> > > > E: tomasz.urbas...@polidea.com > > > > > > Unique Tech > > > Check out our projects! <https://www.polidea.com/our-work> > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Proposed roadmap for Airflow 2.0

t; Production Docker image + Helm chart > >> >>> We at Astronomer are committing to work on these in roughly this > order > >> >> if no one else gets to them first. I also propose that we create SIGs > >> >> (Special Interest Groups) in slack with weekly/fortnightly (every 14 > >> days) > >> >> "calls"/update sessions. We already have #sig-ui and > >> #sig-dag-serialisation. > >> >>> > >> >>> This roadmap is also not a promise that all of these will be done > >> before > >> >> Airflow 2.0 - we may decide later to push something back to v2.1 etc. > >> >>> > >> >>> Does anyone disagree strongly with these priorities, or have > anything > >> >> they want to add that you are willing to work on? > >> >>> > >> >>> Thanks, > >> >>> Ash > >> >> > >> >> > >> > > >> > -- > >> > > >> > Tomasz Urbaszek > >> > Polidea <https://www.polidea.com/> | Junior Software Engineer > >> > > >> > M: +48 505 628 493 <+48505628493> > >> > E: tomasz.urbas...@polidea.com > >> > > >> > Unique Tech > >> > Check out our projects! <https://www.polidea.com/our-work> > >> > >> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

AIP-21 - grouping google operators

n 2.0.0 we can still change it without hassle so we thought it's a good idea to do it now. Does anyone have anything against it ? If not we will soon move the operators there. J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129

Re: AIP-21 - grouping google operators

Agree with Ash. After doing the gcp move and seeing the result we agreed that 'provider' is better as additional prefix. If no-one objects (Lazy Consensus ) till Monday 3.20 CEST, we will update AIP-21 and move the gcp operators to *prov

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

s it easier and addresses my concerns. (Although it took me a > > while to realize those tables can be scrolled horizontally!). > > > I'm working on redesign of documentation theme. It's part of AIP-11 > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-11+Create+a+Landing+Page+for+Apache+Airflow > We are currently at the stage of collecting comments from the first > phase - we sent materials to the community, but also conducted tests > with real users > > https://lists.apache.org/thread.html/6fa1cdceb97ed17752978a8d4202bf1ff1a86c6b50bbc9d09f694166@%3Cdev.airflow.apache.org%3E > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

case calling > > this a S3ToMsSqlOperator and putting it under AWS seems silly, even > though > > you could say s3 is "source" here. > > > > I think in most of these cases we say "let's use source" because source > is > > where the actual

Re: ApacheCon EU ticket

heCon in Berlin later in October? I have one > > > ticket to give away the a community member of anyone is interested and > > can > > > travel to Berlin. > > > > > > > > > -- > > > > Tomasz Urbaszek > > Polidea <https://www.pol

Re: AIP-21 - grouping google operators

; > > > This seems unnecessary to me. > > > > > > Is everything going to be under some 'provider' or just certain sets of > > > operators, and if so what differentiates when something should be > under a > > > provider or not? For example,

Re: AIP-21 - grouping google operators

separate module. I could imagine that other > companies that provide managed services > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations > > > might > evolve to a separate package, but maybe we should come up with some > crite

[PROPOSAL] Production-ready Airflow docker image and helm chart proposal

/discussion in the AIP-26 document are welcome ! J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

* becomes airflow/providers/google/gcp/operators/* *mysql_to_gcs_operator.py* *SSH operator:* - *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* *operators/ssh_operator.py* On Fri, Oct 4, 2019 at 6:22 PM Jarek Potiuk wrote: > Yeah. I think the important point is that

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

run Kubernetes pods from Airflow using > in_cluster mode. Not sure if it allows to deploy Kubernetes executor > though. > > On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk > wrote: > > > I just created "AIP-26 - Production-ready Airflow Docker Image and helm > >

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

comments. J. On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk wrote: > Sorry - I missed that comment - it was not that you got ignored, I just > missed it in the flood of emails I had :(. sorry. > > The chart (and corresponding puckel image) is quite ok for the past but if > we want

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

t, I believe it's better to upgrade/fix the existing chart than > create a new one. > > On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, wrote: > > > Some more comments about Production-ready image and security/patches > > aspect. > > > > I updated the motivation sec

[PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

eing an example for other providers how they can do it. Let me know what you think - and whether I should make it into an official AIP maybe? J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

https://hub.helm.sh/charts?q=airflow - > to help deal some of the scale challenges of managing so many charts in > a monorepo, it might also be worth considering moving the chart to its > own repo. It'd be easier to couple the chart to official Airflow > versions and even have mu

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

Issue created! https://github.com/helm/charts/issues/17933 . Thanks Jonathan for feedback and bringing this up! On Mon, Oct 14, 2019 at 8:29 AM Jarek Potiuk wrote: > Thanks! Good points Jonathan. I will loop in the OWNERS indeed. I am also > in contact with Daniel from Bitnami who manag

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

should I start a vote on it ? J. On Mon, Oct 14, 2019 at 8:42 AM Jarek Potiuk wrote: > Issue created! https://github.com/helm/charts/issues/17933 . Thanks > Jonathan for feedback and bringing this up! > > On Mon, Oct 14, 2019 at 8:29 AM Jarek Potiuk > wrote: > >> Thanks!

Re: [PROPOSAL] Production-ready Airflow docker image and helm chart proposal

> So, tell you how could I help. > I think reviews, comments in the PR and in the discussion here is the best way to help. Also I think once we have the image more-or-less ready i will ask for help with testing the image in various scenarios, so here I'd appreciate your help here. > &g

Re: [PROPOSAL] Migrate to Pytest

ally that it > seems > > that we only need to fix 16 tests to obtain green CI build. > > > > Your considerations and feedback are highly welcomed! > > > > Bests, > > Tomek > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

his email is formally calling for a vote to accept the AIP and PR. Please > note that we will update the PR / feature to fix bugs if we find any. > > Cheers, > Kaxil > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

(binding) On Tue, Oct 15, 2019 at 9:43 AM Jarek Potiuk wrote: > > I looked over your shoulders while you were implementing it. Big +1 from > me. > > On Tue, Oct 15, 2019 at 2:57 AM Kaxil Naik wrote: > >> Hello, Airflow community, >> >> This email calls for

Kubernetes tests failing

running master build now to verify that it's OK and I cancelled the builds in progress to make it faster - once this is passing I will ask you to rebase your builds to get them working. J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48

Re: Kubernetes tests failing

the PR building from forks is implemented (they actively work on this change). J. On Tue, Oct 15, 2019 at 1:09 PM Jarek Potiuk wrote: > Today we started to have kubernetes tests failing due to some dependency > change. > > Luckily Gerardo just completed a change which changes th

Re: [Notes] KNative executor sync

: > >>> https://apache-airflow.slack.com/archives/CCPRP7943/p1571081889034400 > >>> > >>> > >>> Bests, > >>> Tomek > >>> > >>> -- > >>> > >>> Tomasz Urbaszek > >>> Polidea <https://www.polidea.com/> | Junior Software Engineer > >>> > >>> M: +48 505 628 493 <+48505628493> > >>> E: tomasz.urbas...@polidea.com > >>> > >>> Unique Tech > >>> Check out our projects! <https://www.polidea.com/our-work> > >> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [Discuss] EMR Step operator/sensor

> > There is still value in keeping those 2 other classes around when we > don't > > care about the result of an EMR step or we are polling for the completion > > of an EMR step we did not start from Airflow, but for most tasks > wouldn't a > > "merged operator" make sense? > > > > Thanks! > > Daniel > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

e number of DAGs. > > > >- A JSON Schema has been defined and we validate the serialized > dag > > > >before writing it to the database > > > > > > > > [image: image.png] > > > > > > > > A PR (https://github.com/apache/airflow/pull/5743) is ready for > review > > > > from the committers and community. > > > > > > > > We also have a WIP PR (https://github.com/apache/airflow/pull/5992) > to > > > > backport this feature to 1.10.* branch. > > > > > > > > A big thank you to Zhou and Ash for their continuous help in > improving > > > > this feature/PR. > > > > > > > > This email is formally calling for a vote to accept the AIP and PR. > > > Please > > > > note that we will update the PR / feature to fix bugs if we find any. > > > > > > > > Cheers, > > > > Kaxil > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Dockerhub issues

Seems that we have continuous DockerHub incident that impacts nearly everyone https://twitter.com/search?q=dockerhub&src=typed_query If you have better things to do, I'd say let's call it a day ;) . J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Softw

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

day of every month > > > > following > > > > > until this executor is released (meeting frequency may be updated > if > > > > there > > > > > is a need). > > > > > > > > > > We’ll be going over what we’ve done so far, what is yet to be done, > > and > > > > &g

First Warsaw Apache Meetup - Live Streamed in 10 minutes

Hello Everyone, In about 10 minutes we start out first Apache Airflow Meetup. It will be live-streamed at https://youtu.be/Nr4Pp1SNXeU J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Pol

Please rebase to latest master -> we have workaround for Kubernetes failures

Kubernetes. This will give us a chance to fix the tests while not blocking other changes. J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [DISCUSS] Smarter test execution for CI (trivial changes without full tests)

but there are frequent Doc only changes. > > In this case we should not run "real-test" but only static checks - mypy, > > pylint. flake8, doc generation. > > > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk > > wrote: > > > > > Hello everyone,

Re: [DISCUSS] Smarter test execution for CI (trivial changes without full tests)

3 PM Daniel Imberman wrote: > I’m still somewhat hesitant on this as it could allow regressions to peak > through, though as long as we’re still doing the daily build, and with our > soon-to-be-created prerelease load testing I think we should be okay. > > On Fri, Oct 18, 2019 at 10:30

Re: New PMC Member: Jarek Potiuk

t; > > > > > > Congratulations Jarek! You've been extremely to even us novices > learning > > > > the ins and outs of Airflow :). > > > > > > > > Damian > > > > > > > > -Original Message- > > > > From: Kaxil Na

Re: [VOTE] Release 1.10.6 from 1.10.6rc1

date to new logo (#6066) > > - [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291) > > - [AIRFLOW-XXX] Add message about breaking change in > > DAG#get_task_instances in 1.10.4 (#6226) > > - [AIRFLOW-XXX] Fix incorrect units in docs for metrics using Timers > > (#6152) > > - [AIRFLOW-XXX] Fix backtick issues in .rst files & Add Precommit hook > > (#6162) > > - [AIRFLOW-XXX] Update documentation about variables forcing answer > (#6158) > > - [AIRFLOW-XXX] Add a third way to configure authorization (#6134) > > - [AIRFLOW-XXX] Add example of running pre-commit hooks on single file > > (#6143) > > - [AIRFLOW-XXX] Add information about default pool to docs (#6019) > > - [AIRFLOW-XXX] Make Breeze The default integration test environment > > (#6001) > > > > Misc/Internal > > """"""""""""" > > - [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline (#6358) > (#6361) > > - [AIRFLOW-5533] Fixed failing CRON build (#6167) > > - [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant from library > > (#5744) > > - [AIRFLOW-5369] Adds interactivity to pre-commits (#5976) > > - [AIRFLOW-5531] Replace deprecated log.warn() with log.warning() (#6165) > > - [AIRFLOW-4686] Make dags Pylint compatible (#5753) > > - [AIRFLOW-4864] Remove calls to load_test_config (#5502) > > - [AIRFLOW-XXX] Pin version of mypy so we are stable over time (#6198) > > - [AIRFLOW-XXX] Add tests that got missed from #5127 > > - [AIRFLOW-4928] Move config parses to class properties inside DagBag > > (#5557) > > - [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627) > > - [AIRFLOW-5580] Add base class for system test (#6229) > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [VOTE] Release 1.10.6 from 1.10.6rc1

update a few docs and build a CI test for it, and > if it cannot just work outta box or requires a lot work we can give up > eariler :D). > > Cheers, > Kevin Y > > On Fri, Oct 18, 2019 at 7:26 PM Jarek Potiuk > wrote: > > > This is really what's intended. 1.10 su

Re: Proposed roadmap for Airflow 2.0

> > > > bharath...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > >Hi, > > > > > > > > > > > > > > > >Is there any discussion thread on adding prior

Re: [VOTE] Release 1.10.6 from 1.10.6rc1

pg: Good signature from "Ash Berlin-Taylor " [unknown] > gpg: WARNING: This key is not certified with a trusted signature! > gpg: There is no indication that the signature belongs to the > owner. > Primary key fingerprint: 5CCA EAC7 58ED 64CA 323F 053B 807C 731A 8C

Re: [VOTE] AIP-24: Persisting serialized DAG in DB for webserver scalability

ed is a very frequently accessed > > >> endpoint > > >>> and for large DAG files( we have DAG files taking multiple mins to > > parse) > > >>> it would take too long to load, block the webserver workers if we got > > too > > >>&

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

ion number excludes the `rcX` string, so it's > > now > > > simply 1.10.6. This will allow us to rename the artifact without > > modifying > > > the artifact checksums when we actually release. > > > > > > The changes since RC1 are to fix License issues, ensure tests

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

Hello - any comments on that? I am happy to make it into an AIP :)? On Sun, Oct 13, 2019 at 5:53 PM Jarek Potiuk wrote: > *Motivation* > > I think we really should start thinking about making it easier to migrate > to 2.0 for our users. After implementing some recent changes relat

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

release those (without removing the old operators yet - just raise deprecation warnings and inform that for python3 the new "airflow-google", "airflow-aws" etc. packages can be installed and users can switch to it). J. > > Cheers, > Bas > > On 27 Oct 2019, at

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

es/c9b9312a60f891475d3072584171c2af56246829#r339287344 > > > > > > > > So we either need to release a 1.10.7 and force users to migrate to > > that > > > > version first, before going to 2.0.0, otherwise, it won't work. I've > > > ope

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

licts: # only query for pool conflicts if one may exist pool = ( session .query(Pool) .filter(Pool.slots == 1) .filter(Pool.pool == self.pool) .first() ) On Sun, Oct 27, 2019 at 5:33 PM Jarek Potiuk wrote: >

Re: [VOTE] Release 1.10.6 from 1.10.6rc2

I see further migrations > after the error. Don't forget that DagBag loading handles errors, and, > further migrations run after the error is printed, and `echo $?` reports > success. > > -ash > > > On 27 Oct 2019, at 16:33, Jarek Potiuk wrote: > > > > I also

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

utilizing namespaces should reduce a lot of problems raised by > > using separate repos (who will manage it? how to release? where should be > > the repo?). > > > > Bests, > > Tomek > > > > On Sun, Oct 27, 2019 at 11:54 AM Jarek Potiuk > > wrote: > &g

Re: Apache Airflow 1.10.6 released!

rflow`): > > https://pypi.python.org/pypi/apache-airflow > > The documentation is available on: > https://airflow.apache.org/ > https://airflow.apache.org/1.10.6/ > > Find the CHANGELOG here for more details: > > https://airflow.apache.org/changelog.html#ai

Re: [RESULT] [VOTE] Release 1.10.6 from 1.10.6rc2

Fantastic! On Mon, Oct 28, 2019 at 10:18 PM Ash Berlin-Taylor wrote: > Hello, > > Apache Airflow 1.10.6 (based on RC2) has been accepted. > > 4 “+1” binding votes received: > - Ash Berlin-Taylor (binding) > - Kaxil Naik (binding) > - Fokko Driesprong (binding) > - Ja

Re: [RESULT] [VOTE] Release 1.10.6 from 1.10.6rc2

ted. > >> > >> 4 “+1” binding votes received: > >> - Ash Berlin-Taylor (binding) > >> - Kaxil Naik (binding) > >> - Fokko Driesprong (binding) > >> - Jarek Potiuk (binding) > >> > >> No other votes were cast. > >> > >

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Amazon Web Services, GCP: Google Cloud Platform, Service integrations, Software integrations, Protocol integrations). I think it would make sense to use the same approach in the code: We could have fundamentals/asf/azure(microsoft/azure?)/aws(amazon/aws?)/google/services/software/protocols)

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

damentals” and “software” are meant to be :-) > > While we’re at it: we should really move the BaseOperator out of models. > The BaseOperator has no representation in the DB and should be placed > together with other scripts where it belongs, i.e. something like > airflow.operators.base_oper

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

ly I would omit the Apache one. We should not step into the > > fallacy as before with not being sure if it was in contrib or not. I > would > > even consider merging software and protocols, as it not entirely clear > what > > a protocol is or not. In

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

lready talked about how it > can benefit us. IMO on a high level, having clearly seperation on core vs. > hooks/operators would make the project much more scalable and the gains > would outweigh the cost we pay. > > That being said, I'm supportive to this moving towards AIP-8 while l

Re: Donating code to add Common Workflow Language import to Airflow

I have not looked at the details of the code yet, it I would like to understand few things: 1) is the CWL package more of a converter of CWL to Python DAG files (that can then be scheduled as usual) or whether it is running alongside of the scheduler and schedules tasks and operators separately us

Re: Tomorrow is our first SIG-Knative meeting

endar.google.com%26scp%3DALL > > ] > > > > The event will also be recorded and posted to #sig-knative, > > #airflow-kubernetes and the mailing list afterwards > > > > See you there! > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Inconsistent class names for AWS integrations

<mailto:mingram...@gmail.com>> wrote: > > > > Hi all. > > > > While I was working on AIRFLOW-5803, I found there are inconsistencies > > between class names for each AWS operators/hooks. > > > > For example, Athena hook is named as 'AWSAthenaH

Re: [PROPOSAL] Migrate to Pytest

ProtonMail Mobile > > > > > > > > > > On Mon, Oct 14, 2019 at 16:39, Daniel Imberman < > > > > > dimberman.opensou...@gmail.com> wrote: > > > > > > > > > > > Huge +1 from me on this one! > > > > > > > >

Re: New promotional materials for Airflow

ps://cwiki.apache.org/confluence/display/AIRFLOW/Promo+stuff > > Best regards, > Kamil > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Inconsistent class names for AWS integrations

gt; Felix > > Sent from ProtonMail Mobile > > On Wed, Oct 30, 2019 at 10:05, Jarek Potiuk > wrote: > > > Few comments: > > > > - I like the rules > > - For google we add additional "google/gcp" , "google/gsuite". > > "google/m

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

. On Tue, Oct 29, 2019 at 4:51 PM Kaxil Naik wrote: > The namespace feature looks promising and from your tests, it looks like it > would work well from Airflow 2.0 and onwards. > > I will look at it in-depth and see if I have more suggestions or opinion on > it > >

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

.py| |-tests -> tests for the "apache-airflow-integrations-protocols" package J. On Thu, Oct 31, 2019 at 3:38 PM Kaxil Naik wrote: > So create another package in a different repo? or the same repo with a > separate setup.py file that has airflow has dependency? &

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

age| | > >|-__init__.py| |-protocols > >| |-setup.py -> setup.py for the > >"apache-airflow-integrations-protocols" package > >| |-airflow_integrations > >||-protocols > >| |-__init__.py| |-tests -> tests for th

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

/site-packages/notairflow/providers/gcp/__init__.py > module level code running > > > > Let me test it again in a few different cases etc. > > > > -a > > > > On 4 November 2019 14:00:24 GMT, Jarek Potiuk > wrote: > > Hey Ash, > > > > Thanks for t

Wrong Urllib/Urlquote imports in v1-10-test

airflow/contrib/operators/sftp_to_s3_operator.py - airflow/sensors/s3_key_sensor.py - airflow/utils/log/gcs_task_handler.py J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: Wrong Urllib/Urlquote imports in v1-10-test

M Ash Berlin-Taylor wrote: > > In [9]: urllib.parse > Out[9]: '/Users/ash/.homebrew/lib/python2.7/site-packages/future/backports/urllib/parse.pyc'> > > So it's coming form the future module we use in py2. > > So nothing is broken thankfully. > &

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

eem to have a working solution now. I will cast a final vote for what I think is a consensus now as update to AIP-21 (there is no point in creating a separate AIP). J. On Tue, Nov 5, 2019 at 11:34 AM Kaxil Naik wrote: > Yes let's just do (1) for now. > > > > On Tue, Nov 5

[VOTE] AIP-21 update for Airflow 1.10.* backportability

GCS-> S3 should be in "amazon". 4) One-side provider transfer operators should be kept at the "provider" regardless if they are target or source. For example GCS-> SFTP or SFTP -> GCS should be in "google" provider. 5) If in doubt we will discuss individ

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

mplexity, and the shared code, are in the > gsheet component -- not into the storage destination. > > > On Tue, Nov 5, 2019 at 5:46 PM Jarek Potiuk > wrote: > > > Hello Airflow Community, > > > > The email calls for a vote to update AIP-21 Changes in import

[VOTE] AIP-26 Production ready Airflow Docker image and helm chart

pache/airflow/pull/6501 The vote will last a bit longer than 72 hours as it will include the weekend - till Monday 11th of November 11pm CET. At least three binding votes are needed. Non-binding (non-committers) votes are still strongly encouraged. J. -- Jarek Potiuk Polidea <https://www.pol

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

ntal" operators/hooks etc.. * Airflow 2.0 is still going to be installed as a single package with all operators (so we are not yet implementing AIP-8) J. On Wed, Nov 6, 2019 at 10:07 AM Jarek Potiuk wrote: > I think all this cases are valid but maybe I was not super-clear. It's > on

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

r wrote: > Do we need to include `-backport,`? What was the thinking behind that? > > I think software and protocol should be merged. I would also say > _everything_ is a provider, so airflow.providers.ssh.SSHOperator for > instance is what I would prefer > > -a > > On 8 Nov

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

... > > > microsoft > > > azure > > > operators > > > ... > > > hadoop > > > hdfs > > > operators > > > ... > > > > > > We can

Re: [Final review] Airflow website + need your input to the web copy

t; > > > > > > > > > >> On 8 November 2019 20:08:41 GMT, Aizhamal Nurmamat kyzy < > > >> aizha...@apache.org> wrote: > > >>> > > >>> Hi everybody, > > >>> > > >>> We are super close to the final version of the Airflow website and we > > >>> really need YOU to play with the prototype to make sure that > > everything is > > >>> working properly. > > >>> > > >>> Here is the link: https://airflow-website-s5z26d2t7a-ew.a.run.app/ > > >>> > > >>> Please submit your feedback on the prototype in this thread before > > Tuesday, > > >>> 10.00am CEST. After that it will be Commit-then-Review process. > > >>> > > >>> We also need YOUR help to create a website copy. Marta has created > this > > >>> document [1] where you can put the information, instructions or > > >>> guidelines that need to appear in each section. It doesn't need to be > > >>> perfect, just needs to convey the knowledge that you already have on > > how to > > >>> install Airflow or how to submit a bug, for example. I will be adding > > >>> things myself too. > > >>> > > >>> Let's make sure that we have the necessary and relevant > > >>> information available for our users and contributors before we > publish > > it. > > >>> Here is an article on how to write a web copy if anyone wants to > take a > > >>> look [2]. > > >>> > > >>> [1] > > >>> > > > https://docs.google.com/document/d/1Y4jBjtQ13zeLZ2e3V4foa51pN-168p1DHJ5zc7cbyGg/edit?ts=5dbc78cd > > >>> [2] https://expresswriters.com/how-to-write-web-copy/ > > >>> > > >>> > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

8, 2019 at 5:47 PM Jarek Potiuk wrote: > Let me then cancel this vote and I will restart it next week. > > Yeah. It's a bit like re-opening the Pandora's box but now that we know > that we can do it, and we are unblocked in moving to google (which is now > the biggest mo

Re: Donating code to add Common Workflow Language import to Airflow

required to put .py file with DAG (CWLDAG > in > > our case) to the dag directory. I would like just to put .cwl file into > DAG > > directory to simplify the usage > > > > I'm ready to develop what is necessary, but I'm not quite sure (I'm not a > > big expert in airflow code) which way to go, plugin or some native core > > code, or ... > > > > The project by itself lives https://github.com/Barski-lab/cwl-airflow, > > there are tons of CWL tests > > https://ci.commonwl.org/job/airflow-conformance/ > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>

Re: [VOTE] AIP-21 update for Airflow 1.10.* backportability

r integration into our products with other products. > > On Mon, Nov 11, 2019 at 12:28 AM Jarek Potiuk > wrote: > > > > Just to select the "packages" for this update. Anyone has objections for > > this structure (details including transfer operators in

  1   2   3   4   5   6   7   8   9   10   >