Re: [PLEASE PARTICIPATE][AIP-11] UX phase of Airflow website

=== > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > ===

Re: [PLEASE PARTICIPATE][AIP-11] UX phase of Airflow website

t; > > > > My only other thing to note was to make sure that the new site is > > > accessible so that Airflow is inclusive to all regardless of ability > :) - I > > > ran the Chrome accessibility audit on this and it went meh but am not > > > tot

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

think? Anyone strongly against this? Or maybe we can follow > > lazy consensus rule for this? Or maybe someone can come up with a better > > name :) ? > > > > J. > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/>

Re: Proposed roadmap for Airflow 2.0

rflow 2.0 - we may decide later to push something back to v2.1 etc. > > > > Does anyone disagree strongly with these priorities, or have anything > they want to add that you are willing to work on? > > > > Thanks, > > Ash > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Issues for Hacktoberfest 2019

Hi all, It's nearly time for the Hacktoberfest 2019. For those who do not know - it's a global event encouraging developers around the world to contribute to open source projects. For more information check official web page: https://hacktoberfest.digitalocean.com . I am wondering if there's any

Re: Warsaw Apache Airflow Meetup

etup.com/Warsaw-Airflow-Meetup/events/264867971/ > > > > > > > > Please send your talk proposal if you wish to share a talk or your > use > > > > case at this event. > > > > > > > > Looking forward to seeing you all! > > > > > > > > Thanks, > > > > Aizhamal > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [AIP-11] Airflow Brandbook

Hello, > > > > > >The Polidea team has prepared a Brandbook for us. > > > > > >https://drive.google.com/open?id=1ErNyfR62eQP2KTPW-9ULx5glYoYgkT-r > > > > > >I hope that it will be helpful. > > > > > >Best regards, > > >Kamil > >

Re: [PLEASE PARTICIPATE][AIP-11] UI phase of Airflow website

g to this email. Preferably till 7th October 11:00 am > > > (UTC+01:00). > > > https://www.timeanddate.com/worldclock/fixedtime.html?iso=20191007T11 > > > > > > Best regards, > > > Kamil > > > > > > > > > > > >

Re: ApacheCon EU ticket

e away the a community member of anyone is interested and can > travel to Berlin. > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: AIP-21 - grouping google operators

ote: > > > > > > > > > > > > This seems unnecessary to me. > > > > > > > > > > > > Is everything going to be under some 'provider' or just certain > > sets > > > of > > > > > > op

Re: ApacheCon EU ticket

Sure! Thank you very much! Tomek On Wed, Oct 9, 2019 at 9:51 AM Aizhamal Nurmamat kyzy wrote: > @Tomasz Urbaszek I got another ticket for > the ApacheCon EU that you can use :) Are you still interested? > > On Fri, Oct 4, 2019 at 6:26 PM Jarek Potiuk > wrote: > >&

[PROPOSAL] Migrate to Pytest

Hi all, I've just created AIP-27 Migrate to pytest . I think this proposal will improve our test suite especially that it seems that we only need to fix 16 tests to obtain green CI build. Your considerations and feedbac

[Notes] KNative executor sync

of regular meetings to talk about the future of airflow proposed by Kamil Breguła in this thread: https://apache-airflow.slack.com/archives/CCPRP7943/p1571081889034400 Bests, Tomek -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48

Re: [Notes] KNative executor sync

ld think about archiving these types of documents. What > do you think about copying this document to the cwiki? > > Best regards, > > On Tue, Oct 15, 2019 at 5:35 PM Tomasz Urbaszek > wrote: > > > > Hi all, > > > > Today Astronomer and Polidea have a very

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

; >>>>> following > >>>>>> until this executor is released (meeting frequency may be updated > >> if > >>>>> there > >>>>>> is a need). > >>>>>> > >>>>>> We’ll be going over what we’ve done so far, what is yet to be done, > >>> and > >>>>>> what opportunities are available to members of the community > >>> interested > >>>>> in > >>>>>> breaking into the open source world :). > >>>>>> > >>>>>> Add yourself to the monthly event here > >>>>>> < > >>>>> > >>>> > >>> > >> > https://calendar.google.com/event?action=TEMPLATE&tmeid=b2I4Yml2MWdhcXRtdGRtZmkzcXNxdmY5cThfMjAxOTEwMzBUMTUwMDAwWiBkYW5pZWxAYXN0cm9ub21lci5pbw&tmsrc=daniel%40astronomer.io&scp=ALL > >>>>>> , > >>>>>> join us at #sig-knative and see you all in two weeks! > >>>>>> > >>>>> > >>>> > >>> > >> > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: New PMC Member: Jarek Potiuk

ject. > > Congratulations Jarek, keep doing the great work !! > > Cheers, > Kaxil Naik, > on behalf of Airflow PMC > > > > === > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com

Re: [VOTE] Release 1.10.6 from 1.10.6rc1

; > >they > > >> can't > > >> > > > heartbeat (#6284) > > >> > > > - [AIRFLOW-5572] Clear task reschedules when clearing task > > >instances > > >> > > > (#6217) > > >> > > > - [AIRFLOW-5543] Fix tooltip disappears in tree and graph view > > >(RBAC > > >> > UI) > > >> > > > (#6174) > > >> > > > - [AIRFLOW-5444] Fix action_logging so that request.form for > > >POST is > > >> > > > logged (#6064) > > >> > > > - [AIRFLOW-5484] fix PigCliHook has incorrect named parameter > > >(#6112) > > >> > > > - [AIRFLOW-5342] Fix MSSQL breaking task_instance db migration > > >> (#6014) > > >> > > > - [AIRFLOW-5556] Add separate config for timeout from scheduler > > >dag > > >> > > > processing (#6186) > > >> > > > - [AIRFLOW-4858] Deprecate "Historical convenience functions" > > >in > > >> > > > airflow.configuration (#5495) (#6144) > > >> > > > - [AIRFLOW-774] Fix long-broken DAG parsing Statsd metrics > > >(#6157) > > >> > > > - [AIRFLOW-5419] Use ``sudo`` to kill cleared tasks when > > >running with > > >> > > > impersonation (#6026) (#6176) > > >> > > > - [AIRFLOW-5537] Yamllint is not needed as dependency on host > > >> > > > - [AIRFLOW-5536] Better handling of temporary output files > > >> > > > - [AIRFLOW-5535] Fix name of VERBOSE parameter > > >> > > > - [AIRFLOW-5519] Fix sql_to_gcs operator missing multi-level > > >default > > >> > args > > >> > > > by adding apply_defaults decorator (#6146) > > >> > > > - [AIRFLOW-5210] Make finding template files more efficient > > >(#5815) > > >> > > > - [AIRFLOW-5447] Scheduler stalls because second watcher thread > > >in > > >> > > default > > >> > > > args (#6129) > > >> > > > > > >> > > > Doc-only changes > > >> > > > """""""""""""""" > > >> > > > - [AIRFLOW-5574] Fix Google Analytics script loading (#6218) > > >> > > > - [AIRFLOW-5588] Add Celery's architecture diagram (#6247) > > >> > > > - [AIRFLOW-5521] Fix link to GCP documentation (#6150) > > >> > > > - [AIRFLOW-5398] Update contrib example DAGs to context manager > > >> (#5998) > > >> > > > - [AIRFLOW-5268] Apply same DAG naming conventions as in > > >literature > > >> > > (#5874) > > >> > > > - [AIRFLOW-5101] Fix inconsistent owner value in examples > > >(#5712) > > >> > > > - [AIRFLOW-XXX] Fix typo - AWS DynamoDB Hook (#6319) > > >> > > > - [AIRFLOW-XXX] Fix Documentation for adding extra Operator > > >Links > > >> > (#6301) > > >> > > > - [AIRFLOW-XXX] Add section on task lifecycle & correct casing > > >in > > >> docs > > >> > > > (#4681) > > >> > > > - [AIRFLOW-XXX] Make it clear that 1.10.5 wasn't accidentally > > >omitted > > >> > > from > > >> > > > UPDATING.md (#6240) > > >> > > > - [AIRFLOW-XXX] Improve format in code-block directives (#6242) > > >> > > > - [AIRFLOW-XXX] Format Sendgrid docs (#6245) > > >> > > > - [AIRFLOW-XXX] Update to new logo (#6066) > > >> > > > - [AIRFLOW-XXX] Typo in FAQ - schedule_interval (#6291) > > >> > > > - [AIRFLOW-XXX] Add message about breaking change in > > >> > > > DAG#get_task_instances in 1.10.4 (#6226) > > >> > > > - [AIRFLOW-XXX] Fix incorrect units in docs for metrics using > > >Timers > > >> > > > (#6152) > > >> > > > - [AIRFLOW-XXX] Fix backtick issues in .rst files & Add > > >Precommit > > >> hook > > >> > > > (#6162) > > >> > > > - [AIRFLOW-XXX] Update documentation about variables forcing > > >answer > > >> > > (#6158) > > >> > > > - [AIRFLOW-XXX] Add a third way to configure authorization > > >(#6134) > > >> > > > - [AIRFLOW-XXX] Add example of running pre-commit hooks on > > >single > > >> file > > >> > > > (#6143) > > >> > > > - [AIRFLOW-XXX] Add information about default pool to docs > > >(#6019) > > >> > > > - [AIRFLOW-XXX] Make Breeze The default integration test > > >environment > > >> > > > (#6001) > > >> > > > > > >> > > > Misc/Internal > > >> > > > """"""""""""" > > >> > > > - [AIRFLOW-5687] Upgrade pip to 19.0.2 in CI build pipeline > > >(#6358) > > >> > > (#6361) > > >> > > > - [AIRFLOW-5533] Fixed failing CRON build (#6167) > > >> > > > - [AIRFLOW-5130] Use GOOGLE_APPLICATION_CREDENTIALS constant > > >from > > >> > library > > >> > > > (#5744) > > >> > > > - [AIRFLOW-5369] Adds interactivity to pre-commits (#5976) > > >> > > > - [AIRFLOW-5531] Replace deprecated log.warn() with > > >log.warning() > > >> > (#6165) > > >> > > > - [AIRFLOW-4686] Make dags Pylint compatible (#5753) > > >> > > > - [AIRFLOW-4864] Remove calls to load_test_config (#5502) > > >> > > > - [AIRFLOW-XXX] Pin version of mypy so we are stable over time > > >> (#6198) > > >> > > > - [AIRFLOW-XXX] Add tests that got missed from #5127 > > >> > > > - [AIRFLOW-4928] Move config parses to class properties inside > > >DagBag > > >> > > > (#5557) > > >> > > > - [AIRFLOW-5003] Making AWS Hooks pylint compatible (#5627) > > >> > > > - [AIRFLOW-5580] Add base class for system test (#6229) > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > -- > > >> > > > >> > Jarek Potiuk > > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > > >> > M: +48 660 796 129 <+48660796129> > > >> > [image: Polidea] <https://www.polidea.com/> > > >> > > > >> > > > > > > > > >-- > > > > > >Jarek Potiuk > > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > >M: +48 660 796 129 <+48660796129> > > >[image: Polidea] <https://www.polidea.com/> > > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

st in 1.10). That might lead to confusion of > > users and problems with cooperation between different operators/hooks > > - Having new operators in 1.10 python 3 might keep people from > > migrating to 2.0 > > - It will require some maintenance and separate release overhead. > > > > I already spoke to Composer team @Google and they are very positive about > > this. I also spoke to Ash and seems it might also be OK for Astronomer > > team. We have Google's backing and support, and we can provide > maintenance > > and support for those packages - being an example for other providers how > > they can do it. > > > > Let me know what you think - and whether I should make it into an > official > > AIP maybe? > > > > J. > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: Apache Airflow 1.10.6 released!

;> > > > apache-airflow`): > >> > > > > >> > > > https://pypi.python.org/pypi/apache-airflow > >> > > > > >> > > > The documentation is available on: > >> > > > https://airflow.apache.o

Re: Inconsistent class names for AWS integrations

WS > integrations like GCP ( > https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb72dfd# > ) > > What do you think about it? > > Thank you. > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628

Re: [PROPOSAL] Migrate to Pytest

gt;> > > > > >> > On 14 Oct 2019, at 15:29, Jarek Potiuk < > jarek.pot...@polidea.com> > > > > wrote: > > > > >> > > > > > >> > Fantastic! Pytest is sooo much better than nosetests! > > > > >>

Re: New promotional materials for Airflow

lidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [Final review] Airflow website + need your input to the web copy

ace > >> >>>>> > >> >>>> The idea was to have a documentation page, that supports > >> images/graphs, > >> >>>> etc. And if we end up adding more to it in the future. It can be > one > >> >> page > >> >>>> only as well. > >> >>>> > >> >>>> - "learn more" from meetup page doesn't go anywhere useful > >> >>>>> > >> >>>> Learn more will lead to the Meetup.com registration page. > >> >>>> > >> >>>> > >> >>>>> - There are two section "collapsers" on the community page: a blue > >> >> arrow > >> >>>>> on the right and a black one on the left (again, see screenshot) > >> >>>>> > >> >>>> Again, can't see the screenshots :( > >> >>>> > >> >>>> > >> >>>>> - we should probably point most users towards users@ not dev@ > >> mailing > >> >>> list > >> >>>>> > >> >>>> Agree. Is user@ list currently active? I am just wondering if > people > >> >> are > >> >>>> following that list, and user's questions are getting enough > >> attention. > >> >>>> > >> >>>> - "Primary Members Commiters" heading on Community page should > >> probably > >> >>> say > >> >>>>> "Project Management Committee" > >> >>>> > >> >>>> +1 > >> >>>> > >> >>>> Hope this list was useful :) > >> >>>>> > >> >>>> +100 :) Thank you. > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>>> On 8 November 2019 20:08:41 GMT, Aizhamal Nurmamat kyzy < > >> >>>>> aizha...@apache.org> wrote: > >> >>>>>> > >> >>>>>> Hi everybody, > >> >>>>>> > >> >>>>>> We are super close to the final version of the Airflow website > and > >> we > >> >>>>>> really need YOU to play with the prototype to make sure that > >> >>> everything is > >> >>>>>> working properly. > >> >>>>>> > >> >>>>>> Here is the link: > https://airflow-website-s5z26d2t7a-ew.a.run.app/ > >> >>>>>> > >> >>>>>> Please submit your feedback on the prototype in this thread > before > >> >>> Tuesday, > >> >>>>>> 10.00am CEST. After that it will be Commit-then-Review process. > >> >>>>>> > >> >>>>>> We also need YOUR help to create a website copy. Marta has > created > >> >> this > >> >>>>>> document [1] where you can put the information, instructions or > >> >>>>>> guidelines that need to appear in each section. It doesn't need > to > >> be > >> >>>>>> perfect, just needs to convey the knowledge that you already > have on > >> >>> how to > >> >>>>>> install Airflow or how to submit a bug, for example. I will be > >> adding > >> >>>>>> things myself too. > >> >>>>>> > >> >>>>>> Let's make sure that we have the necessary and relevant > >> >>>>>> information available for our users and contributors before we > >> >> publish > >> >>> it. > >> >>>>>> Here is an article on how to write a web copy if anyone wants to > >> >> take a > >> >>>>>> look [2]. > >> >>>>>> > >> >>>>>> [1] > >> >>>>>> > >> >>> > >> >> > >> > https://docs.google.com/document/d/1Y4jBjtQ13zeLZ2e3V4foa51pN-168p1DHJ5zc7cbyGg/edit?ts=5dbc78cd > >> >>>>>> [2] https://expresswriters.com/how-to-write-web-copy/ > >> >>>>>> > >> >>>>>> > >> >>> > >> >>> > >> >> > >> > > >> > > >> > -- > >> > > >> > Jarek Potiuk > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > > >> > M: +48 660 796 129 <+48660796129> > >> > [image: Polidea] <https://www.polidea.com/> > >> > >> -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [VOTE] 2nd attempt for AIP-21 update for Airflow 1.10.* backportability

ttps://lists.apache.org/thread.html/2c9559184045e772acd21cbdd7435f6bf89c76eb9311311d58d16e5f@%3Cdev.airflow.apache.org%3E > > > Consider this my +1 (binding) vote. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer >

Re: [Discuss] Airflow Summits 2020

vent please let > us know. If you know of any suitable venues, let us know as well. We are > thinking of 200-300 people for each summit. > > Thanks, > Aizhamal > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+4850562

Re: Closing JIRA Issue for Merged PRs

e sure to close the Jira issues if the related PRs are > > >> merged. > > >> > > > > >> > > I am going through the Jira Reports (Image: > > https://imgur.com/n50Ticx > > >> ) > > >> > and > > >> > > was concerned

Re: Closing JIRA Issue for Merged PRs

kport a PR, but this could be achieved with > Milestones in GitHub. > > (A fringe benefit is that most people won't have an ASF Jira account so > opening issues to ask questions is harder. "Benefit" as it avoids issue > spam for question.) > > Is there anything else?

Re: [VOTE] Accept new Airflow website contribution

allow this PR to move forward so we > > can > > >> finalize the project; and work with us to realize any gradual > > improvements > > >> after the current version is merged. > > >> > > >> To see what the website looks like, pleas

Re: [Discuss] Airflow Summits 2020

> >> > > > > > > > > >> >> > > > > > > > Hi Airflow community, > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > I’d like to kick of a thread to discuss the > organization > >> of > >> >> two > >> >> > > > > Airflow > >> >> > > > > > > > summits in 2020, one in North America, one in Europe. > >> This > >> >> idea > >> >> > > has > >> >> > > > > > been > >> >> > > > > > > > brought to the attention of the PMC, and with their > >> >> consensus I > >> >> > > am > >> >> > > > > > > sharing > >> >> > > > > > > > it here. > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > These summits will be community events, and an > >> opportunity > >> >> to > >> >> > > bring > >> >> > > > > > > > together users and contributors of Apache Airflow, and > >> >> > > collaborate > >> >> > > > on > >> >> > > > > > the > >> >> > > > > > > > development of the project. > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > Proposed dates and locations are as follows, but we can > >> >> discuss > >> >> > > any > >> >> > > > > > other > >> >> > > > > > > > options as well: > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > - NA: Bay Area, late May or early June > >> >> > > > > > > > - Europe: London or Amsterdam or Warsaw mid Sep or > early > >> Oct > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > If you’d like to help with the organization or sponsor > >> the > >> >> > event > >> >> > > > > please > >> >> > > > > > > > let us know. If you know of any suitable venues, let us > >> >> know as > >> >> > > > well. > >> >> > > > > > We > >> >> > > > > > > > are thinking of 200-300 people for each summit. > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > Thanks, > >> >> > > > > > > > Aizhamal > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > -- > >> >> > > > > > > >> >> > > > > > Jarek Potiuk > >> >> > > > > > Polidea <https://www.polidea.com/> | Principal Software > >> >> Engineer > >> >> > > > > > > >> >> > > > > > M: +48 660 796 129 <+48%20660%20796%20129> > >> <+48%20660%20796%20129> > >> >> > <+48%20660%20796%20129> <+48660796129 <+48%20660%20796%20129> > >> <+48%20660%20796%20129> > >> >> > > > <+48%20660%20796%20129>> > >> >> > > > > > [image: Polidea] <https://www.polidea.com/> > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > > >> >> > -- > >> >> > > >> >> > Jarek Potiuk > >> >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> >> > > >> >> > M: +48 660 796 129 <+48%20660%20796%20129> <+48%20660%20796%20129> > < > >> +48660796129 <+48%20660%20796%20129> > >> >> > <+48%20660%20796%20129>> > >> >> > [image: Polidea] <https://www.polidea.com/> > >> >> > > >> >> > >> > > >> > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [Discuss] Airflow Summits 2020

+1 for Airflow Current On Wed, 20 Nov 2019 at 17:09, Philippe Gagnon wrote: > Airflow Current? > > On Wed, Nov 20, 2019 at 5:52 AM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > wrote: > > > Airflow Wind? Airflow Breath? > > > > On Wed, Nov 20

Re: KubernetesPodOperator usage sharing

ight be good idea to share it here as well: > > https://medium.com/dailymotion/bring-machine-learning-models-faster-to-production-with-airflow-and-kubernetes-e9d47ca3bee5 > > Your comments are most welcome ! > > Cheers, > > Eric > -- Tomasz Urbaszek Polidea <https

Re: [VOTE] Accept donation of Airflow k8s Operator to Apache Airflow

gt; > > > > > > [ ] +1 : Accept contribution of Airflow k8s operator > > > > > > [ ] 0 : No opinion > > > > > > [ ] -1 : Reject contribution because... > > > > > > > > > > > > The vote will be open for at least 72 hours. > > > > > > >

Re: New PMC members: Aizhamal and Kevin

is much more to > an open-source project than just the code! > > Being a PMC member enables assistance with the management and to guide the > direction of the project. > > Congratulations Kevin and Aizhamal, and keep doing the great work!! > > Cheers, > Ash and the Airflow P

Airflow user survey

Hi all, On slack there was an idea to run another survey among Airflow users, one that will be similar to the one done by Ash earlier this year. The survey will allow us to get more users insight before Airflow 2.0. This tim

Re: [DISCUSS] Using shared memory for inter-task communication

that: > > > > > > 1) they are all executed on the same worker machine > > > 2) the processes should remain in-memory until all tasks finish for > data > > > sharing (even if there is a dependency between the tasks) > > > 3) back-filling should act on the w

Re: [PROPOSAL] Migrate to Pytest

> isn't that big a change, right? > > >>> > > > >>> > (+1 though. Love py-test) > > >>> > > > >>> > -a > > >>> > > > >>> > > On 30 Oct 2019, at 14:49, Jarek Potiuk > > &

Re: [PROPOSAL] Migrate to Pytest

ge means mostly that you should use pytest instead of nose / run-tests script. If you run test from IDE it will mean that you need to have to adjust test runner. The mentioned PR: https://github.com/apache/airflow/pull/6472 Bests, Tomek On Wed, Nov 27, 2019 at 5:11 PM Tomasz Urbaszek wrote:

Re: [DISCUSS] Using asserts in airflow code

gt; > > > > > > > > We would like to discuss those points in community and have a > > > > community - > > > > > > driven decision on: > > > > > > > > > > > > 1) whether we should use asserts? > > > > > > 2) in which cases > > > > > > 3) in which cases we should NOT use asserts. > > > > > > > > > > > > J. > > > > > > > > > > > > The links here: > > > > > > > > > > > > Slack Discussion: > > > > > > > > > https://apache-airflow.slack.com/archives/CCQ7EGB1P/p157536466404130 > > > 0 > > > > > > > > > > > > Github threads: > > > > > > > > > > > >- > > > https://github.com/apache/airflow/pull/6596#discussion_r352916409 > > > > > >- > > > https://github.com/apache/airflow/pull/6596#discussion_r352914727 > > > > > >- > > > > > > > > > > > > https://github.com/apache/airflow/pull/3690#pullrequestreview-14337662 > > 9 > > > > > > > > > > > > Stack overflow link for asserts: > > > > > > > > > > > >- https://stackoverflow.com/a/1838411/5691525 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software > > > > > > Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > -- > > > > > _ > > > > > > > > > > С уважением, Юлия Волкова > > > > > Тел. : +7 (911) 116-71-82 > > > > > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > -- > > _ > > > > С уважением, Юлия Волкова > > Тел. : +7 (911) 116-71-82 > > > > > > === > > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > === > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [VOTE] Allow using asserts in Airflow code

r". > >> > >> ** [-1] Forbidding using asserts is mainly due to ambiguities when to > >> use/when to not use asserts. If we agree to that, we will forbid using > >> asserts via pre-commits and remove all assertions in our code. > >> > >> J. > >

Re: [DISCUSS] Revenge of the asserts

t;Demo of common assertion errors produced with pytest: > > >http://doc.pytest.org/en/latest/example/reportingdemo.html#tbreportdemo > > > > > >J. > > > > > >-- > > > > > >Jarek Potiuk > > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > >M: +48 660 796 129 <+48660796129> > > >[image: Polidea] <https://www.polidea.com/> > > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [DISCUSS] Revenge of the asserts

tomatically. > > How about we try: https://github.com/pytest-dev/unittest2pytest > > J. > > On Mon, Dec 9, 2019 at 5:48 PM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> wrote: > >> I agree. Should we encourage people to use asserts when adding new tests? >>

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Hi all, It sometime since we last discussed using other CI than Travis. One of the main reasons behind considering Gitlab CI was its ability to work on self-hosted runner. However, over time of few long weeks Github Actions matured enough to allow using self-hosted runners! Github Actions are

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

tty cool! It helps that they're github native. Where does the > compute power come from? Is it provided by github? > > On Mon, Dec 9, 2019 at 1:57 PM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > wrote: > > > Hi all, > > > > It sometime since we

Grouping tests using pytest markers

Hi everyone, Since we run our tests using pytest we are able to use test markers [1]. Using them will give use some useful things: - additional information of test type (ex. when used for system test) - easy way to select test by types (ex. pytest -v -m "not system") - way to split our test suite

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

; >>>>> I am all for it! GitLab has been less-than helpful so far and > > >>>> recently it > > >>>>> seems that running PRs from forks will only be run in Enterrprise > > >>>> Edition, > > >>>>> which is less than welcome. I

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

> > > > I’d rather we do it all at once and not need to maintain two builds. > > > Most/all of our CI/CD is dockerized at this point so there shouldn’t > be a > > > huge difference. > > > > > > > > > On Fri, Dec 13, 2019 at 1:23 AM, Tomasz

Re: [DISCUSS] Revenge of the asserts

agster. > > > > > > > > Best regards, > > > > Kamil > > > > > > > > On Thu, Dec 12, 2019 at 7:58 PM Daniel Huang > > wrote: > > > > > > > > > Big yes to asserts in pytest! > > > > > > > > > > On Tue

Re: Airflow user survey

ny voices as > > possible :). > > > > I will certainly use all social media I have influence on! > > > > J. > > > > On Sat, Nov 23, 2019 at 1:08 PM Tomasz Urbaszek < > > tomasz.urbas...@polidea.com> wrote: > > > >> Hi all, > >

Re: [VOTE] Add Probot Integrations to Airflow Github Repo

; > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [DISCUSS] Packaging DAG/operator dependencies in wheels

I like the idea of a DagFetcher (https://github.com/apache/airflow/pull/3138). I think it's a good and simple starting point to fetch .py files from places like local file system, S3 or GCS (that's what Composer actually do under the hood). As the next step we can think about wheels, zip and other

Re: Call to organize more First time Airflow Contributor's Workshops

Hi all, Just for your information ASF has established idea of Apache Local Community (ALC) : "ALC comprises local groups of Apache (Open Source) enthusiasts, called an 'ALC Chapter'. There will be a single ALC chapter per town/city. ALC is similar to Google Developer Group, Facebook Developer Cir

Re: [AirFlow]: Pandas DataFrame Between Tasks

> > > XD > > > > On Tue, Dec 24, 2019 at 15:00 David Muñoz > wrote: > > > > > Hi, > > > > > > Excuse me, I am new to this and maybe this topic has already been > treated. > > > > > > I would like to know if there

Re: Grouping tests using pytest markers

> >> > I am all-in for markers. > >> > > >> > I think we should start with small set of useful markers, which should > >> have > >> > a useful purpose from the beginning and implement them first - to > learn > >> how > >> >

Re: Grouping tests using pytest markers

ame marker > (but a module and/or class name could serve this purpose too). > > -- Darren > > PS, I stumbled in this after creating > https://github.com/apache/airflow/pull/6876 to mark a slow test > > On 2019/12/27 11:30:09, Tomasz Urbaszek wrote: > > +1 for integrations

Re: [NON-TECHNICAL] [DISCUSS] Being even more welcoming community ?

who has > > difficulties speaking english or you can have someone who will be your > > local mentor for that). > > > > I do not have concrete proposals yet, or I do not ask you to have them > > immediately. I don't even know yet if we should do something or not. But

Re: Test style convention after migration to pytest

Thanks Kamil for raising this issue! I think we can split this migration in many simple steps but the moment of "chaos" how we do things is unavoidable. But AIP-21 (contrib -> core) created some confusion too. But I think we are doing fine with it. When should we migrate? I dunno, I would like to

Re: [VOTE] Add Probot Integrations to Airflow Github Repo

t's super easy when you > type "Depends on #" - > you can start typing JIRA_ID or part of description and it will auto-show > the list of matching PRs. We do not have multiple repos > so the issue https://github.com/z0al/dep/issues/18 is not really relevant > in

Re: [DISCUSS] Airflow Website Blog Guidelines

+1 for publishing whole blog posts on the Airflow website. T. On Fri, Jan 3, 2020 at 3:37 PM Kaxil Naik wrote: > Hey all, > > I wanted to raise a point regarding our Blog on the Airflow website. > > There is an open PR: https://github.com/apache/airflow-site/pull/231 from > Chandu Kavar to add

Re: Killing Webserver/Scheduler gracefully

>From some time I think about adding "stop" commands like "airflow scheduler stop", "airflow celery worker stop". What do you think? I have already done this in native executor POC and it's helpful. T. On Sat, Jan 4, 2020 at 12:22 PM Kaxil Naik wrote: > Systemd integrations have worked nicely f

Re: Test style convention after migration to pytest

; > > 5) Do we want to use class-less tests? > > > > > > No. > > > > 6) Do we want to use pytest function instead of current? > > > > > > Yes > > > > 7) Do we want to use monkeypatch fixture? > > > > > > I also p

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

, Dec 17, 2019 at 3:28 PM Philippe Gagnon wrote: > We have been using Actions on the prestosql project for a little while as a > Travis replacement and we like it a lot. > > On Tue, Dec 17, 2019 at 9:08 AM Jarek Potiuk > wrote: > > > :( > > > > On Tue

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

t of Travis. > > On Fri, Jan 10, 2020 at 2:54 PM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > wrote: > > > Hi all, > > > > Just a small update: > > - after Jarek's changes from https://github.com/apache/airflow/pull/6516 > > kubernetes &

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Sponsors" project and start getting some money through that > https://github.com/sponsors)? I think having several sources of funding > for > our compute resources might be a good idea. WDYT? > > J > > On Sat, Jan 11, 2020 at 11:41 AM Kaxil Naik wrote: > > > Yeah, th

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

; J. > > > On Sat, Jan 11, 2020 at 12:41 PM Tomasz Urbaszek > wrote: > > > I have checked and Github Actions have unlimited minutes for open > > repositories. > > > > T > > > > On Sat, 11 Jan 2020 at 12:27, Jarek Potiuk > > wrote: > &g

Re: Test style convention after migration to pytest

:42 PM Jarek Potiuk > > > wrote: > > > > > > > Good idea! > > > > > > > > On Mon, Jan 6, 2020 at 3:12 PM Kamil Breguła < > > kamil.breg...@polidea.com> > > > > wrote: > > > > > > > >> Hello, >

Re: Often failing tests in CI (and a way to fix them quickly and future-proof)

-mocked/integration tests > too? > >>>>>>> > >>>>>> > >>>>>> I would not remove them just yet. Let's see how it works when I > >>>>>> separate it out. I have a feeling that we have very little number > of those > >>>>>> integration tests overall so maybe it will be stable and fast > enough when > >>>>>> we only run those in a separate job. I think it's good to have > different > >>>>>> levels of tests (unit/integration/system) as they find different > types of > >>>>>> problems. As long as we can have integration/system tests clearly > >>>>>> separated, stable and easy to disable/enable - I am all for having > >>>>>> different types of tests. There is this old and well established > concept of > >>>>>> Test Pyramid https://martinfowler.com/bliki/TestPyramid.html which > >>>>>> applies very accurately to our case. By adding markers/categorising > the > >>>>>> tests and seeing how many of those tests we have, how stable they > are, how > >>>>>> long they are and (eventtually) how much it costs us - we can make > better > >>>>>> decisions. > >>>>>> > >>>>>> J. > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> Jarek Potiuk > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>> > >>>>> M: +48 660 796 129 <+48660796129> > >>>>> [image: Polidea] <https://www.polidea.com/> > >>>>> > >>>>> > >>>> > >>>> -- > >>>> > >>>> Jarek Potiuk > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>> > >>>> M: +48 660 796 129 <+48660796129> > >>>> [image: Polidea] <https://www.polidea.com/> > >>>> > >>>> > >>> > >>> -- > >>> > >>> Jarek Potiuk > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > >>> M: +48 660 796 129 <+48660796129> > >>> [image: Polidea] <https://www.polidea.com/> > >>> > >>> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > >> > >> > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [NON-TECHNICAL] [DISCUSS] Being even more welcoming community ?

Hi all, There's an interesting discussion on the devcomm list regarding bidirectional integration between Slack and mailing list: "We have set up Mahout's slack space to forward directly to d...@mahout.apache.org. We will now be able to plan publicly on slack. This a bi-directional connection, all

Re: [DISCUSS] Forwarding Slack Threads/Communication to mailing list

Thanks Kaxil for starting new this thread! In my opinion, we should use dev@ for this integration. I am afraid that using slack@ with selective forwarding in a bidirectional way will be hard to achieve. In my opinion, the most important aspect of this integration is to bring devlist discussion

Re: [DISCUSS] Forwarding Slack Threads/Communication to mailing list

> > UI etc. > > > > Forwarding those to "users" might make more sense I feel but I am very open > > to suggestions and discussions. > > > > Regards, > > Kaxil > > > > On Mon, Jan 27, 2020, 15:10 Tomasz Urbaszek wrote: > > > >

Re: [DISCUSS] Forwarding Slack Threads/Communication to mailing list

l > >for. > > > >J. > > > > > >On Mon, Jan 27, 2020 at 10:50 AM Kaxil Naik > >wrote: > > > >> One 1 concern I have with forwarding everything to Dev would the > >"how-to" > >> question. > >> > >> For ex

Re: [DISCUSS] Forwarding Slack Threads/Communication to mailing list

n Mon, Jan 27, 2020 at 8:33 PM Kaxil Naik wrote: > > > >> I think we should first discuss on our main objective on sending slack > >> messages to a list ( it can be a separate list) i.e. answer the very > basic > >> question : why do we want to forward slack messa

Re: [DISCUSS] Forwarding Slack Threads/Communication to mailing list

laptop :) > > -ash > > > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com Unique Tech Check out our projects! <https://www.polidea.com/our-work>

Re: [DISCUSS] Airflow functional DAGs

/corrent/tree/master/dags > > > > This approach is highly inspired from some alternatives that have lately > > appeared on ETL pipelines world (ex: Dagster) that support more functional > > ways of defining DAGs. > > > > I believe this approach could benefit the Airflow community to grow. What > > do you think? Any thoughts on functional DAGs? Would love to hear what are > > pros and cons of this expansion. To me, it seems highly linked to the > > Lineage discussion happening at the same time, as both discussions are > > around improving data pipelines. > > > > Thanks! > > > > Gerard Casas Saez > > Twitter | Cortex | @casassaez > > -- Tomasz Urbaszek Polidea | Software Engineer M: +48 505 628 493 E: tomasz.urbas...@polidea.com Unique Tech Check out our projects!

Re: [DISCUSS] Airflow functional DAGs

re > that examples are there - but maybe we could even somewhat automate it and > generate some part of the "functional variants" for the examples we have? > > WDYT Dan, others ? > > J. -- Tomasz Urbaszek Polidea | Software Engineer M: +48 505 628 493 E: tomasz.urbas...@polidea.com Unique Tech Check out our projects!

Re: [DISCUSS] Airflow functional DAGs

> > > > Gerard Casas Saez > > Twitter | Cortex | @casassaez > > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk , > > wrote: > > > +1 short design doc would be cool. > > > > > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek < > > >

Promo materials for Airflow Meetups

Hi all, I've created a simple tool that generates pictures that can be used for promoting Airflow Meetups: https://github.com/nuclearpinguin/airflow-meetups This include event background (for facebook , meetups, etc.) and speakers cards. It's not a big thing if you have designers or proper softwa

Re: [PROPOSAL] Approach for releasing the backported "providers" packages

might work best (.MM.DD). And to make it > simple we should release one "big" providers package with all providers in. > We can have roughly monthly cadence for it. > > But I am also open to any suggestions here. > > Please let me know what you think. > > J. > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> -- Tomasz Urbaszek Polidea | Software Engineer M: +48 505 628 493 E: tomasz.urbas...@polidea.com Unique Tech Check out our projects!

Re: Backported "providers" packages for Airflow 1.10.*

rom google > > cloud > > > > > platform operators - the example dags are used to both - provide > > examples > > > > > in the docs but also can be run (with appropriate environment) to > > run the > > > > > example dag automatically with a real external system (GCP in this > > > > > > > > case). I > > > > > proposed this approach a long time ago in AIP-4 > > > > > < > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-4+Support+for+System+Tests+for+external+systems > > > > > > > > > > > and > > > > > while it has not been "universally" accepted yet, we followed it > > with GCP > > > > > operator implementations (and we have all GCP operators automatically > > > > > testable with system tests), With this PR I made the system test > > approach > > > > > nicely integrated with Pytest markers, Breeze and our test > > environment - > > > > > > > > so > > > > > it is now very easy to run system tests semi-automatically (and in > > the > > > > > future we can fully automate it when we switch to GitHub actions). > > > > > > > > > > We are planning to run all the system tests for all GCP operators, > > but > > > > when > > > > > it's there it's also rather easy to add tests for other groups of > > > > > > > > operators > > > > > so I am planning to have a community-driven effort to add more of > > those > > > > > system-tests (and make sure that backported packages can be safely > > used > > > > > > > > in > > > > > 1.10.* environment). > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > -- > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > -- Tomasz Urbaszek Polidea | Software Engineer M: +48 505 628 493 E: tomasz.urbas...@polidea.com Unique Tech Check out our projects!

Re: [PROPOSAL] Approach for releasing the backported "providers" packages

dev list and thus aware of > > the conditions placed on unreleased materials. > On Feb 10 2020, at 9:49 am, Tomasz Urbaszek > wrote: > > As per the frequency of releases maybe we can consider "nightly > > builds" for providers? In this way any contributed h

Re: [PROPOSAL] Approach for releasing the backported "providers" packages

upposed to be used by users not developers. > > J. > > On Mon, Feb 10, 2020 at 11:16 AM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> wrote: > > > I think as long as we follow: > > > The only people who are supposed to know about such developer reso

Re: How delete mailinglist subscription

; > > > > You can try to send a mail to dev-unsubscr...@airflow.apache.org > > > > > Hope you can still keep in touch with the community in other ways > :) > > > > > > > > > > XD > > > > > On Tue, Feb 11, 2020 at 07:55 Claudio >

Re: [DISCUSS] AIP-4 System tests

+1 for introducing system tests. Lack of them is a big pain. I would like also to suggest to mark some actual tests (those running DAGs, etc) as system tests. Then we can simplify our units and probably speed up CI builds (not to mention the reduction of side effects). The approach used for GCP sy

Re: Airflow Celery Executor - Some Task instances turning to failed without running and exporting log

Can you please tell me more about your environment? Especially how do you sync your DAGs / logs from celery workers? I know one setup where I've seen I/O error when wiritting to a log... T. On Wed, Feb 19, 2020 at 10:56 AM Mehmet Ersoy wrote: > > Hello Friends, > > I'm new to Airflow and I'm us

Re: Airflow Celery Executor - Some Task instances turning to failed without running and exporting log

of task instances that > > # a worker will take, so size up your workers based on the resources on > > # your worker box and the nature of your tasks > > *worker_concurrency = 32* > > > > > > # The maximum number of active DAG runs per DAG > > *max_active_runs_per

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

tch db operations in jobs > https://github.com/apache/airflow/pull/7370 > > This is not the last improvement of performance. We still keep working > and other changes will appear in the future. > > Many thanks to friends from Databand [https://databand.ai/] for support. > > Be

Re: Proposal: SIG-Kubernetes

initiatives like the airflow helm > chart, airflow custom controller, official docker image, and writing > documentation for best practices on Kubernetes. This could also be a great > forum for users to discuss their production experiences and use-cases. > > Would love any thoughts

Re: Proposal: SIG-Kubernetes

gt; > > > > > > > > > > >> On Wed, Feb 26, 2020 at 10:58 AM Xinbin Huang < > > > bin.huan...@gmail.com> > > > > > wrote: > > > > > >> > > > > > >> +1 would love to join. > > > > &g

Re: [VOTE] Backport package release process

> > > > https://github.com/apache/airflow/pull/7506/files#diff-337e7c8a1a7c5e1e9be3c5d8063d7d4cR81 > > > >> > > > >> The current list of cross-dependencies (extras) is available here. > > > >> > > > > https://github.com/apache/airflow/pull/7506/files#diff-ea1f65a8eaac0cec7ae151

Re: Stateless Webserver with DAG Serialization

Thanks Kaxil, Ash, Anita and Zhou! This is a big step forward :) T. On Sat, Mar 14, 2020 at 12:03 PM Sumit Maheshwari wrote: > Great stuff!! Thanks to everyone who contributed 👏 > > On Sat, Mar 14, 2020 at 2:26 PM Deng Xiaodong wrote: > > > It is really great news and would be a big plus. > >

Re: [DISCUSS] AIP-33 Creds Backend

7;t > >> have to > >> > > worry about loading creds. > >> > > And if you don't have access to the airflow CLI (because you are on a > >> > cloud > >> > > platform perhaps) then this can provide an easy way to load and manage

Re: [VOTE] AIP-33 Secrets Backend

+1 binding On Mon, Mar 16, 2020 at 1:16 PM Driesprong, Fokko wrote: > +1 > > Op ma 16 mrt. 2020 om 12:53 schreef Bas Harenslak > : > > > +1 > > > > > On 16 Mar 2020, at 12:27, Ash Berlin-Taylor wrote: > > > > > > +1 binding > > > > > > On Mar 16 2020, at 11:20 am, Kaxil Naik wrote: > > > > > >

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

+1 for Github issues. Github allows creating issue template (feature, bug, custom) so this should help. And I have a feeling that GH issues are indexed better than JIRA tickets. JIRA gives the possibility to interlink between ASF projects but I don't think is something important for us. I've also

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

"why are we > bothering with this" today. > It seems like we have fairly broad agreement this time. AIP worth, or just > a VOTE and switch over? > -a > On Mar 16 2020, at 2:02 pm, Tomasz Urbaszek wrote: > > +1 for Github issues. Github allows creating issue template (

Re: [VOTE] Switch from using Jira to Github Issues

+1 binding T. On Mon, Mar 16, 2020 at 3:42 PM Kaxil Naik wrote: > > +1 binding > > BTW, we can enable Github Issues by just adding the following block in > *airflow/.asf.yaml* > > github: > features: > # Enable issues management > issues: true > # Enable projects for project manage

Re: [VOTE] AIP-32: Airflow REST API

. > > This vote will last for 72 hours until 2020-03-22T10:30Z, and until at > least 3 votes have been cast. > > > https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200322T1030&p1=262 > > This is my +1 vote. > > Thanks, > Kamil > -- Tomasz Urbasze

Re: Helm Chart for Airflow

An official helm chart is something our community needs! Using your chart as the official makes a lot of sens to me because as you mentioned - it's battle tested. One question: what Airflow image do you use? Also, would you mind sharing a link to the chart? Tomek On Tue, Mar 24, 2020 at 2:07 PM

Re: SIG-Autoscaling meeting

+1 Tomek On Wed, Mar 25, 2020 at 5:14 PM Kaxil Naik wrote: > > Sounds good, thanks Daniel. > > > > On Wed, Mar 25, 2020 at 3:57 PM Daniel Imberman > wrote: > > > Hello all! > > > > Since the KEDA autoscaler has been released, for the moment there’s not > > too much going on with SIG-Autoscalin

Re: [DISCUSS] AIP-31: Airflow functional DAG API

Hi all, Should we proceed with the vote? T. On Fri, Mar 20, 2020 at 1:37 AM Gerard Casas Saez wrote: > Hi everyone, > > Sorry for leaving this unfinished. I was commenting an extension that > Tomasz suggested in the AIP doc, to add fields in the operator that serve > as templated args. This sh

[VOTE] AIP-31: Airflow functional DAG definition

Hello everyone! This email calls for a vote on the design proposed in AIP-31: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-31:+Airflow+functional+DAG+definition Discussion threads: https://lists.apache.org/thread.html/ra3bf1b20584ef1e4e0fb9b00f5b246d7f18b7c796edde8c807a8d86a%40%3Cdev

Re: How about switching from Celery to RQ?

I think we once tried to abandon Celery for other stuff (kombu, knative)... If there's no reasonable advantage of RQ I see no reason in switching to it. However, rewriting the "Queue worker" can help a lot :) Also a few days ago one of Celery maintainers expressed interest in contributing to Airfl

  1   2   3   4   >