I want to express my sincere gratitude to WealthSimple for hosting us and extend my appreciation to all the attendees for their active participation. This event has been a truly enriching experience.
One aspect that stood out to me during our time together was how we came together as a community to address a common challenge. We engaged in brainstorming sessions, exploring various potential solutions to expedite the resolution of issues that have been plaguing our community. In my case, I had the privilege of participating in discussions focused on enhancing the performance of Airflow when dealing with task orders that exceed the usual limits. I am optimistic that we can continue to foster these collaborative opportunities on a larger scale, propelling us forward in our endeavors. My heartfelt congratulations to everyone involved for their contributions and commitment. Thanks & Regards, Amogh Desai On Fri, Sep 22, 2023 at 11:25 PM Mehta, Shubham <shu...@amazon.com> wrote: > Thank you for taking the time to summarize the workshop and provide > details around all the discussions we had. I appreciate you and Julien > taking ownership and driving it. I also really liked the transparency > you've maintained by bringing this forward to the rest of the community. > > > > One of my key takeaways from the workshop is the shared challenges that > many of us face, whether we're operating self-hosted Airflow instances or > offering managed Airflow services. The workshop was a great venue for > coming together to hear from each other about what problems we face and > figure out how we can bring these forward in the future. It's not about > finding immediate solutions; it's about deepening our common understanding, > which is a crucial first step towards improving the product. While async > channels like mailing lists, Slack, and GitHub are super useful, they often > can't replace the depth of real-time, focused discussions. IMO, these > interactions, whether face-to-face or virtual, are highly efficient and can > help us accelerate innovation and growth in Airflow. > > > > I completely agree that no matter where these discussions occur, it's > crucial to funnel the insights back into the broader Airflow community via > the devlist. This ensures that we maintain a democratic process and drive > consensus on the next steps. > > > > Kudos to the Wealthsimple team for their excellent hospitality and event > organization. The attention to detail, from "kettle-bell" power banks to > accommodating the breakout session needs, was outstanding. Their efforts > certainly contributed to the overall success of the workshop. > > > > To me, this workshop was a really successful "POC," and I'm hoping we > continue to have similar discussions on a larger scale in the future. > > > > Shubham > > > > > > *From: *Jarek Potiuk <ja...@potiuk.com> > *Date: *Friday, September 22, 2023 at 9:18 AM > *To: *"dev@airflow.apache.org" <dev@airflow.apache.org>, Julien Le Dem < > jul...@astronomer.io>, Mandy Gu <m...@wealthsimple.com>, " > hch...@wealthsimple.com" <hch...@wealthsimple.com> > *Cc: *Amogh Desai <amoghdesai....@gmail.com>, "astep...@wealthsimple.org" > <astep...@wealthsimple.org>, Bartosz Jankiewicz <bjankiew...@google.com>, > Daniel Standish <daniel.stand...@astronomer.io>, Diederik van Liere < > dvanli...@gmail.com>, Elad Kalif <elad...@apache.org>, Ephraim Anierobi < > ephr...@astronomer.io>, "i...@sayapin.biz" <i...@sayapin.biz>, John > Jackson <jacn...@amazon.com>, Jed Cunningham <j...@astronomer.io>, " > kna...@google.com" <kna...@google.com>, Nikolas Oliveira < > oliveira...@gmail.com>, Pierre Jeambrun <pierrejb...@gmail.com>, Rafał > Biegacz <rafalbieg...@google.com>, "Bishundeo, Rajeshwar" < > rbish...@amazon.com>, Rich Bowen <rbo...@rcbowen.com>, "Mehta, Shubham" < > shu...@amazon.com>, Sam Talasila <stalas...@wealthsimple.com>, Sung Yun < > syu...@bloomberg.net>, "Balog, Tarus" <tarus...@amazon.com>, Deng > Xiaodong <xd.den...@gmail.com> > *Subject: *RE: [EXTERNAL] [COURRIEL EXTERNE] Summary of the In-person > informal workshops we had in Toronto > > > > *CAUTION*: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > *AVERTISSEMENT*: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > Resending to devlist again with a link to the picture instead of the image > (our mailing list rejected the message as too big). > > > > Here is the picture https://pasteboard.co/c23tOnopp8l9.png > > > > BTW. I am going for two weeks on vacation - still today I might be > reachable, but as of Monday I somewhere in an RV in the middle of wild > Eeast Canada, without the intention of checking my github or slack or email > messages :) > > > > > > On Fri, Sep 22, 2023 at 11:58 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > Hey everyone, > > > > I would like to share a short information about a little informal workshop > we held at Toronto with a number of folks who were here due to the Airflow > Summit. > > > > First of all - explanation why I am writing about it only now and that it > has not been public/ on the devlist so far and the workshops was > invite-only. > > > > The idea I had about it was rather an "ad-hoc" one and I had hard time to > organise it (together with Julien) over the last few weeks. Adding devlist > discussion to the list would be complicating things a lot and I was afraid > people might get FOMO syndrome (needlessly). > > > > Also - I am not really used to organising things like this - this was WAAY > outside of my comfort zone and I had a really hard time mentally to keep > myself "in a grip" so to speak, so adding devlist discussion would be a > huge burden for me personally. > > > > I hope one of the results of this meeting will be that we will organize > such meetings more often/maybe via conference calls and in more 'open' > fashion. I look forward to that - I hope some people got inspired to lead > that part. > > > > The idea of meeting came from my discussion with Julien Le Dem - > OpenLineage creator based on some ideas in the talk of his from the summit > https://airflowsummit.org/sessions/2023/nurturing-open-source-community/. > We had been discussing the idea over the last few weeks and iterated on it > together. > > > > I invited PMC members, some of the representatives of the stakeholders, > a few people from Wealthsimple (my customer) who kindly agreed to host the > meeting at their offices and few other people who I thought might be good > to have. The idea of mine heavily evolved during the workshop and we did > quite a different thing that I planned originally, but well, that was also > part of the plan - to propose something and let it form itself by having > smart and passionate and vested people around who would collectively guide > where the discussions will lead us to. > > > > We ended up with ~20 people. And we finally split into three groups. I > will not go into details - but we have not made any decisions of course, > and other than discussion itself you have not missed being part of what's > going to happen next. > > > > All of the things we discussed will have to find its way to the devlist > and I hope they will - but I will leave it up to the people who are > passionate about it and would like to follow up on those subjects > individually. > > > > Broadly speaking those were the three things that appeared to be the most > important topics: > > > > 1) We feel the need of improving product management of Airflow - in the > sense that we need to have Product Management-focused people to help our > community to better organise planning, releases and "roadmap"/direction. > > > > 2) We feel the need to build robust performance and benchmarking solutions > for Airflow. It seems that all big vendors - Google, Astronomer, Amazon > feel the same need and are committed to contribute a lot to make it a > reality. > > > > 3) We feel the need for airflow to be more and more relevant in the future > of data, to expand the current lineage information and add more semantic > schema to the kinds of data that's being processed by and we feel that > Airflow is a perfect platform to implement some open standards there. > > > > Those are broad "big" subjects we discussed and I am super excited and > thrilled on what this might bring in the future. > > > > Once again - great thanks to Wealthsimple - especially Mandy Gu who was an > absolutely fantastic host and Mark Cheng who helped us with making the > workshop run smoothly. Mark came to the office early especiallybfor is and > make us all feel welcome and taken care of - big thanks for that :). And > thanks to Diederik van Liere for being the "sponsor" of that work and a > very active participant in the discussions. > > > > I am really thankful for all the help and openness and responsiveness of > Wealthsimple. I think Wealthsimple is a great community member and I look > forward to the future collaboration :). > > > > Thanks to Julien who helped me with the workahop (and helped me to get > through some difficult times where I had serious doubts if this whole > thing makes sense at all). > > > > And special thanks to Rich Bowen - the board director of the Apache > Software Foundation and VP conferencing there - he provided super valuable > input from the Foundation history and present that I think helped to > streamline the Product discussion in the right direction. > > > > See the picture on how we looked after the workshops. Looks like we all > survived :) . > > > > I barely, but I did :). > > > > J. > >
