Thank you for taking the time to summarize the workshop and provide details around all the discussions we had. I appreciate you and Julien taking ownership and driving it. I also really liked the transparency you've maintained by bringing this forward to the rest of the community.
One of my key takeaways from the workshop is the shared challenges that many of us face, whether we're operating self-hosted Airflow instances or offering managed Airflow services. The workshop was a great venue for coming together to hear from each other about what problems we face and figure out how we can bring these forward in the future. It's not about finding immediate solutions; it's about deepening our common understanding, which is a crucial first step towards improving the product. While async channels like mailing lists, Slack, and GitHub are super useful, they often can't replace the depth of real-time, focused discussions. IMO, these interactions, whether face-to-face or virtual, are highly efficient and can help us accelerate innovation and growth in Airflow. I completely agree that no matter where these discussions occur, it's crucial to funnel the insights back into the broader Airflow community via the devlist. This ensures that we maintain a democratic process and drive consensus on the next steps. Kudos to the Wealthsimple team for their excellent hospitality and event organization. The attention to detail, from "kettle-bell" power banks to accommodating the breakout session needs, was outstanding. Their efforts certainly contributed to the overall success of the workshop. To me, this workshop was a really successful "POC," and I'm hoping we continue to have similar discussions on a larger scale in the future. Shubham From: Jarek Potiuk <ja...@potiuk.com> Date: Friday, September 22, 2023 at 9:18 AM To: "dev@airflow.apache.org" <dev@airflow.apache.org>, Julien Le Dem <jul...@astronomer.io>, Mandy Gu <m...@wealthsimple.com>, "hch...@wealthsimple.com" <hch...@wealthsimple.com> Cc: Amogh Desai <amoghdesai....@gmail.com>, "astep...@wealthsimple.org" <astep...@wealthsimple.org>, Bartosz Jankiewicz <bjankiew...@google.com>, Daniel Standish <daniel.stand...@astronomer.io>, Diederik van Liere <dvanli...@gmail.com>, Elad Kalif <elad...@apache.org>, Ephraim Anierobi <ephr...@astronomer.io>, "i...@sayapin.biz" <i...@sayapin.biz>, John Jackson <jacn...@amazon.com>, Jed Cunningham <j...@astronomer.io>, "kna...@google.com" <kna...@google.com>, Nikolas Oliveira <oliveira...@gmail.com>, Pierre Jeambrun <pierrejb...@gmail.com>, Rafał Biegacz <rafalbieg...@google.com>, "Bishundeo, Rajeshwar" <rbish...@amazon.com>, Rich Bowen <rbo...@rcbowen.com>, "Mehta, Shubham" <shu...@amazon.com>, Sam Talasila <stalas...@wealthsimple.com>, Sung Yun <syu...@bloomberg.net>, "Balog, Tarus" <tarus...@amazon.com>, Deng Xiaodong <xd.den...@gmail.com> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] Summary of the In-person informal workshops we had in Toronto CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Resending to devlist again with a link to the picture instead of the image (our mailing list rejected the message as too big). Here is the picture https://pasteboard.co/c23tOnopp8l9.png BTW. I am going for two weeks on vacation - still today I might be reachable, but as of Monday I somewhere in an RV in the middle of wild Eeast Canada, without the intention of checking my github or slack or email messages :) On Fri, Sep 22, 2023 at 11:58 AM Jarek Potiuk <ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote: Hey everyone, I would like to share a short information about a little informal workshop we held at Toronto with a number of folks who were here due to the Airflow Summit. First of all - explanation why I am writing about it only now and that it has not been public/ on the devlist so far and the workshops was invite-only. The idea I had about it was rather an "ad-hoc" one and I had hard time to organise it (together with Julien) over the last few weeks. Adding devlist discussion to the list would be complicating things a lot and I was afraid people might get FOMO syndrome (needlessly). Also - I am not really used to organising things like this - this was WAAY outside of my comfort zone and I had a really hard time mentally to keep myself "in a grip" so to speak, so adding devlist discussion would be a huge burden for me personally. I hope one of the results of this meeting will be that we will organize such meetings more often/maybe via conference calls and in more 'open' fashion. I look forward to that - I hope some people got inspired to lead that part. The idea of meeting came from my discussion with Julien Le Dem - OpenLineage creator based on some ideas in the talk of his from the summit https://airflowsummit.org/sessions/2023/nurturing-open-source-community/. We had been discussing the idea over the last few weeks and iterated on it together. I invited PMC members, some of the representatives of the stakeholders, a few people from Wealthsimple (my customer) who kindly agreed to host the meeting at their offices and few other people who I thought might be good to have. The idea of mine heavily evolved during the workshop and we did quite a different thing that I planned originally, but well, that was also part of the plan - to propose something and let it form itself by having smart and passionate and vested people around who would collectively guide where the discussions will lead us to. We ended up with ~20 people. And we finally split into three groups. I will not go into details - but we have not made any decisions of course, and other than discussion itself you have not missed being part of what's going to happen next. All of the things we discussed will have to find its way to the devlist and I hope they will - but I will leave it up to the people who are passionate about it and would like to follow up on those subjects individually. Broadly speaking those were the three things that appeared to be the most important topics: 1) We feel the need of improving product management of Airflow - in the sense that we need to have Product Management-focused people to help our community to better organise planning, releases and "roadmap"/direction. 2) We feel the need to build robust performance and benchmarking solutions for Airflow. It seems that all big vendors - Google, Astronomer, Amazon feel the same need and are committed to contribute a lot to make it a reality. 3) We feel the need for airflow to be more and more relevant in the future of data, to expand the current lineage information and add more semantic schema to the kinds of data that's being processed by and we feel that Airflow is a perfect platform to implement some open standards there. Those are broad "big" subjects we discussed and I am super excited and thrilled on what this might bring in the future. Once again - great thanks to Wealthsimple - especially Mandy Gu who was an absolutely fantastic host and Mark Cheng who helped us with making the workshop run smoothly. Mark came to the office early especiallybfor is and make us all feel welcome and taken care of - big thanks for that :). And thanks to Diederik van Liere for being the "sponsor" of that work and a very active participant in the discussions. I am really thankful for all the help and openness and responsiveness of Wealthsimple. I think Wealthsimple is a great community member and I look forward to the future collaboration :). Thanks to Julien who helped me with the workahop (and helped me to get through some difficult times where I had serious doubts if this whole thing makes sense at all). And special thanks to Rich Bowen - the board director of the Apache Software Foundation and VP conferencing there - he provided super valuable input from the Foundation history and present that I think helped to streamline the Product discussion in the right direction. See the picture on how we looked after the workshops. Looks like we all survived :) . I barely, but I did :). J.
