OK. Let me clarify then, 1) First of all - yes - i completely missed cadwyn and API versioning we agreed to in the AIP. We had so many of those AIPs that I simply forgot this tiny-little detail. Totally my fault and sorry for calling that one out. Sorry Ash for that if you felt you've been called out needlessly. 2) Secondly - I am amazed how well we are delivering Airflow 3 together. This an amazing feat to deliver it with such a speed, and coordination and the way how we work together - exactly via combination of those different, well balanced communication methods:
* having a grand plan and timeline that Vikram and Kaxil are keeping updating and we synchronize and plan our deliveries on a weekly basis according to Airflow 3 plan https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Development+Milestones * earlier agreeing and discussing HUGE impact decisions in AIPs * making decisions on things impacting the whole project and all people in DEVLIST * making quick decisions and discussions in PRs where particular problems are solved and need a discussion from involved people * quick discussions on slack where things need to be decided and reacted on-the-spot This all works wonderfully. I would even say miraculously in the distributed environment of ours, where we a) work remotely, b) in disconnected teams without hierarchy c) each of us focuses on their part while following what others are doing and trying to fit-in This is - and I am not afraid to say it - an ultimate proof where "community over code" is showing its real power, where various stakeholders are involved and we all work in one direction, despite each of us working separately. Chapeau bax to everyone involved - everyone puts their own efforts as a separate brick and we build the building together, and - surprisingly the new building we build looks already fantastic and very coherent. In this context, a little more explanation on the reasoning here. I just wanted to make sure that the "packaging' and "repository layout" comes into the picture. The thread here was sparked with a slack discussion where TP commented essentially "we should change our layout of folder, I do not like the lack of consistency" - in a random slack conversation, that many of us could have missed. And I was called out essentially for saying "We've been discussing it for weeks on the devlist, please bring it to devlist and let's discuss it there". When we talk about packaging and connected layout of our repository has been "crossing" all that - aiming to improve "contributor's journey" and supporting the new architecture by splitting airflow into actually independent pieces, I personally very carefully planned timeline and delivery of multi-step packaging changes to fit-in and to put my own brick where it fits (and exactly when it was planned according to the milestones and a grand plan for Airflow 3 timeline we agreed to. Initially we were agreeing high level changes with - slack discussions and general ideas, some discussions with Ash and few other people who were involved in packaging, Heck - it was not even only me - Ash was initially separating providers as a whole, then added `task_sdk", eventually last week we agreed to the new "airflow-ctl" separation as something that has been moved out of Airflow 3.0 - so Jed and Bugra did it "quickly" to fit in the beta timeline, so yeah that was done "quick-and-dirty" way now to enable Airflow 3 timeline. In the meantime I spent a lot of time preparing and testing and communicating all the plans to make the biggest and potentially most disruptive move - moving all airflow code. It's been many weeks in preparations, discussions in slack, earlier provider's separation was treated as a "playground" where we solved a number of teething issues, learned how uv works for us, spend a lot of time on teaching people how to work with it, documenting and trying to communicate on all available channels - slack, devlist, devcall demos etc. on what those changes are going to bring. We discussed (on devlist this time) the new structure, proposed and agreed to on various names, and this HUGE move (I think the biggest PR ever with ~ 3000 files moved) was a result of all that preparatory work and countless hours of testing and making things work after the move. It's not as easy as "move stuff to a new directory". Ash knows it best when he moved the "providers" to a new "providers" folder that this is a LOT of effort - mainly because our test harness that we use daily is heavily affected by such a move. At all the stages when we knew (everyone involved in such moves) - the how and what has been announced and discussed in the devlist (and slack) weeks in advance to prepare everyone to what's coming and to get feedback - and a lot of feedback from multiple people involved we had and applied. All was good. Until suddenly, after all those discussions, agreement, tests, communication, right at the start of the busiest period of testing for RC candidate where packaging was supposed to be done exactly **now** according to the plans that we all agreed and followed, when I hear "I do not like the new layout, please change" my natural reaction is "bring it to devlist". We need to discuss it, realise the consequences, understand why want to make such a - again - potentially hugely disruptive and costly change, and first of all we need to understand if this is the right time to again disrupt the work of pretty much everyone working on Airflow 3 by changing the layout of our repo (which I'd estimate as again at least few weeks of preparation, communication of work) is the right thing to do now. I just think that "bringing this to a devlist" in this context rather than "let's discuss it in this random slack thread" was not too outrageous of a proposal? I am not sure if I did something wrong by proposing it to TP? Or maybe we have a different understanding of what should and what should not be discussed on the devlist? This question of mine here was mostly - do I understand the way we communicate here wrongly? Should we really agree and discuss such hugely impactful and heavy changes on slack? Is that wrong to ask someone to bring such a discussion to devlist? Those are my questions really. J. On Sat, Mar 22, 2025 at 8:37 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > I completely agree regarding transparency, having options expressed in > writing, and tracking decisions. > > I also support the ASF philosophy that voting should only happen on the dev > list. > > However, treating the dev list as the only acceptable mechanism for > discussion feels overly burdensome, in my humble opinion. > In practice, I don't believe that email is not structured enough for all > types of interactions. > Here are two simple examples: > 1. *Github issues* > We have a template for reporting issues that enforces structure - capturing > key elements such as Airflow version, affected area, reproduction steps, > and so on. This level of enforced structure is hard to achieve in email. > > We also frequently respond to Github issues with decisions - explicitly or > implicitly - which, in an abstract sense, are still decisions. > > 2. *AIPs (Airflow Improvement Proposals)* > We have a template for AIPs. While many improvement ideas start as a simple > thought in a Github issue or email, we often ask contributors to write them > up as AIPs. The structure helps guide discussion, enables threaded > comments, and supports versioning. Personally, I find the AIP process quite > helpful. > We do reference the AIP in the voting thread on the dev list, but the bulk > of the discussion happens in Confluence, within the context of the written > document. > > I may be misunderstanding the intent of this discussion, but one of the > referenced examples - specifically "the versioning rules for the API" - was > already covered in the original AIP > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK > >, > in the following paragraph: > *"This API should be strongly versioned using CalVer, and we propose an > approach such as https://docs.cadwyn.dev/ <https://docs.cadwyn.dev/> to > allow us to easily support many versions in parallel (tl;dr of this > project: calver + request "migrations" lets you easily handle multiple > versions API clients without needing parallel support, even what might > otherwise be a breaking change, as long as change from one version to the > next does not throw away information.) Each change, no matter how small, to > this API will result in a new CalVer."* > > I have always viewed ASF processes as encouraging written, open > communication to foster transparency in decision making, but not as > precluding the use of openly accessible applications such as Github and > Wiki. > > So far, I believe the Github and AIP processes have worked well alongside > the dev list, and I am unsure what the motivation is for requiring a change > here. > > Vikram > > > On Sat, Mar 22, 2025 at 10:20 AM Michał Modras > <michalmod...@google.com.invalid> wrote: > > > +1, I fully agree with the mechanism - it is open, transparent, gives > > everyone an opportunity to participate, and keeps track of how the > > decisions are made. > > > > On Sat, Mar 22, 2025 at 5:34 PM Shahar Epstein <sha...@apache.org> > wrote: > > > > > Overall I agree that decisions should be made in the devlist. > > > One improvement that I'd like to suggest, though, is to have a > dedicated > > > page on cwiki that summarizes all recent and upcoming decisions (votes > + > > > lazy consensus) in a table, including links to their respective devlist > > > threads. We could have a page for important discussions as well. > > > By doing it, we'll gain: > > > 1. Better transparency - instead of searching within many emails for > > > past decisions in client/Apache mail archives, we'll just look them up > at > > > the table. > > > 2. Better noticeability - people would be more easily aware of > > > upcoming votes. Personally I often miss emails as I redirect them to a > > > subdirectory, and I get disappointed if I miss important votes. > > > > > > We already attach links to threads for AIPs in their wiki page, but I > > think > > > that having a more generalized page would benefit us all. > > > That's my two ₪ anyway :) > > > > > > > > > Shahar > > > > > > On Sat, Mar 22, 2025 at 12:09 AM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > I also would like to start another thread regarding decision making > in > > > our > > > > projects. We had some discussions about decision making in our > project. > > > > > > > > As an ASF project we are supposed to make important decisions on the > > > > devlist. Full stop. "If it did not happen at the devlist, it did not > > > > happen" is something that you can hear often in the ASF - even if the > > > only > > > > place you can actually see it written this way is > > > > https://www.apache.org/press/highlights.html#2017 > > > > > > > > The Apache Software Foundation has a very clear notion of important > > > > decision making: > > > > > > > > * official voting rules > https://www.apache.org/foundation/voting.html > > > > * community development guidelines > > > > https://community.apache.org/committers/decisionMaking.html > > > > > > > > Both of those refer to decision making at the devlist. > > > > > > > > We had a few - unnecessarily heated - discussions on how decisions > are > > > made > > > > in the project. I personally think that our decision making should > be > > > done > > > > at the devlist - where anyone can participate. Things like versioning > > > rules > > > > for API (hinted by Ash at the last dev call) or how our repo is > > > structured > > > > (a new proposal raised today by TP) should - IMHO - be discussed > here, > > at > > > > the devlist. > > > > > > > > Not in a slack thread, not in private discussion, not when two or > more > > > > people talk to each other in a call. It's fine to discuss things > > outside > > > - > > > > of course, but then any proposals for a change of things that we > > already > > > > discussed for weeks and either implicitly (by non-objection) or > > > explicitly > > > > (by not responding to [LAZY CONSENSUS] or [VOTE] thread in the > devlist) > > > are > > > > just discussions. If someone wants a change, starting a thread in > > devlist > > > > is the right way of proposing a change. Especially for things that > were > > > > discussed before - sometimes for weeks, and no concerns were raised. > > > > > > > > IMHO - it's very simple - want a change - start and lead a [DISCUSS]. > > > [LAZY > > > > CONSENSUS]. or [VOTE] (depending on the level of disagreement and > > number > > > of > > > > different opinions). Not very complex - it just requires to start and > > > lead > > > > a discussion mailing list thread. Super inclusive and follows the > > > archiving > > > > and all other requirements of the ASF (for example you do not have to > > > agree > > > > TOC of Slack to participate) > > > > > > > > I would love to hear if others disagree with it and think that this > > > process > > > > is overly complex or problematic. I think it's quite clear, > reasonable > > > and > > > > great to keep community decision making in check and follow all the > > rules > > > > and expectations of the ASF, but maybe there are some concerns with > the > > > > process and we would like to improve it. > > > > > > > > I would love to hear what others think about it. > > > > > > > > J. > > > > > > > > > >
