Hi, I want to flip this discussion to NOT talk about anything we are debating in VOTE, but talk about the "abnormal process" being done with my migration logic PRs, and propose to fix it.
Here is the timeline: 1. We merged the PR to "remove" the incorrect SQL config to master/4.0, "with the concern of upgradability from Spark 3.5 to Spark 4". 2. We have migration logic PRs for master/4.0/3.5. 3. We merged the PR for 3.5, but stopped merging the PRs for master/4.0, without any discussion/decision/justification of such an action. 4. Hence we had to have a super noisy DISCUSS and VOTE to continue to merge the migration logic PRs to master/4.0. Arguably, unless we clarify that the PR which was merged to the older version line is not needed to newer versions, we always merge the PR to the newer versions. In addition, it is quite natural to merge master first and go down the version. These are not respected during the process. I argue 1 must be done "later" than 3. At this point we did 1, this was a breaking change, but this was done without VOTE simply relying on the belief that we will address backward compatibility sooner. Arguably, we didn't address it yet - both proposals we have are neither got the community consensus nor not having VETO. That said, 1 is definitely incorrect to be merged, because we never solved the problem of backward compatibility. 1 can only be merged after we reach the consensus of the solution of backward compatibility, or make a decision via VOTE that the community is OK to introduce a breaking change. Again, none of them is done as of now. Here I propose to fix this by following steps: 1) reverting the removal of config in master/4.0 (reverting 1) 2) merging the migration logic PRs for master/4.0 (follow the normal process for 3) 3) removing the config again Worth noting that 3 is not strictly in scope, but this is "roughly" made consensus, so adding this explicitly to avoid concerns. For our convenience, we can probably merge 1 to 3 altogether to have a single PR. After fixing the process, people who are concerned about the existing codebase will initiate with DISCUSS and eventually reach the community consensus with VOTE. This will be used as a baseline of "what if we didn't have these proposals". We identified that both proposals are conflicting and it's really hard and/or noisy to push one forward. Arguably, both proposals are yet to get community consensus. IMO, this is simply a mandatory process to change the codebase to be neutral, because my proposal follows our normal procedure of dealing with multiple versions of PR, merging master -> 4.0 -> 3.5. I haven't heard any strong reason to not follow the normal procedure, so I'd rather say this is what we should have done, rather than what happened now. Looking forward to hearing the voice. Thanks, Jungtaek Lim (HeartSaVioR)
