Regardless of this vote result, we (including me) had too many
mistakes/faults during handling this (I'd rather say "incident", or even
"disaster").
I really think we should have a retrospective, making a rough timeline of
the events with making people involved to be anonymous, and brainstorm how
we could improve the mindset / process / etc (even BYLAW) to avoid this
from happening again. Personally, I was seriously considering escalating
this based on the events that happened to me from this incident. But
instead of doing that, discussing in public and thinking about how we could
prevent bad things from happening again, sounds to me a lot beneficial to
everyone, because anyone can be "someone" in the situation. I mean, this is
the last time I am open to finding a better solution, and if this happens
again I am willing to immediately escalate the problem.
For sure, it's not perfect and there is zero way to completely anonymize
this (someone might be willing to waste their time to correlate the
anonymized event and the real event), but if we don't do that because we
might indirectly point out someone, I think we will never be able to bring
the problem the community and individual make to the public discussion, and
then, the only way to resolve this is to escalate which directly jumps to
the last resort. Writing a postmortem is like that - the content has to
call out people, but the mindset is just to brainstorm how we can prevent
this from happening again. If the postmortem is too heavy, maybe just call
it a retrospective.
I understand people being involved in this event (including me) might just
want to forget about it and move on, but arguably this event was a perfect
example of how things could get messed up and make some people
to lose faith of someone/community, and we should revisit what we did and
be sure not to do that again. Otherwise, this will definitely happen again.
On Tue, Mar 18, 2025 at 7:27 AM Hyukjin Kwon <gurwls...@apache.org> wrote:
> I would like to conclude this vote with my own -1, for the same reasons as
> Holden.
> Personally, I feel that this approach has been overly excessive in several
> ways - such as overriding a veto - and it would have been better to explore
> alternative solutions first.
> That said, it seems that this vote has passed by a simple majority, and
> the vote in the other thread remains valid.
> Since it has now been 48 hours, as mentioned earlier in the thread, I hope
> we can formally conclude this vote.
>
> On Tue, 18 Mar 2025 at 06:47, Yuanjian Li <xyliyuanj...@gmail.com> wrote:
>
>> +1
>>
>> I would like to add my perspective to this discussion. It's important
>> that we maintain a fair and objective technical evaluation process,
>> especially when a veto is cast. A veto should be backed by concrete
>> technical reasoning.
>>
>> From my reading of the discussion, the key concern is whether the
>> justification for the veto is based on an objective technical issue with
>> the proposed change. If the concern is not rooted in a clear technical
>> limitation or regression, then it should be open to further discussion and
>> reconsideration. It is crucial for us as a community to ensure that
>> decisions are made transparently and with technical merit as the primary
>> driver.
>>
>> I also want to emphasize that maintaining a welcoming and inclusive
>> community is vital to the success of Apache Spark. If contributors feel
>> isolated or that decisions are being swayed by affiliations rather than
>> technical arguments, that would set a concerning precedent. We should
>> strive to engage in discussions constructively and focus on what is best
>> for the project and its users.
>>
>> Jungtaek Lim <kabhwan.opensou...@gmail.com> 于2025年3月16日周日 23:11写道：
>>
>>> The problem you are mentioning is arguably resolvable if we leave a
>>> config name as it is, and add
>>> withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
>>> Let's not nitpick just from reverting the PR. We have to revert the PR
>>> "semantically".
>>>
>>> Btw, from what I understand of dealing with backport PR is, we mostly
>>> merge in master to begin with, and down the version line. When we handle my
>>> migration PR, we do not follow this practice without any discussion. I
>>> submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I
>>> asked for merging to 4.0, I was pushed back. I don't think this is an
>>> ordinary practice of dealing with multiple versions of PRs, especially
>>> since the author never agreed with the way of processing.
>>>
>>> If we were following the practice, we should already have migration
>>> logic for master/4.0/3.5, since we should have the same fix in master/4.0.
>>> We probably had a discussion about removing config from master/4.0 based on
>>> the discussion, and we probably agreed to remove the config since we still
>>> have a migration logic. W.r.t migration logic, based on the discussion we
>>> are having, we probably can't make an agreement to take it out, then
>>> arguably the migration logic is left as it is.
>>>
>>> This way I never needed to drive such a long and sensitive DISCUSSION
>>> and VOTE. But the PR for master and 4.0 weren't merged because of the
>>> individual's belief of the rollout plan, which we are seeing does not fit a
>>> majority of voices now.
>>>
>>> Shall we fix the broken process in which we made a huge mistake before
>>> moving on? We should have merged the same content in 3.5 to master/4.0 as
>>> well, and then have a PR to remove the config. This is totally swapped
>>> which does not make sense to me.
>>>
>>> On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <dongj...@apache.org>
>>> wrote:
>>>
>>>> I reviewed Wenchen's reverting PR. Although it's a proposal for
>>>> discussion, it is another breaking change against Apache Spark 3.5.5, isn't
>>>> it? If we consider Apache Spark 3.5.4 users, I believe we need to consider
>>>> Apache Spark 3.5.5 users too which uses
>>>> `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
>>>>
>>>> https://github.com/apache/spark/pull/50291
>>>>
>>>> - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>>> +
>>>> buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>>>
>>>> Dongjoon.
>>>>
>>>> On 2025/03/17 02:36:42 Wenchen Fan wrote:
>>>> > I've created the revert PR for branch-4.0:
>>>> > https://github.com/apache/spark/pull/50291 . We can merge PRs with
>>>> lazy
>>>> > consensus but it's clear that this breaking change PR has failed to
>>>> achieve
>>>> > consensus.
>>>> >
>>>> > I hope we now have a clear foundation for discussing solutions. As it
>>>> > stands, the misnamed configuration will be released in 4.0.0. I like
>>>> > Jungtaek’s proposal to deprecate it, but the decision is up to the
>>>> > community.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>>
>>>>
