+1
I would like to add my perspective to this discussion. It's important that
we maintain a fair and objective technical evaluation process, especially
when a veto is cast. A veto should be backed by concrete technical
reasoning.
>From my reading of the discussion, the key concern is whether the
justification for the veto is based on an objective technical issue with
the proposed change. If the concern is not rooted in a clear technical
limitation or regression, then it should be open to further discussion and
reconsideration. It is crucial for us as a community to ensure that
decisions are made transparently and with technical merit as the primary
driver.
I also want to emphasize that maintaining a welcoming and inclusive
community is vital to the success of Apache Spark. If contributors feel
isolated or that decisions are being swayed by affiliations rather than
technical arguments, that would set a concerning precedent. We should
strive to engage in discussions constructively and focus on what is best
for the project and its users.
Jungtaek Lim <kabhwan.opensou...@gmail.com> 于2025年3月16日周日 23:11写道：
> The problem you are mentioning is arguably resolvable if we leave a config
> name as it is, and add
> withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan").
> Let's not nitpick just from reverting the PR. We have to revert the PR
> "semantically".
>
> Btw, from what I understand of dealing with backport PR is, we mostly
> merge in master to begin with, and down the version line. When we handle my
> migration PR, we do not follow this practice without any discussion. I
> submitted PRs for master/4.0/3.5, and 3.5 was merged "first", and when I
> asked for merging to 4.0, I was pushed back. I don't think this is an
> ordinary practice of dealing with multiple versions of PRs, especially
> since the author never agreed with the way of processing.
>
> If we were following the practice, we should already have migration logic
> for master/4.0/3.5, since we should have the same fix in master/4.0. We
> probably had a discussion about removing config from master/4.0 based on
> the discussion, and we probably agreed to remove the config since we still
> have a migration logic. W.r.t migration logic, based on the discussion we
> are having, we probably can't make an agreement to take it out, then
> arguably the migration logic is left as it is.
>
> This way I never needed to drive such a long and sensitive DISCUSSION and
> VOTE. But the PR for master and 4.0 weren't merged because of the
> individual's belief of the rollout plan, which we are seeing does not fit a
> majority of voices now.
>
> Shall we fix the broken process in which we made a huge mistake before
> moving on? We should have merged the same content in 3.5 to master/4.0 as
> well, and then have a PR to remove the config. This is totally swapped
> which does not make sense to me.
>
> On Mon, Mar 17, 2025 at 1:57 PM Dongjoon Hyun <dongj...@apache.org> wrote:
>
>> I reviewed Wenchen's reverting PR. Although it's a proposal for
>> discussion, it is another breaking change against Apache Spark 3.5.5, isn't
>> it? If we consider Apache Spark 3.5.4 users, I believe we need to consider
>> Apache Spark 3.5.5 users too which uses
>> `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
>>
>> https://github.com/apache/spark/pull/50291
>>
>> - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>> +
>> buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
>>
>> Dongjoon.
>>
>> On 2025/03/17 02:36:42 Wenchen Fan wrote:
>> > I've created the revert PR for branch-4.0:
>> > https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy
>> > consensus but it's clear that this breaking change PR has failed to
>> achieve
>> > consensus.
>> >
>> > I hope we now have a clear foundation for discussing solutions. As it
>> > stands, the misnamed configuration will be released in 4.0.0. I like
>> > Jungtaek’s proposal to deprecate it, but the decision is up to the
>> > community.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
