I reviewed Wenchen's reverting PR. Although it's a proposal for discussion, it is another breaking change against Apache Spark 3.5.5, isn't it? If we consider Apache Spark 3.5.4 users, I believe we need to consider Apache Spark 3.5.5 users too which uses `spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan` already.
https://github.com/apache/spark/pull/50291 - buildConf("spark.sql.optimizer.pruneFiltersCanPruneStreamingSubplan") + buildConf("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan") Dongjoon. On 2025/03/17 02:36:42 Wenchen Fan wrote: > I've created the revert PR for branch-4.0: > https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy > consensus but it's clear that this breaking change PR has failed to achieve > consensus. > > I hope we now have a clear foundation for discussing solutions. As it > stands, the misnamed configuration will be released in 4.0.0. I like > Jungtaek’s proposal to deprecate it, but the decision is up to the > community. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
