Starting from my +1 (non-binding). In addition, I propose to retain migration logic till Spark 4.1.x and remove it in Spark 4.2.0.
On Mon, Mar 10, 2025 at 9:44 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Hi dev, > > Please vote to retain migration logic of incorrect `spark.databricks.*` > configuration in Spark 4.0.x. > > - DISCUSSION: > https://lists.apache.org/thread/xzk9729lsmo397crdtk14f74g8cyv4sr > ([DISCUSS] Handling spark.databricks.* config being exposed in 3.5.4 in > Spark 4.0.0+) > > Specifically, please review this post > https://lists.apache.org/thread/xtq1kjhsl4ohfon78z3wld2hmfm78t9k which > explains pros and cons about the proposal - proposal is about "Option 1". > > Simply speaking, this vote is to allow streaming queries which had been > ever run in Spark 3.5.4 to be upgraded with Spark 4.0.x, "without having to > be upgraded with Spark 3.5.5+ in prior". If the vote passes, we will help > users to have a smooth upgrade from Spark 3.5.4 to Spark 4.0.x, which would > be almost 1 year. > > The (only) cons in this option is having to retain the incorrect > configuration name as "string" in the codebase a bit longer. The code > complexity of migration logic is arguably trivial. (link > <https://github.com/apache/spark/blob/4231d58245251a34ae80a38ea4bbf7d720caa439/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala#L174-L183> > ) > > This VOTE is for Spark 4.0.x, but if someone supports including migration > logic to be longer than Spark 4.0.x, please cast +1 here and leave the > desired last minor version of Spark to retain this migration logic. > > The vote is open for the next 72 hours and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Retain migration logic of incorrect `spark.databricks.*` > configuration in Spark 4.0.x > [ ] -1 Remove migration logic of incorrect `spark.databricks.*` > configuration in Spark 4.0.0 because... > > Thanks! > Jungtaek Lim (HeartSaVioR) >
