Let me reformulate your suggestions and my interpretation.
Option 1 "Adding back `spark.databricks.*` in Spark codebase and keep
forever"
If we follow the proposed logic and reasoning, it means there is no safe
version to remove that configuration because Apache Spark 3.5.4 users can
jump to any
I think it is how to handle the deprecation and removal.
If we leave the migration path for Spark 4.1.x, it will take more than "1
year of upgrade path" to be successful. From our release cadence, Spark
4.2.0 would probably be released March next year or later. And Spark 3.5.4
was released in Dece
Here is one more problem I found during RC2 verification:
https://github.com/apache/spark/pull/50173
This one is just a test issue.
Chris Nauroth
On Tue, Mar 4, 2025 at 2:55 PM Jules Damji wrote:
> - 1 (non-binding)
>
> A ran into number of installation and launching problems. May be it’s my
