Spark has a lot of strange behaviors already that we don't fix in patch releases. And bugs aren't usually fixed with a configuration flag to turn on the fix.
That said, I don't have a problem with this commit making it into a patch release. This is a small change and looks safe enough to me. I was just a little surprised since I was expecting a correctness issue if this is prompting a release. I'm definitely on the side of case-by-case judgments on what to allow in patch releases and this looks fine. On Tue, Apr 16, 2019 at 4:27 PM Michael Armbrust <mich...@databricks.com> wrote: > I would argue that its confusing enough to a user for options from > DataFrameWriter to be silently dropped when instantiating the data source > to consider this a bug. They asked for partitioning to occur, and we are > doing nothing (not even telling them we can't). I was certainly surprised > by this behavior. Do you have a different proposal about how this should > be handled? > > On Tue, Apr 16, 2019 at 4:23 PM Ryan Blue <rb...@netflix.com> wrote: > >> Is this a bug fix? It looks like a new feature to me. >> >> On Tue, Apr 16, 2019 at 4:13 PM Michael Armbrust <mich...@databricks.com> >> wrote: >> >>> Hello All, >>> >>> I know we just released Spark 2.4.1, but in light of fixing SPARK-27453 >>> <https://issues.apache.org/jira/browse/SPARK-27453> I was wondering if >>> it might make sense to follow up quickly with 2.4.2. Without this fix its >>> very hard to build a datasource that correctly handles partitioning without >>> using unstable APIs. There are also a few other fixes that have trickled >>> in since 2.4.1. >>> >>> If there are no objections, I'd like to start the process shortly. >>> >>> Michael >>> >> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> > -- Ryan Blue Software Engineer Netflix
