[VOTE][RESULT][FINAL] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-18 Thread Jungtaek Lim
The vote passes with 8 +1s (4 binding +1s) and no -1s. Thanks to all who helped with the vote! (* = binding) +1: - Sean R. Owen * - Jungtaek Lim - Nicholas Chammas - Wenchen Fan * - Adam Binford - Russell Jurney - Yang Jie * - Mridul Muralidharan * NOTE1: The community discussed the type of vote

Re: [VOTE][RESULT] Single-pass Analyzer for Catalyst

2025-03-18 Thread Ángel Álvarez Pascua
This SPIP looked like a great idea. Does anybody know if there's actually someone working on it? El vie, 4 oct 2024 a las 10:09, Vladimir Golubev () escribió: > Hi folks! > > The vote for 'SPIP: Single-pass Analyzer for Catalyst' passed with 14 +1s > (8 bindings, * = binding): > > +1: > Reynold X

[RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

2025-03-18 Thread Hyukjin Kwon
The vote passes with 5 +1s (4 binding +1s) and 3 -1s (3 binding -1s). (* = binding) +1: - Mark Hamstra * - Jungtaek Lim - Wenchen Fan * - Reynold Xin * - Yuanjian Li * -1: - Holden Karau * - Hyukjin Kwon * - Dongjoon Hyun * Thanks.

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

2025-03-18 Thread Jungtaek Lim
Regardless of this vote result, we (including me) had too many mistakes/faults during handling this (I'd rather say "incident", or even "disaster"). I really think we should have a retrospective, making a rough timeline of the events with making people involved to be anonymous, and brainstorm how

Re: [RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

2025-03-18 Thread Jungtaek Lim
I'm definitely OK with modifying migration logic to exclude "databricks" if people think it is better. I'm even having a code change locally. The reason I didn't ask killing the VOTE despite I have the other way around is, I think we made a huge mistake/fault w.r.t. this event, and I don't want my

Re: [RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

2025-03-18 Thread Hyukjin Kwon
>From my understanding, yes. On Tue, Mar 18, 2025 at 4:03 PM Wenchen Fan wrote: > Do I understand correctly that now we can merge > https://github.com/apache/spark/pull/49984 and unblock Spark 4.0? > > To make sure that everyone is on the same page as not all the people have > read all the discu

Re: [RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

2025-03-18 Thread Wenchen Fan
Do I understand correctly that now we can merge https://github.com/apache/spark/pull/49984 and unblock Spark 4.0? To make sure that everyone is on the same page as not all the people have read all the discussion threads: 1. This PR *DOES* *NOT* add back the misnamed configuration. People cannot se

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

2025-03-18 Thread Jungtaek Lim
I came up with the PR - https://github.com/apache/spark/pull/50314 Maybe it's less performant than the original one since we no longer do get directly and we iterate the keys, but hopefully, offset log is expected to maintain a static number of configs (subset of SQL config), and it's just 10-ish

RE: Re: [Discuss] SPIP: Support NanoSecond Timestamps

2025-03-18 Thread Qi Tan
Hello Reynold, I truly appreciate your time and attention to this feature. For the performance, here are my thoughts: * As Serge mentioned above, Apache Spark needs to be aligned with other competitive products. We should not overlook potential benefits just because of performance regression.