Re: Re: [Discuss] SPIP: Support NanoSecond Timestamps

2025-03-14 Thread Szehon Ho
+1 to the idea as well, as Iceberg V3 is coming with time with nanos, and Spark would not be able to read this type without this. Thanks Szehon On Fri, Mar 14, 2025 at 3:34 PM Wenchen Fan wrote: > In general, I think it's good for Spark to support the common data types > in the ecosystem, as it

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Jungtaek Lim
That said, if I understand correctly, you weren’t intended to “block” the vote, right? You say you expected the vote to be finished. Could you please cast the vote to -0.x since some people views this as code change vote, or clarify explicitly that you think this is not a code change vote? This wi

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Jungtaek Lim
Dongjoon, Please look at what I got from the community. It is quite different whether "we strongly recommend" vs "we force". We never have an agreement that the latter would make sense. The DISCUSSION and VOTE were to gain consensus that the latter does not make sense, and I think we get consensu

Re: [VOTE][RESULT] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Jungtaek Lim
And the criteria of justifying -1 must be whether he answered all 4 questions from me. https://lists.apache.org/thread/kdtto3poz28q4yrqdqk6839y965sfn5c Where is the evidence that having a vendor name in the codebase is > violating ASF policy? Again, I see "Apple" to be used as a vendor name in >

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Dongjoon Hyun
Thank you all. The vote is finished in an intended way with the expected result. We have enough time to discuss and I have been sticking to my original technical justification from the beginning (including this). 1. Helping renaming the conf via SPARK-51172 (by approving it) 2. Banning `spark.dat

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Jungtaek Lim
Thanks for the update. Though I have to clarify that "What all of us agree on is that the previous code base is okay." is not true. Wenchen summarized what happened in other thread which I think it's more proper, like following: 1. A mistake was made, leading to a vendor name being included in t

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Jungtaek Lim
If we were not intended to block the VOTE but just to express the disagreement, please say "-1" instead of representing it as "veto". When saying veto, you intend to kill the process unless you are not persuaded or you are not having proper technical justification. On Fri, Mar 14, 2025 at 6:27 PM

Re: [VOTE] Retain migration logic of incorrect `spark.databricks.*` configuration in Spark 4.0.x

2025-03-14 Thread Jungtaek Lim
Sorry, I was missing the type of the vote - this totally depends on the type of the vote. If we weren't intended to block the VOTE which could have been interpreted as code change, maybe -0 or -0.5 or -0.99 should have been used rather than -1 to block the process. On Fri, Mar 14, 2025 at 7:01 PM