Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

The problem you are mentioning is arguably resolvable if we leave a config name as it is, and add withAlternative("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan"). Let's not nitpick just from reverting the PR. We have to revert the PR "semantically". Btw, from what I understa

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I reviewed Wenchen's reverting PR. Although it's a proposal for discussion, it is another breaking change against Apache Spark 3.5.5, isn't it? If we consider Apache Spark 3.5.4 users, I believe we need to consider Apache Spark 3.5.5 users too which uses `spark.sql.optimizer.pruneFiltersCanPrune

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

If we are really wanting to make a "correct" discussion going forward, I believe the revert PR has to be merged. After that, either my proposal gets not accepted, or he starts to DISCUSS and eventually reaches the VOTE pass, or we just leave the config to be kept deprecated instead of removed. We

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Btw, I don't think reverting the PR of removing config is violating community's consensus, because we never had a VOTE for this breaking change (this is definitely a breaking change without migration logic), so we did not really get explicit consensus with that. The real neutral state is that we do

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Holden, I think I have some workaround like I posted in dev@ (link ), which is definitely less better than this proposal so I still want to see this VOTE to go forward, but it's somewhat better in this situation that we no longer tal

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Thanks Mark for starting this. +1 and agree with your reasoning. Wearing the Apache Spark PMC hat, I think having a few lines of straightforward logic to ease users' migrations is a no-brainer to do. Imagine how confused a user would be when they upgraded to 4.0 and things stopped working in a way

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I'm delighted to see folks talking about a compromise. However, instead of just asking Dongjoon to withdraw the VETO perhaps folks can suggest alternatives that that might meet some of both parties goals? On Sun, Mar 16, 2025 at 7:41 PM Wenchen Fan wrote: > I agree with Holden that withdrawing a

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Holden and Dongjoon, Let me make this vote super simple. I never got the answer from Dongjoon about this question. This is super important because if he's casting veto "to block", it is a strong indication that he was intended to play with me, which I am seriously considering escalating the proble

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I agree with Holden that withdrawing a veto is always better than overriding it: it's healthier for the community. Dongjoon, would you be willing to reconsider your veto given the current as-is state of the 4.0.0 release (the breaking change will be reverted)? On Mon, Mar 17, 2025 at 10:36 AM Wenc

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I've created the revert PR for branch-4.0: https://github.com/apache/spark/pull/50291 . We can merge PRs with lazy consensus but it's clear that this breaking change PR has failed to achieve consensus. I hope we now have a clear foundation for discussing solutions. As it stands, the misnamed confi

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

OK, let's be super honest. Again, I think you agree that *"both" proposals are "technically" correct (or one side can't have a strong theoretical evidence to counter the other side)*. So this naturally has a fate to have more supporters to get to the end. It's very easy for me to VETO to his propo

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

First let me start with my key hope: We find a way to compromise and have the veto withdrawn rather than overridden. >From what I understand of the change in question: So my understanding, and I may be over simplifying here but there are (at least) three technical paths forward (migration guide,

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Holden, I believe you should already know "both" approaches are "technically" correct. It's not about which one you have a preference for, no, this VOTE is not intended to extend the debate. Again, what you are encouraged to do here is, not exposing your preference of two approaches, but exposing

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

Hi Holden, > I think I disagree with Mark on the assertion that the veto needs to have “substantial technical concern,” but rather a valid concern. I think in addition to the veto they’ve also gone above and beyond providing alternative ways to accomplish this. I think this is not what the Apache

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I was trying hard to stay away from this VOTE, but I should have reminded everyone about "what" we are going to VOTE. Dongjoon casted a VETO against code change VOTE. That VETO is described in ASF Voting Process page: https://www.apache.org/foundation/voting.html#Veto A -1 vote by a qualified vo

Unsubscribe

- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Discuss] SPIP: Support NanoSecond Timestamps

+1 for supporting NanoSecond Timestamps. Thank you, Qi. Dongjoon. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

It's OK, I just wanted to let him give the link whenever he makes the argument. It is NOT important that he brings up a new reasoning at this point. It must have been there in VOTE. Also, I think we all agree that this is NOT a place of debate about having vendor names in the codebase. I think I g

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I believe he intended to VETO this change for Spark 4, of course if Dongjoon did not (or no longer intends to) then this VOTE becomes moot. I think bringing up 3.5.5 confuses the issue -- this vote thread is very clearly about VETOing the change for Spark 4. I think that accusing each-other of act

Re: [DISCUSS] New Spark Connect Client repository for Swift language

+1 Sounds like a plan Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile On Sun, 16 Mar 2025 at 21:10, Martin Grund wrote: > So I was just playing with the Swift cl

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

I second this point of Holden. In fairness anything published in a public forum like this one is fair game for analysis or criticism. That is the practice of analyzing, classifying, interpreting, or evaluating the technical matter. If someone makes a claim on a technical or procedure matter in an

Re: [VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

-1 (binding) — to me it doesn’t matter that the cost is low if the objection is technical then I think we need to respect the veto. There is a fundamental disagreement as to what the correct technical way to address this problem is (removal + documentation vs legacy config) and a PMC member has vet

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Hi Jungtaek. With regard to your point below "...Hi dev, I'm really tired of the discussion which does not move forward because the argument is not backed by strict ASF policy" Regardless, we all appreciate your efforts and your tenacity. cheers Dr Mich Talebzadeh, Architect | Data Scienc

Re: [DISCUSS] New Spark Connect Client repository for Swift language

Thank you for the suggestion, Martin. I prefer to follow `spark` and `spark-kubernetes-operator` repository way as the traditional Apache Spark way. In addition, apparently, it seems that I'm going to maintain `spark-connect-swift` repository as the main contributor and main reviewer in most c

Re: [DISCUSS] New Spark Connect Client repository for Swift language

So I was just playing with the Swift client to build a Mac app and everything worked nicely! However, similar to the Go repository, my suggestion would be to handle issues directly in Github and not in Jira because the vast majority of initial issues will be simply compatibility issues. And creatin

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Yeah... maybe 1 is simpler if there is no side effect, and probably the latter pattern we have is long enough to figure out aliases without full text matching. On Sun, Mar 16, 2025 at 5:15 PM Mark Hamstra wrote: > Doing something like pattern matching on > u0064\u0061\u0074\u0061\u0062\u0072\u00

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Doing something like pattern matching on u0064\u0061\u0074\u0061\u0062\u0072\u0069\u0063\u006b\u0073 instead of “databricks” might also be an option if including “databricks” in the code is believed to be so offensive. On Sun, Mar 16, 2025 at 12:52 AM Jungtaek Lim wrote: > Hi dev, > > I'm reall

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Apologize for quick fix, This won't go to VOTE "unless" there are "valid" objections. The message was sent accidentally before I double checked. On Sun, Mar 16, 2025 at 5:00 PM Jungtaek Lim wrote: > Just to clarify: This won't go to VOTE as long as there are "valid" > objections. Let's not waste

Re: [DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Just to clarify: This won't go to VOTE as long as there are "valid" objections. Let's not waste more time on our end. On Sun, Mar 16, 2025 at 4:52 PM Jungtaek Lim wrote: > Hi dev, > > I'm really tired of the discussion which does not move forward because the > argument is not backed by strict AS

[DISCUSS] Involve any hack / workaround to not include vendor name in migration logic

Hi dev, I'm really tired of the discussion which does not move forward because the argument is not backed by strict ASF policy. We debate based on the interpretation of ASF policy by individuals, which I think makes zero sense. I really thought about this a lot how to resolve this, and now I'm op

[ANNOUNCE] Apache Sedona 1.7.1 released

Dear all, We are happy to report that we have released Apache Sedona 1.7.1. Thank you again for your help. Apache Sedona is a cluster computing system for processing large-scale spatial data on top of Apache Spark, Flink and Snowflake. Vote thread (Permalink from https://lists.apache.org/list.ht