Re: setuptools 78.0.0 does not work with pyspark 3.x releases

2025-04-04 Thread Bjørn Jørgensen
Yes, I did make that PR a long time ago. It was merged on 20.11.2023. I can backport it.. it is only changing - to _ but I don't think that we have anyone to make a new spark version 3.5.X today and release it? Or do we have anyone that will make a new spark 3.5 release very soon? https://github.

Re: Revert of [SPARK-51229][BUILD][CONNECT] Fix dependency:analyze goal on connect common

2025-04-04 Thread Rozov, Vlad
Hi Hyukjin, Can you clarify what is broken in SBT? As I already mentioned in my previous e-mail, I tested both maven and SBT builds and both works. Is there a test that I can run that shows what is broken? Please file JIRA, so we can communicate on the JIRA and document what is broken. Thank y

Re: [RESULT][VOTE] Technical Justification for the veto of the "Retain migration logic..." code change proposal is not valid

2025-04-04 Thread Jungtaek Lim
Maybe I will just update the VOTE result, since the rationale of this VOTE, and the VOTE result is public. On Tue, Mar 18, 2025 at 10:00 PM Jungtaek Lim wrote: > I'm definitely OK with modifying migration logic to exclude "databricks" > if people think it is better. I'm even having a code change

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

2025-04-04 Thread Ángel Álvarez Pascua
Well ... and then? When are we going to tackle this? I could help. El mié, 12 mar 2025, 15:50, Mich Talebzadeh escribió: > Agreed. Hive upgrade is more time consuming as it involves backing up Hive > schema on your metastore and then running Hive provided upgrade schema > scripts against Hive sc

Re: Spark 3.5.2 and Hadoop 3.4.1 slow performance

2025-04-04 Thread Steve Loughran
Create the JIRA and we can look at it. if it is just write performance, then I am confident that that hadoo 3.4.1 is way faster writing code, with some extra parameters available to make things faster. https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/performance.html#Options_to_T