Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

Well ... and then? When are we going to tackle this? I could help. El mié, 12 mar 2025, 15:50, Mich Talebzadeh escribió: > Agreed. Hive upgrade is more time consuming as it involves backing up Hive > schema on your metastore and then running Hive provided upgrade schema > scripts against Hive sc

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

This is what my WIP PR targets. It will help to identify any compatibility or breaking issues with the new dependency. Thank you, Vlad On Mar 26, 2025, at 3:14 AM, Mich Talebzadeh wrote: Because of dependencies we need to ensure that the underlying artifacts (Hive 4.0.1) is also stable enoug

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

Because of dependencies we need to ensure that the underlying artifacts (Hive 4.0.1) is also stable enough. We should aim to establish that first and look for release timelines and where it fits cheers Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR v

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

I started working on it. See https://github.com/apache/spark/pull/50213. Review and comments on the PR will help a lot. +1 for 4.1. It won’t be ready for 4.0 and will require extensive testing. I have few more local changes that fixes some tests in sql/hive and should publish another revision s

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

I agree, 4.0 is already in the RC stage and I think it's too late to do such a big version bump for the Hive dependency. We definitely need to do this upgrade and thanks for working on it! On Mon, Mar 24, 2025 at 1:31 PM Ángel Álvarez Pascua < angel.alvarez.pas...@gmail.com> wrote: > That's grea

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

That's great news! If you need anything from me, just ask... We should also check and update other non-Hive third-party libraries with high/critical vulnerabilities, as someone mentioned in another email thread. Since this is a major change, I think we should leave it for Spark 4.1. What do you t

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

For now, I am testing apache-hive-4.0.1-bin which is the latest release version from https://dlcdn.apache.org/hive/hive-4.0.1/ apache-hive-4.0.1-bin.tar My metastore is Oracle and upgrade scripts are provided.. My previous version is Hive 3.1.1 and the metastore upgrade went OK without any maj

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

Not an easy task, I guess, but I'm totally for it too. The issue SPARK-49910 is related to this. El mar, 11 mar 2025 a las 23:06, Mich Talebzadeh () escribió: > Yes I am all for it, as I use Hive with Oracle as its metastore > extensively. >

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

Agreed. Hive upgrade is more time consuming as it involves backing up Hive schema on your metastore and then running Hive provided upgrade schema scripts against Hive schema that could be problematic,but needs to be done one way or another. HTH Dr Mich Talebzadeh, Architect | Data Science | Finan

Re: [DISCUSS] Upgrade Hive compile time dependency to 4.0

Yes I am all for it, as I use Hive with Oracle as its metastore extensively. Case in point, on 6th March A Hive user alluded to it and I quote "I just wanted to highlight that Hive 3.x line is EOL. It has various known security v

[DISCUSS] Upgrade Hive compile time dependency to 4.0

Hi All, As Apache Hive announced EOL for Hive 2.x [1] and 3.x [2], should Spark be compiled against Hive 4.x and use it as default? Thank you, Vlad [1] https://lists.apache.org/thread/4ctrzfw60jkhc0hq2xoh1jpqxgt2zd93 [2] https://lists.apache.org/thread/99h6wr7nk4684r6tkcbm8ydfytgqy6f3 [3] http