i started working on this by upgrading my hadoop to
Hadoop 3.4.1
My Hive is
Driver: Hive JDBC (version 4.0.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Running init script /home/hduser/dba/bin/add_jars.hql
25/06/05 23:33:44 [main]: WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
0: jdbc:hive2://rhes75:10099/default> *set hive.support.concurrency=false;*
No rows affected (0.027 seconds)
0: jdbc:hive2://rhes75:10099/default>
0: jdbc:hive2://rhes75:10099/default> Beeline version 4.0.1 by Apache Hive
Now we hive transactional with Hive 4. which we did not have with the prior
versions. Compounded my metastore is on Oracle 12.
messing around with
set hive.support.concurrency=false;
show databases
. . . . . . . . . . . . . . . . . . > Error: Error running query
(state=,code=0)
*set hive.support.concurrency=true*
. . . . . . . . . . . . . . . . . . > No rows affected (0.002 seconds)
0: jdbc:hive2://rhes75:10099/default> show databases
. . . . . . . . . . . . . . . . . . > +----------------+
| database_name |
+----------------+
| access |
| accounts |
So the last time I worked on it I was trying to sort out the concurrency
issues
Running simple queries on Hive
FAILED: Error in acquiring locks: Error communicating with the metastore
ERROR : FAILED: Error in acquiring locks: Error communicating with the
metastore
org.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with
the metastore
at
org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:183)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:475)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:509)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:296)
at
org.apache.hadoop.hive.ql.lockmgr.HiveTxnManagerImpl.acquireLocks(HiveTxnManagerImpl.java:81)
at
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:101)
at
org.apache.hadoop.hive.ql.DriverTxnHandler.acquireLocksInternal(DriverTxnHandler.java:328)
at
org.apache.hadoop.hive.ql.DriverTxnHandler.acquireLocks(DriverTxnHandler.java:232)
at
org.apache.hadoop.hive.ql.DriverTxnHandler.acquireLocksIfNeeded(DriverTxnHandler.java:144)
at org.apache.hadoop.hive.ql.Driver.lockAndRespond(Driver.java:356)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:197)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at
org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.TApplicationException: Internal error
processing lock
Unfortunately something is missing somewhere
They have seen this error with postgres Hive metastore DB as well. I need
to work on it when I have a chance
HTH
Dr Mich Talebzadeh,
Architect | Data Science | Financial Crime | Forensic Analysis | GDPR
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
On Thu, 5 Jun 2025 at 22:20, Rozov, Vlad <vro...@amazon.com.invalid> wrote:
> Hi All,
>
> I want to start a discussion thread on the SPIP titled "Upgrade Apache
> Hive to 4.x” [JIRA <https://issues.apache.org/jira/browse/SPARK-52408>][
> Doc
> <https://docs.google.com/document/d/1ejaGpuBvwBz2cD3Xj-QysShauBrdgYSh5yTxfAGvS1c/edit?usp=sharing>]
> that I was researching for some time now.
>
> The SPIP proposes upgrading the Apache Hive version used in Apache Spark
> builds from 2.3.10 to version 4.x. It also proposes discontinuing support
> for Apache Hive 2.x and 3.x, as these versions are no longer maintained by
> the Apache Hive community and have reached end-of-life (EOL).
>
> The key objectives of this proposal are to:
>
>
> - Maintain all existing functionality currently supported in Apache
> Hive 2.x that is compatible with Apache Hive 4.x
> - Ensure no functional or performance regressions occur
> - Provide the best upgrade path for current Apache Spark users,
> minimizing prerequisites and manual steps for those using Hive 2.x or 3.x
>
>
> I'd greatly appreciate your feedback, thoughts, and suggestions on this
> proposal!
>
> Thank you,
>
> Vlad
>
