pan3793 commented on PR #46521:
URL: https://github.com/apache/spark/pull/46521#issuecomment-2706965438
> For this one PR, I believe we need a verification for different HMS
versions to make it sure.
@dongjoon-hyun I managed to set up an env to test the IsolatedClassLoader,
it works as expected.
The basic test steps:
- Hadoop Cluster with HDFS and YARN (v3.3.6)
- HMS 3.1.2
- Spark 4.0.0 RC2 and delete `hive-jacksons` folder
Verify built-in Hive 2.3.10 works well without CodeHaus Jackson jars
```
$ bin/spark-shell
scala> spark.sql("show databases").show()
+---------+
|namespace|
+---------+
| default|
+---------+
scala> spark.sql("create temporary function hive_uuid as
'org.apache.hadoop.hive.ql.udf.UDFUUID'")
val res1: org.apache.spark.sql.DataFrame = []
scala> spark.sql("select hive_uuid()").show
+--------------------+
| hive_uuid()|
+--------------------+
|3ad7e110-2ad9-4f0...|
+--------------------+
```
Verify Hive 3.1.3 metastore jars also works well without CodeHaus Jackson
jars
```
$ bin/spark-shell --conf spark.sql.hive.metastore.version=3.1.3 --conf
spark.sql.hive.metastore.jars=maven
scala> spark.sql("show databases").show()
... (triggers jar downloading include CodeHaus Jackson jars)
downloading
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar
...
[SUCCESSFUL ]
org.codehaus.jackson#jackson-core-asl;1.9.13!jackson-core-asl.jar (1178ms)
downloading
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar
...
[SUCCESSFUL ]
org.codehaus.jackson#jackson-mapper-asl;1.9.13!jackson-mapper-asl.jar (3245ms)
downloading
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar
...
[SUCCESSFUL ]
org.codehaus.jackson#jackson-jaxrs;1.9.13!jackson-jaxrs.jar (429ms)
downloading
https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar
...
[SUCCESSFUL ] org.codehaus.jackson#jackson-xc;1.9.13!jackson-xc.jar
(454ms)
...
downloading
https://repo1.maven.org/maven2/org/apache/hive/hive-llap-client/3.1.3/hive-llap-client-3.1.3.jar
...
[SUCCESSFUL ]
org.apache.hive#hive-llap-client;3.1.3!hive-llap-client.jar (484ms)
downloading
https://repo1.maven.org/maven2/org/apache/hive/hive-llap-common/3.1.3/hive-llap-common-3.1.3.jar
...
[SUCCESSFUL ]
org.apache.hive#hive-llap-common;3.1.3!hive-llap-common.jar (851ms)
...
:: retrieving ::
org.apache.spark#spark-submit-parent-a3e58de0-7045-4d08-ba86-3d7b1fc03a46
confs: [default]
209 artifacts copied, 0 already retrieved (236164kB/372ms)
2025-03-07T16:51:48.371441489Z main INFO Starting configuration
XmlConfiguration[location=/etc/spark/conf/log4j2.xml,
lastModified=2025-03-07T15:59:46.435Z]...
2025-03-07T16:51:48.371577157Z main INFO Start watching for changes to
/etc/spark/conf/log4j2.xml every 0 seconds
2025-03-07T16:51:48.371673867Z main INFO Configuration
XmlConfiguration[location=/etc/spark/conf/log4j2.xml,
lastModified=2025-03-07T15:59:46.435Z] started.
2025-03-07T16:51:48.371843828Z main INFO Stopping configuration
org.apache.logging.log4j.core.config.DefaultConfiguration@4926e32c...
2025-03-07T16:51:48.371982996Z main INFO Configuration
org.apache.logging.log4j.core.config.DefaultConfiguration@4926e32c stopped.
2025-03-07 16:51:48 INFO HiveConf: Found configuration file
file:/etc/hive/conf/hive-site.xml
Hive Session ID = 1bd666c3-8064-43e5-9b8d-5b680ccc5e6a
2025-03-07 16:51:48 INFO SessionState: Hive Session ID =
1bd666c3-8064-43e5-9b8d-5b680ccc5e6a
2025-03-07 16:51:48 INFO HiveMetaStoreClient: Trying to connect to metastore
with URI thrift://hadoop-master1.orb.local:9083
2025-03-07 16:51:48 INFO HiveMetaStoreClient: Opened a connection to
metastore, current connections: 1
2025-03-07 16:51:48 INFO HiveMetaStoreClient: Connected to metastore.
2025-03-07 16:51:48 INFO RetryingMetaStoreClient: RetryingMetaStoreClient
proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0
+---------+
|namespace|
+---------+
+---------+
scala> spark.sql("create temporary function hive_uuid as
'org.apache.hadoop.hive.ql.udf.UDFUUID'")
val res0: org.apache.spark.sql.DataFrame = []
scala> spark.sql("select hive_uuid()").show
+--------------------+
| hive_uuid()|
+--------------------+
|8ec74979-7fba-47e...|
+--------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
