Cheng,
You are right that Spark's IsolatedClientLoader is only used by its
HiveExternalCatalog.
In any case, that only adds to the argument to continue supporting
iceberg-hive-metastore on Hive 2.
In yesterday's community sync, there seems to be agreement to support
publishing iceberg-hive-metastor
> Spark can be configured to use a different version
It’s not true. IsolatedClassLoader only supports Spark Hive session catalog and
Iceberg always uses Spark built-in Hive jars, namely, Hive 2.3.9 for Spark 3.x
and Hive 2.3.10 for Spark 4.0
Thanks,
Cheng Pan
> On May 21, 2025, at 03:06, Den
I don't believe continuing to build iceberg-hive-metastore jars for Hive 2 is
well justified.
Hive 2 is deprecated and has known security vulnerabilities, which should
encourage users to upgrade.
As you pointed out, Spark can be configured to use a different version, but
it's ultimately up to t
Denys,
As I have mentioned before, Spark uses a built-in version of the Hive
metastore (currently 2.3.10), but can be configured to use a different
version and be pointed to a path containing Hive metastore jars for the
different version. However, the highest Hive version that can be configured
fo
Wing Yew, how about Hive 3.x HMS? Is it compatible with Spark 3.x? If yes, why
keep support for v2?
FYI Hive 2.x has many unresolved security vulnerabilities.
Peter and Denys, thank you for your responses.
First, I'd like to get agreement on producing separate
iceberg-hive-metastore jars for Hive 2, 3 and 4.
Peter, do you agree with that?
Second, if we do that, I listed three options for the runtime jars of
engines, and it appears that you prefer option
Thanks for working on this @Wing Yew Poon !
As I have mentioned on the PR, I would prefer unbundling the hive-metastore
dependency from the runtime jars.
This would help in multiple ways:
- Decrease the size of the runtime jars - nowadays more-and-more people use
Iceberg against a REST catalog, so
Hi Wing Yew,
Hive-2.x and Hive 3.x are EOL. We encourage everyone to upgrade instead of
keeping compatibility with the legacy versions.
Note: Spark started migration to Hive-4.0, and we are helping here:
https://github.com/apache/spark/pull/50213
Regards,
Denys
Hi,
I'd like to bring attention to this again.
As you know, the hive-metastore module (which provides the HiveCatalog) is
built against Hive 2.
I have a PR, https://github.com/apache/iceberg/pull/12721, for building and
testing the hive-metastore module against Hive 2, 3 and 4. Daniel Weeks
looked
Hey Wing,
There are a few concerning issues I have with the current PR:
1) We need to update LICENSE/NOTICE for the hive3/hive4 dependencies
because I believe we only looked at the referenced version
2) We're producing artifacts for hive3 and hive4 modules which I think we
want to exclude (we sho
Hi,
I created https://github.com/apache/iceberg/pull/12721 as an alternative to
my earlier PR, https://github.com/apache/iceberg/pull/12681. With very
helpful pointers from Peter Vary and his persistent prodding, we have an
implementation where a single source set can be used to build and test wit
Hi Wing Yew,
Thanks for taking a look at this.
After the removal of the Hive runtime code, we only depend on HMS in the
HiveCatalog module in the production code. Since HMS API is supposed to be
backward compatible, I would prefer to keep a single hive-metastore module
and a single source dir for
Hi Peter,
Thank you for your thoughts on this.
My PR, https://github.com/apache/iceberg/pull/12681, achieves the following
objectives:
(1) Build and test the hive-metastore module against Hive 2, 3, and 4. (As
you suggested in the earlier thread, we build and test against 2.3.10,
3.1.3, and 4.0.1
13 matches
Mail list logo
