[
https://issues.apache.org/jira/browse/HIVE-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739830#comment-17739830
]
Stamatis Zampetakis commented on HIVE-27475:
--------------------------------------------
[~ayushtkn] Using 4.0.0 and 4.0.0-beta-1 is fine at the moment. Once I cut the
new RC I will move everything that is fixed in 4.0.0 to 4.0.0-beta-1. Using
4.0.0 might be a better choice cause if someone uses 4.0.0-beta-1 after I cut
the branch but before it is actually released the ticket may get lost from the
release notes of the next release.
> Downgrade Kafka version to 2.5.0 to stabilise Kafka handler
> ------------------------------------------------------------
>
> Key: HIVE-27475
> URL: https://issues.apache.org/jira/browse/HIVE-27475
> Project: Hive
> Issue Type: Task
> Components: kafka integration
> Affects Versions: 4.0.0-alpha-2
> Reporter: Akshat Mathur
> Assignee: Akshat Mathur
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> It was needed to shade a version of zstd-jni that is compatible with parquet
> in hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't
> work with parquet and zstd encryption. Parquet and kafka-client both use
> zstd-jni. In hive-ql, zstd-jni is coming as a transitive dependency from
> kafka-client. The zstd-jni version in kafka-client 2.5.0 was not compatible
> with parquet, so in [PR-4082|https://github.com/apache/hive/pull/4082]
> kafka-client was upgraded to latest version which used zstd-jni version close
> to the version in parquet.
> While the upgrade fixed the zstd-ini issue, It introduced multiple
> compatibility issue in the Kafka-handler and due to lack to test cases and
> disabled tests these issues went unidentified.
> With the recent refactor in [HIVE-27402|http://example.com/], Kafka-client
> dependency was moved out of hive-ql, Which fixed the original zstd-ini
> issue.(confirmed by [~difin])
> Hence , It makes more sense to downgrade kafka versions back to 2.5.0,
> stabalize kafka-handler, enable test and then upgrade.
> Ref to discussion can be found here:
> [#4436|https://github.com/apache/hive/pull/4436]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
