MartijnVisser commented on code in PR #21128:
URL: https://github.com/apache/flink/pull/21128#discussion_r1001704804
##########
flink-connectors/flink-connector-hive/pom.xml:
##########
@@ -39,6 +39,11 @@ under the License.
<reflections.version>0.9.8</reflections.version>
<derby.version>10.10.2.0</derby.version>
<hive.avro.version>1.8.2</hive.avro.version>
+ <!--
+ Hive 2.3.4 relies on Hadoop 2.7.2 and later versions.
+ For Hadoop 2.7, the minor Hadoop version supported for
flink-shaded-hadoop-2-uber is 2.7.5
+ -->
+
<hivemetastore.hadoop.version>2.7.5</hivemetastore.hadoop.version>
Review Comment:
You're right but at https://hive.apache.org/downloads.html is stated:
```
7 November 2018: release 2.3.4 available
This release works with Hadoop 2.x.y. You can look at the complete [JIRA
change log for this
release](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344319&styleName=Text&projectId=12310843).
```
So my conclusion is:
1. This comment is actually incorrect, because the Hive version that we're
pulling in is Hive 2.3.9
https://github.com/apache/flink/blob/master/pom.xml#L171
2. Hive 2.3.9 is also compatible with Hadoop 2.x.y. So we should be able to
completely remove all references to `hivemetastore.hadoop.version`, let the
Hive connector rely on the Hadoop version from the parent POM and then it's
still supported
3. The hive3 profile that we have refers to Hadoop 2.8.2
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hive/pom.xml#L1103,
but that doesn't make sense since hive3 works with Hadoop 3.x.y (per the same
download page). I would argue that this value should be set Hadoop 3.1.3 since
that's the version that we're using for the `cron_hadoop313` profile.
@luoyuxia You're the Hive expert, can you give me your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
