Hello, with 1.2.0 `WritableTypeInfo` got moved into its own artifact (flink-hadoop-compatibility_2.10-1.2.0.jar). Unlike with 1.1.0, the distribution jar `flink-dist_2.10-1.2.0.jar` does not include the hadoop compatibility classes anymore. However, `TypeExtractor` which is part of the distribution jar tries to load `WritableTypeInfo` using it was loaded itself from:
> Class<?> typeInfoClass;
> try {
> typeInfoClass =
> Class.forName(HADOOP_WRITABLE_TYPEINFO_CLASS, false,
> TypeExtractor.class.getClassLoader());
> }
> catch (ClassNotFoundException e) {
> throw new RuntimeException("Could not load the
> TypeInformation for the class '"
> + HADOOP_WRITABLE_CLASS + "'. You may
> be missing the 'flink-hadoop-compatibility' dependency.");
> }
Adding `flink-hadoop-compatibility` to my application jar leads to the
following stack trace on yarn (running `bin/flink run -m yarn-cluster...`):
> Caused by: java.lang.RuntimeException: Could not load the TypeInformation for
> the class 'org.apache.hadoop.io.Writable'. You may be missing the
> 'flink-hadoop-compatibility' dependency.
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createHadoopWritableTypeInfo(TypeExtractor.java:2025)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1649)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.privateGetForClass(TypeExtractor.java:1591)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:778)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createSubTypesInfo(TypeExtractor.java:998)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:679)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:629)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfo(TypeExtractor.java:595)
> at
> org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfo(TypeExtractor.java:588)
> at
> org.apache.flink.api.common.typeinfo.TypeHint.<init>(TypeHint.java:47)
> at cz.seznam.euphoria.benchmarks.flink.Util$2.<init>(Util.java:80)
I guess I'm supposed to customize my flink installation by adding the
hadoop-compatibility jar to flink's `lib` dir, correct? If so, is this
documented? I couldn't find any hints on [1] nor [2] and, thus, suppose
this is maybe an unintentional change between 1.1 and 1.2.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/batch/hadoop_compatibility.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/migration.html
P.
signature.asc
Description: OpenPGP digital signature
