[
https://issues.apache.org/jira/browse/FLINK-17989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Israel Ekpo updated FLINK-17989:
--------------------------------
Description:
In the POM.xml classes from certain packages are relocated and filtered out of
the final jar In the POM.xml classes from certain packages are relocated and
filtered out of the final jar
This is causing errors for customers and users that are using the
StreamingFileSink with Azure Blob Storage in Flink versions 1.9.x, 1.10.x and
possibly 1.11.x
[https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
[https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
I would like to know why the relocation is happening and the reasoning behind
the exclusion and filtering of the classes
It seems to affect just the Azure file systems in my sample implementations
{code:java}
String outputPath =
"wasbs://contai...@account.blob.core.windows.net/streaming-output/";
final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new
Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
stream.addSink(sink);
// execute programenv.execute(StreamingJob.class.getCanonicalName());
{code}
{code:java}
//Exception Details Below
2020-05-27 17:23:16java.lang.NoClassDefFoundError:
org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at
org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at
java.lang.Thread.run(Thread.java:748)Caused by:
java.lang.ClassNotFoundException:
org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351){code}
was:
In the POM.xml classes from certain packages are relocated and filtered out of
the final jar In the POM.xml classes from certain packages are relocated and
filtered out of the final jar
This is causing errors for customers and users that are using the
StreamingFileSink with Azure Blob Storage in Flink version 1.9.x, 1.10.x and
possibly 1.11.x
[https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
I would like to know why the relocation is happening and the reasoning behind
the exclusion and filtering of the classes
It seems to affect just the Azure file systems in my sample implementations
{code:java}
String outputPath =
"wasbs://contai...@account.blob.core.windows.net/streaming-output/";
final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new
Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
stream.addSink(sink);
// execute programenv.execute(StreamingJob.class.getCanonicalName());
{code}
{code:java}
//Exception Details Below
2020-05-27 17:23:16java.lang.NoClassDefFoundError:
org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at
org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at
java.lang.Thread.run(Thread.java:748)Caused by:
java.lang.ClassNotFoundException:
org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351){code}
> java.lang.NoClassDefFoundError:
> org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-17989
> URL: https://issues.apache.org/jira/browse/FLINK-17989
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, FileSystems
> Affects Versions: 1.9.3, 1.10.1
> Environment: Ubuntu 18
> Java 1.8
> Flink 1.9.x and 1.10.x
> Reporter: Israel Ekpo
> Priority: Critical
> Fix For: 1.9.3, 1.10.1
>
>
> In the POM.xml classes from certain packages are relocated and filtered out
> of the final jar In the POM.xml classes from certain packages are relocated
> and filtered out of the final jar
> This is causing errors for customers and users that are using the
> StreamingFileSink with Azure Blob Storage in Flink versions 1.9.x, 1.10.x and
> possibly 1.11.x
> [https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
> [https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
>
> I would like to know why the relocation is happening and the reasoning behind
> the exclusion and filtering of the classes
> It seems to affect just the Azure file systems in my sample implementations
>
> {code:java}
> String outputPath =
> "wasbs://contai...@account.blob.core.windows.net/streaming-output/";
> final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new
> Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
> stream.addSink(sink);
> // execute programenv.execute(StreamingJob.class.getCanonicalName());
> {code}
> {code:java}
> //Exception Details Below
> 2020-05-27 17:23:16java.lang.NoClassDefFoundError:
> org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at
> org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202)
> at
> org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at
> java.lang.Thread.run(Thread.java:748)Caused by:
> java.lang.ClassNotFoundException:
> org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
> org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
