[
https://issues.apache.org/jira/browse/FLINK-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139114#comment-15139114
]
ASF GitHub Bot commented on FLINK-3271:
---------------------------------------
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1543#discussion_r52328751
--- Diff: flink-shaded-hadoop/pom.xml ---
@@ -111,6 +111,7 @@ under the License.
<include>io.netty:netty:*</include>
<include>org.apache.curator:*</include>
<include>org.apache.hadoop:*</include>
+
<include>org.mortbay.jetty:*</include>
--- End diff --
This PR does not change that the classes are not hidden. It only puts them
into the prepared Hadoop dependency for Flink (which later goes into the Flink
fat jar).
> Using webhdfs in a flink topology throws classnotfound exception
> ----------------------------------------------------------------
>
> Key: FLINK-3271
> URL: https://issues.apache.org/jira/browse/FLINK-3271
> Project: Flink
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.10.1
> Reporter: Abhishek Agarwal
> Assignee: Abhishek Agarwal
>
> I was just trying to run a storm topology on flink using flink-storm. I got
> this exception -
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/mortbay/util/ajax/JSON
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:727)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:610)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
> {noformat}
> My topology list some files on hdfs using webhdfs API.
> org.mortbay.util.ajax.JSON was included in the application uber jar. I
> noticed that flink loads the application jar in a child classloader. This is
> what most likely happened -
> 1. WebHdfsFileSystem class was loaded through parent class loader since it is
> included in flink-dist.jar.
> 2. WebHdfsFileSystem has reference to the org.mortbay.util.ajax.JSON but
> since it is loaded through parent class loader, WebHdfsFileSystem can't read
> a class in child class loader.
> Ideally all the referenced classes should be available in the distribution
> jar so that these sort of issues may not occur.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
