GitHub user GraceH opened a pull request:
https://github.com/apache/spark/pull/2892
New FsPermission instance w/o FsPermission.createImmutable in eventlog
By default, Spark builds its package against Hadoop 1.0.4 version. In that
version, it has some FsPermission bug (see [HADOOP-7629]
(https://issues.apache.org/jira/browse/HADOOP-7629) by Todd Lipcon). This bug
got fixed since 1.1 version. By using that FsPermission.createImmutable() API,
end-user may see some RPC exception like below (if turn on eventlog over HDFS).
Here proposes a quick fix to avoid certain exception for all hadoop versions.
```
Exception in thread "main" java.io.IOException: Call to
sr484/10.1.2.84:54310 failed on local exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1150)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at $Proxy6.setPermission(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at $Proxy6.setPermission(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:1285)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:572)
at
org.apache.spark.util.FileLogger.createLogDir(FileLogger.scala:138)
at org.apache.spark.util.FileLogger.start(FileLogger.scala:115)
at
org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:74)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:324)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/GraceH/spark eventlog-rpc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2892.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2892
----
commit 58ea0385939e3c3244d1e8a91dcc3bdef58dd816
Author: Grace <[email protected]>
Date: 2014-10-22T13:59:13Z
new FsPermission Instance w/o FsPermission.createImmutable
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
