Re: Switching to Java 7

2014-12-08 Thread Steve Loughran 
yes, bumped them up to

export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
export ANT_OPTS=$MAVEN_OPTS

also extended test runs times.



On 8 December 2014 at 00:58, Ted Yu  wrote:

> Looking at the test failures of
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk 1.7:
>
> e.g.
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
>
> java.lang.OutOfMemoryError: Java heap space
> at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> at
> io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> at
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
>
>
> Should more heap be given to the tests ?
>
>
> Cheers
>
>
> On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran 
> wrote:
>
> > The latest migration status:
> >
> >   if the jenkins builds are happy then the patch will go in -I do that
> > monday morning 10:00 UTC
> >
> > https://builds.apache.org/view/H-L/view/Hadoop/
> >
> > Getting jenkins to work has been "surprisingly difficult"...it turns out
> > that those builds which we thought were java7 or java8 weren't, as
> setting
> >   export JAVA_HOME=${TOOLS_HOME}/java/latest
> >
> > meant that they picked up a java 6 machine
> >
> > Now the trunk precommit/postcommit and scheduled branches should have
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
> >
> > the Java 8 builds have more changes
> >
> > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> > export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> > and  -Dmaven.javadoc.skip=true  on the mvn builds
> >
> > without these javadocs fails and test runs OOM.
> >
> > We need to have something resembling the nightly build env setup again,
> > git/Svn stored file with something for java8 alongside the normal env
> vars.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-7487) NPE in MetricsSourceAdapter

2014-12-08 Thread Brahma Reddy Battula (JIRA) 
Brahma Reddy Battula created HDFS-7487:
--

 Summary: NPE in MetricsSourceAdapter
 Key: HDFS-7487
 URL: https://issues.apache.org/jira/browse/HDFS-7487
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Brahma Reddy Battula


{noformat}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateAttrCache(MetricsSourceAdapter.java:247)
at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:177)
at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getAttribute(MetricsSourceAdapter.java:102)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hadoop-Hdfs-trunk - Build # 1964 - Still Failing

2014-12-08 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 6358 lines...]
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE [  02:29 h]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  2.164 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:29 h
[INFO] Finished at: 2014-12-08T12:40:14+00:00
[INFO] Final Memory: 58M/804M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating YARN-1492
Updating YARN-2927
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
22 tests failed.
REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.testFailureToReadEditsOnTransitionToActive[0]

Error Message:
org/apache/hadoop/test/GenericTestUtils

Stack Trace:
java.lang.NoClassDefFoundError: org/apache/hadoop/test/GenericTestUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.testFailureToReadEditsOnTransitionToActive(TestFailureToReadEdits.java:295)


REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.testCheckpointStartingMidEditsFile[1]

Error Message:
java.util.zip.ZipException: invalid block type

Stack Trace:
java.lang.RuntimeException: java.util.zip.ZipException: invalid block type
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at 
org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown 
Source)
at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown 
Source

Build failed in Jenkins: Hadoop-Hdfs-trunk #1964

2014-12-08 Thread Apache Jenkins Server 
See 

Changes:

[kasha] YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent. 
(Ray Chiang via kasha)

--
[...truncated 6165 lines...]
Running org.apache.hadoop.hdfs.TestHFlush
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.559 sec - in 
org.apache.hadoop.hdfs.TestHFlush
Running org.apache.hadoop.hdfs.TestDFSFinalize
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.057 sec - in 
org.apache.hadoop.hdfs.TestDFSFinalize
Running org.apache.hadoop.hdfs.TestDatanodeRegistration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.293 sec - in 
org.apache.hadoop.hdfs.TestDatanodeRegistration
Running org.apache.hadoop.hdfs.TestRenameWhileOpen
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.206 sec - in 
org.apache.hadoop.hdfs.TestRenameWhileOpen
Running org.apache.hadoop.hdfs.TestAbandonBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.803 sec - in 
org.apache.hadoop.hdfs.TestAbandonBlock
Running org.apache.hadoop.hdfs.TestDFSRollback
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.474 sec - in 
org.apache.hadoop.hdfs.TestDFSRollback
Running org.apache.hadoop.hdfs.TestReservedRawPaths
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.838 sec - in 
org.apache.hadoop.hdfs.TestReservedRawPaths
Running org.apache.hadoop.hdfs.TestLeaseRecovery
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.301 sec - in 
org.apache.hadoop.hdfs.TestLeaseRecovery
Running org.apache.hadoop.hdfs.TestEncryptionZones
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.315 sec - 
in org.apache.hadoop.hdfs.TestEncryptionZones
Running org.apache.hadoop.hdfs.TestSmallBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.887 sec - in 
org.apache.hadoop.hdfs.TestSmallBlock
Running org.apache.hadoop.hdfs.TestLeaseRecovery2
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 180.945 sec - 
in org.apache.hadoop.hdfs.TestLeaseRecovery2
Running org.apache.hadoop.hdfs.TestDFSMkdirs
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.653 sec - in 
org.apache.hadoop.hdfs.TestDFSMkdirs
Running org.apache.hadoop.hdfs.TestSetTimes
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.634 sec - in 
org.apache.hadoop.hdfs.TestSetTimes
Running org.apache.hadoop.hdfs.TestFileStatus
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.802 sec - in 
org.apache.hadoop.hdfs.TestFileStatus
Running org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.662 sec - in 
org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
Running org.apache.hadoop.hdfs.TestRollingUpgradeRollback
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.456 sec - in 
org.apache.hadoop.hdfs.TestRollingUpgradeRollback
Running org.apache.hadoop.hdfs.TestDFSStartupVersions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.792 sec - in 
org.apache.hadoop.hdfs.TestDFSStartupVersions
Running org.apache.hadoop.hdfs.TestDFSShellGenericOptions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.563 sec - in 
org.apache.hadoop.hdfs.TestDFSShellGenericOptions
Running org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.907 sec - in 
org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Running org.apache.hadoop.hdfs.TestLeaseRenewer
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.354 sec - in 
org.apache.hadoop.hdfs.TestLeaseRenewer
Running org.apache.hadoop.hdfs.TestDatanodeBlockScanner
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 158.008 sec - 
in org.apache.hadoop.hdfs.TestDatanodeBlockScanner
Running org.apache.hadoop.hdfs.TestDFSRemove
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.152 sec - in 
org.apache.hadoop.hdfs.TestDFSRemove
Running org.apache.hadoop.hdfs.TestFileAppend4
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.072 sec - in 
org.apache.hadoop.hdfs.TestFileAppend4
Running org.apache.hadoop.hdfs.TestParallelRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.189 sec - in 
org.apache.hadoop.hdfs.TestParallelRead
Running org.apache.hadoop.hdfs.TestClose
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.932 sec - in 
org.apache.hadoop.hdfs.TestClose
Running org.apache.hadoop.hdfs.TestDFSAddressConfig
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.214 sec - in 
org.apache.hadoop.hdfs.TestDFSAddressConfig
Running org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.156 sec - in 
org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
Running org.apache.hadoop.hdfs.TestLargeBlock
Tests run: 1, Failure

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #32

2014-12-08 Thread Apache Jenkins Server 
See 

Changes:

[kasha] YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent. 
(Ray Chiang via kasha)

[harsh] MAPREDUCE-6177. Minor typo in the EncryptedShuffle document about 
ssl-client.xml. Contributed by Yangping Wu. (harsh)

--
[...truncated 6185 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.833 sec - in 
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
Running org.apache.hadoop.hdfs.server.mover.TestStorageMover
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 186.136 sec - 
in org.apache.hadoop.hdfs.server.mover.TestStorageMover
Running org.apache.hadoop.hdfs.server.mover.TestMover
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 129.188 sec - 
in org.apache.hadoop.hdfs.server.mover.TestMover
Running org.apache.hadoop.hdfs.TestDatanodeReport
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.107 sec - in 
org.apache.hadoop.hdfs.TestDatanodeReport
Running org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.765 sec - in 
org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Running org.apache.hadoop.hdfs.TestEncryptedTransfer
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 107.181 sec - 
in org.apache.hadoop.hdfs.TestEncryptedTransfer
Running org.apache.hadoop.hdfs.TestPipelines
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.214 sec - in 
org.apache.hadoop.hdfs.TestPipelines
Running org.apache.hadoop.hdfs.TestHttpPolicy
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.42 sec - in 
org.apache.hadoop.hdfs.TestHttpPolicy
Running org.apache.hadoop.hdfs.TestLeaseRenewer
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.177 sec - in 
org.apache.hadoop.hdfs.TestLeaseRenewer
Running org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.7 sec - in 
org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
Running org.apache.hadoop.hdfs.TestWriteRead
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.275 sec - in 
org.apache.hadoop.hdfs.TestWriteRead
Running org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.497 sec - in 
org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
Running org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.258 sec - in 
org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
Running org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.107 sec - in 
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
Running org.apache.hadoop.hdfs.TestPersistBlocks
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.338 sec - in 
org.apache.hadoop.hdfs.TestPersistBlocks
Running org.apache.hadoop.hdfs.TestFSInputChecker
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.986 sec - in 
org.apache.hadoop.hdfs.TestFSInputChecker
Running org.apache.hadoop.fs.TestFcHdfsSetUMask
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.224 sec - in 
org.apache.hadoop.fs.TestFcHdfsSetUMask
Running org.apache.hadoop.fs.TestFcHdfsPermission
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.015 sec - in 
org.apache.hadoop.fs.TestFcHdfsPermission
Running org.apache.hadoop.fs.TestGlobPaths
Tests run: 35, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 4.149 sec - in 
org.apache.hadoop.fs.TestGlobPaths
Running org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.384 sec - in 
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
Running org.apache.hadoop.fs.TestSymlinkHdfsDisable
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.749 sec - in 
org.apache.hadoop.fs.TestSymlinkHdfsDisable
Running org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
Tests run: 72, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 7.885 sec - in 
org.apache.hadoop.fs.TestSymlinkHdfsFileSystem
Running org.apache.hadoop.fs.TestUrlStreamHandler
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.128 sec - in 
org.apache.hadoop.fs.TestUrlStreamHandler
Running org.apache.hadoop.fs.TestXAttr
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.067 sec - in 
org.apache.hadoop.fs.TestXAttr
Running org.apache.hadoop.fs.TestFcHdfsCreateMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.989 sec - in 
org.apache.hadoop.fs.TestFcHdfsCreateMkdir
Running org.apache.hadoop.fs.TestEnhancedByteBufferAccess
Tests run: 10, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 14.674 sec - 
in org.apache.hadoop.fs.TestEnhancedByteBufferAccess
Running org.apache.hadoop.fs

Hadoop-Hdfs-trunk-Java8 - Build # 32 - Still Failing

2014-12-08 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/32/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 6378 lines...]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE [  02:41 h]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  1.649 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:41 h
[INFO] Finished at: 2014-12-08T14:15:39+00:00
[INFO] Final Memory: 48M/227M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk-Java8/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating MAPREDUCE-6177
Updating YARN-1492
Updating YARN-2927
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
22 tests failed.
REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.testDecommissionDeadDN

Error Message:
Problem binding to [localhost:46782] java.net.BindException: Address already in 
use; For more details see:  http://wiki.apache.org/hadoop/BindException

Stack Trace:
java.net.BindException: Problem binding to [localhost:46782] 
java.net.BindException: Address already in use; For more details see:  
http://wiki.apache.org/hadoop/BindException
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:414)
at sun.nio.ch.Net.bind(Net.java:406)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.apache.hadoop.ipc.Server.bind(Server.java:410)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:576)
at org.apache.hadoop.ipc.Server.(Server.java:2291)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:935)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:780)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.initIpcServer(DataNode.java:722)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1083)
at 
org.apache.hadoop.hdfs.server.datanode.Data

Re: Switching to Java 7

2014-12-08 Thread Ted Yu 
Looks like there was still OutOfMemoryError :

https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/

FYI

On Mon, Dec 8, 2014 at 2:42 AM, Steve Loughran 
wrote:

> yes, bumped them up to
>
> export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> export ANT_OPTS=$MAVEN_OPTS
>
> also extended test runs times.
>
>
>
> On 8 December 2014 at 00:58, Ted Yu  wrote:
>
> > Looking at the test failures of
> > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/ which uses jdk
> 1.7:
> >
> > e.g.
> >
> >
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1963/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameFileAndDeleteSnapshot/
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at
> sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
> > at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
> > at
> >
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
> > at
> > io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
> > at
> io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
> > at
> >
> io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
> > at
> >
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
> >
> >
> > Should more heap be given to the tests ?
> >
> >
> > Cheers
> >
> >
> > On Sun, Dec 7, 2014 at 2:09 PM, Steve Loughran 
> > wrote:
> >
> > > The latest migration status:
> > >
> > >   if the jenkins builds are happy then the patch will go in -I do that
> > > monday morning 10:00 UTC
> > >
> > > https://builds.apache.org/view/H-L/view/Hadoop/
> > >
> > > Getting jenkins to work has been "surprisingly difficult"...it turns
> out
> > > that those builds which we thought were java7 or java8 weren't, as
> > setting
> > >   export JAVA_HOME=${TOOLS_HOME}/java/latest
> > >
> > > meant that they picked up a java 6 machine
> > >
> > > Now the trunk precommit/postcommit and scheduled branches should have
> > > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.7.0_55
> > >
> > > the Java 8 builds have more changes
> > >
> > > export JAVA_HOME=${TOOLS_HOME}/java/jdk1.8.0
> > > export MAVEN_OPTS="-Xmx3072m -XX:MaxPermSize=768m"
> > > and  -Dmaven.javadoc.skip=true  on the mvn builds
> > >
> > > without these javadocs fails and test runs OOM.
> > >
> > > We need to have something resembling the nightly build env setup again,
> > > git/Svn stored file with something for java8 alongside the normal env
> > vars.
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Hadoop-Hdfs-trunk - Build # 1965 - Still Failing

2014-12-08 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1965/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 6326 lines...]
[INFO] Deleting 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE [  02:39 h]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  2.292 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:39 h
[INFO] Finished at: 2014-12-08T15:24:34+00:00
[INFO] Final Memory: 52M/788M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating MAPREDUCE-6177
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
35 tests failed.
FAILED:  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots.testRenameFileAndDeleteSnapshot

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:120)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:68)
at 
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)
at io.netty.channel.nio.NioEventLoop.(NioEventLoop.java:120)
at 
io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:87)
at 
io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
at 
io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
at 
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:61)
at 
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:52)
at 
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:44)
at 
io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
at 
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:75)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:680)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1072)
at 
org.a

Build failed in Jenkins: Hadoop-Hdfs-trunk #1965

2014-12-08 Thread Apache Jenkins Server 
See 

Changes:

[harsh] MAPREDUCE-6177. Minor typo in the EncryptedShuffle document about 
ssl-client.xml. Contributed by Yangping Wu. (harsh)

--
[...truncated 6133 lines...]
Running org.apache.hadoop.hdfs.TestReservedRawPaths
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.674 sec - in 
org.apache.hadoop.hdfs.TestReservedRawPaths
Running org.apache.hadoop.hdfs.TestLeaseRecovery
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.277 sec - in 
org.apache.hadoop.hdfs.TestLeaseRecovery
Running org.apache.hadoop.hdfs.TestEncryptionZones
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 36.285 sec - 
in org.apache.hadoop.hdfs.TestEncryptionZones
Running org.apache.hadoop.hdfs.TestSmallBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.988 sec - in 
org.apache.hadoop.hdfs.TestSmallBlock
Running org.apache.hadoop.hdfs.TestLeaseRecovery2
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 179.968 sec - 
in org.apache.hadoop.hdfs.TestLeaseRecovery2
Running org.apache.hadoop.hdfs.TestDFSMkdirs
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.847 sec - in 
org.apache.hadoop.hdfs.TestDFSMkdirs
Running org.apache.hadoop.hdfs.TestSetTimes
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.804 sec - in 
org.apache.hadoop.hdfs.TestSetTimes
Running org.apache.hadoop.hdfs.TestFileStatus
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.038 sec - in 
org.apache.hadoop.hdfs.TestFileStatus
Running org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.97 sec - in 
org.apache.hadoop.hdfs.TestEncryptionZonesWithHA
Running org.apache.hadoop.hdfs.TestRollingUpgradeRollback
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.212 sec - in 
org.apache.hadoop.hdfs.TestRollingUpgradeRollback
Running org.apache.hadoop.hdfs.TestDFSStartupVersions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.146 sec - in 
org.apache.hadoop.hdfs.TestDFSStartupVersions
Running org.apache.hadoop.hdfs.TestDFSShellGenericOptions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.582 sec - in 
org.apache.hadoop.hdfs.TestDFSShellGenericOptions
Running org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.945 sec - in 
org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Running org.apache.hadoop.hdfs.TestLeaseRenewer
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.4 sec - in 
org.apache.hadoop.hdfs.TestLeaseRenewer
Running org.apache.hadoop.hdfs.TestDatanodeBlockScanner
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 158.319 sec - 
in org.apache.hadoop.hdfs.TestDatanodeBlockScanner
Running org.apache.hadoop.hdfs.TestDFSRemove
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.114 sec - in 
org.apache.hadoop.hdfs.TestDFSRemove
Running org.apache.hadoop.hdfs.TestFileAppend4
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.112 sec - in 
org.apache.hadoop.hdfs.TestFileAppend4
Running org.apache.hadoop.hdfs.TestParallelRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.017 sec - in 
org.apache.hadoop.hdfs.TestParallelRead
Running org.apache.hadoop.hdfs.TestClose
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.076 sec - in 
org.apache.hadoop.hdfs.TestClose
Running org.apache.hadoop.hdfs.TestDFSAddressConfig
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.38 sec - in 
org.apache.hadoop.hdfs.TestDFSAddressConfig
Running org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.774 sec - in 
org.apache.hadoop.hdfs.TestParallelShortCircuitLegacyRead
Running org.apache.hadoop.hdfs.TestLargeBlock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.181 sec - in 
org.apache.hadoop.hdfs.TestLargeBlock
Running org.apache.hadoop.hdfs.TestHDFSTrash
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.688 sec - in 
org.apache.hadoop.hdfs.TestHDFSTrash
Running org.apache.hadoop.hdfs.TestClientReportBadBlock
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.558 sec - in 
org.apache.hadoop.hdfs.TestClientReportBadBlock
Running org.apache.hadoop.hdfs.TestWriteRead
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.854 sec - in 
org.apache.hadoop.hdfs.TestWriteRead
Running org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.132 sec - in 
org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
Running org.apache.hadoop.hdfs.TestBalancerBandwidth
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.792 sec - in 
org.apache.hadoop.hdfs.TestBalancerBandwidt

Re: Switching to Java 7

2014-12-08 Thread Steve Loughran 
On 8 December 2014 at 14:58, Ted Yu  wrote:

> Looks like there was still OutOfMemoryError :
>
>
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/
>

Well, I'm going to ignore that for now as it's a java 8 problem, surfacing
this weekend once the builds were actually switched to Java 8. memory size
tuning can continue.

I have now committed the Java 7+ only patch to branch-2 and up: new code
does not have to worry about java 6 compatibility unless they plan to
backport to Java 2.6 or earlier. Having written some Java 7 code, the <>
constructor for typed classes are a convenience, the multiple-catch entries
more useful, as they eliminate duplicate code in exception handling.

Getting this patch in has revealed that the Jenkins builds of hadoop are
(a) a bit of a mess and (b) prone to race conditions related to the m2
repository if >1 project builds simultaneously. The way the nightly builds
are staggered means this doesn't usually surface, but it may show up during
precommit/postcommit builds.

The switch to Java 7 as the underlying JDK appears to be triggering
failures, these are things that the projects themselves are going to have
to look at.


This then, is where we are with builds right now. This is not a consequence
of the changes to the POM; this list predates that patch. This is Jenkins
running Hadoop builds and tests with Java 7u55


*Working: *

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/

*failing tests*

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-2-Build/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk/
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/

*failing tests on Java 8 (may include OOM)*

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/


*failing with maven internal dependency problems*

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/


*failing even though it appears to work in the logs*

https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-trunk/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-7488) HDFS Windows CIFS Gateway

2014-12-08 Thread Hari Sekhon (JIRA) 
Hari Sekhon created HDFS-7488:
-

 Summary: HDFS Windows CIFS Gateway
 Key: HDFS-7488
 URL: https://issues.apache.org/jira/browse/HDFS-7488
 Project: Hadoop HDFS
  Issue Type: New Feature
Affects Versions: 2.4.0
 Environment: HDP 2.1
Reporter: Hari Sekhon


Stakeholders are pressuring for native Windows file share access to our Hadoop 
clusters.

I've used NFS gateway several times and while it's theoretically viable for 
users now UID mapping is implemented in 2.5... insecure NFS makes our fully 
Kerberized clusters security pointless.

We really need CIFS gateway access to enforce authentication which NFSv3 
doesn't (NFSv4?).

I've even tried Samba over NFS gateway loopback mount point (don't laugh - they 
want it that badly), and enabled hdfs atime precision to an hour to prevent 
FSNamesystem.setTimes() java exceptions in gw logs, but the NFS server still 
doesn't like the Windows CIFS client actions:

{code}2014-12-08 16:31:38,053 ERROR nfs3.RpcProgramNfs3 
(RpcProgramNfs3.java:setattr(346)) - Setting file size is not supported when 
setattr, fileId: 25597
2014-12-08 16:31:38,065 INFO  nfs3.WriteManager 
(WriteManager.java:handleWrite(136)) - No opened stream for fileId:25597
2014-12-08 16:31:38,122 INFO  nfs3.OpenFileCtx 
(OpenFileCtx.java:receivedNewWriteInternal(624)) - Have to change stable write 
to unstable write:FILE_SYNC
{code}

A debug of the Samba server shows it's trying to set metadata timestamps which 
hangs indefinitely, resulting in the creation of a zero byte file when trying 
to copy a file in to HDFS /tmp via the Windows mapped drive.

{code}
...
 smb_set_file_time: setting utimes to modified values.
file_ntime: actime: Thu Jan  1 01:00:00 1970
file_ntime: modtime: Mon Dec  8 16:31:38 2014
file_ntime: ctime: Thu Jan  1 01:00:00 1970
file_ntime: createtime: Thu Jan  1 01:00:00 1970
{code}

This is the traceback from NFS gw log when hdfs precision was set to 0:

{code}org.apache.hadoop.ipc.RemoteException(java.io.IOException): Access time 
for hdfs is not configured.  Please set dfs.namenode.accesstime.precision 
configuration parameter.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:1960)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setTimes(NameNodeRpcServer.java:950)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:833)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
...
{code}

Regards,

Hari Sekhon
http://www.linkedin.com/in/harisekhon



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7489) Slow FsVolumeList checkDirs can hang datanodes

2014-12-08 Thread Noah Lorang (JIRA) 
Noah Lorang created HDFS-7489:
-

 Summary: Slow FsVolumeList checkDirs can hang datanodes
 Key: HDFS-7489
 URL: https://issues.apache.org/jira/browse/HDFS-7489
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Affects Versions: 2.6.0, 2.5.0
Reporter: Noah Lorang


Starting after upgrading to 2.5.0 (CDH 5.2.1), we started to see datanodes 
hanging their heartbeat and requests from clients. After some digging, I 
identified the culprit as being the checkDiskError() triggered by catching 
IOExceptions (in our case, SocketExceptions being triggered on one datanode by 
ReplicaAlreadyExistsExceptions on another datanode).

Thread dumps reveal that the checkDiskErrors() thread is holding a lock on the 
FsVolumeList:
{code}
"Thread-409" daemon prio=10 tid=0x7f4e50200800 nid=0x5b8e runnable 
[0x7f4e2f855000]
   java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.list(Native Method)
at java.io.File.list(File.java:973)
at java.io.File.listFiles(File.java:1051)
at org.apache.hadoop.util.DiskChecker.checkDirs(DiskChecker.java:89)
at org.apache.hadoop.util.DiskChecker.checkDirs(DiskChecker.java:91)
at org.apache.hadoop.util.DiskChecker.checkDirs(DiskChecker.java:91)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.checkDirs(BlockPoolSlice.java:257)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.checkDirs(FsVolumeImpl.java:210)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.checkDirs(FsVolumeList.java:180)
- locked <0x00063b182ea0> (a 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.checkDataDir(FsDatasetImpl.java:1396)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode$5.run(DataNode.java:2832)
at java.lang.Thread.run(Thread.java:662)
{code}

Other things would then lock the FsDatasetImpl while waiting for the 
FsVolumeList, e.g.:

{code}
"DataXceiver for client  at /10.10.0.52:46643 [Receiving block 
BP-1573746465-127.0.1.1-1352244533715:blk_1073770670_106962574]" daemon 
prio=10 tid=0x7f4e55561000 nid=0x406d waiting for monitor entry 
[0x7f4e3106d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.getNextVolume(FsVolumeList.java:64)
- waiting to lock <0x00063b182ea0> (a 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:927)
- locked <0x00063b1f9a48> (a 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:101)
at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:167)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:604)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:126)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:72)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:225)
at java.lang.Thread.run(Thread.java:662)
{code}

That lock on the FsDatasetImpl then causes other threads to block:

{code}
"Thread-127" daemon prio=10 tid=0x7f4e4c67d800 nid=0x2e02 waiting for 
monitor entry [0x7f4e3339]
   java.lang.Thread.State: BLOCKED (on object monitor)
at 
org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:228)
- waiting to lock <0x00063b1f9a48> (a 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
at 
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:436)
at 
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:523)
at 
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:684)
at 
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:650)
at 
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)

"DataNode: [[[DISK]file:/data/0/dfs/dn/, [DISK]file:/data/1/dfs/dn/, 
[DISK]file:/data/2/dfs/dn/, [DISK]file:/data/3/dfs/dn/, 
[DISK]file:/data/4/dfs/dn/, [DISK]file:/data/5/dfs/dn/, 
[DISK]file:/data/6/dfs/dn/, [DISK]file:/data/7/dfs/dn/, 
[DISK]file:/data/8/dfs/dn/, [DISK]file:/data/9/dfs/dn/, 
[DISK]file:/data/10/dfs/dn/, [DISK]file:/data/11/dfs/dn/, 
[DISK]file:/data/12/dfs/dn/, [DISK]file:/data/13/df

Re: Thinking ahead to hadoop-2.7

2014-12-08 Thread Colin McCabe 
On Fri, Dec 5, 2014 at 11:15 AM, Karthik Kambatla  wrote:
> It would be nice to cut the branch for the next "feature" release (not just
> Java 7) in the first week of January, so we can get the RC out by the end
> of the month?
>
> Yesterday, this came up in an offline discussion on ATS. Given people can
> run 2.6 on Java 7, is there merit to doing 2.7 with the exact same bits
> targeting Java 7? I am okay with going through with it, as long as it
> doesn't delay the next feature release.
>
> Thoughts?

That's a good point.  I think it's important to point out that most of
our users are already on JDK7.  We shouldn't think of this decision as
adding support for something new, we should think about it as taking
something away (JDK6 support).  I think it's good that we are finally
moving away from supporting JDK6, but I'm not completely sure that we
need to devote a whole release to that.  Are there a lot of open JDK7
issues that would require a release to straighten out?

best,
Colin


>
> On Wed, Dec 3, 2014 at 8:59 AM, Sangjin Lee  wrote:
>
>> Late January sounds fine to me. I think we should be able to wrap it up
>> much earlier than that (hopefully).
>>
>> Thanks,
>> Sangjin
>>
>> On Tue, Dec 2, 2014 at 5:19 PM, Arun C Murthy  wrote:
>>
>> > Sangjin/Karthik,
>> >
>> >  How about planning on hadoop-2.8 by late Jan? Thoughts?
>> >
>> > thanks,
>> > Arun
>> >
>> > On Dec 2, 2014, at 11:09 AM, Sangjin Lee  wrote:
>> >
>> > > If 2.7 is being positioned as the JDK7-only release, then it would be
>> > good
>> > > to know how 2.8 lines up in terms of timing. Our interest is landing
>> the
>> > > shared cache feature (YARN-1492)... Thanks.
>> > >
>> > > Sangjin
>> > >
>> > > On Mon, Dec 1, 2014 at 2:55 PM, Karthik Kambatla 
>> > wrote:
>> > >
>> > >> Thanks for starting this thread, Arun.
>> > >>
>> > >> Your proposal seems reasonable to me. I suppose we would like new
>> > features
>> > >> and improvements to go into 2.8 then? If yes, what time frame are we
>> > >> looking at for 2.8? Looking at YARN, it would be nice to get a release
>> > with
>> > >> shared-cache and a stable version of reservation work. I believe they
>> > are
>> > >> well under way and should be ready in a few weeks.
>> > >>
>> > >> Regarding 2.7 release specifics, do you plan to create a branch off of
>> > >> current branch-2.6 and update all issues marked fixed for 2.7 to be
>> > fixed
>> > >> for 2.8?
>> > >>
>> > >> Thanks
>> > >> Karthik
>> > >>
>> > >> On Mon, Dec 1, 2014 at 2:42 PM, Arun Murthy 
>> > wrote:
>> > >>
>> > >>> Folks,
>> > >>>
>> > >>> With hadoop-2.6 out it's time to think ahead.
>> > >>>
>> > >>> As we've discussed in the past, 2.6 was the last release which
>> supports
>> > >>> JDK6.
>> > >>>
>> > >>> I'm thinking it's best to try get 2.7 out in a few weeks (maybe by
>> the
>> > >>> holidays) with just the switch to JDK7 (HADOOP-10530) and possibly
>> > >>> support for JDK-1.8 (as a runtime) via HADOOP-11090.
>> > >>>
>> > >>> This way we can start with the stable base of 2.6 and switch over to
>> > >>> JDK7 to allow our downstream projects to use either for a short time
>> > >>> (hadoop-2.6 or hadoop-2.7).
>> > >>>
>> > >>> I'll update the Roadmap wiki accordingly.
>> > >>>
>> > >>> Thoughts?
>> > >>>
>> > >>> thanks,
>> > >>> Arun
>> > >>>
>> > >>> --
>> > >>> CONFIDENTIALITY NOTICE
>> > >>> NOTICE: This message is intended for the use of the individual or
>> > entity
>> > >> to
>> > >>> which it is addressed and may contain information that is
>> confidential,
>> > >>> privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > >>> of this message is not the intended recipient, you are hereby
>> notified
>> > >> that
>> > >>> any printing, copying, dissemination, distribution, disclosure or
>> > >>> forwarding of this communication is strictly prohibited. If you have
>> > >>> received this communication in error, please contact the sender
>> > >> immediately
>> > >>> and delete it from your system. Thank You.
>> > >>>
>> > >>
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/hdp/
>> >
>> >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or entity
>> to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>
>
>
> --
> -- Karthik Kambatla, Software Engineer, Cloudera
> 
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es

Re: Switching to Java 7

2014-12-08 Thread Colin McCabe 
On Mon, Dec 8, 2014 at 7:46 AM, Steve Loughran  wrote:
> On 8 December 2014 at 14:58, Ted Yu  wrote:
>
>> Looks like there was still OutOfMemoryError :
>>
>>
>> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/
>>
>
> Well, I'm going to ignore that for now as it's a java 8 problem, surfacing
> this weekend once the builds were actually switched to Java 8. memory size
> tuning can continue.
>
> I have now committed the Java 7+ only patch to branch-2 and up: new code
> does not have to worry about java 6 compatibility unless they plan to
> backport to Java 2.6 or earlier. Having written some Java 7 code, the <>
> constructor for typed classes are a convenience, the multiple-catch entries
> more useful, as they eliminate duplicate code in exception handling.
>
> Getting this patch in has revealed that the Jenkins builds of hadoop are
> (a) a bit of a mess and (b) prone to race conditions related to the m2
> repository if >1 project builds simultaneously. The way the nightly builds
> are staggered means this doesn't usually surface, but it may show up during
> precommit/postcommit builds.

It would be nice if we could have a separate .m2 directory per test executor.

It seems like that would eliminate these race conditions once and for
all, at the cost of storing a few extra jars (proportional to the # of
simultaneous executors)

best,
Colin


>
> The switch to Java 7 as the underlying JDK appears to be triggering
> failures, these are things that the projects themselves are going to have
> to look at.
>
>
> This then, is where we are with builds right now. This is not a consequence
> of the changes to the POM; this list predates that patch. This is Jenkins
> running Hadoop builds and tests with Java 7u55
>
>
> *Working: *
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/
>
> *failing tests*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-2-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/
>
> *failing tests on Java 8 (may include OOM)*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/
>
>
> *failing with maven internal dependency problems*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/
>
>
> *failing even though it appears to work in the logs*
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-trunk/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

[jira] [Created] (HDFS-7490) HDFS tests OOM on Java7+

2014-12-08 Thread Steve Loughran (JIRA) 
Steve Loughran created HDFS-7490:


 Summary: HDFS tests OOM on Java7+
 Key: HDFS-7490
 URL: https://issues.apache.org/jira/browse/HDFS-7490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build, test
Affects Versions: 3.0.0
 Environment: Jenkins on Java 7+
Reporter: Steve Loughran
Assignee: Steve Loughran


The HDFS tests are running out of memory with the switch to Java7; HADOOP-11363 
covers the patch; this is in HDFS to force test it there



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7491) Add incremental blockreport latency to DN metrics

2014-12-08 Thread Ming Ma (JIRA) 
Ming Ma created HDFS-7491:
-

 Summary: Add incremental blockreport latency to DN metrics
 Key: HDFS-7491
 URL: https://issues.apache.org/jira/browse/HDFS-7491
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ming Ma
Priority: Minor


In a busy cluster, IBR processing could be delayed due to NN FSNamesystem lock 
and cause NN to throw NotReplicatedYetException to DFSClient and thus increase 
the overall application latency.

This will be taken care of when we address the NN FSNamesystem lock contention 
issue.

It is useful if we can provide IBR latency metrics from DN's point of view.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7492) If multiple threads call FsVolumeList#checkDirs at the same time, we should only do checkDirs once and give the results to all waiting threads

2014-12-08 Thread Colin Patrick McCabe (JIRA) 
Colin Patrick McCabe created HDFS-7492:
--

 Summary: If multiple threads call FsVolumeList#checkDirs at the 
same time, we should only do checkDirs once and give the results to all waiting 
threads
 Key: HDFS-7492
 URL: https://issues.apache.org/jira/browse/HDFS-7492
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Colin Patrick McCabe


checkDirs is called when we encounter certain I/O errors.  It's rare to get 
just a single I/O error... normally you start getting many errors when a disk 
is going bad.  For this reason, we shouldn't start a new checkDirs scan for 
each error.  Instead, if multiple threads call FsVolumeList#checkDirs at around 
the same time, we should only do checkDirs once and give the results to all the 
waiting threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7493) removedDst should be checked against null in finally block of FSDirRenameOp#unprotectedRenameTo()

2014-12-08 Thread Ted Yu (JIRA) 
Ted Yu created HDFS-7493:


 Summary: removedDst should be checked against null in finally 
block of FSDirRenameOp#unprotectedRenameTo()
 Key: HDFS-7493
 URL: https://issues.apache.org/jira/browse/HDFS-7493
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  removedDst = dstIIP.getLastINode();
  undoRemoveDst = true;
{code}
If removedDst is null, the following code in finally block may result in NPE:
{code}
if (dstParent.isDirectory() &&
dstParent.asDirectory().isWithSnapshot()) {
  dstParent.asDirectory().undoRename4DstParent(removedDst,
  dstIIP.getLatestSnapshotId());
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7494) Checking of closed in DFSInputStream#pread() should be protected by synchronization

2014-12-08 Thread Ted Yu (JIRA) 
Ted Yu created HDFS-7494:


 Summary: Checking of closed in DFSInputStream#pread() should be 
protected by synchronization
 Key: HDFS-7494
 URL: https://issues.apache.org/jira/browse/HDFS-7494
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  private int pread(long position, byte[] buffer, int offset, int length)
  throws IOException {
// sanity checks
dfsClient.checkOpen();
if (closed) {
{code}
Checking of closed should be protected by holding lock on "DFSInputStream.this"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7495) Lock inversion in DFSInputStream#getBlockAt()

2014-12-08 Thread Ted Yu (JIRA) 
Ted Yu created HDFS-7495:


 Summary: Lock inversion in DFSInputStream#getBlockAt()
 Key: HDFS-7495
 URL: https://issues.apache.org/jira/browse/HDFS-7495
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


There're two locks: one on DFSInputStream.this , one on DFSInputStream.infoLock
Normally lock is obtained on infoLock, then on DFSInputStream.infoLock

However, such order is not observed in DFSInputStream#getBlockAt() :
{code}
synchronized(infoLock) {
...
  if (updatePosition) {
// synchronized not strictly needed, since we only get here
// from synchronized caller methods
synchronized(this) {
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7496) Fix FsVolume removal race conditions on the DataNode

2014-12-08 Thread Colin Patrick McCabe (JIRA) 
Colin Patrick McCabe created HDFS-7496:
--

 Summary: Fix FsVolume removal race conditions on the DataNode 
 Key: HDFS-7496
 URL: https://issues.apache.org/jira/browse/HDFS-7496
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe


We discussed a few FsVolume removal race conditions on the DataNode in 
HDFS-7489.  We should figure out a way to make removing an FsVolume safe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Switching to Java 7

2014-12-08 Thread Steve Loughran 
On 8 December 2014 at 19:58, Colin McCabe  wrote:

> It would be nice if we could have a separate .m2 directory per test
> executor.
>
> It seems like that would eliminate these race conditions once and for
> all, at the cost of storing a few extra jars (proportional to the # of
> simultaneous executors)
>
> best,
> Colin
>

all we should really need to do is to force a unique build ID on every
build so that it's not the 3.0.0-SNAPSHOT that is needed. We'd have the
problem of repository-cruft-bloat instead.

Returning to the build, hadoop-hdfs is going OOM on test runs even on java
7; there is a patch which should fix this

https://issues.apache.org/jira/browse/HADOOP-11363

Jenkins is down right now & can't verify —could someone have a look @ this
and apply if they are happy.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Thinking ahead to hadoop-2.7

2014-12-08 Thread Steve Loughran 
On 8 December 2014 at 19:48, Colin McCabe  wrote:

> Are there a lot of open JDK7
> issues that would require a release to straighten out?
>

I don't think so —the 2.6 release was tested pretty aggressively on JDK7,
and solely on it for Windows. Pushing out a 2.7 release would be more
symbolic than of use to people

there's one exception: the fix for java 8 security. I do also think there
may be Java 8 test failures; these need to be identified and fixed ...
which would point more to a 2.8 release for that

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-7497) Inconsistent report of decommissioning DataNodes between dfsadmin and NameNode webui

2014-12-08 Thread Yongjun Zhang (JIRA) 
Yongjun Zhang created HDFS-7497:
---

 Summary: Inconsistent report of decommissioning DataNodes between 
dfsadmin and NameNode webui
 Key: HDFS-7497
 URL: https://issues.apache.org/jira/browse/HDFS-7497
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, namenode
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang


It's observed that dfsadmin report list DNs in the decomm state while NN UI 
list DNs in dead state.

I found what happens is:

NN webui uses two steps to get the result:

* first collect a list of all alive DNs, 
* traverse through all live  DNs to find decommissioning DNs. 

It calls the following method to decide whether a DN is dead or alive:
{code}
  /** Is the datanode dead? */
  boolean isDatanodeDead(DatanodeDescriptor node) {
return (node.getLastUpdate() <
(Time.now() - heartbeatExpireInterval));
  }
{code}

On the other hand, dfsadmin traverse all DNs to find to all decommissioning DNs 
(check whether a DN is in {{AdminStates.DECOMMISSION_INPROGRESS}} state), 
without checking whether a DN is dead or alive like above.

The problem is, when a DN is determined to be dead, its state may still be 
{{AdminStates.DECOMMISSION_INPROGRESS}} .










--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Upgrading findbugs

2014-12-08 Thread Haohui Mai 
Hi,

The recent changes on moving to Java 7 triggers a bug in findbug (
http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
runs (e.g., HADOOP-11287) to fail.

The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
Given that:

(1) The current bug that we hit are fixed by a later version of findbug.
(2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
against Java 8.
(3) Newer findbugs are capable of catching more bugs. :-)

Is it a good time to consider upgrading findbugs, which gives us better
tools on ensuring the quality of the code case?

I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
NPEs, resource leaks, and ignored exception which are indeed bugs and are
worthwhile to address.

However, one issue that needs to be considered is that how to deal with the
additional warnings reported by the newer findbugs without breaking the
Jenkins pre-commit runs.

Personally I can see three possible routes if we decide to upgrade findbugs:

(1) Fix all warnings before upgrading to newer findbugs.
(2) Add all new warnings to the exclude list and fix them slowly.
(3) Update test-patch.sh to make sure that new code won't introduce any new
findbugs warnings.

I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
accelerate the effort if it is required.

Thoughts?

Regards,
Haohui

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Upgrading findbugs

2014-12-08 Thread Karthik Kambatla 
Thanks for initiating this, Haohui. +1 to upgrading findbugs version.

Inline.

On Mon, Dec 8, 2014 at 9:57 PM, Haohui Mai  wrote:

> Hi,
>
> The recent changes on moving to Java 7 triggers a bug in findbug (
> http://sourceforge.net/p/findbugs/bugs/918), which causes all pre-commit
> runs (e.g., HADOOP-11287) to fail.
>
> The current version of findbugs (1.3.9) used by Hadoop is released in 2009.
> Given that:
>
> (1) The current bug that we hit are fixed by a later version of findbug.
> (2) A newer findbug (3.0.0) is required to analyze Hadoop that is compiled
> against Java 8.
> (3) Newer findbugs are capable of catching more bugs. :-)
>
> Is it a good time to consider upgrading findbugs, which gives us better
> tools on ensuring the quality of the code case?
>
> I ran findbugs 3.0.0 against trunk today. It reported 111 warnings for
> hadoop-common, 44 for HDFS and 40+ for YARN. Many of them are possible
> NPEs, resource leaks, and ignored exception which are indeed bugs and are
> worthwhile to address.
>
> However, one issue that needs to be considered is that how to deal with the
> additional warnings reported by the newer findbugs without breaking the
> Jenkins pre-commit runs.
>
> Personally I can see three possible routes if we decide to upgrade
> findbugs:
>
> (1) Fix all warnings before upgrading to newer findbugs.
>

This might take a while. We might want to use the newer findbugs sooner?


> (2) Add all new warnings to the exclude list and fix them slowly.
>

I have my doubts on how soon we fix these warnings unless we make the
associated JIRAs (assuming we have one per exclude) blockers for the next
release. A findbugs "Fix It" day would be ideal to get this done.


> (3) Update test-patch.sh to make sure that new code won't introduce any new
> findbugs warnings.
>

Seems the best, especially if test-patch.sh shows the warnings, but doesn't
-1 unless there are new findbugs warnings. This way, the contributor can
choose to fix "related" warnings at the least.


>
> I proposed upgrading to findbugs 2.0.2 and fixing new warnings in
> HADOOP-10476, which could be dated backed to April, 2014. I volunteer to
> accelerate the effort if it is required.


> Thoughts?
>
> Regards,
> Haohui
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Karthik Kambatla
Software Engineer, Cloudera Inc.

http://five.sentenc.es

[jira] [Created] (HDFS-7498) Simplify the logic in INodesInPath

2014-12-08 Thread Jing Zhao (JIRA) 
Jing Zhao created HDFS-7498:
---

 Summary: Simplify the logic in INodesInPath
 Key: HDFS-7498
 URL: https://issues.apache.org/jira/browse/HDFS-7498
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Jing Zhao
Assignee: Jing Zhao


Currently we have relatively complicated logic in INodesInPath:
1) It can contain null elements in its INode array, and in {{mkdirRecursively}} 
these null INodes are replaced with new directories.
2) Operations like rename may also replace the inode in its INode array
3) {{getINodes}} requires trimming inodes array if the INodesInPath is derived 
from a dot-snapshot path
4) A lot of methods directly use/manipulate its INode array

We aim to simplify the logic of INodesInPath in this jira. Specifically, we can
make INodesInPath an immutable data structure and move the inode trimming logic 
to path resolving.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)