Re: [VOTE] Release Apache Hadoop 2.5.0 RC2

[Steve Loughran]

+1 binding, same tests as before: purging mvn repo, slider test suite
including fault injection, followed by full test of S3N and openstack FS
contracts (because I know not enough people test those)


On 6 August 2014 21:59, Karthik Kambatla  wrote:

> Hi folks,
>
> I have put together a release candidate (rc2) for Hadoop 2.5.0.
>
> The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/
> The RC tag in svn is here:
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1009/
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the now usual 5
> days.
>
> Thanks
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-6833) DirectoryScanner should not register a deleting block with memory of DataNode

[Shinichi Yamashita (JIRA)]

Shinichi Yamashita created HDFS-6833:


 Summary: DirectoryScanner should not register a deleting block 
with memory of DataNode
 Key: HDFS-6833
 URL: https://issues.apache.org/jira/browse/HDFS-6833
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode
Affects Versions: 3.0.0
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita


When a block is deleted in DataNode, the following messages are usually output.

{code}
2014-08-07 17:53:11,606 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
 Scheduling blk_1073741825_1001 file 
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
 for deletion
2014-08-07 17:53:11,617 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
 Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file 
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
{code}

However, DirectoryScanner may be executed when DataNode deletes the block in 
the current implementation. And the following messsages are output.

{code}
2014-08-07 17:53:30,519 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
 Scheduling blk_1073741825_1001 file 
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
 for deletion
2014-08-07 17:53:31,426 INFO 
org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool 
BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata 
files:0, missing block files:0, missing blocks in memory:1, mismatched blocks:0
2014-08-07 17:53:31,426 WARN 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added 
missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
  getNumBytes() = 21230663
  getBytesOnDisk()  = 21230663
  getVisibleLength()= 21230663
  getVolume()   = /hadoop/data1/dfs/data/current
  getBlockFile()= 
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
  unlinked  =false
2014-08-07 17:53:31,531 INFO 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
 Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file 
/hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
{code}

Deleting block information is registered in DataNode's memory.
And when DataNode sends a block report, NameNode receives wrong block 
information.

For example, when we execute recommission or change the number of replication, 
NameNode may delete the right block as "ExcessReplicate" by this problem.
And "Under-Replicated Blocks" and "Missing Blocks" occur.

When DataNode run DirectoryScanner, DataNode should not register a deleting 
block.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Build failed in Jenkins: Hadoop-Hdfs-trunk #1830

[Apache Jenkins Server]

See 

Changes:

[szetszwo] HDFS-6812. Remove addBlock and replaceBlock from DatanodeDescriptor.

[szetszwo] HDFS-6809. Move Balancer's inner classes MovedBlocks and Matcher as 
to standalone classes and separates KeyManager from NameNodeConnector.

[zjshen] YARN-2388. Fixed TestTimelineWebServices failure due to HADOOP-10791. 
Contributed by Zhijie Shen.

[kasha] YARN-2359. Application hangs when it fails to launch AM container. 
(Zhihai Xu via kasha)

[jing9] HDFS-6791. A block could remain under replicated if all of its replicas 
are on decommissioned nodes. Contributed by Ming Ma.

[jianhe] YARN-2374. Fixed TestDistributedShell#testDSShell failure due to 
hostname dismatch. Contributed by Varun Vasudev

[aw] HDFS-6517. Remove hadoop-metrics2.properties from hdfs project (Akira 
AJISAKA via aw)

[aw] HDFS-6517. Remove hadoop-metrics2.properties from hdfs project (Akira 
AJISAKA via aw)

--
[...truncated 13528 lines...]
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 sec - in 
org.apache.hadoop.hdfs.util.TestLightWeightLinkedSet
Running org.apache.hadoop.hdfs.util.TestXMLUtils
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.067 sec - in 
org.apache.hadoop.hdfs.util.TestXMLUtils
Running org.apache.hadoop.hdfs.util.TestBestEffortLongFile
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.216 sec - in 
org.apache.hadoop.hdfs.util.TestBestEffortLongFile
Running org.apache.hadoop.hdfs.util.TestLightWeightHashSet
Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.173 sec - in 
org.apache.hadoop.hdfs.util.TestLightWeightHashSet
Running org.apache.hadoop.hdfs.util.TestDiff
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.653 sec - in 
org.apache.hadoop.hdfs.util.TestDiff
Running org.apache.hadoop.hdfs.util.TestExactSizeInputStream
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.075 sec - in 
org.apache.hadoop.hdfs.util.TestExactSizeInputStream
Running org.apache.hadoop.hdfs.protocol.TestExtendedBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec - in 
org.apache.hadoop.hdfs.protocol.TestExtendedBlock
Running org.apache.hadoop.hdfs.protocol.TestAnnotations
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.126 sec - in 
org.apache.hadoop.hdfs.protocol.TestAnnotations
Running org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081 sec - in 
org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Running org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.96 sec - in 
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
Running org.apache.hadoop.hdfs.protocol.datatransfer.TestPacketReceiver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.364 sec - in 
org.apache.hadoop.hdfs.protocol.datatransfer.TestPacketReceiver
Running org.apache.hadoop.hdfs.TestDatanodeRegistration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.928 sec - in 
org.apache.hadoop.hdfs.TestDatanodeRegistration
Running org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.815 sec - in 
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
Running org.apache.hadoop.hdfs.TestReadWhileWriting
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.128 sec - in 
org.apache.hadoop.hdfs.TestReadWhileWriting
Running org.apache.hadoop.hdfs.TestDFSStartupVersions
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.666 sec - in 
org.apache.hadoop.hdfs.TestDFSStartupVersions
Running org.apache.hadoop.hdfs.TestDefaultNameNodePort
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.829 sec - in 
org.apache.hadoop.hdfs.TestDefaultNameNodePort
Running org.apache.hadoop.hdfs.TestFSInputChecker
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.642 sec - in 
org.apache.hadoop.hdfs.TestFSInputChecker
Running org.apache.hadoop.hdfs.TestDFSRollback
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.781 sec - in 
org.apache.hadoop.hdfs.TestDFSRollback
Running org.apache.hadoop.hdfs.TestSafeMode
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.143 sec - in 
org.apache.hadoop.hdfs.TestSafeMode
Running org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.491 sec - 
in org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
Running 
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.681 sec - in 
org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl
Running org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
T

Hadoop-Hdfs-trunk - Build # 1830 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1830/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 13721 lines...]
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE [  02:13 h]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [  2.253 s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 02:13 h
[INFO] Finished at: 2014-08-07T13:48:07+00:00
[INFO] Final Memory: 62M/882M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating YARN-2374
Updating HADOOP-10791
Updating HDFS-6791
Updating HDFS-6809
Updating YARN-2388
Updating HDFS-6812
Updating HDFS-6517
Updating YARN-2359
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.testEnd2End

Error Message:
Rebalancing expected avg utilization to become 0.2, but on datanode 
127.0.0.1:57515 it remains at 0.08 after more than 4 msec.

Stack Trace:
java.util.concurrent.TimeoutException: Rebalancing expected avg utilization to 
become 0.2, but on datanode 127.0.0.1:57515 it remains at 0.08 after more than 
4 msec.
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForBalancer(TestBalancer.java:321)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:580)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.integrationTest(TestBalancer.java:658)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.testEnd2End(TestBlockTokenWithDFS.java:592)

Re: [VOTE] Release Apache Hadoop 2.5.0 RC2

[Masatake Iwasaki]

+1 (non-binding)

+ verified MD5 for tarball and source tarball
+ built from source tarball
+ ran example jobs such as
  nutchindexing, wordcount, dfsioe, hivebench, kmeans, pagerank, bayes, sort, 
terasort
  with HiBench on the cluster with 3 slave nodes.


(8/6/14, 13:59), Karthik Kambatla wrote:

Hi folks,

I have put together a release candidate (rc2) for Hadoop 2.5.0.

The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/
The RC tag in svn is here:
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1009/

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the now usual 5
days.

Thanks

[jira] [Created] (HDFS-6834) Improve the configuration guidance in DFSClient when there are no Codec classes found in configs

[Uma Maheswara Rao G (JIRA)]

Uma Maheswara Rao G created HDFS-6834:
-

 Summary: Improve the configuration guidance in DFSClient when 
there are no Codec classes found in configs
 Key: HDFS-6834
 URL: https://issues.apache.org/jira/browse/HDFS-6834
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor


This is the comment in HADOOP-10886 from Andrew. 
 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6821) Atomicity of multi file operations

[Chris Nauroth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth resolved HDFS-6821.
-

Resolution: Won't Fix

Hi, [~samera].

Ideas similar to this have been proposed several times.  The consensus has 
always been that pushing a recursive operation all the way to the NameNode for 
atomicity would impact throughput too severely.  The implementation would 
require holding the write lock while updating every inode in a subtree.  During 
that time, all other RPC caller threads would block waiting for release of the 
write lock.  A finer-grained locking implementation would help mitigate this, 
but it wouldn't eliminate the problem completely.

It's typical behavior in many file systems that recursive operations are driven 
from user space, and the syscalls modify a single inode at a time.  HDFS isn't 
different in this respect.

I'm going to resolve this as won't fix.

> Atomicity of multi file operations
> --
>
> Key: HDFS-6821
> URL: https://issues.apache.org/jira/browse/HDFS-6821
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Samer Al-Kiswany
>Priority: Minor
>
> Looking how HDFS updates the log files in case of chmod –r or chown –r 
> operations. In these operations, HDFS name node seems to update each file 
> separately; consequently the strace of the operation looks as follows.
> append(edits)
> fsync(edits)
> append(edits)
> fsync(edits)
> ---
> append(edits)
> fsync(edits)
> append(edits)
> fsync(edits)
> If a crash happens in the middle of this operation (e.g. at the dashed line 
> in the trace), the system will end up with part of the files updates with the 
> new owner or permissions and part still with the old owner.
> Isn’t it better to log the whole operations (chown -r) as one entry in the 
> edit file?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6835) Archival Storage: Add a new API to set storage policy

[Tsz Wo Nicholas Sze (JIRA)]

Tsz Wo Nicholas Sze created HDFS-6835:
-

 Summary: Archival Storage: Add a new API to set storage policy
 Key: HDFS-6835
 URL: https://issues.apache.org/jira/browse/HDFS-6835
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: hdfs-client, namenode
Reporter: Tsz Wo Nicholas Sze
Assignee: Jing Zhao


The new data migration tool proposed HDFS-6801 will determine if the storage 
policy of files needs to be updated.  The tool needs a new API to set storage 
policy.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.5.0 RC2

[Andrew Wang]

+1 binding

* verified mds
* built tarball from source
* checked the CHANGES.txt files, they have dates
* ran apache-rat:check
* ran pseudo cluster from the resulting tarball, ran teragen, sort,
validate on uncached and cached files



On Thu, Aug 7, 2014 at 8:58 AM, Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> +1 (non-binding)
>
> + verified MD5 for tarball and source tarball
> + built from source tarball
> + ran example jobs such as
>   nutchindexing, wordcount, dfsioe, hivebench, kmeans, pagerank, bayes,
> sort, terasort
>   with HiBench on the cluster with 3 slave nodes.
>
>
>
> (8/6/14, 13:59), Karthik Kambatla wrote:
>
>> Hi folks,
>>
>> I have put together a release candidate (rc2) for Hadoop 2.5.0.
>>
>> The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/
>> The RC tag in svn is here:
>> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/
>> The maven artifacts are staged at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1009/
>>
>> You can find my public key at:
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>
>> Please try the release and vote. The vote will run for the now usual 5
>> days.
>>
>> Thanks
>>
>>
>