[jira] [Resolved] (HDFS-3942) Backport HDFS-3495: Update balancer policy for Network Topology with additional 'NodeGroup' layer

[Tsz Wo (Nicholas), SZE (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-3942.
--

   Resolution: Fixed
Fix Version/s: 1-win
   1.2.0
 Hadoop Flags: Reviewed

+1 on the new patch.

I have committed this.  Thanks Junping!

Also thanks Jing for testing the patch.

> Backport HDFS-3495: Update balancer policy for Network Topology with 
> additional 'NodeGroup' layer
> -
>
> Key: HDFS-3942
> URL: https://issues.apache.org/jira/browse/HDFS-3942
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: balancer
>Affects Versions: 1.0.0
>Reporter: Junping Du
>Assignee: Junping Du
> Fix For: 1.2.0, 1-win
>
> Attachments: HDFS-3942.patch, HDFS-3942-v2.patch, HDFS-3942-v3.patch, 
> HDFS-3942-v4.patch, HDFS-3942-v5.patch
>
>
> This is the backport work for HDFS-3495 and HDFS-4234.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-4317) Rear

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-4317:


 Summary: Rear
 Key: HDFS-4317
 URL: https://issues.apache.org/jira/browse/HDFS-4317
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


This is a subtask separated from HDFS-4103:
- Change Diff to support undo.
- Add createSnapshotCopy() for inode
- Reorganize the INode and its subclasses constructors and some other methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #465

[Apache Jenkins Server]

See 

--
[...truncated 12075 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Hadoop-Hdfs-trunk - Build # 1256 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1256/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 9754 lines...]
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.439 sec
Running org.apache.hadoop.hdfs.web.TestOffsetUrlInputStream
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.162 sec
Running org.apache.hadoop.hdfs.web.TestWebHDFS
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 296.443 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsUrl
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.075 sec
Running org.apache.hadoop.hdfs.web.TestJsonUtil
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.166 sec
Running org.apache.hadoop.hdfs.web.TestAuthFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.38 sec
Running org.apache.hadoop.hdfs.TestConnCache
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.608 sec
Running org.apache.hadoop.hdfs.TestDFSClientRetries
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 131.977 sec
Running org.apache.hadoop.hdfs.TestListPathServlet
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.142 sec
Running org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 118.404 sec
Running org.apache.hadoop.hdfs.TestFileCreationEmpty
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.019 sec
Running org.apache.hadoop.hdfs.TestSetrepIncreasing

Results :

Tests run: 1112, Failures: 0, Errors: 0, Skipped: 5

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE 
[1:38:53.906s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:38:54.678s
[INFO] Finished at: Sun Dec 16 13:14:18 UTC 2012
[INFO] Final Memory: 43M/399M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on 
project hadoop-hdfs: ExecutionException; nested exception is 
java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked 
VM terminated without saying properly goodbye. VM crash or System.exit called ? 
-> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Updating MAPREDUCE-4807
Updating HADOOP-9041
Updating HDFS-4275
Updating HDFS-4274
Updating MAPREDUCE-4809
Updating HADOOP-8561
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
No tests ran.

Build failed in Jenkins: Hadoop-Hdfs-trunk #1256

[Apache Jenkins Server]

See 

Changes:

[llu] Move to the right section in CHANGES.txt

[llu] Update CHANGES.txt separately for clean cherry-picks.

[llu] HADOOP-9041. FsUrlStreamHandlerFactory could cause an infinite loop in 
FileSystem initialization. (Yanbo Liang and Radim Kolar via llu)

[llu] HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in 
child hadoop client processes. (Yu Gao via llu)

[tucu] MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu)

[tucu] MAPREDUCE-4809. Change visibility of classes for pluggable sort changes. 
(masokan via tucu)

[suresh] HDFS-4275. MiniDFSCluster-based tests fail on Windows due to failure 
to delete test namenode directory. Contributed by Chris Nauroth.

[suresh] HDFS-4274. BlockPoolSliceScanner does not close verification log 
during shutdown. Contributed by Chris Nauroth.

--
[...truncated 9561 lines...]
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.02 sec
Running org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.083 sec
Running org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.064 sec
Running org.apache.hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.073 sec
Running org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.027 sec
Running org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.506 sec
Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.917 sec
Running org.apache.hadoop.hdfs.server.namenode.TestTransferFsImage
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.644 sec
Running org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.466 sec
Running org.apache.hadoop.hdfs.server.namenode.TestAllowFormat
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.966 sec
Running org.apache.hadoop.hdfs.server.namenode.TestEditLog
Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.29 sec
Running 
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.343 sec
Running org.apache.hadoop.hdfs.server.namenode.TestEditsDoubleBuffer
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.059 sec
Running org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.195 sec
Running org.apache.hadoop.hdfs.server.namenode.TestGetImageServlet
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.706 sec
Running org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.237 sec
Running org.apache.hadoop.hdfs.server.namenode.TestFSDirectory
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.829 sec
Running org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.894 sec
Running org.apache.hadoop.hdfs.server.namenode.TestFileLimit
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.75 sec
Running org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.225 sec
Running org.apache.hadoop.hdfs.server.namenode.TestNameCache
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.13 sec
Running org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.894 sec
Running org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourcePolicy
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.282 sec
Running org.apache.hadoop.hdfs.server.namenode.TestCorruptFilesJsp
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.878 sec
Running org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.9 sec
Running org.apache.hadoop.hdfs.server.namenode.TestFSImageStorageInspector
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.338 sec
Running org.apache.hadoop.hdfs.server.namenode.TestFsck
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.967 sec
Running org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.623 sec
Running org.apache.hadoop.hdfs.serv

[jira] [Created] (HDFS-4318) validateBlockMetadata reduce the success rate of block recover

[Li Junjun (JIRA)]

Li Junjun created HDFS-4318:
---

 Summary: validateBlockMetadata reduce the success rate of block 
recover
 Key: HDFS-4318
 URL: https://issues.apache.org/jira/browse/HDFS-4318
 Project: Hadoop HDFS
  Issue Type: Wish
  Components: datanode
Affects Versions: 1.0.1
Reporter: Li Junjun
Priority: Minor


some logs like "java.io.IOException: Block blk_3272028001529756059_11883841 
length is 20480 does not match block file length 21376 "
when recovery block

when datanode perform startBlockRecovery  it call validateBlockMetadata( in 
FSDataset.startBlockRecovery  ),
check the file lenth match the block's numBytes 

so , let us see how  block's numBytes was updated in datanode 
when write block in BlockReceiver.receivePacket , 
write->flush->setVisibleLength, that means 
it is  normal and reasonable  that the file length > the block's numBytes if 
write or flush throw exception . 
In startBlockRecovery( or other situation,to be check) 
we just need to guarantee  the file length < the block's numBytes never happens 
.


I suggest change the validateBlockMetadata , because it  reduced the success 
rate of block recover.

when you have a pipline , a->b->c  when a got error in network , b got error in 
write->flush , we can only 
count on c!



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4316) branch-trunk-win contains test code accidentally added during work on fixing tests on Windows

[Suresh Srinivas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HDFS-4316.
---

   Resolution: Fixed
Fix Version/s: trunk-win

Committed to branch-trunk-win. Thank you Chris for fixing this issue.

> branch-trunk-win contains test code accidentally added during work on fixing 
> tests on Windows
> -
>
> Key: HDFS-4316
> URL: https://issues.apache.org/jira/browse/HDFS-4316
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: test
>Affects Versions: trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: trunk-win
>
> Attachments: HDFS-4316-branch-trunk-win.1.patch
>
>
> branch-trunk-win currently contains some test code that was not meant to be 
> committed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4317) Change INode to support HDFS-4103

[Tsz Wo (Nicholas), SZE (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4317.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)
 Hadoop Flags: Reviewed

I have committed this.

> Change INode to support HDFS-4103
> -
>
> Key: HDFS-4317
> URL: https://issues.apache.org/jira/browse/HDFS-4317
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: Snapshot (HDFS-2802)
>
> Attachments: 4317.INodeDirectoryWithSnapshot.patch, 
> h4317_20121216b.patch, h4317_20121216.patch
>
>
> This is a subtask separated from HDFS-4103:
> - Change Diff to support undo.
> - Add createSnapshotCopy() for inode
> - Reorganize the INode and its subclasses constructors and some other methods.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira