Jenkins build is back to normal : Hadoop-Hdfs-trunk #1679

[Apache Jenkins Server]

See 

[jira] [Created] (HDFS-5983) TestSafeMode#testInitializeReplQueuesEarly fails

[Kihwal Lee (JIRA)]

Kihwal Lee created HDFS-5983:


 Summary: TestSafeMode#testInitializeReplQueuesEarly fails
 Key: HDFS-5983
 URL: https://issues.apache.org/jira/browse/HDFS-5983
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Kihwal Lee


It was seen from one of the precommit build of HDFS-5962.  The test case 
creates 15 blocks and then shuts down all datanodes. Then the namenode is 
restarted with a low safe block threshold and one datanode is restarted. The 
idea is that the initial block report from the restarted datanode will make the 
namenode leave the safemode and initialize the replication queues.

According to the log, the datanode reported 3 blocks, but slightly before that 
the namenode did repl queue init with 1 block.  I will attach the log.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5984) Fix TestEditLog and TestStandbyCheckpoints

[Jing Zhao (JIRA)]

Jing Zhao created HDFS-5984:
---

 Summary: Fix TestEditLog and TestStandbyCheckpoints
 Key: HDFS-5984
 URL: https://issues.apache.org/jira/browse/HDFS-5984
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-5984.000.patch

This jira aims to fix current test failures in TestEditLog, 
TestStandbyCheckpoints, and TestBookKeeperHACheckpoints. These failures are 
caused by changes in the NameNode side.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5984) Fix TestEditLog and TestStandbyCheckpoints

[Tsz Wo (Nicholas), SZE (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-5984.
--

   Resolution: Fixed
Fix Version/s: HDFS-5535 (Rolling upgrades)

I have committed this.  Thanks, Jing!

> Fix TestEditLog and TestStandbyCheckpoints
> --
>
> Key: HDFS-5984
> URL: https://issues.apache.org/jira/browse/HDFS-5984
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>Priority: Minor
> Fix For: HDFS-5535 (Rolling upgrades)
>
> Attachments: HDFS-5984.000.patch
>
>
> This jira aims to fix current test failures in TestEditLog, 
> TestStandbyCheckpoints, and TestBookKeeperHACheckpoints. These failures are 
> caused by changes in the NameNode side.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5985) SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException

[Jing Zhao (JIRA)]

Jing Zhao created HDFS-5985:
---

 Summary: SimulatedFSDataset#disableAndPurgeTrashStorage should not 
throw UnsupportedOperationException
 Key: HDFS-5985
 URL: https://issues.apache.org/jira/browse/HDFS-5985
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao


Instead, we can let the method do nothing. This can fix part of the failed unit 
tests in 
https://issues.apache.org/jira/browse/HDFS-5535?focusedCommentId=13906717&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13906717



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5985) SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee resolved HDFS-5985.
--

   Resolution: Fixed
Fix Version/s: HDFS-5535 (Rolling upgrades)
 Hadoop Flags: Reviewed

> SimulatedFSDataset#disableAndPurgeTrashStorage should not throw 
> UnsupportedOperationException
> -
>
> Key: HDFS-5985
> URL: https://issues.apache.org/jira/browse/HDFS-5985
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, ha, hdfs-client, namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
>Priority: Minor
> Fix For: HDFS-5535 (Rolling upgrades)
>
> Attachments: HDFS-5985.000.patch
>
>
> Instead, we can let the method do nothing. This can fix part of the failed 
> unit tests in 
> https://issues.apache.org/jira/browse/HDFS-5535?focusedCommentId=13906717&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13906717



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Thinking ahead to 2.4

[Suresh Srinivas]

Arun,

Some of the previously 2.4 targeted features were made available in 2.3:
- Heterogeneous storage support
- Datanode cache

The following are being targeted for 2.4:
- Use protobuf for fsimge (already in)
- ACLs (in trunk. In a week or so, this will be merged to branch-2.4)
- Rolling upgrades (last bunch of jiras being worked in feature branch.
Will be in 2.4 in around two weeks. Currently testing is in progress)

So HDFS features should be ready in two weeks.


On Sat, Feb 15, 2014 at 4:47 PM, Azuryy  wrote:

> Hi，
> I think you omit some key pieces in 2.4
>
> Protobuf fsimage, rolling upgrade are also targeting 2.4
>
>
>
> Sent from my iPhone5s
>
> > On 2014年2月16日, at 6:59, Arun C Murthy  wrote:
> >
> > Folks,
> >
> > With hadoop-2.3 nearly done, I think it's time to think ahead to
> hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we
> finished up pieces that didn't make it in such as HDFS Caching & Support
> for Heterogenous Storage.
> >
> > Now, most of the key pieces incl. Resource Manager Automatic Failover
> (YARN-149), Application History Server (YARN-321) & Application Timeline
> Server (YARN-1530) are either complete or very close to done, and I think
> we will benefit with an extended test-cycle for 2.4 - similar to what
> happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks
> of extended testing and it really helped us push out a very stable release.
> >
> > I think it will be good to create a 2.4 branch ASAP and start testing.
> As such, I plan to cut the branch early next week. With this, we should be
> good shape sometime to release 2.4 in mid-March.
> >
> > I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this.
> >
> > Also, we should start thinking ahead to 2.5 and what folks would like to
> see in it. If we continue our 6-week cycles, we could shoot to get that out
> in April.
> >
> > Thoughts?
> >
> > thanks,
> > Arun
> >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI

[Suresh Srinivas (JIRA)]

Suresh Srinivas created HDFS-5986:
-

 Summary: Capture the number of blocks pending deletion on namenode 
webUI
 Key: HDFS-5986
 URL: https://issues.apache.org/jira/browse/HDFS-5986
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Suresh Srinivas


When a directory that has large number of directories and files are deleted, 
the namespace deletes the corresponding inodes immediately. However it is hard 
to to know when the invalidated blocks are actually deleted on the datanodes, 
which could take a while.

I propose adding on namenode webUI, along with under replicated blocks, the 
number of blocks that are pending deletion.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5586) Add quick-restart option for datanode

[Kihwal Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee resolved HDFS-5586.
--

Resolution: Duplicate

Most of the planned changes will be covered after HDFS-5498. There are some 
missing, but I don't think it is critical at this point. To name a few for 
later reference,

- Quick registration with NN. When NN get a registration request from a 
datanode that isn't "dead" (i.e. restart), the blocks on the node will be 
removed from the blocksmap and readded when the initial block report is 
received. If DN isn't going to change its content significantly and the 
identity (storage ID) stays the same, NN may be better off keeping the block 
list for the DN and update it few minutes later when the block report is 
received.

- DN to persist more state so that it can start serving sooner. Even if a DN is 
up, it won't be able to serve clients before registering with NN, because it 
cannot verify the block token. Saving the shared secret is risky though.

The quick DN registration change will lower the DN restart overhead on NN, but 
reasonably paced DN rolling upgrades should still be acceptable even without 
this.  This will be more useful in the case where DNs are restarted en masse. 
So I will not call it a necessary improvement for rolling upgrades.

> Add quick-restart option for datanode
> -
>
> Key: HDFS-5586
> URL: https://issues.apache.org/jira/browse/HDFS-5586
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, ha, hdfs-client, namenode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>
> This feature, combined with the graceful shutdown feature, will enable data 
> nodes to come back up and start serving quickly.  This is likely a command 
> line option for data node, which triggers it to look for saved state 
> information in its local storage.  If the information is present and 
> reasonably up-to-date, data node would skip some of the startup steps.
> Ideally it should be able to do quick registration without requiring removal 
> of all blocks from the date node descriptor on the name node and 
> reconstructing it with the initial full block report. This implies that all 
> RBW blocks are recorded during shutdown and on start-up they are not turned 
> into RWR. Other than the quick registration, name node should treat the 
> restart as if few heart beats were lost from the node. There should be no 
> unexpected replica state changes.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5987) Fix findbugs warnings in Rolling Upgrade branch

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-5987:


 Summary: Fix findbugs warnings in Rolling Upgrade branch
 Key: HDFS-5987
 URL: https://issues.apache.org/jira/browse/HDFS-5987
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5977) FSImageFormatPBINode does not respect "-renameReserved" upgrade flag

[Haohui Mai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haohui Mai resolved HDFS-5977.
--

Resolution: Later

> FSImageFormatPBINode does not respect "-renameReserved" upgrade flag
> 
>
> Key: HDFS-5977
> URL: https://issues.apache.org/jira/browse/HDFS-5977
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.4.0
>Reporter: Andrew Wang
>  Labels: protobuf
>
> HDFS-5709 added a new upgrade flag "-renameReserved" which can be used to 
> automatically rename reserved paths like "/.reserved" encountered during 
> upgrade. The new protobuf loading code does not have a similar facility, so 
> future reserved paths cannot be automatically renamed via "-renameReserved".



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5988) Bad fsimage generated after upgrade

[Andrew Wang (JIRA)]

Andrew Wang created HDFS-5988:
-

 Summary: Bad fsimage generated after upgrade
 Key: HDFS-5988
 URL: https://issues.apache.org/jira/browse/HDFS-5988
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Blocker


Internal testing revealed an issue where, after upgrading from an earlier 
release, we always fail to save a correct PB-based fsimage (namely, missing 
inodes leading to an inconsistent namespace). This results in substantial data 
loss, since the upgraded fsimage is broken, as well as the fsimages generated 
by saveNamespace and checkpointing.

This ended up being a bug in the old fsimage loading code, patch coming.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5987) Fix findbugs warnings in Rolling Upgrade branch

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-5987.
-

  Resolution: Fixed
   Fix Version/s: HDFS-5535 (Rolling upgrades)
Target Version/s: HDFS-5535 (Rolling upgrades)
Hadoop Flags: Reviewed

+1 for the patch.

Thanks for fixing these Nicholas!

> Fix findbugs warnings in Rolling Upgrade branch
> ---
>
> Key: HDFS-5987
> URL: https://issues.apache.org/jira/browse/HDFS-5987
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, ha, hdfs-client, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Fix For: HDFS-5535 (Rolling upgrades)
>
> Attachments: h5987_20140220.patch
>
>
> {noformat}
> RV
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.restoreBlockFilesFromTrash(File)
>  ignores exceptional return value of java.io.File.mkdirs()
> RV
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.restoreBlockFilesFromTrash(File)
>  ignores exceptional return value of java.io.File.renameTo(File)
> RV
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$ReplicaFileDeleteTask.moveFiles()
>  ignores exceptional return value of java.io.File.mkdirs()
> ISInconsistent synchronization of 
> org.apache.hadoop.hdfs.qjournal.server.Journal.committedTxnId; locked 92% of 
> time
> NPDereference of the result of readLine() without nullcheck in 
> org.apache.hadoop.hdfs.util.MD5FileUtils.renameMD5File(File, File)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5989) merge of HDFS-4685 to trunk introduced trunk test failure

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HDFS-5989:
---

 Summary: merge of HDFS-4685 to trunk introduced trunk test failure
 Key: HDFS-5989
 URL: https://issues.apache.org/jira/browse/HDFS-5989
 Project: Hadoop HDFS
  Issue Type: Bug
 Environment: CentOS release 6.5 (Final)
cpe:/o:centos:linux:6:GA

Reporter: Yongjun Zhang


HI,

I'm seeing trunk branch test failure locally (centOs6) today. And I identified 
it's this commit that caused the failure. 

Author: Chris Nauroth   2014-02-19 10:34:52
Committer: Chris Nauroth   2014-02-19 10:34:52
Parent: 7215d12fdce727e1f4bce21a156b0505bd9ba72a (YARN-1666. Modified RM HA 
handling of include/exclude node-lists to be available across RM failover by 
making using of a remote configuration-provider. Contributed by Xuan Gong.)
Parent: 603ebb82b31e9300cfbf81ed5dd6110f1cb31b27 (HDFS-4685. Correct minor 
whitespace difference in FSImageSerialization.java in preparation for trunk 
merge.)
Child:  ef8a5bceb7f3ce34d08a5968777effd40e0b1d0f (YARN-1171. Add default queue 
properties to Fair Scheduler documentation (Naren Koneru via Sandy Ryza))
Branches: remotes/apache/HDFS-5535, remotes/apache/trunk, testv10, testv3, 
testv4, testv7
Follows: testv5
Precedes: 

Merge HDFS-4685 to trunk.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1569870 
13f79535-47bb-0310-9956-ffa450edef68


I'm not sure whether other folks are seeing the same, or maybe related to my 
environment. But prior to chis change, I don't see this problem.

The failures are in TestWebHDFS:

Running org.apache.hadoop.hdfs.web.TestWebHDFS
Tests run: 5, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 3.687 sec <<< 
FAILURE! - in org.apache.hadoop.hdfs.web.TestWebHDFS
testLargeDirectory(org.apache.hadoop.hdfs.web.TestWebHDFS)  Time elapsed: 2.478 
sec  <<< ERROR!
java.lang.IllegalArgumentException: length != 
10(unixSymbolicPermission=drwxrwxr-x.)
at 
org.apache.hadoop.fs.permission.FsPermission.valueOf(FsPermission.java:323)
at 
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:572)
at 
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:540)
at 
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:129)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:146)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:1835)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1877)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1859)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
at 
org.apache.hadoop.hdfs.web.TestWebHDFS.testLargeDirectory(TestWebHDFS.java:229)

testNamenodeRestart(org.apache.hadoop.hdfs.web.TestWebHDFS)  Time elapsed: 
0.342 sec  <<< ERROR!
java.lang.IllegalArgumentException: length != 
10(unixSymbolicPermission=drwxrwxr-x.)
at 
org.apache.hadoop.fs.permission.FsPermission.valueOf(FsPermission.java:323)
at 
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:572)
at 
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:540)
at 
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:129)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:146)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:1835)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:1877)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1859)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
at 
org.apache.hadoop.hdfs.TestDFSClientRetries.namenodeRestartTest(TestDFSClientRetries.java:886)
at 
org.apache.hadoop.hdfs.web.TestWebHDFS.testNamenode

[jira] [Created] (HDFS-5990) Create options to search files/dirs in OfflineImageViewer

[Akira AJISAKA (JIRA)]

Akira AJISAKA created HDFS-5990:
---

 Summary: Create options to search files/dirs in OfflineImageViewer
 Key: HDFS-5990
 URL: https://issues.apache.org/jira/browse/HDFS-5990
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Akira AJISAKA
Priority: Minor


The enhancement of HDFS-5975.
I suggest options to search files/dirs in OfflineImageViewer.
An example command is as follows:
{code}
hdfs oiv -i input -o output -p Ls -owner theuser -group supergroup -minSize 
1024 -maxSize 1048576
{code}




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Thinking ahead to 2.4

[Arun C Murthy]

Thanks Azuryy & Suresh. I've updated the roadmap wiki to reflect this.

Arun

On Feb 20, 2014, at 2:01 PM, Suresh Srinivas  wrote:

> Arun,
> 
> Some of the previously 2.4 targeted features were made available in 2.3:
> - Heterogeneous storage support
> - Datanode cache
> 
> The following are being targeted for 2.4:
> - Use protobuf for fsimge (already in)
> - ACLs (in trunk. In a week or so, this will be merged to branch-2.4)
> - Rolling upgrades (last bunch of jiras being worked in feature branch.
> Will be in 2.4 in around two weeks. Currently testing is in progress)
> 
> So HDFS features should be ready in two weeks.
> 
> 
> On Sat, Feb 15, 2014 at 4:47 PM, Azuryy  wrote:
> 
>> Hi，
>> I think you omit some key pieces in 2.4
>> 
>> Protobuf fsimage, rolling upgrade are also targeting 2.4
>> 
>> 
>> 
>> Sent from my iPhone5s
>> 
>>> On 2014年2月16日, at 6:59, Arun C Murthy  wrote:
>>> 
>>> Folks,
>>> 
>>> With hadoop-2.3 nearly done, I think it's time to think ahead to
>> hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we
>> finished up pieces that didn't make it in such as HDFS Caching & Support
>> for Heterogenous Storage.
>>> 
>>> Now, most of the key pieces incl. Resource Manager Automatic Failover
>> (YARN-149), Application History Server (YARN-321) & Application Timeline
>> Server (YARN-1530) are either complete or very close to done, and I think
>> we will benefit with an extended test-cycle for 2.4 - similar to what
>> happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks
>> of extended testing and it really helped us push out a very stable release.
>>> 
>>> I think it will be good to create a 2.4 branch ASAP and start testing.
>> As such, I plan to cut the branch early next week. With this, we should be
>> good shape sometime to release 2.4 in mid-March.
>>> 
>>> I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this.
>>> 
>>> Also, we should start thinking ahead to 2.5 and what folks would like to
>> see in it. If we continue our 6-week cycles, we could shoot to get that out
>> in April.
>>> 
>>> Thoughts?
>>> 
>>> thanks,
>>> Arun
>>> 
>>> 
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>> 
>>> 
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>> 
> 
> 
> 
> -- 
> http://hortonworks.com/download/
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.3.0

[Arun C Murthy]

With 22 +1s (7 binding) and no -1s the vote passes.

Thanks to all who tried and verified the release, I'll start the process to 
push the bits out.

Arun

On Feb 11, 2014, at 6:49 AM, Arun C Murthy  wrote:

> Folks,
> 
> I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to 
> get released.
> 
> The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
> The RC tag in svn is here: 
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0
> 
> The maven artifacts are available via repository.apache.org.
> 
> Please try the release and vote; the vote will run for the usual 7 days.
> 
> thanks,
> Arun
> 
> PS: Thanks to Andrew, Vinod & Alejandro for all their help in various release 
> activities.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-5991) TestLoadGenerator#testLoadGenerator fails on trunk

[Akira AJISAKA (JIRA)]

Akira AJISAKA created HDFS-5991:
---

 Summary: TestLoadGenerator#testLoadGenerator fails on trunk
 Key: HDFS-5991
 URL: https://issues.apache.org/jira/browse/HDFS-5991
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Akira AJISAKA


>From https://builds.apache.org/job/PreCommit-HDFS-Build/6194//testReport/

{code}
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:97)
at java.io.BufferedReader.readLine(BufferedReader.java:292)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at 
org.apache.hadoop.fs.loadGenerator.LoadGenerator.loadScriptFile(LoadGenerator.java:511)
at 
org.apache.hadoop.fs.loadGenerator.LoadGenerator.init(LoadGenerator.java:418)
at 
org.apache.hadoop.fs.loadGenerator.LoadGenerator.run(LoadGenerator.java:324)
at 
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator.testLoadGenerator(TestLoadGenerator.java:231)
{code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5992) Fix NPE in MD5FileUtils

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-5992:


 Summary: Fix NPE in MD5FileUtils
 Key: HDFS-5992
 URL: https://issues.apache.org/jira/browse/HDFS-5992
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


MD5FileUtils.readStoredMd5(File md5File)  may return null but the callers may 
not check it.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5993) org.apache.hadoop.fs.loadGenerator.TestLoadGenerator failure in trunk

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HDFS-5993:
---

 Summary: org.apache.hadoop.fs.loadGenerator.TestLoadGenerator 
failure in trunk
 Key: HDFS-5993
 URL: https://issues.apache.org/jira/browse/HDFS-5993
 Project: Hadoop HDFS
  Issue Type: Bug
 Environment: CentOS release 6.5 (Final)
cpe:/o:centos:linux:6:GA

Reporter: Yongjun Zhang


With today's latest trunk at
commit d926e51bdc27f08e916534567a1edcfd994e2784

When running it locally, I can consistently see the following test fails as:

---
 T E S T S
---

---
 T E S T S
---
Running org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.888 sec <<< 
FAILURE! - in org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
testLoadGenerator(org.apache.hadoop.fs.loadGenerator.TestLoadGenerator)  Time 
elapsed: 14.285 sec  <<< ERROR!
java.io.IOException: Stream closed
at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
at java.io.BufferedReader.readLine(BufferedReader.java:310)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at 
org.apache.hadoop.fs.loadGenerator.LoadGenerator.loadScriptFile(LoadGenerator.java:511)
at 
org.apache.hadoop.fs.loadGenerator.LoadGenerator.init(LoadGenerator.java:418)
at 
org.apache.hadoop.fs.loadGenerator.LoadGenerator.run(LoadGenerator.java:324)
at 
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator.testLoadGenerator(TestLoadGenerator.java:231)

Results :

Tests in error:
  TestLoadGenerator.testLoadGenerator:231 » IO Stream closed

This failure is also reported in one upstream test for HDFS-5939 patch.
(I can see the same problem locally without applying this patch).




--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5993) org.apache.hadoop.fs.loadGenerator.TestLoadGenerator failure in trunk

[Jing Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao resolved HDFS-5993.
-

Resolution: Duplicate

Closed as duplicate of HADOOP-10355. But thanks for the report [~yzhangal]!

> org.apache.hadoop.fs.loadGenerator.TestLoadGenerator failure in trunk
> -
>
> Key: HDFS-5993
> URL: https://issues.apache.org/jira/browse/HDFS-5993
> Project: Hadoop HDFS
>  Issue Type: Bug
> Environment: CentOS release 6.5 (Final)
> cpe:/o:centos:linux:6:GA
>Reporter: Yongjun Zhang
>
> With today's latest trunk at
> commit d926e51bdc27f08e916534567a1edcfd994e2784
> When running it locally, I can consistently see the following test fails as:
> ---
>  T E S T S
> ---
> ---
>  T E S T S
> ---
> Running org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.888 sec 
> <<< FAILURE! - in org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
> testLoadGenerator(org.apache.hadoop.fs.loadGenerator.TestLoadGenerator)  Time 
> elapsed: 14.285 sec  <<< ERROR!
> java.io.IOException: Stream closed
> at java.io.BufferedReader.ensureOpen(BufferedReader.java:115)
> at java.io.BufferedReader.readLine(BufferedReader.java:310)
> at java.io.BufferedReader.readLine(BufferedReader.java:382)
> at 
> org.apache.hadoop.fs.loadGenerator.LoadGenerator.loadScriptFile(LoadGenerator.java:511)
> at 
> org.apache.hadoop.fs.loadGenerator.LoadGenerator.init(LoadGenerator.java:418)
> at 
> org.apache.hadoop.fs.loadGenerator.LoadGenerator.run(LoadGenerator.java:324)
> at 
> org.apache.hadoop.fs.loadGenerator.TestLoadGenerator.testLoadGenerator(TestLoadGenerator.java:231)
> Results :
> Tests in error:
>   TestLoadGenerator.testLoadGenerator:231 » IO Stream closed
> This failure is also reported in one upstream test for HDFS-5939 patch.
> (I can see the same problem locally without applying this patch).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)