Re: HDFS read/write data throttling

2013-11-12 Thread Andrew Wang
Hey Steve, My research project (Cake, published at SoCC '12) was trying to provide SLAs for mixed workloads of latency-sensitive and throughput-bound applications, e.g. HBase running alongside MR. This was challenging because seeks are a real killer. Basically, we had to strongly limit MR I/O to k

block access token

2013-11-12 Thread lei liu
When client read block from DataNode, the block access token is used for authorization on DataNode. But if the block access token is stolen by impostor, the impostor can read the block, I think this is one security hole. I think we can use the replay cache mechanism in Kerberos to resolve the que

[jira] [Resolved] (HDFS-5501) Fix pendingReceivedRequests tracking in BPServiceActor

2013-11-12 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5501. - Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Thanks Nicholas, I commi

[jira] [Resolved] (HDFS-5508) Fix compilation error after merge

2013-11-12 Thread Arpit Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5508. - Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Hadoop Flags: Review

[jira] [Created] (HDFS-5509) TestPathBasedCacheRequests#testReplicationFactor is flaky

2013-11-12 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-5509: - Summary: TestPathBasedCacheRequests#testReplicationFactor is flaky Key: HDFS-5509 URL: https://issues.apache.org/jira/browse/HDFS-5509 Project: Hadoop HDFS Issue T

[jira] [Created] (HDFS-5508) Fix compilation error after merge

2013-11-12 Thread Tsz Wo (Nicholas), SZE (JIRA)
Tsz Wo (Nicholas), SZE created HDFS-5508: Summary: Fix compilation error after merge Key: HDFS-5508 URL: https://issues.apache.org/jira/browse/HDFS-5508 Project: Hadoop HDFS Issue Typ

[jira] [Created] (HDFS-5507) Consider maximum DN memory, stale status when scheduling recaching

2013-11-12 Thread Colin Patrick McCabe (JIRA)
Colin Patrick McCabe created HDFS-5507: -- Summary: Consider maximum DN memory, stale status when scheduling recaching Key: HDFS-5507 URL: https://issues.apache.org/jira/browse/HDFS-5507 Project: H

[jira] [Created] (HDFS-5506) Use URLConnectionFactory in DelegationTokenFetcher

2013-11-12 Thread Haohui Mai (JIRA)
Haohui Mai created HDFS-5506: Summary: Use URLConnectionFactory in DelegationTokenFetcher Key: HDFS-5506 URL: https://issues.apache.org/jira/browse/HDFS-5506 Project: Hadoop HDFS Issue Type: Sub-

Re: Next releases

2013-11-12 Thread Todd Lipcon
On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote: > To be honest, I'm not aware of anything in 2.2.1 that shouldn't be > there. However, I have only been following the HDFS and common side > of things so I may not have the full picture. Arun, can you give a > specific example of something you

[jira] [Created] (HDFS-5505) DataStorage.format() should ignore NamespaceID

2013-11-12 Thread Eric Sirianni (JIRA)
Eric Sirianni created HDFS-5505: --- Summary: DataStorage.format() should ignore NamespaceID Key: HDFS-5505 URL: https://issues.apache.org/jira/browse/HDFS-5505 Project: Hadoop HDFS Issue Type: Bu

[jira] [Created] (HDFS-5504) In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold, leads to NN safemode.

2013-11-12 Thread Vinay (JIRA)
Vinay created HDFS-5504: --- Summary: In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold, leads to NN safemode. Key: HDFS-5504 URL: https://issues.apache.org/jira/browse/HDFS-5504 Project: H

Re: HDFS read/write data throttling

2013-11-12 Thread Steve Loughran
I've looked at it a bit within the context of YARN. YARN containers are where this would be ideal, as then you'd be able to request IO capacity as well as CPU and RAM. For that to work, the throttling would have to be outside the App, as you are trying to limit code whether or not it wants to be,

[jira] [Created] (HDFS-5503) Datanode#checkDiskError also should check for ClosedChannelException

2013-11-12 Thread Vinay (JIRA)
Vinay created HDFS-5503: --- Summary: Datanode#checkDiskError also should check for ClosedChannelException Key: HDFS-5503 URL: https://issues.apache.org/jira/browse/HDFS-5503 Project: Hadoop HDFS Issue T