[jira] [Created] (HDFS-5510) Fix a findbug warning in DataStorage.java on HDFS-2832 branch

[Junping Du (JIRA)]

Junping Du created HDFS-5510:


 Summary: Fix a findbug warning in DataStorage.java on HDFS-2832 
branch
 Key: HDFS-5510
 URL: https://issues.apache.org/jira/browse/HDFS-5510
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Junping Du
Assignee: Junping Du


The access to datanodeUuid should be synchronized as findBugs warning says.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: HDFS read/write data throttling

[Steve Loughran]

this is interesting -I've moved my comments over to the JIRA and it would
be good for yours to go there too.

is there a URL for your paper?


On 13 November 2013 06:27, Andrew Wang  wrote:

> Hey Steve,
>
> My research project (Cake, published at SoCC '12) was trying to provide
> SLAs for mixed workloads of latency-sensitive and throughput-bound
> applications, e.g. HBase running alongside MR. This was challenging because
> seeks are a real killer. Basically, we had to strongly limit MR I/O to keep
> worst-case seek latency down, and did so by putting schedulers on the RPC
> queues in HBase and HDFS to restrict queuing in the OS and disk where we
> lacked preemption.
>
> Regarding citations of note, most academics consider throughput-sharing to
> be a solved problem. It's not dissimilar from normal time slicing, you try
> to ensure fairness over some coarse timescale. I think cgroups [1] and
> ioprio_set [2] essentially provide this.
>
> Mixing throughput and latency though is difficult, and my conclusion is
> that there isn't a really great solution for spinning disks besides
> physical isolation. As we all know, you can get either IOPS or bandwidth,
> but not both, and it's not a linear tradeoff between the two. If you're
> interested in this though, I can dig up some related work from my Cake
> paper.
>
> However, since it seems that we're more concerned with throughput-bound
> apps, we might be okay just using cgroups and ioprio_set to do
> time-slicing. I actually hacked up some code a while ago which passed a
> client-provided priority byte to the DN, which used it to set the I/O
> priority of the handling DataXceiver accordingly. This isn't the most
> outlandish idea, since we've put QoS fields in our RPC protocol for
> instance; this would just be another byte. Short-circuit reads are outside
> this paradigm, but then you can use cgroup controls instead.
>
> My casual conversations with Googlers indicate that there isn't any special
> Borg/Omega sauce either, just that they heavily prioritize DFS I/O over
> non-DFS. Maybe that's another approach: if we can separate block management
> in HDFS, MR tasks could just write their output to a raw HDFS block, thus
> bringing a lot of I/O back into the fold of "datanode as I/O manager" for a
> machine.
>
> Overall, I strongly agree with you that it's important to first define what
> our goals are regarding I/O QoS. The general case is a tarpit, so it'd be
> good to carve off useful things that can be done now (like Lohit's
> direction of per-stream/FS throughput throttling with trusted clients) and
> then carefully grow the scope as we find more usecases we can confidently
> solve.
>
> Best,
> Andrew
>
> [1] cgroups blkio controller
> https://www.kernel.org/doc/Documentation/cgroups/blkio-controller.txt
> [2] ioprio_set http://man7.org/linux/man-pages/man2/ioprio_set.2.html
>
>
> On Tue, Nov 12, 2013 at 1:38 AM, Steve Loughran  >wrote:
>
> > I've looked at it a bit within the context of YARN.
> >
> > YARN containers are where this would be ideal, as then you'd be able to
> > request IO capacity as well as CPU and RAM. For that to work, the
> > throttling would have to be outside the App, as you are trying to limit
> > code whether or not it wants to be, and because you probably (*) want to
> > give it more bandwidth if the system is otherwise idle. Self-throttling
> > doesn't pick up spare IO
> >
> >
> >1. you can use cgroups in YARN to throttle local disk IO through the
> >file:// URLs or the java filesystem APIs -such as for MR temp data
> >2. you can't c-group throttle HDFS per YARN container, which would be
> >the ideal use case for it. The IO is taking place in the DN, and
> cgroups
> >only limits IO in the throttled process group.
> >3. implementing it in the DN would require a lot more complex code
> there
> >to prioritise work based on block ID (sole identifier that goes around
> >everywhere) or input source (local sockets for HBase IO vs TCP stack)
> >4. One you go to a heterogenous filesystem you need to think about IO
> >load per storage layer as well as/alongside per-volume
> >5. There's also generic RPC request throttle to prevent DoS against
> the
> >NN and other HDFS services. That would need to be server side, but
> once
> >implemented in the RPC code be universal.
> >
> > You also need to define what is the load you are trying to throttle, pure
> > RPCs/second, read bandwidth, write bandwidth, seeks or IOPs. Once a file
> is
> > lined up for sequential reading, you'd almost want it to stream through
> the
> > next blocks until a high priority request came through, but operations
> like
> > a seek which would involve a disk head movement backwards would be
> > something to throttle (hence you need to be storage type aware as SSD
> seeks
> > costs less). You also need to consider that although the cost of writes
> is
> > high, it's usually being done with the goal of preserving data -and yo

Hadoop-Hdfs-0.23-Build - Build # 789 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/789/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 7885 lines...]
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3313,27]
 cannot find symbol
[ERROR] symbol  : method 
setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto)
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3319,8]
 cannot find symbol
[ERROR] symbol  : method makeExtensionsImmutable()
[ERROR] location: class 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3330,10]
 cannot find symbol
[ERROR] symbol  : method 
ensureFieldAccessorsInitialized(java.lang.Class,java.lang.Class)
[ERROR] location: class com.google.protobuf.GeneratedMessage.FieldAccessorTable
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3335,31]
 cannot find symbol
[ERROR] symbol  : class AbstractParser
[ERROR] location: package com.google.protobuf
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3344,4]
 method does not override or implement a method from a supertype
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[4098,12]
 cannot find symbol
[ERROR] symbol  : method 
ensureFieldAccessorsInitialized(java.lang.Class,java.lang.Class)
[ERROR] location: class com.google.protobuf.GeneratedMessage.FieldAccessorTable
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[4371,104]
 cannot find symbol
[ERROR] symbol  : method getUnfinishedMessage()
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5264,8]
 getUnknownFields() in 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto 
cannot override getUnknownFields() in com.google.protobuf.GeneratedMessage; 
overridden method is final
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5284,19]
 cannot find symbol
[ERROR] symbol  : method 
parseUnknownField(com.google.protobuf.CodedInputStream,com.google.protobuf.UnknownFieldSet.Builder,com.google.protobuf.ExtensionRegistryLite,int)
[ERROR] location: class 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5314,15]
 cannot find symbol
[ERROR] symbol  : method 
setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto)
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5317,27]
 cannot find symbol
[ERROR] symbol  : method 
setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto)
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5323,8]
 cannot find symbol
[ERROR] symbol  : method makeExtensionsImmutable()
[ERROR] location: class 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto
[ERROR] 
/home/jenk

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #789

[Apache Jenkins Server]

See 

Changes:

[jeagles] YARN-1386. NodeManager mistakenly loses resources and relocalizes 
them (Jason Lowe via jeagles)

--
[...truncated 7692 lines...]
[ERROR] location: package com.google.protobuf
[ERROR] 
:[270,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[281,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[10533,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[10544,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[8357,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[8368,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[12641,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[12652,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[9741,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[9752,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[1781,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[1792,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[5338,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[5349,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[6290,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 

Re: Next releases

[Arun C Murthy]

On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:

> On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote:
> 
>> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
>> there.  However, I have only been following the HDFS and common side
>> of things so I may not have the full picture.  Arun, can you give a
>> specific example of something you'd like to "blow away"?

There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, 
similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* 
in CHANGES.txt which aren't necessary for a patch release, plus things like:

HADOOP-9623 
Update jets3t dependency to 0.9.0







 

Having said that, the HDFS devs know their code the best.

> I agree with Colin. If we've been backporting things into a patch release
> (third version component) which don't belong, we should explicitly call out
> those patches, so we can learn from our mistakes and have a discussion
> about what belongs.

Good point.

Here is a straw man proposal:


A patch (third version) release should only include *blocker* bugs which are 
critical from an operational, security or data-integrity issues.

This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) 
is always release-able, and more importantly, deploy-able at any point in time.



Sandy did bring up a related point about timing of releases and the urge for 
everyone to cram features/fixes into a dot release.

So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 
etc.) and keep the patch releases limited to blocker bugs.

Thoughts?

thanks,
Arun





-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Next releases

[Arun C Murthy]

On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:

> On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote:
> 
>> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
>> there.  However, I have only been following the HDFS and common side
>> of things so I may not have the full picture.  Arun, can you give a
>> specific example of something you'd like to "blow away"?

There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, 
similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* 
in CHANGES.txt which aren't necessary for a patch release, plus things like:

HADOOP-9623 
Update jets3t dependency to 0.9.0







 

Having said that, the HDFS devs know their code the best.

> I agree with Colin. If we've been backporting things into a patch release
> (third version component) which don't belong, we should explicitly call out
> those patches, so we can learn from our mistakes and have a discussion
> about what belongs.

Good point.

Here is a straw man proposal:


A patch (third version) release should only include *blocker* bugs which are 
critical from an operational, security or data-integrity issues.

This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) 
is always release-able, and more importantly, deploy-able at any point in time.



Sandy did bring up a related point about timing of releases and the urge for 
everyone to cram features/fixes into a dot release.

So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 
etc.) and keep the patch releases limited to blocker bugs.

Thoughts?

thanks,
Arun





-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Next releases

[Steve Loughran]

On 13 November 2013 18:11, Arun C Murthy  wrote:

>
>
HADOOP-9623
> Update jets3t dependency to 0.9.0
>
>
I saw that change -I don't think its a bad one, but I do think we need more
testing of blobstores & especially big operations, like 6Gb uploads (which
should now work with the 0.9.0 jets3t).

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-5511) improve CacheManipulator interface to allow better unit testing

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HDFS-5511:
--

 Summary: improve CacheManipulator interface to allow better unit 
testing
 Key: HDFS-5511
 URL: https://issues.apache.org/jira/browse/HDFS-5511
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: datanode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe


The CacheManipulator interface has been helpful in allowing us to stub out 
{{mlock}} in cases where we don't want to test it.  We should move  the 
{{getMemlockLimit}} and {{getOperatingSystemPageSize}} functions into this 
interface as well so that we don't have to skip these tests on machines where 
these methods would ordinarily not work for us.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Next releases

[Arun C Murthy]

On Nov 13, 2013, at 12:38 PM, Sandy Ryza  wrote:

> Here are few patches that I put into 2.2.1 and are minimally invasive, but
> I don't think are blockers:
> 
>  YARN-305. Fair scheduler logs too many "Node offered to app" messages.
>  YARN-1335. Move duplicate code from FSSchedulerApp and
> FiCaSchedulerApp into SchedulerApplication
>  YARN-1333. Support blacklisting in the Fair Scheduler
>  YARN-1109. Demote NodeManager "Sending out status for container" logs
> to debug (haosdent via Sandy Ryza)
>  YARN-1388. Fair Scheduler page always displays blank fair share
> 
> +1 to doing releases at some fixed time interval.

To be clear, I still think we should be *very* clear about what features we 
target for each release (2.3, 2.4, etc.).

Except, we don't wait infinitely for any specific feature - if we miss a 4-6 
week window a feature goes to the next train.

Makes sense?

thanks,
Arun

> 
> -Sandy
> 
> 
> On Wed, Nov 13, 2013 at 10:10 AM, Arun C Murthy  wrote:
> 
>> 
>> On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:
>> 
>>> On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe >> wrote:
>>> 
 To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
 there.  However, I have only been following the HDFS and common side
 of things so I may not have the full picture.  Arun, can you give a
 specific example of something you'd like to "blow away"?
>> 
>> There are bunch of issues in YARN/MapReduce which clearly aren't
>> *critical*, similarly in HDFS a cursory glance showed up some
>> *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a
>> patch release, plus things like:
>> 
>>HADOOP-9623
>> Update jets3t dependency to 0.9.0
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Having said that, the HDFS devs know their code the best.
>> 
>>> I agree with Colin. If we've been backporting things into a patch release
>>> (third version component) which don't belong, we should explicitly call
>> out
>>> those patches, so we can learn from our mistakes and have a discussion
>>> about what belongs.
>> 
>> Good point.
>> 
>> Here is a straw man proposal:
>> 
>> 
>> A patch (third version) release should only include *blocker* bugs which
>> are critical from an operational, security or data-integrity issues.
>> 
>> This way, we can ensure that a minor series release (2.2.x or 2.3.x or
>> 2.4.x) is always release-able, and more importantly, deploy-able at any
>> point in time.
>> 
>> 
>> 
>> Sandy did bring up a related point about timing of releases and the urge
>> for everyone to cram features/fixes into a dot release.
>> 
>> So, we could remedy that situation by doing a release every 4-6 weeks
>> (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs.
>> 
>> Thoughts?
>> 
>> thanks,
>> Arun
>> 
>> 
>> 
>> 
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: block access token

[Luke Lu]

Block access token is only valid for a short period of time, as the NN/DN
shared secrets are rolled periodically. As long as you cannot steal block
token easily (besides using zero-day bugs), there is really no security
hole here (by design). If you know of a way to steal block tokens without
root access, let us know.


On Tue, Nov 12, 2013 at 7:30 PM, lei liu  wrote:

> When client read block from DataNode, the block access token is used for
> authorization on DataNode. But if the block access token is stolen by
> impostor, the  impostor can read the block,
> I think this is one security hole.
>
> I think we can use the replay cache mechanism in Kerberos to resolve the
> question, example below explaining:
>
> The possibility exists for an impostor to simultaneously steal both the
> ticket and the authenticator and use them during the 2 minutes the
> authenticator is valid. This is very difficult but not impossible. To solve
> this problem with Kerberos 5, Replay Cache has been introduced. In
> application servers (but also in TGS), there exists the capacity to
> remember authenticators which have arrived within the last 2 minutes, and
> to reject them if they are replicas. With this the problem is resolved as
> long as the impostor is not smart enough to copy the ticket and
> authenticator and make them arrive at the application server before the
> legitimate request arrives. This really would be a hoax, since the
> authentic user would be rejected while the impostor would have access to
> the service.
>
>
> Thanks,
>
> LiuLei
>

Re: Next releases

[Alejandro Abdelnur]

Sound goods, just a little impedance between what seem to be 2 conflicting
goals:

* what features we target for each release
* train releases

If we want to do train releases at fixed times, then if a feature is not
ready, it catches the next train, no delays of the train because of a
feature. If a bug is delaying the train and a feature becomes ready in the
mean time and it does not stabilize the release, it can jump on board, if
it breaks something, it goes out of the window until the next train.

Also we have do decide what we do with 2.2.1. I would say start wrapping up
the current 2.2 branch and make it the first train.

thx




On Wed, Nov 13, 2013 at 12:55 PM, Arun C Murthy  wrote:

>
> On Nov 13, 2013, at 12:38 PM, Sandy Ryza  wrote:
>
> > Here are few patches that I put into 2.2.1 and are minimally invasive,
> but
> > I don't think are blockers:
> >
> >  YARN-305. Fair scheduler logs too many "Node offered to app" messages.
> >  YARN-1335. Move duplicate code from FSSchedulerApp and
> > FiCaSchedulerApp into SchedulerApplication
> >  YARN-1333. Support blacklisting in the Fair Scheduler
> >  YARN-1109. Demote NodeManager "Sending out status for container" logs
> > to debug (haosdent via Sandy Ryza)
> >  YARN-1388. Fair Scheduler page always displays blank fair share
> >
> > +1 to doing releases at some fixed time interval.
>
> To be clear, I still think we should be *very* clear about what features
> we target for each release (2.3, 2.4, etc.).
>
> Except, we don't wait infinitely for any specific feature - if we miss a
> 4-6 week window a feature goes to the next train.
>
> Makes sense?
>
> thanks,
> Arun
>
> >
> > -Sandy
> >
> >
> > On Wed, Nov 13, 2013 at 10:10 AM, Arun C Murthy 
> wrote:
> >
> >>
> >> On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:
> >>
> >>> On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe  >>> wrote:
> >>>
>  To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
>  there.  However, I have only been following the HDFS and common side
>  of things so I may not have the full picture.  Arun, can you give a
>  specific example of something you'd like to "blow away"?
> >>
> >> There are bunch of issues in YARN/MapReduce which clearly aren't
> >> *critical*, similarly in HDFS a cursory glance showed up some
> >> *enhancements*/*improvements* in CHANGES.txt which aren't necessary for
> a
> >> patch release, plus things like:
> >>
> >>HADOOP-9623
> >> Update jets3t dependency to 0.9.0
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Having said that, the HDFS devs know their code the best.
> >>
> >>> I agree with Colin. If we've been backporting things into a patch
> release
> >>> (third version component) which don't belong, we should explicitly call
> >> out
> >>> those patches, so we can learn from our mistakes and have a discussion
> >>> about what belongs.
> >>
> >> Good point.
> >>
> >> Here is a straw man proposal:
> >>
> >> 
> >> A patch (third version) release should only include *blocker* bugs which
> >> are critical from an operational, security or data-integrity issues.
> >>
> >> This way, we can ensure that a minor series release (2.2.x or 2.3.x or
> >> 2.4.x) is always release-able, and more importantly, deploy-able at any
> >> point in time.
> >>
> >> 
> >>
> >> Sandy did bring up a related point about timing of releases and the urge
> >> for everyone to cram features/fixes into a dot release.
> >>
> >> So, we could remedy that situation by doing a release every 4-6 weeks
> >> (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs.
> >>
> >> Thoughts?
> >>
> >> thanks,
> >> Arun
> >>
> >>
> >>
> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Alejandro

Re: Next releases

[Jason Lowe]

I think a lot of confusion comes from the fact that the 2.x line is 
starting to mature.  Before this there wasn't such a big contention of 
what went into patch vs. minor releases and often the lines were blurred 
between the two.  However now we have significant customers and products 
starting to use 2.x as a base, which means we need to start treating it 
like we treat 1.x.  That means getting serious about what we should put 
into a patch release vs. what we postpone to a minor release.


Here's my $0.02 on recent proposals:

+1 to releasing more often in general.  A lot of the rush to put changes 
into a patch release is because it can be a very long time between any 
kind of release.  If minor releases are more frequent then I hope there 
would be less of a need to rush something or hold up a release.


+1 to limiting checkins of patch releases to Blockers/Criticals.  If 
necessary committers check into trunk/branch-2 only and defer to the 
patch release manager for the patch release merge.  Then there should be 
fewer surprises for everyone what ended up in a patch release and less 
likely the patch release becomes destabilized from the sheer amount of 
code churn.  Maybe this won't be necessary if everyone understands that 
the patch release isn't the only way to get a change out in timely manner.


As for 2.2.1, again I think it's expectations for what that release 
means.  If it's really just a patch release then there shouldn't be 
features in it and tons of code churn, but I think many were treating it 
as the next vehicle to deliver changes in general.  If we think 2.2.1 is 
just as good or better than 2.2.0 then let's wrap it up and move to a 
more disciplined approach for subsequent patch releases and more 
frequent minor releases.


Jason

On 11/13/2013 12:10 PM, Arun C Murthy wrote:

On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:


On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote:


To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
there.  However, I have only been following the HDFS and common side
of things so I may not have the full picture.  Arun, can you give a
specific example of something you'd like to "blow away"?

There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, 
similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* 
in CHANGES.txt which aren't necessary for a patch release, plus things like:

HADOOP-9623 
Update jets3t dependency to 0.9.0







  


Having said that, the HDFS devs know their code the best.


I agree with Colin. If we've been backporting things into a patch release
(third version component) which don't belong, we should explicitly call out
those patches, so we can learn from our mistakes and have a discussion
about what belongs.

Good point.

Here is a straw man proposal:


A patch (third version) release should only include *blocker* bugs which are 
critical from an operational, security or data-integrity issues.

This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) 
is always release-able, and more importantly, deploy-able at any point in time.



Sandy did bring up a related point about timing of releases and the urge for 
everyone to cram features/fixes into a dot release.

So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 
etc.) and keep the patch releases limited to blocker bugs.

Thoughts?

thanks,
Arun

[jira] [Created] (HDFS-5512) CacheAdmin -listPools fails with NPE when user lacks permissions to view all pools

[Stephen Chu (JIRA)]

Stephen Chu created HDFS-5512:
-

 Summary: CacheAdmin -listPools fails with NPE when user lacks 
permissions to view all pools
 Key: HDFS-5512
 URL: https://issues.apache.org/jira/browse/HDFS-5512
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 3.0.0
Reporter: Stephen Chu


https://issues.apache.org/jira/browse/HDFS-5471 (CacheAdmin -listPools fails 
when user lacks permissions to view all pools) was recently resolved, but on a 
build with this fix, I am running into another error when using cacheadmin 
-listPools.

Now, when the user does not have permissions to view all the pools, the 
cacheadmin -listPools command will throw a NullPointerException.

On a system with a single pool "root" with a mode of 750, I see this as user 
schu:
{code}
[schu@hdfs-c5-nfs ~]$ hdfs cacheadmin -listPools
Exception in thread "main" java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.tools.CacheAdmin$ListCachePoolsCommand.run(CacheAdmin.java:745)
at org.apache.hadoop.hdfs.tools.CacheAdmin.run(CacheAdmin.java:82)
at org.apache.hadoop.hdfs.tools.CacheAdmin.main(CacheAdmin.java:87)
{code}

After we modify the root pool to 755, then -listPools works properly.
{code}
[schu@hdfs-c5-nfs ~]$ hdfs cacheadmin -listPools
Found 1 result.
NAME  OWNER  GROUP  MODE   WEIGHT 
root  root   root   rwxr-xr-x  100
[schu@hdfs-c5-nfs ~]$ 
{code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5513) CacheAdmin commands fail when using . as the path

[Stephen Chu (JIRA)]

Stephen Chu created HDFS-5513:
-

 Summary: CacheAdmin commands fail when using . as the path
 Key: HDFS-5513
 URL: https://issues.apache.org/jira/browse/HDFS-5513
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 3.0.0
Reporter: Stephen Chu
Priority: Minor


The hdfs CLI commands generally accept "." as a path argument.
e.g.
{code}
hdfs dfs -rm .
hdfs dfsadmin -allowSnapshot .
{code}

I don't think it's very common to use the path "." but the CacheAdmin commands 
will fail saying that it cannot create a Path from an empty string.

{code}
[schu@hdfs-c5-nfs ~]$ hdfs cacheadmin -removeDirectives -path .
Exception in thread "main" java.lang.IllegalArgumentException: Can not create a 
Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
at org.apache.hadoop.fs.Path.(Path.java:184)
at 
org.apache.hadoop.hdfs.protocol.PathBasedCacheDirective$Builder.(PathBasedCacheDirective.java:66)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.listPathBasedCacheDirectives(DistributedFileSystem.java:1639)
at 
org.apache.hadoop.hdfs.tools.CacheAdmin$RemovePathBasedCacheDirectivesCommand.run(CacheAdmin.java:365)
at org.apache.hadoop.hdfs.tools.CacheAdmin.run(CacheAdmin.java:82)
at org.apache.hadoop.hdfs.tools.CacheAdmin.main(CacheAdmin.java:87)
[schu@hdfs-c5-nfs ~]$ hdfs cacheadmin -addDirective -path . -pool schu
Exception in thread "main" java.lang.IllegalArgumentException: Can not create a 
Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
at org.apache.hadoop.fs.Path.(Path.java:184)
at 
org.apache.hadoop.hdfs.protocol.PathBasedCacheDirective$Builder.(PathBasedCacheDirective.java:66)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.addPathBasedCacheDirective(DistributedFileSystem.java:1598)
at 
org.apache.hadoop.hdfs.tools.CacheAdmin$AddPathBasedCacheDirectiveCommand.run(CacheAdmin.java:180)
at org.apache.hadoop.hdfs.tools.CacheAdmin.run(CacheAdmin.java:82)
at org.apache.hadoop.hdfs.tools.CacheAdmin.main(CacheAdmin.java:87)
{code}




--
This message was sent by Atlassian JIRA
(v6.1#6144)