[jira] [Resolved] (HDFS-1497) Write pipeline sequence numbers should be sequential with no skips or duplicates

2015-03-01 Thread Allen Wittenauer (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HDFS-1497.

Resolution: Unresolved

Cancelling the patch and closing this issue as stale given the amount of time 
that has passed and the refactoring of the code involved.

> Write pipeline sequence numbers should be sequential with no skips or 
> duplicates
> 
>
> Key: HDFS-1497
> URL: https://issues.apache.org/jira/browse/HDFS-1497
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs-client
>Affects Versions: 0.20-append, 0.22.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: hdfs-1497.txt, hdfs-1497.txt, hdfs-1497.txt, 
> hdfs-1497.txt, hdfs-1497.txt
>
>
> In HDFS-895 we discovered that multiple hflush() calls in a row without 
> intervening writes could cause a skip in sequence number. This doesn't seem 
> to have any direct consequences, but we should maintain and assert the 
> invariant that sequence numbers have no gaps or duplicates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HDFS-1348) Improve NameNode reponsiveness while it is checking if datanode decommissions are complete

2015-03-01 Thread Allen Wittenauer (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HDFS-1348.

Resolution: Unresolved

Cancelling the patch and closing this issue as stale given the amount of time 
that has passed and the refactoring of the code involved.


> Improve NameNode reponsiveness while it is checking if datanode decommissions 
> are complete
> --
>
> Key: HDFS-1348
> URL: https://issues.apache.org/jira/browse/HDFS-1348
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Reporter: Hairong Kuang
>Assignee: Hairong Kuang
> Attachments: decomissionImp1.patch, decomissionImp2.patch, 
> decommission.patch, decommission1.patch
>
>
> NameNode normally is busy all the time. Its log is full of activities every 
> second. But once for a while, NameNode seems to pause for more than 10 
> seconds without doing anything, leaving a blank in its log even though no 
> garbage collection is happening.  All other requests to NameNode are blocked 
> when this is happening.
> One culprit is DecommionManager. Its monitor holds the fsynamesystem lock 
> during the whole process of checking if decomissioning DataNodes are finished 
> or not, during which it checks every block of up to a default of 5 datanodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Jenkins build is back to normal : Hadoop-Hdfs-trunk #2051

2015-03-01 Thread Apache Jenkins Server
See 



Jenkins build is back to normal : Hadoop-Hdfs-trunk-Java8 #110

2015-03-01 Thread Apache Jenkins Server
See 



Re: 2.7 status

2015-03-01 Thread Vinod Kumar Vavilapalli
Hi Ravi,

Only one patch on that umbrella JIRA is PA. And others don't have patches. Are 
they being actively worked on and can they get in a week or so? Given the delay 
in 2.7 already, does it make sense to roll this umbrella over to 2.8?

Thanks,
+Vinod

On Feb 14, 2015, at 8:56 AM, Ravi Prakash  wrote:

> I would like the improvements to the Namenode UI be included in 2.7 too. 
> HDFS-7588. All the code is ready and we can try to get as much of it in as 
> possible piecemeal.
> 
> 
> On Saturday, February 14, 2015 3:52 AM, Steve Loughran 
>  wrote:
> 
> 
> 
> 
> 
> On 14 February 2015 at 00:37:07, Karthik Kambatla 
> (ka...@cloudera.com) wrote:
> 
> 2 weeks from now (end of Feb) sounds reasonable. The one feature I would
> like for to be included is shared-cache: we are pretty close - two more
> main items to take care of.
> 
> In an offline conversation, Steve mentioned building Windows binaries for
> our releases. Do we want to do that for 2.7? If so, can anyone with Windows
> expertise setup a Jenkins job to build these artifacts, and may be hook it
> up to https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/
> 
> 
> 
> 
> 
> someone will have to first fix MiniYarnCluster to come up on the ASF jenkins 
> machines; it currently fails with directory permission setup problems that 
> may matter in production, but not in test runs.
> 
> 



Re: 2.7 status

2015-03-01 Thread Vinod Kumar Vavilapalli
Allen, can you make that JIRA a blocker with that rationale so that it can be 
discussed so on the JIRA?

Thanks
+Vinod

On Feb 14, 2015, at 9:30 AM, Allen Wittenauer  wrote:

> 
> IMO, HDFS-5796 (in some form or another) is a blocker for 2.7. 
> 
> Right now, DFS browsing is pretty much broken on secure systems when a 
> hadoop-auth-compatible plugin is in use.  The "fix" (HDFS-5716) introduced 
> (what appears to be) an incompatible and undocumented method to provide auth 
> versus the rest of Hadoop.
> 
> On Feb 14, 2015, at 8:56 AM, Ravi Prakash  wrote:
> 
>> I would like the improvements to the Namenode UI be included in 2.7 too. 
>> HDFS-7588. All the code is ready and we can try to get as much of it in as 
>> possible piecemeal.
>> 
>> 
>>On Saturday, February 14, 2015 3:52 AM, Steve Loughran 
>>  wrote:
>> 
>> 
>> 
>> 
>> 
>> On 14 February 2015 at 00:37:07, Karthik Kambatla 
>> (ka...@cloudera.com) wrote:
>> 
>> 2 weeks from now (end of Feb) sounds reasonable. The one feature I would
>> like for to be included is shared-cache: we are pretty close - two more
>> main items to take care of.
>> 
>> In an offline conversation, Steve mentioned building Windows binaries for
>> our releases. Do we want to do that for 2.7? If so, can anyone with Windows
>> expertise setup a Jenkins job to build these artifacts, and may be hook it
>> up to https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/
>> 
>> 
>> 
>> 
>> 
>> someone will have to first fix MiniYarnCluster to come up on the ASF jenkins 
>> machines; it currently fails with directory permission setup problems that 
>> may matter in production, but not in test runs.
>> 
>> 
> 



Re: 2.7 status

2015-03-01 Thread Arun Murthy
Sounds good, thanks for the help Vinod!

Arun


From: Vinod Kumar Vavilapalli
Sent: Sunday, March 01, 2015 11:43 AM
To: Hadoop Common; Jason Lowe; Arun Murthy
Subject: Re: 2.7 status

Agreed. How about we roll an RC end of this week? As a Java 7+ release with 
features, patches that already got in?

Here's a filter tracking blocker tickets - 
https://issues.apache.org/jira/issues/?filter=12330598. Nine open now.

+Arun
Arun, I'd like to help get 2.7 out without further delay. Do you mind me taking 
over release duties?

Thanks,
+Vinod

From: Jason Lowe 
Sent: Friday, February 13, 2015 8:11 AM
To: common-...@hadoop.apache.org
Subject: Re: 2.7 status

I'd like to see a 2.7 release sooner than later.  It has been almost 3 months 
since Hadoop 2.6 was released, and there have already been 634 JIRAs committed 
to 2.7.  That's a lot of changes waiting for an official release.
https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2Chdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolution%3DFixed
Jason

  From: Sangjin Lee 
 To: "common-...@hadoop.apache.org" 
 Sent: Tuesday, February 10, 2015 1:30 PM
 Subject: 2.7 status

Folks,

What is the current status of the 2.7 release? I know initially it started
out as a "java-7" only release, but looking at the JIRAs that is very much
not the case.

Do we have a certain timeframe for 2.7 or is it time to discuss it?

Thanks,
Sangjin



Re: 2.7 status

2015-03-01 Thread Tsuyoshi Ozawa
Allen and Steve,

Should we mark HADOOP-11602 as a blocker issue?

Thanks,
- Tsuyoshi

On Mon, Mar 2, 2015 at 4:58 AM, Arun Murthy  wrote:
> Sounds good, thanks for the help Vinod!
>
> Arun
>
> 
> From: Vinod Kumar Vavilapalli
> Sent: Sunday, March 01, 2015 11:43 AM
> To: Hadoop Common; Jason Lowe; Arun Murthy
> Subject: Re: 2.7 status
>
> Agreed. How about we roll an RC end of this week? As a Java 7+ release with 
> features, patches that already got in?
>
> Here's a filter tracking blocker tickets - 
> https://issues.apache.org/jira/issues/?filter=12330598. Nine open now.
>
> +Arun
> Arun, I'd like to help get 2.7 out without further delay. Do you mind me 
> taking over release duties?
>
> Thanks,
> +Vinod
> 
> From: Jason Lowe 
> Sent: Friday, February 13, 2015 8:11 AM
> To: common-...@hadoop.apache.org
> Subject: Re: 2.7 status
>
> I'd like to see a 2.7 release sooner than later.  It has been almost 3 months 
> since Hadoop 2.6 was released, and there have already been 634 JIRAs 
> committed to 2.7.  That's a lot of changes waiting for an official release.
> https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2Chdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolution%3DFixed
> Jason
>
>   From: Sangjin Lee 
>  To: "common-...@hadoop.apache.org" 
>  Sent: Tuesday, February 10, 2015 1:30 PM
>  Subject: 2.7 status
>
> Folks,
>
> What is the current status of the 2.7 release? I know initially it started
> out as a "java-7" only release, but looking at the JIRAs that is very much
> not the case.
>
> Do we have a certain timeframe for 2.7 or is it time to discuss it?
>
> Thanks,
> Sangjin
>


Re: 2.7 status

2015-03-01 Thread Yongjun Zhang
Hi,

Thanks for working on 2.7 release.

Currently the fallback from KerberosAuthenticator to
PseudoAuthenticator  is enabled by default in a hardcoded way. HAOOP-10895
changes the default and requires applications (such as oozie) to set a
config property or call an API to enable the fallback.

This jira has been reviewed, and "almost" ready to get in. However, there
is a concern that we have to change the relevant applications. Please see
my comment here:

https://issues.apache.org/jira/browse/HADOOP-10895?focusedCommentId=14321823&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14321823

Any of your comments will be highly appreciated. This jira was postponed
from 2.6. I think it should be no problem to skip 2.7. But your comments
would help us to decide what to do with this jira for future releases.

Thanks.

--Yongjun


On Sun, Mar 1, 2015 at 11:58 AM, Arun Murthy  wrote:

> Sounds good, thanks for the help Vinod!
>
> Arun
>
> 
> From: Vinod Kumar Vavilapalli
> Sent: Sunday, March 01, 2015 11:43 AM
> To: Hadoop Common; Jason Lowe; Arun Murthy
> Subject: Re: 2.7 status
>
> Agreed. How about we roll an RC end of this week? As a Java 7+ release
> with features, patches that already got in?
>
> Here's a filter tracking blocker tickets -
> https://issues.apache.org/jira/issues/?filter=12330598. Nine open now.
>
> +Arun
> Arun, I'd like to help get 2.7 out without further delay. Do you mind me
> taking over release duties?
>
> Thanks,
> +Vinod
> 
> From: Jason Lowe 
> Sent: Friday, February 13, 2015 8:11 AM
> To: common-...@hadoop.apache.org
> Subject: Re: 2.7 status
>
> I'd like to see a 2.7 release sooner than later.  It has been almost 3
> months since Hadoop 2.6 was released, and there have already been 634 JIRAs
> committed to 2.7.  That's a lot of changes waiting for an official release.
>
> https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2Chdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolution%3DFixed
> Jason
>
>   From: Sangjin Lee 
>  To: "common-...@hadoop.apache.org" 
>  Sent: Tuesday, February 10, 2015 1:30 PM
>  Subject: 2.7 status
>
> Folks,
>
> What is the current status of the 2.7 release? I know initially it started
> out as a "java-7" only release, but looking at the JIRAs that is very much
> not the case.
>
> Do we have a certain timeframe for 2.7 or is it time to discuss it?
>
> Thanks,
> Sangjin
>
>


[jira] [Resolved] (HDFS-5420) Not need to launch Secondary namenode for NN HA mode?

2015-03-01 Thread Allen Wittenauer (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer resolved HDFS-5420.

Resolution: Duplicate

 I'm going to merge this into HADOOP--11590.  Closing as a dupe.  

> Not need to launch Secondary namenode for NN HA mode?
> -
>
> Key: HDFS-5420
> URL: https://issues.apache.org/jira/browse/HDFS-5420
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 3.0.0
>Reporter: Raymond Liu
>Assignee: Allen Wittenauer
>Priority: Minor
> Attachments: HDFS-5420-00.patch
>
>
> For Hadoop 2, When deploying with NN HA, the wiki says that it is an error to 
> start a secondary namenode. While the sbin/start-dfs.sh still launch a 
> secondary namenode even nothing related to the secondary namenode is 
> configured. Should this be fixed? or people just don't use this scripts to 
> start HA hdfs?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HDFS-7864) Erasure Coding: Update safemode calculation for striped blocks

2015-03-01 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-7864:
---

 Summary: Erasure Coding: Update safemode calculation for striped 
blocks
 Key: HDFS-7864
 URL: https://issues.apache.org/jira/browse/HDFS-7864
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao


We need to update the safemode calculation for striped blocks. Specifically, 
each striped block now consists of multiple data/parity blocks stored in 
corresponding DataNodes. The current code's calculation is thus inconsistent: 
each striped block is only counted as 1 expected block, while each of its 
member block may increase the number of received blocks by 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HDFS-7865) NullPointerException in SimulatedFSDataset

2015-03-01 Thread Tsz Wo Nicholas Sze (JIRA)
Tsz Wo Nicholas Sze created HDFS-7865:
-

 Summary: NullPointerException in SimulatedFSDataset
 Key: HDFS-7865
 URL: https://issues.apache.org/jira/browse/HDFS-7865
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Tsz Wo Nicholas Sze
Priority: Minor


https://builds.apache.org/job/PreCommit-HDFS-Build/9690//testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancer/testUnknownDatanode/
{noformat}
java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset$BInfo.access$400(SimulatedFSDataset.java:126)
at 
org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset.getPinning(SimulatedFSDataset.java:1319)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.copyBlock(DataXceiver.java:969)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opCopyBlock(Receiver.java:244)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:80)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:249)
at java.lang.Thread.run(Thread.java:745)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)