[jira] [Resolved] (HDFS-2819) Document new HA-related configs in hdfs-default.xml

[Eli Collins (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins resolved HDFS-2819.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Thanks for the reviews Uma and Todd. I've committed this.

> Document new HA-related configs in hdfs-default.xml
> ---
>
> Key: HDFS-2819
> URL: https://issues.apache.org/jira/browse/HDFS-2819
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation, ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Eli Collins
> Attachments: hdfs-2819.txt, hdfs-2819.txt, hdfs-2819.txt
>
>
> We've added a few configs, like shared edits dir, dfs.ha.namenodes, etc - we 
> should probably add these to hdfs-default.xml so they get documented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2752) HA: exit if multiple shared dirs are configured

[Eli Collins (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins resolved HDFS-2752.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Thanks for the review Todd. Fixed the nit and committed.

> HA: exit if multiple shared dirs are configured
> ---
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an 
> error in this case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2894) HA: automatically determine the nameservice Id if only one nameservice is configured

[Eli Collins (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins resolved HDFS-2894.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Thanks for the review Todd. I've committed this.

> HA: automatically determine the nameservice Id if only one nameservice is 
> configured
> 
>
> Key: HDFS-2894
> URL: https://issues.apache.org/jira/browse/HDFS-2894
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-2894.txt, hdfs-2894.txt
>
>
> The SecondaryNameNode should log a message and refuse to start if HA is 
> enabled since the StandbyNode checkpoints by default and IIRC we have not yet 
> enabled the ability to have multiple checkpointers in the NN.
> On the HA branch the 2NN does not currently start from start-dfs.sh because 
> getconf -secondarynamenodes claims the http-address is not configured even 
> though it is, though this seems like a bug, in branch 23 getconf will 
> correctly return localhost:50090.
> {noformat}
>  
>dfs.namenode.secondary.http-address
>localhost:50090
>  
> 
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs getconf -secondarynamenodes
> Incorrect configuration: secondary namenode address 
> dfs.namenode.secondary.http-address is not configured.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2893) The start/stop scripts don't start/stop the 2NN when using the default configuration

[Eli Collins (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins resolved HDFS-2893.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

Thanks Todd and ATM. I committed this and merged. Didn't run test-patch since 
it just updates the start/stop scripts. I tested the generated tarball from 
branch-23 by hand.

> The start/stop scripts don't start/stop the 2NN when using the default 
> configuration
> 
>
> Key: HDFS-2893
> URL: https://issues.apache.org/jira/browse/HDFS-2893
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Minor
> Attachments: hdfs-2893.txt
>
>
> HDFS-1703 changed the behavior of the start/stop scripts so that the masters 
> file is no longer used to indicate which hosts to start the 2NN on. The 2NN 
> is now started, when using start-dfs.sh, on hosts only when 
> dfs.namenode.secondary.http-address is configured with a non-wildcard IP. 
> This means you can not start a NN using an http-address specified using a 
> wildcard IP. We should allow a 2NN to be started with the default config, ie 
> start-dfs.sh should start a NN, 2NN and DN. The packaging already works this 
> way (it doesn't use start-dfs.sh, it uses hadoop-daemon.sh directly w/o first 
> checking getconf) so let's bring start-dfs.sh in line with this behavior.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #161

[Apache Jenkins Server]

See 

Changes:

[eli] HDFS-2893. svn merge -c 1240928 from trunk

[harsh] Fix CHANGES.txt after 1240899 (HDFS-2868)

[harsh] Fix CHANGES.txt after 1240899 (HDFS-2868)

[harsh] HDFS-2868. svn merge -c 1240897 from trunk

[acmurthy] Merge -c 1240886 from trunk to branch-0.23 to fix MAPREDUCE-3746. 
Initialize queue metrics upfront and added start/finish time to RM Web-UI.

--
[...truncated 14328 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

[jira] [Created] (HDFS-2899) Service protocol change to support multiple storages added in HDFS-2880

[Suresh Srinivas (Created) (JIRA)]

Service protocol change to support multiple storages added in HDFS-2880
---

 Key: HDFS-2899
 URL: https://issues.apache.org/jira/browse/HDFS-2899
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 0.24.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
 Attachments: HDFS-2899.txt

This jira makes changes in the service protocol corresponding to protobuf 
protocol changes made in HDFS-2880.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2900) Move ProtocolSignatureWritable from HDFS to common

[Suresh Srinivas (Created) (JIRA)]

Move ProtocolSignatureWritable from HDFS to common
--

 Key: HDFS-2900
 URL: https://issues.apache.org/jira/browse/HDFS-2900
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 0.24.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas


ProtocolSignatureWritable is more appropriate in common as it is common to RPC 
implemenations.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2848) hdfs corruption appended to blocks is not detected by fs commands or fsck

[Ravi Prakash (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash resolved HDFS-2848.


Resolution: Not A Problem

Closing as discussed.

> hdfs corruption appended to blocks is not detected by fs commands or fsck
> -
>
> Key: HDFS-2848
> URL: https://issues.apache.org/jira/browse/HDFS-2848
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>
> Courtesy Pat White [~patwhitey2007]
> {quote}
> Appears that there is a regression in corrupt block detection by both fsck 
> and fs cmds like 'cat'. Testcases for
> pre-block and block-overwrite corruption of all replicas is correctly 
> reporting errors however post-block corruption is
> not, fsck on the filesystem reports it's Healthy and 'cat' returns without 
> error. Looking at the DN blocks themselves,
> they clearly contain the injected corruption pattern.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

How to start to try NN HA?

[Mingjie Lai]

Hi hdfs devs.

I'd like to start to evaluate the nn ha feature.

I know it's still a work in progress, but, is there any 
installation/configuration document available for me to start from?


Is there an expected ``feature complete'' date?

I watched the nn ha presentation at hw2011, and it was mentioned to be 
included in 0.23.1. Is it still the plan?


Thanks,
Mingjie

Re: How to start to try NN HA?

[Harsh J]

Hi,

Check out https://issues.apache.org/jira/browse/HDFS-2733

On Mon, Feb 6, 2012 at 2:40 PM, Mingjie Lai  wrote:
> Hi hdfs devs.
>
> I'd like to start to evaluate the nn ha feature.
>
> I know it's still a work in progress, but, is there any
> installation/configuration document available for me to start from?
>
> Is there an expected ``feature complete'' date?
>
> I watched the nn ha presentation at hw2011, and it was mentioned to be
> included in 0.23.1. Is it still the plan?
>
> Thanks,
> Mingjie



-- 
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

Re: How to start to try NN HA?

[Eli Collins]

Hey Mingjie,

It's in progress, slated for a 23 release.

The latest docs are here:
https://issues.apache.org/jira/secure/attachment/12513412/HDFSHighAvailability.html

Thanks,
Eli

On Mon, Feb 6, 2012 at 1:10 AM, Mingjie Lai  wrote:
> Hi hdfs devs.
>
> I'd like to start to evaluate the nn ha feature.
>
> I know it's still a work in progress, but, is there any
> installation/configuration document available for me to start from?
>
> Is there an expected ``feature complete'' date?
>
> I watched the nn ha presentation at hw2011, and it was mentioned to be
> included in 0.23.1. Is it still the plan?
>
> Thanks,
> Mingjie

Re: How to start to try NN HA?

[Aaron T. Myers]

Very slightly updated docs are here:

https://issues.apache.org/jira/secure/attachment/12513420/HDFSHighAvailability.html

Aaron

On Feb 6, 2012, at 8:41 AM, Eli Collins  wrote:

> Hey Mingjie,
> 
> It's in progress, slated for a 23 release.
> 
> The latest docs are here:
> https://issues.apache.org/jira/secure/attachment/12513412/HDFSHighAvailability.html
> 
> Thanks,
> Eli
> 
> On Mon, Feb 6, 2012 at 1:10 AM, Mingjie Lai  wrote:
>> Hi hdfs devs.
>> 
>> I'd like to start to evaluate the nn ha feature.
>> 
>> I know it's still a work in progress, but, is there any
>> installation/configuration document available for me to start from?
>> 
>> Is there an expected ``feature complete'' date?
>> 
>> I watched the nn ha presentation at hw2011, and it was mentioned to be
>> included in 0.23.1. Is it still the plan?
>> 
>> Thanks,
>> Mingjie

Re: How to start to try NN HA?

[Mingjie Lai]

Thanks guys.

I've checked out hdfs-123 branch and gonna have a try.

Thanks,
Mingjie

On 02/06/2012 08:50 AM, Aaron T. Myers wrote:

Very slightly updated docs are here:

https://issues.apache.org/jira/secure/attachment/12513420/HDFSHighAvailability.html

Aaron

On Feb 6, 2012, at 8:41 AM, Eli Collins  wrote:


Hey Mingjie,

It's in progress, slated for a 23 release.

The latest docs are here:
https://issues.apache.org/jira/secure/attachment/12513412/HDFSHighAvailability.html

Thanks,
Eli

On Mon, Feb 6, 2012 at 1:10 AM, Mingjie Lai  wrote:

Hi hdfs devs.

I'd like to start to evaluate the nn ha feature.

I know it's still a work in progress, but, is there any
installation/configuration document available for me to start from?

Is there an expected ``feature complete'' date?

I watched the nn ha presentation at hw2011, and it was mentioned to be
included in 0.23.1. Is it still the plan?

Thanks,
Mingjie

[jira] [Created] (HDFS-2901) HA: Improvements for SBN web UI - not show under-replicated/missing blocks

[Brandon Li (Created) (JIRA)]

HA: Improvements for SBN web UI - not show under-replicated/missing blocks
--

 Key: HDFS-2901
 URL: https://issues.apache.org/jira/browse/HDFS-2901
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Brandon Li
Assignee: Brandon Li


Since the Standby Namenode doesn't compute replication queues, we shouldn't 
show under-replicated/missing blocks or corrupt files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2902) Allow new edit logs dir to be configured while NN is running

[Bikas Saha (Created) (JIRA)]

Allow new edit logs dir to be configured while NN is running


 Key: HDFS-2902
 URL: https://issues.apache.org/jira/browse/HDFS-2902
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Bikas Saha




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2903) NPE if shared edits directory is not available during failover

[Hari Mankude (Created) (JIRA)]

NPE if shared edits directory is not available during failover
--

 Key: HDFS-2903
 URL: https://issues.apache.org/jira/browse/HDFS-2903
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Hari Mankude
Assignee: Hari Mankude


Journalset has a null pointer exception if shared edits directory is not 
available during HA state transition.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2904) HA: Client support for getting delegation tokens to an HA cluster

[Todd Lipcon (Created) (JIRA)]

HA: Client support for getting delegation tokens to an HA cluster
-

 Key: HDFS-2904
 URL: https://issues.apache.org/jira/browse/HDFS-2904
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha, hdfs client, name-node, security
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical


Currently we have server-side support for delegation tokens in HA, and some 
tests to verify it, but the client throws NPEs when trying to fetch a DT. This 
is because the cluster doesn't have a single hostname, but instead a logical 
nameservice name.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2733) Document HA configuration and CLI

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HDFS-2733.
--

   Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
 Hadoop Flags: Reviewed

Thanks a lot for the reviews, everyone. I've just committed this to the HA 
branch.

> Document HA configuration and CLI
> -
>
> Key: HDFS-2733
> URL: https://issues.apache.org/jira/browse/HDFS-2733
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation, ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Aaron T. Myers
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2733-HDFS-1623.patch, HDFS-2733-HDFS-1623.patch, 
> HDFS-2733-HDFS-1623.patch, HDFS-2733-HDFS-1623.patch, 
> HDFS-2733-HDFS-1623.patch, HDFS-2733-HDFS-1623.patch, 
> HDFSHighAvailability.html, HDFSHighAvailability.html, 
> HDFSHighAvailability.html, HDFSHighAvailability.html, 
> HDFSHighAvailability.html, HDFSHighAvailability.html
>
>
> We need to document the configuration changes in HDFS-2231 and the new CLI 
> introduced by HADOOP-7774.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2905) Standby NN NPE when shared edits dir is deleted

[Bikas Saha (Created) (JIRA)]

Standby NN NPE when shared edits dir is deleted
---

 Key: HDFS-2905
 URL: https://issues.apache.org/jira/browse/HDFS-2905
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Bikas Saha




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2794) HA: Active NN may purge edit log files before standby NN has a chance to read them

[Todd Lipcon (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-2794.
---

   Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
 Hadoop Flags: Reviewed

Thanks for the reviews, committed to the HA branch

> HA: Active NN may purge edit log files before standby NN has a chance to read 
> them
> --
>
> Key: HDFS-2794
> URL: https://issues.apache.org/jira/browse/HDFS-2794
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2794.txt, hdfs-2794.txt
>
>
> Given that the active NN is solely responsible for purging finalized edit log 
> segments, and given that the active NN has no way of knowing when the standby 
> reads edit logs, it's  possible that the standby NN could fail to read all 
> edits it needs before the active purges the files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2906) Null Pointer execption in blockreceiver

[Brahma Reddy Battula (Created) (JIRA)]

Null Pointer execption in blockreceiver
---

 Key: HDFS-2906
 URL: https://issues.apache.org/jira/browse/HDFS-2906
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
Reporter: Brahma Reddy Battula


Stared with two DN's and one DN is keep on restarting while write inprogress
{noformat}
2012-02-07 01:03:45,897 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: 
HOST-10-18-40-23:50010:DataXceiver error processing WRITE_BLOCK operation  src: 
/10.18.40.20:23862 dest: /10.18.40.23:50010
java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:151)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:340)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:167)
at java.lang.Thread.run(Thread.java:619)
2012-02-07 01:03:46,083 INFO org.apache.hadoop.hdfs.server.common.Storage: 
Locking is disabled

{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira