[jira] [Created] (HDFS-2951) Block reported as corrupt while running multi threaded client program that performs write and read operation on a set of files

2012-02-15 Thread J.Andreina (Created) (JIRA) 
Block reported as corrupt while running multi threaded client program that 
performs write and read operation on a set of files
--

 Key: HDFS-2951
 URL: https://issues.apache.org/jira/browse/HDFS-2951
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.0
Reporter: J.Andreina
 Fix For: 0.24.0


Block incorrectly detected as bad in the following scenario:
Running multi threaded client program which performs write and read operation 
on a set of files
One block detected as bad by DN


Multiple recoveries where triggered from the NN side(It was happening every 1 
hr)
After around 6 hrs the recovery was successful(Commitblocksynchronization 
successful at NN side)
At the DN side around the same time when Commitblocksynchronization happened 
one more NN recovery call has come and this was subsequently failig as already 
the block was recovered and the generation timestamp is updated.
At the DN side block verification failed and the block was reported as bad.
FSCk report is indicating that the block is corrupt

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hadoop-Hdfs-trunk #956

2012-02-15 Thread Apache Jenkins Server 
See 

Changes:

[tucu] MAPREDUCE-3736. Variable substitution depth too large for 
fs.default.name causes jobs to fail (ahmed via tucu)

[mahadev] MAPREDUCE-3858. Task attempt failure during commit results in task 
never completing. (Tom White via mahadev)

[vinodkv] MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's 
environmental variables in TestMiniMRChildTask. (Tom White via vinodkv)

[todd] HADOOP-8071. Avoid an extra packet in client code when nagling is 
disabled. Contributed by Todd Lipcon.

[todd] HADOOP-8070. Add a standalone benchmark for RPC call performance. 
Contributed by Todd Lipcon.

[vinodkv] MAPREDUCE-3802. Added test to validate that AM can crash multiple 
times and still can recover successfully after MAPREDUCE-3846. (vinodkv)

--
[...truncated 11596 lines...]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$1.class(org/apache/hadoop/fs:FileContext$1.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FSError.class(org/apache/hadoop/fs:FSError.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/DelegateToFileSystem.class(org/apache/hadoop/fs:DelegateToFileSystem.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/ChecksumFs.class(org/apache/hadoop/fs:ChecksumFs.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/ChecksumFileSystem.class(org/apache/hadoop/fs:ChecksumFileSystem.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/ChecksumFileSystem$1.class(org/apache/hadoop/fs:ChecksumFileSystem$1.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/BufferedFSInputStream.class(org/apache/hadoop/fs:BufferedFSInputStream.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/BlockLocation$1.class(org/apache/hadoop/fs:BlockLocation$1.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/AvroFSInput.class(org/apache/hadoop/fs:AvroFSInput.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/AbstractFileSystem$2.class(org/apache/hadoop/fs:AbstractFileSystem$2.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/AbstractFileSystem$1.class(org/apache/hadoop/fs:AbstractFileSystem$1.class)]
  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading

Hadoop-Hdfs-trunk - Build # 956 - Still Failing

2012-02-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/956/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 11789 lines...]
  [javadoc] Loading source files for package org.apache.hadoop.hdfs.tools...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.tools.offlineEditsViewer...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.tools.offlineImageViewer...
  [javadoc] Loading source files for package org.apache.hadoop.hdfs.util...
  [javadoc] Loading source files for package org.apache.hadoop.hdfs.web...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.web.resources...
  [javadoc] 2 errors
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE [1:04.406s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:05.192s
[INFO] Finished at: Wed Feb 15 11:35:49 UTC 2012
[INFO] Final Memory: 40M/435M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project 
hadoop-hdfs: An Ant BuildException has occured: input file 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/findbugsXml.xml
 does not exist -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
+ /home/jenkins/tools/maven/latest/bin/mvn test 
-Dmaven.test.failure.ignore=true -Pclover 
-DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license
Archiving artifacts
Recording test results
Publishing Javadoc
ERROR: Publisher hudson.tasks.JavadocArchiver aborted due to exception
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/site/api
 does not exist.
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:474)
at hudson.FilePath$35.hasMatch(FilePath.java:1905)
at hudson.FilePath$35.invoke(FilePath.java:1814)
at hudson.FilePath$35.invoke(FilePath.java:1805)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2099)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Recording fingerprints
Updating MAPREDUCE-3736
Updating MAPREDUCE-3854
Updating MAPREDUCE-3846
Updating HADOOP-8070
Updating HADOOP-8071
Updating MAPREDUCE-3858
Updating MAPREDUCE-3802
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
All tests passed

Hadoop-Hdfs-0.23-Build - Build # 169 - Still Failing

2012-02-15 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/169/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 11306 lines...]
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.server.namenode.web.resources...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.server.protocol...
  [javadoc] Loading source files for package org.apache.hadoop.hdfs.tools...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.tools.offlineEditsViewer...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.tools.offlineImageViewer...
  [javadoc] Loading source files for package org.apache.hadoop.hdfs.util...
  [javadoc] Loading source files for package org.apache.hadoop.hdfs.web...
  [javadoc] Loading source files for package 
org.apache.hadoop.hdfs.web.resources...
  [javadoc] 2 errors
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE [1:08.573s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:09.239s
[INFO] Finished at: Wed Feb 15 11:36:30 UTC 2012
[INFO] Final Memory: 34M/379M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-antrun-plugin:1.6:run (site) on project 
hadoop-hdfs: An Ant BuildException has occured: input file 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/findbugsXml.xml
 does not exist -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
+ /home/jenkins/tools/maven/latest/bin/mvn test 
-Dmaven.test.failure.ignore=true -Pclover 
-DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license
Archiving artifacts
Recording test results
Publishing Javadoc
ERROR: Publisher hudson.tasks.JavadocArchiver aborted due to exception
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/site/api
 does not exist.
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:474)
at hudson.FilePath$35.hasMatch(FilePath.java:1905)
at hudson.FilePath$35.invoke(FilePath.java:1814)
at hudson.FilePath$35.invoke(FilePath.java:1805)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2099)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Recording fingerprints
Updating MAPREDUCE-3736
Updating MAPREDUCE-3854
Updating MAPREDUCE-3846
Updating HADOOP-8071
Updating MAPREDUCE-3858
Updating MAPREDUCE-3802
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
All tests passed

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #169

2012-02-15 Thread Apache Jenkins Server 
See 

Changes:

[tucu] Merge -r 1244263:1244264 from trunk to branch. FIXES: MAPREDUCE-3736

[mahadev] MAPREDUCE-3858. Task attempt failure during commit results in task 
never completing. (Tom White via mahadev) - Merging r1244254 from trunk.

[vinodkv] MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's 
environmental variables in TestMiniMRChildTask. (Tom White via vinodkv)
svn merge --ignore-ancestry -c 1244223 ../../trunk/

[todd] HADOOP-8071. Avoid an extra packet in client code when nagling is 
disabled. Contributed by Todd Lipcon.

[vinodkv] MAPREDUCE-3802. Added test to validate that AM can crash multiple 
times and still can recover successfully after MAPREDUCE-3846. (vinodkv)
svn merge --ignore-ancestry -c 1244178 ../../trunk/

--
[...truncated 3 lines...]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$23.class(org/apache/hadoop/fs:FileContext$23.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$22.class(org/apache/hadoop/fs:FileContext$22.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$21.class(org/apache/hadoop/fs:FileContext$21.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$20.class(org/apache/hadoop/fs:FileContext$20.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$2.class(org/apache/hadoop/fs:FileContext$2.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$19.class(org/apache/hadoop/fs:FileContext$19.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$18.class(org/apache/hadoop/fs:FileContext$18.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$17.class(org/apache/hadoop/fs:FileContext$17.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$16.class(org/apache/hadoop/fs:FileContext$16.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$15.class(org/apache/hadoop/fs:FileContext$15.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$14.class(org/apache/hadoop/fs:FileContext$14.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$13.class(org/apache/hadoop/fs:FileContext$13.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$12.class(org/apache/hadoop/fs:FileContext$12.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$11.class(org/apache/hadoop/fs:FileContext$11.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$10.class(org/apache/hadoop/fs:FileContext$10.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FileContext$1.class(org/apache/hadoop/fs:FileContext$1.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/FSError.class(org/apache/hadoop/fs:FSError.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/DelegateToFileSystem.class(org/apache/hadoop/fs:DelegateToFileSystem.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/ChecksumFs.class(org/apache/hadoop/fs:ChecksumFs.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/ChecksumFileSystem.class(org/apache/hadoop/fs:ChecksumFileSystem.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/ChecksumFileSystem$1.class(org/apache/hadoop/fs:ChecksumFileSystem$1.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/BufferedFSInputStream.class(org/apache/hadoop/fs:BufferedFSInputStream.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/BlockLocation$1.class(org/apache/hadoop/fs:BlockLocation$1.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/AvroFSInput.class(org/apache/hadoop/fs:AvroFSInput.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/AbstractFileSystem$2.class(org/apache/hadoop/fs:AbstractFileSystem$2.class)]
  [javadoc] [loading 
org/apache/hadoop/fs/AbstractFileSystem$1.class(org/apache/hadoop/fs:AbstractFileSystem$1.class)]
  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [loading 

  [javadoc] [

[jira] [Created] (HDFS-2952) HA: NN should not start with upgrade option or with a pending an unfinalized upgrade

2012-02-15 Thread Aaron T. Myers (Created) (JIRA) 
HA: NN should not start with upgrade option or with a pending an unfinalized 
upgrade


 Key: HDFS-2952
 URL: https://issues.apache.org/jira/browse/HDFS-2952
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


For simplicity, we should require that upgrades be done with HA disabled. We 
might support this in future versions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2953) The DN should stop running it's background threads fail

2012-02-15 Thread Eli Collins (Created) (JIRA) 
The DN should stop running it's background threads fail
---

 Key: HDFS-2953
 URL: https://issues.apache.org/jira/browse/HDFS-2953
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: data-node
Affects Versions: 1.0.0, 0.23.0
Reporter: Eli Collins


The DN will currently will startup and run if one of its background threads (eg 
the block scanner, async disk service) fails while trying to start. Which in 
this cases gives you a running DN w/o a block scanner. The DN should fail to 
start if any of its necessary background threads (eg not xfer threads) fail to 
start (or fail later on during execution).

In trunk this should be per block pool, ie if one block pool can start but 
another cannot we should probably allow the DN to run with the block pools that 
can start. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2954) Remove implementing Writable interface in the datatypes used at the client and the server

2012-02-15 Thread Suresh Srinivas (Created) (JIRA) 
Remove implementing Writable interface in the datatypes used at the client and 
the server
-

 Key: HDFS-2954
 URL: https://issues.apache.org/jira/browse/HDFS-2954
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: data-node, hdfs client, name-node
Affects Versions: 0.24.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas


With the completion of translator layer in HDFS, the separation of 
server/client datatypes and the wire types is complete. The server/client 
datatypes no longer need to implement Writables.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2909) HA: Inaccessible shared edits dir not getting removed from FSImage storage dirs upon error

2012-02-15 Thread Jitendra Nath Pandey (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey resolved HDFS-2909.


  Resolution: Fixed
Hadoop Flags: Reviewed

Committed to the branch. Thanks to Bikas!

> HA: Inaccessible shared edits dir not getting removed from FSImage storage 
> dirs upon error
> --
>
> Key: HDFS-2909
> URL: https://issues.apache.org/jira/browse/HDFS-2909
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Attachments: HDFS-2909-HDFS-1623.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2934) HA: Allow configs to be scoped to all NNs in the nameservice

2012-02-15 Thread Todd Lipcon (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-2934.
---

   Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
 Hadoop Flags: Reviewed

Thanks for the review. I addressed the nits on commit.

> HA: Allow configs to be scoped to all NNs in the nameservice
> 
>
> Key: HDFS-2934
> URL: https://issues.apache.org/jira/browse/HDFS-2934
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2934.txt
>
>
> Currently, for namenode-specific keys in HA, one must configure them as 
> keyfoo.nameserviceid.namenodeid. However, in many cases all of the NNs in a 
> nameservice would share the same value. So we should allow the configuration 
> of "keyfoo.nameserviceid" to apply to all NNs. The resolution path for these 
> keys would then be:
> keyfoo.nameserviceid.nnid
> keyfoo.nameserviceid (if above not set)
> keyfoo (if neither of above set)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2935) HA: Shared edits dir property should be suffixed with nameservice and namenodeID

2012-02-15 Thread Todd Lipcon (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-2935.
---

   Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
 Hadoop Flags: Reviewed

Committed to HA branch.

> HA: Shared edits dir property should be suffixed with nameservice and 
> namenodeID
> 
>
> Key: HDFS-2935
> URL: https://issues.apache.org/jira/browse/HDFS-2935
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Vinithra Varadharajan
>Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2935.txt
>
>
> Similar to the NameNode's name dirs, we should also be able to specify the 
> shared edits dir as dfs.namenode.shared.edits.dir.nameserviceId.nnId.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2955) IllegalStateException during standby startup in getCurSegmentTxId

2012-02-15 Thread Hari Mankude (Created) (JIRA) 
IllegalStateException during standby startup in getCurSegmentTxId
-

 Key: HDFS-2955
 URL: https://issues.apache.org/jira/browse/HDFS-2955
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Hari Mankude
Assignee: Hari Mankude


During standby restarts, a new routine getTransactionsSinceLastLogRoll() has 
been introduced for metrics which is calling getCurSegmentTxId(). checkstate() 
in getCurSegmentTxId() assumes that log is opened for writing and this is not 
the case in standby.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2956) calling fetchdt without a --renewer argument throws NPE

2012-02-15 Thread Todd Lipcon (Created) (JIRA) 
calling fetchdt without a --renewer argument throws NPE
---

 Key: HDFS-2956
 URL: https://issues.apache.org/jira/browse/HDFS-2956
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: security
Affects Versions: 0.24.0
Reporter: Todd Lipcon


If I call "bin/hdfs fetchdt /tmp/mytoken" without a "--renewer foo" argument, 
then it will throw a NullPointerException:

Exception in thread "main" java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:830)

this is because getDelegationToken is being called with a null renewer

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2957) transactionsSinceLastLogRoll metric can throw IllegalStateException

2012-02-15 Thread Todd Lipcon (Created) (JIRA) 
transactionsSinceLastLogRoll metric can throw IllegalStateException
---

 Key: HDFS-2957
 URL: https://issues.apache.org/jira/browse/HDFS-2957
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 0.24.0
Reporter: Todd Lipcon
Assignee: Aaron T. Myers


12/02/15 15:04:36 ERROR lib.MethodMetric: Error invoking method 
getTransactionsSinceLastLogRoll
...
at 
org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:54)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:505)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:435)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:399)
...
Caused by: java.lang.IllegalStateException: Bad state: OPEN_FOR_READING
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.getCurSegmentTxId(FSEditLog.java:417)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getTransactionsSinceLastLogRoll(FSNamesystem.java:3170)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2957) transactionsSinceLastLogRoll metric can throw IllegalStateException

2012-02-15 Thread Todd Lipcon (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-2957.
---

Resolution: Duplicate

oops, this was duplicate of HDFS-2955 filed at almost the same time :)

> transactionsSinceLastLogRoll metric can throw IllegalStateException
> ---
>
> Key: HDFS-2957
> URL: https://issues.apache.org/jira/browse/HDFS-2957
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 0.24.0
>Reporter: Todd Lipcon
>Assignee: Aaron T. Myers
>
> 12/02/15 15:04:36 ERROR lib.MethodMetric: Error invoking method 
> getTransactionsSinceLastLogRoll
> ...
> at 
> org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:54)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:505)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:435)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:399)
> ...
> Caused by: java.lang.IllegalStateException: Bad state: OPEN_FOR_READING
> at 
> com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.getCurSegmentTxId(FSEditLog.java:417)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getTransactionsSinceLastLogRoll(FSNamesystem.java:3170)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HDFS-2948) HA: NN throws NPE during shutdown if it fails to startup

2012-02-15 Thread Todd Lipcon (Reopened) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon reopened HDFS-2948:
---


Oops, this also needs to be fixed on the SBN it appears. Mind if I just commit 
the same null check to the SBN under this same JIRA issue?

{code}

diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
 b/hadoop-hdfs-project/hadoop-hd
index 4e628ce..5c09512 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
@@ -994,7 +994,9 @@ public class NameNode {
 @Override
 public void stopStandbyServices() throws IOException {
   // TODO(HA): Are we guaranteed to be the only active here?
-  namesystem.stopStandbyServices();
+  if (namesystem != null) {
+namesystem.stopStandbyServices();
+  }
 }
 
 @Override
{code}

> HA: NN throws NPE during shutdown if it fails to startup
> 
>
> Key: HDFS-2948
> URL: https://issues.apache.org/jira/browse/HDFS-2948
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2948.txt
>
>
> Last night's nightly build had a bunch of NPEs thrown in NameNode.stop. Not 
> sure which patch introduced the issue, but the problem is that 
> NameNode.stop() is called if an exception is thrown during startup. If the 
> exception is thrown before the namesystem is created, then 
> NameNode.namesystem is null, and {{namesystem.stop}} throws NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2958) Sweep for remaining proxy construction which doesn't go through failover path

2012-02-15 Thread Todd Lipcon (Created) (JIRA) 
Sweep for remaining proxy construction which doesn't go through failover path
-

 Key: HDFS-2958
 URL: https://issues.apache.org/jira/browse/HDFS-2958
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon


There seem to be a couple more places where we construct proxies without going 
through the failover code path. In particular I just tried "hdfs groups" and it 
didn't properly resolve the logical URI. There may be others.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2948) HA: NN throws NPE during shutdown if it fails to startup

2012-02-15 Thread Todd Lipcon (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-2948.
---

Resolution: Fixed

Applied the delta, thanks

> HA: NN throws NPE during shutdown if it fails to startup
> 
>
> Key: HDFS-2948
> URL: https://issues.apache.org/jira/browse/HDFS-2948
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2948.txt
>
>
> Last night's nightly build had a bunch of NPEs thrown in NameNode.stop. Not 
> sure which patch introduced the issue, but the problem is that 
> NameNode.stop() is called if an exception is thrown during startup. If the 
> exception is thrown before the namesystem is created, then 
> NameNode.namesystem is null, and {{namesystem.stop}} throws NPE.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2959) NameNodeProtocolTranslatorPB & ClientNamenodeProtocolTranslatorPB should not be responsible for setting up the underlying proxy.

2012-02-15 Thread Uma Maheswara Rao G (Created) (JIRA) 
NameNodeProtocolTranslatorPB & ClientNamenodeProtocolTranslatorPB should not be 
responsible for setting up the underlying proxy.


 Key: HDFS-2959
 URL: https://issues.apache.org/jira/browse/HDFS-2959
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client, name-node
Affects Versions: 0.24.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor


https://issues.apache.org/jira/browse/HDFS-2928?focusedCommentId=13208969&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13208969
For addressing the Aaron comment in HDFS-2928.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2960) HA: TestDFSUtil is failing

2012-02-15 Thread Uma Maheswara Rao G (Created) (JIRA) 
HA: TestDFSUtil is failing
--

 Key: HDFS-2960
 URL: https://issues.apache.org/jira/browse/HDFS-2960
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha, test
Affects Versions: HA branch (HDFS-1623)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


TestDFSUtil is failing.

{noformat}
org.junit.ComparisonFailure: expected: but 
was:
at org.junit.Assert.assertEquals(Assert.java:123)
at org.junit.Assert.assertEquals(Assert.java:145)
at 
org.apache.hadoop.hdfs.TestDFSUtil.testHANameNodesWithFederation(TestDFSUtil.java:411)
{noformat}



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-2928) HA: ConfiguredFailoverProxyProvider should not create a NameNode proxy with an underlying retry proxy

2012-02-15 Thread Aaron T. Myers (Resolved) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HDFS-2928.
--

   Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)

> HA: ConfiguredFailoverProxyProvider should not create a NameNode proxy with 
> an underlying retry proxy
> -
>
> Key: HDFS-2928
> URL: https://issues.apache.org/jira/browse/HDFS-2928
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ha, hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Uma Maheswara Rao G
>Priority: Minor
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2928.patch, HDFS-2928.patch, HDFS-2928.patch
>
>
> This is to address the following TODO in ConfiguredFailoverProxyProvider:
> {quote}
> // TODO(HA): This will create a NN proxy with an underlying retry
> // proxy. We don't want this.
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira