[jira] [Created] (HDFS-3884) QJM: Journal format() should reset cached values

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3884:
-

 Summary: QJM: Journal format() should reset cached values
 Key: HDFS-3884
 URL: https://issues.apache.org/jira/browse/HDFS-3884
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)
 Attachments: hdfs-3884.txt

Simple bug in the JournalNode: it caches certain values (eg accepted epoch) in 
memory, and the cached values aren't reset when the journal is formatted. So, 
after a format, further calls to the same Journal will see the old value for 
accepted epoch, writer epoch, etc, preventing the journal from being re-used 
until the JN is restarted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3885) QJM: optimize log sync when JN is lagging behind

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3885:
-

 Summary: QJM: optimize log sync when JN is lagging behind
 Key: HDFS-3885
 URL: https://issues.apache.org/jira/browse/HDFS-3885
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon


This is a potential optimization that we can add to the JournalNode: when one 
of the nodes is lagging behind the others (eg because its local disk is slower 
or there was a network blip), it receives edits after they've been committed to 
a majority. It can tell this because the committed txid included in the request 
info is higher than the highest txid in the actual batch to be written. In this 
case, we know that this batch has already been fsynced to a quorum of nodes, so 
we can skip the fsync() on the laggy node, helping it to catch back up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3886) Shutdown requests can possibly check for checkpoint issues (corrupted edits) and save a good namespace copy before closing down?

[Harsh J (JIRA)]

Harsh J created HDFS-3886:
-

 Summary: Shutdown requests can possibly check for checkpoint 
issues (corrupted edits) and save a good namespace copy before closing down?
 Key: HDFS-3886
 URL: https://issues.apache.org/jira/browse/HDFS-3886
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor


HDFS-3878 sorta gives me this idea. Aside of having a method to download it to 
a different location, we can also lock up the namesystem (or deactivate the 
client rpc server) and save the namesystem before we complete up the shutdown.

The init.d/shutdown scripts would have to work with this somehow though, to not 
kill -9 it when in-process. Also, the new image may be stored in a 
shutdown.chkpt directory, to not interfere in the regular dirs, but still allow 
easier recovery.

Obviously this will still not work if all directories are broken. So maybe we 
could have some configs to tackle that as well?

I haven't thought this through, so let me know what part is wrong to do :)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hadoop-Hdfs-0.23-Build - Build # 361 - Still Unstable

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 20895 lines...]
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ 
hadoop-hdfs-project ---
[INFO] Installing 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-hdfs-project/0.23.3-SNAPSHOT/hadoop-hdfs-project-0.23.3-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ 
hadoop-hdfs-project ---
[INFO] Skipped writing classpath file 
'/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/target/classes/mrapp-generated-classpath'.
  No changes found.
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  SUCCESS [5:22.692s]
[INFO] Apache Hadoop HttpFS .. SUCCESS [46.624s]
[INFO] Apache Hadoop HDFS Project  SUCCESS [0.062s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 6:09.985s
[INFO] Finished at: Sat Sep 01 11:39:41 UTC 2012
[INFO] Final Memory: 53M/748M
[INFO] 
+ /home/jenkins/tools/maven/latest/bin/mvn test 
-Dmaven.test.failure.ignore=true -Pclover 
-DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license
Archiving artifacts
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Publishing Javadoc
Recording fingerprints
Updating HADOOP-8727
Updating MAPREDUCE-4612
Updating YARN-66
Updating MAPREDUCE-4611
Updating HDFS-3873
Updating HADOOP-8225
Updating YARN-60
Updating HDFS-3852
Updating MAPREDUCE-4604
Updating YARN-63
Updating MAPREDUCE-4614
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Unstable
Sending email for trigger: Unstable



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.hdfs.TestHftpDelegationToken.testInsecureRemoteCluster

Error Message:
Unable to obtain remote token

Stack Trace:
java.io.IOException: Unable to obtain remote token
at 
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:230)
at org.apache.hadoop.hdfs.HftpFileSystem$2.run(HftpFileSystem.java:237)
at org.apache.hadoop.hdfs.HftpFileSystem$2.run(HftpFileSystem.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
at 
org.apache.hadoop.hdfs.HftpFileSystem.getDelegationToken(HftpFileSystem.java:232)
at 
org.apache.hadoop.hdfs.HftpFileSystem.initDelegationToken(HftpFileSystem.java:185)
at 
org.apache.hadoop.hdfs.HftpFileSystem.initialize(HftpFileSystem.java:174)
at 
org.apache.hadoop.hdfs.HsftpFileSystem.initialize(HsftpFileSystem.java:63)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2190)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:84)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2224)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:2212)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:353)
at 
org.apache.hadoop.hdfs.TestHftpDelegationToken.__CLR3_0_2hvuutv128c(TestHftpDelegationToken.java:147)
at 
org.apache.hadoop.hdfs.TestHftpDelegationToken.tes

Jenkins build is still unstable: Hadoop-Hdfs-0.23-Build #361

[Apache Jenkins Server]

See 

Jenkins build is back to normal : Hadoop-Hdfs-trunk #1152

[Apache Jenkins Server]

See