[jira] [Created] (HDFS-6130) NPE during upgrade using trunk after RU merged

[Fengdong Yu (JIRA)]

Fengdong Yu created HDFS-6130:
-

 Summary: NPE during upgrade using trunk after RU merged
 Key: HDFS-6130
 URL: https://issues.apache.org/jira/browse/HDFS-6130
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.4.0
Reporter: Fengdong Yu


I want upgrade an old cluster(0.20.2-cdh3u1) to trunk instance, 

I can upgrade successfully if I don't configurage HA, but if HA enabled,
there is NPE when I run ' hdfs namenode -initializeSharedEdits'

{code}
14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/03/20 15:06:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of 
total heap and retry cache entry expiry time is 60 millis
14/03/20 15:06:41 INFO util.GSet: Computing capacity for map NameNodeRetryCache
14/03/20 15:06:41 INFO util.GSet: VM type   = 64-bit
14/03/20 15:06:41 INFO util.GSet: 0.02999329447746% max memory 896 MB = 
275.3 KB
14/03/20 15:06:41 INFO util.GSet: capacity  = 2^15 = 32768 entries
14/03/20 15:06:41 INFO namenode.AclConfigFlag: ACLs enabled? false
14/03/20 15:06:41 INFO common.Storage: Lock on 
/data/hadoop/data1/dfs/name/in_use.lock acquired by nodename 7326@10-150-170-176
14/03/20 15:06:42 INFO common.Storage: Lock on 
/data/hadoop/data2/dfs/name/in_use.lock acquired by nodename 7326@10-150-170-176
14/03/20 15:06:42 INFO namenode.FSImage: No edit log streams selected.
14/03/20 15:06:42 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
14/03/20 15:06:42 FATAL namenode.NameNode: Exception in namenode join
java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.isReservedName(FSDirectory.java:2984)
at 
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.addToParent(FSImageFormatPBINode.java:205)
at 
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:162)
at 
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
at 
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
at 
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:120)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:895)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:881)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:704)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:642)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:894)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:653)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initializeSharedEdits(NameNode.java:912)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1276)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1360)
14/03/20 15:06:42 INFO util.ExitUtil: Exiting with status 1
14/03/20 15:06:42 INFO namenode.NameNode: SHUTDOWN_MSG: 
/
SHUTDOWN_MSG: Shutting down NameNode at 10-150-170-176/10.150.170.176
/
{code}




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

[Steve Loughran]

I'm clearly supportive of this, though of course the testing costs needed
to back up the assertion make it more expensive than just a statement.

Two issues

-we'd need to make clear that new cluster features that a client can invoke
won't be available. You can't expect snapshot or symlink support running
against a -2.2.0 cluster, even if the client supports it.

-in YARN, there are no guarantees that an app compiled against later YARN
APIs will work in old clusters. Because YARN apps upload themselves to the
server, and run with their hadoop, hdfs & yarn libraries. We have to do a
bit of introspection in our code already to support this situation. The
compatibility doc would need to be clear on that too: "YARN apps that use
new APIs (including new fields in datastructures) can expect link
exceptions"





On 20 March 2014 04:25, Vinayakumar B  wrote:

> +1, I agree with your point Chris. It depends on the client application
> how they using the hdfs jars in their classpath.
>
> As implementation already supports the compatibility (through protobuf),
> No extra code changes required to support new Client + old server.
>
> I feel it will be good to explicitly mention about the compatibility of
> existing APIs in both versions.
>
> Anyway this is not applicable for the new APIs in latest client and this
> is understood. We can make it explicit in the document though.
>
>
> Regards,
> Vinayakumar B
>
> -Original Message-
> From: Chris Nauroth [mailto:cnaur...@hortonworks.com]
> Sent: 20 March 2014 05:36
> To: common-...@hadoop.apache.org
> Cc: mapreduce-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> Client + Old Server
>
> I think this kind of compatibility issue still could surface for HDFS,
> particularly for custom applications (i.e. something not executed via
> "hadoop jar" on a cluster node, where the client classes ought to be
> injected into the classpath automatically).  Running DistCP between 2
> clusters of different versions could result in a 2.4.0 client calling a
> 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> client as a dependency and try to use it to make HTTP calls to a 2.3.0 HDFS
> cluster.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org
> > wrote:
>
> > It makes sense only for YARN today where we separated out the clients.
> > HDFS is still a monolithic jar so this compatibility issue is kind of
> > invalid there.
> >
> > +vinod
> >
> > On Mar 19, 2014, at 1:59 PM, Chris Nauroth 
> > wrote:
> >
> > > I'd like to discuss clarification of part of our compatibility policy.
> > > Here is a link to the compatibility documentation for release 2.3.0:
> > >
> > >
> > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > /Compatibility.html#Wire_compatibility
> > >
> > > For convenience, here are the specific lines in question:
> > >
> > > Client-Server compatibility is required to allow users to continue
> > > using the old clients even after upgrading the server (cluster) to a
> > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > talking to a Hadoop 2.3.0 cluster.
> > >
> > > Client-Server compatibility is also required to allow upgrading
> > individual
> > > components without upgrading others. For example, upgrade HDFS from
> > version
> > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > >
> > > Server-Server compatibility is required to allow mixed versions
> > > within an active cluster so the cluster may be upgraded without
> > > downtime in a
> > rolling
> > > fashion.
> > >
> > > Notice that there is no specific mention of upgrading the client
> > > ahead of the server.  (There is no clause for "upgraded client + old
> > > server".) Based on my experience, this is a valid use case when a
> > > user wants to
> > pick
> > > up a client-side bug fix ahead of the cluster administrator's
> > > upgrade schedule.
> > >
> > > Is it our policy to maintain client compatibility with old clusters
> > within
> > > the same major release?  I think many of us have assumed that the
> > > answer
> > is
> > > yes and coded our new features accordingly, but it isn't made
> > > explicit in the documentation.  Do we all agree that the answer is
> > > yes, or is it possibly up for debate depending on the change in
> > > question?  In RFC 2119 lingo, is it a MUST or a SHOULD?  Either way,
> > > I'd like to update the
> > policy
> > > text to make our decision clear.  After we have consensus, I can
> > volunteer
> > > to file an issue and patch the text of the policy.
> > >
> > > This discussion started initially in MAPREDUCE-4052, which involved
> > > changing our scripting syntax for MapReduce YARN container submissions.
> >  We
> > > settled the question there by gating the syntax change behind a
> > > configuration option.  By def

Build failed in Jenkins: Hadoop-Hdfs-trunk #1707

[Apache Jenkins Server]

See 

Changes:

[wheat9] HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. Contributed 
by Haohui Mai.

[vinodkv] MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and 
block location calls in parallel. Contributed by Siddharth Seth.

[vinodkv] YARN-1640. Fixed manual failover of ResourceManagers to work 
correctly in secure clusters. Contributed by Xuan Gong.

[wheat9] HDFS-6105. NN web UI for DN list loads the same jmx page multiple 
times. Contributed by Haohui Mai.

[szetszwo] HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException 
and SocketTimeoutException.

[arp] HADOOP-10221. Add file missed in previous checkin, fix typo.

[arp] HADOOP-10221. Add a plugin to specify SaslProperties for RPC protocol 
based on connection properties. (Contributed by Benoy Antony and Daryn Sharp)

[jing9] HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. 
Contributed by Haohui Mai.

--
[...truncated 12950 lines...]
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:974)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:855)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:701)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:108)

testFailuretoReadEdits[1](org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits)
  Time elapsed: 0.818 sec  <<< ERROR!
java.net.BindException: Port in use: localhost:10001
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at 
org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at 
org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:854)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:795)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:614)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:510)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:671)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:656)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1309)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:974)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:855)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:701)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:374)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
at 
org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster.(MiniQJMHACluster.java:95)
at 
org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster.(MiniQJMHACluster.java:36)
at 
org.apache.hadoop.hdfs.qjournal.MiniQJMHACluster$Builder.build(MiniQJMHACluster.java:64)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.setUpCluster(TestFailureToReadEdits.java:116)

testCheckpointStartingMidEditsFile[1](org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits)
  Time elapsed: 0.267 sec  <<< ERROR!
java.net.BindException: Port in use: localhost:10001
at sun.nio.ch.Net.bind(Native Method)
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at 
org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at 
org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:854)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:795)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:132)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:614)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:510)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:671)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:656)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1309)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:974)
at 
org.apache.hadoop.h

Hadoop-Hdfs-trunk - Build # 1707 - Failure

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1707/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 13143 lines...]
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE 
[2:08:27.724s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [3.995s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 2:08:33.290s
[INFO] Finished at: Thu Mar 20 13:44:13 UTC 2014
[INFO] Final Memory: 33M/328M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating HADOOP-10221
Updating HDFS-6123
Updating HDFS-6105
Updating MAPREDUCE-2349
Updating HDFS-6127
Updating HDFS-6100
Updating YARN-1640
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
8 tests failed.
REGRESSION:  
org.apache.hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy.randomizedTest1

Error Message:
Expected 7631 to be in range (6607,7607)

Stack Trace:
java.lang.AssertionError: Expected 7631 to be in range (6607,7607)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.hadoop.test.GenericTestUtils.assertValueWithinRange(GenericTestUtils.java:343)
at 
org.apache.hadoop.test.GenericTestUtils.assertValueNear(GenericTestUtils.java:338)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy.doRandomizedTest(TestAvailableSpaceVolumeChoosingPolicy.java:296)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy.randomizedTest1(TestAvailableSpaceVolumeChoosingPolicy.java:230)


REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.testFailuretoReadEdits[0]

Error Message:

Unfinished stubbing detected here:
-> at 
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits.causeFailureOnEditLogRead(TestFailureToReadEdits.java:302)

E.g. thenReturn() may be missing.
Examples of correct stubbing:
when(mock.isOk()).thenReturn(true);
when(mock.isOk()).thenThrow(exception);
doThrow(exception).when(mock).someVoidMethod();
Hints:
 1. missing thenReturn()
 2. although stubbed methods may return mocks, you cannot inline mock creation 
(mock()) call inside a thenReturn

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

[Chris Nauroth]

Our use of protobuf helps mitigate a lot of compatibility concerns, but
there still can be situations that require careful coding on our part.
 When adding a new field to a protobuf message, the client might need to do
a null check, even if the server-side implementation in the new version
always populates the field.  When adding a whole new RPC endpoint, the
client might need to consider the possibility that the RPC endpoint isn't
there on an old server, and degrade gracefully after the RPC fails.  The
original issue in MAPREDUCE-4052 concerned the script commands passed in a
YARN container submission, where protobuf doesn't provide any validation
beyond the fact that they're strings.

Forward compatibility is harder than backward compatibility, and testing is
a big challenge.  Our test suites in the Hadoop repo don't cover this.
 Does anyone know if anything in Bigtop tries to run with mixed versions?

I agree that we need to make it clear in the language that upgrading client
alone is insufficient to get access to new server-side features, including
new YARN APIs.  Thanks for the suggestions, Steve.

Chris Nauroth
Hortonworks
http://hortonworks.com/



On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran wrote:

> I'm clearly supportive of this, though of course the testing costs needed
> to back up the assertion make it more expensive than just a statement.
>
> Two issues
>
> -we'd need to make clear that new cluster features that a client can invoke
> won't be available. You can't expect snapshot or symlink support running
> against a -2.2.0 cluster, even if the client supports it.
>
> -in YARN, there are no guarantees that an app compiled against later YARN
> APIs will work in old clusters. Because YARN apps upload themselves to the
> server, and run with their hadoop, hdfs & yarn libraries. We have to do a
> bit of introspection in our code already to support this situation. The
> compatibility doc would need to be clear on that too: "YARN apps that use
> new APIs (including new fields in datastructures) can expect link
> exceptions"
>
>
>
>
>
> On 20 March 2014 04:25, Vinayakumar B  wrote:
>
> > +1, I agree with your point Chris. It depends on the client application
> > how they using the hdfs jars in their classpath.
> >
> > As implementation already supports the compatibility (through protobuf),
> > No extra code changes required to support new Client + old server.
> >
> > I feel it will be good to explicitly mention about the compatibility of
> > existing APIs in both versions.
> >
> > Anyway this is not applicable for the new APIs in latest client and this
> > is understood. We can make it explicit in the document though.
> >
> >
> > Regards,
> > Vinayakumar B
> >
> > -Original Message-
> > From: Chris Nauroth [mailto:cnaur...@hortonworks.com]
> > Sent: 20 March 2014 05:36
> > To: common-...@hadoop.apache.org
> > Cc: mapreduce-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> > yarn-...@hadoop.apache.org
> > Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
> > Client + Old Server
> >
> > I think this kind of compatibility issue still could surface for HDFS,
> > particularly for custom applications (i.e. something not executed via
> > "hadoop jar" on a cluster node, where the client classes ought to be
> > injected into the classpath automatically).  Running DistCP between 2
> > clusters of different versions could result in a 2.4.0 client calling a
> > 2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
> > client as a dependency and try to use it to make HTTP calls to a 2.3.0
> HDFS
> > cluster.
> >
> > Chris Nauroth
> > Hortonworks
> > http://hortonworks.com/
> >
> >
> >
> > On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli <
> > vino...@apache.org
> > > wrote:
> >
> > > It makes sense only for YARN today where we separated out the clients.
> > > HDFS is still a monolithic jar so this compatibility issue is kind of
> > > invalid there.
> > >
> > > +vinod
> > >
> > > On Mar 19, 2014, at 1:59 PM, Chris Nauroth 
> > > wrote:
> > >
> > > > I'd like to discuss clarification of part of our compatibility
> policy.
> > > > Here is a link to the compatibility documentation for release 2.3.0:
> > > >
> > > >
> > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
> > > /Compatibility.html#Wire_compatibility
> > > >
> > > > For convenience, here are the specific lines in question:
> > > >
> > > > Client-Server compatibility is required to allow users to continue
> > > > using the old clients even after upgrading the server (cluster) to a
> > > > later version (or vice versa). For example, a Hadoop 2.1.0 client
> > > > talking to a Hadoop 2.3.0 cluster.
> > > >
> > > > Client-Server compatibility is also required to allow upgrading
> > > individual
> > > > components without upgrading others. For example, upgrade HDFS from
> > > version
> > > > 2.1.0 to 2.2.0 without upgrading MapReduce.
> > > >
> > > > Server-Server compatibility i

[jira] [Resolved] (HDFS-6131) Move HDFSHighAvailabilityWithNFS.apt.vm and HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS

[Jing Zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao resolved HDFS-6131.
-

   Resolution: Fixed
Fix Version/s: 2.4.0
 Hadoop Flags: Reviewed

Thanks very much to Arpit for the review! I've committed this to branch-2 and 
branch-2.4.0.

> Move HDFSHighAvailabilityWithNFS.apt.vm and 
> HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS
> 
>
> Key: HDFS-6131
> URL: https://issues.apache.org/jira/browse/HDFS-6131
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.3.0
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.4.0
>
> Attachments: HDFS-6131.000.patch, HDFS-6131.001.patch
>
>
> Currently in branch-2, the document HDFSHighAvailabilityWithNFS.apt.vm and 
> HDFSHighAvailabilityWithQJM.apt.vm are still in the Yarn project. We should 
> move them to HDFS just like in trunk.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6132) TestBlockToken fails on JDK 7

[Siqi Li (JIRA)]

Siqi Li created HDFS-6132:
-

 Summary: TestBlockToken fails on JDK 7
 Key: HDFS-6132
 URL: https://issues.apache.org/jira/browse/HDFS-6132
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Siqi Li






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6133) Make Balancer support don't move blocks belongs to Hbase

[zhaoyunjiong (JIRA)]

zhaoyunjiong created HDFS-6133:
--

 Summary: Make Balancer support don't move blocks belongs to Hbase
 Key: HDFS-6133
 URL: https://issues.apache.org/jira/browse/HDFS-6133
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: balancer, namenode
Reporter: zhaoyunjiong
Assignee: zhaoyunjiong


Currently, run Balancer will destroying Regionserver's data locality.
If getBlocks could exclude blocks belongs to files which have specific path 
prefix, like "/hbase", then we can run Balancer without destroying 
Regionserver's data locality.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (HDFS-5138) Support HDFS upgrade in HA

[Suresh Srinivas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas reopened HDFS-5138:
---


> Support HDFS upgrade in HA
> --
>
> Key: HDFS-5138
> URL: https://issues.apache.org/jira/browse/HDFS-5138
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.1.1-beta
>Reporter: Kihwal Lee
>Assignee: Aaron T. Myers
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, 
> HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, 
> HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, 
> hdfs-5138-branch-2.txt
>
>
> With HA enabled, NN wo't start with "-upgrade". Since there has been a layout 
> version change between 2.0.x and 2.1.x, starting NN in upgrade mode was 
> necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way 
> to get around this was to disable HA and upgrade. 
> The NN and the cluster cannot be flipped back to HA until the upgrade is 
> finalized. If HA is disabled only on NN for layout upgrade and HA is turned 
> back on without involving DNs, things will work, but finaliizeUpgrade won't 
> work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade 
> snapshots won't get removed.
> We will need a different ways of doing layout upgrade and upgrade snapshot.  
> I am marking this as a 2.1.1-beta blocker based on feedback from others.  If 
> there is a reasonable workaround that does not increase maintenance window 
> greatly, we can lower its priority from blocker to critical.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6134) Transparent data at rest encryption

[Alejandro Abdelnur (JIRA)]

Alejandro Abdelnur created HDFS-6134:


 Summary: Transparent data at rest encryption
 Key: HDFS-6134
 URL: https://issues.apache.org/jira/browse/HDFS-6134
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: security
Affects Versions: 2.3.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur


Because of privacy and security regulations, for many industries, sensitive 
data at rest must be in encrypted form. For example: the health­care industry 
(HIPAA regulations), the card payment industry (PCI DSS regulations) or the US 
government (FISMA regulations).

This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be 
used transparently by any application accessing HDFS via Hadoop Filesystem Java 
API, Hadoop libhdfs C library, or WebHDFS REST API.

The resulting implementation should be able to be used in compliance with 
different regulation requirements.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6135) In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back

[Jing Zhao (JIRA)]

Jing Zhao created HDFS-6135:
---

 Summary: In HDFS upgrade with HA setup, JournalNode cannot handle 
layout version bump when rolling back
 Key: HDFS-6135
 URL: https://issues.apache.org/jira/browse/HDFS-6135
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Jing Zhao


While doing HDFS upgrade with HA setup, if the layoutversion gets changed in 
the upgrade, the rollback may trigger the following exception in JournalNodes 
(suppose the new software bumped the layoutversion from -55 to -56):
{code}
14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if roll 
back possible for one or more JournalNodes. 1 exceptions thrown:
Unexpected version of storage directory /grid/1/tmp/journal/mycluster. 
Reported: -56. Expecting = -55.
at 
org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:203)
at 
org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:156)
at 
org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:135)
at 
org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
at 
org.apache.hadoop.hdfs.qjournal.server.JNStorage.(JNStorage.java:73)
at 
org.apache.hadoop.hdfs.qjournal.server.Journal.(Journal.java:142)
at 
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
at 
org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:304)
at 
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
{code}

Looks like for rollback JN with old software cannot handle future layoutversion 
brought by new software.



--
This message was sent by Atlassian JIRA
(v6.2#6252)