[jira] [Created] (HDFS-6360) MiniDFSCluster can cause unexpected side effects due to sharing of config

[Kihwal Lee (JIRA)]

Kihwal Lee created HDFS-6360:


 Summary: MiniDFSCluster can cause unexpected side effects due to 
sharing of config
 Key: HDFS-6360
 URL: https://issues.apache.org/jira/browse/HDFS-6360
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Kihwal Lee


As noted in HDFS-6329 and HDFS-5522, certain use cases of MiniDFSCluster can 
result in unexpected results and falsely failing or passing unit tests.

Since a {{Configuration}} object is shared for all namenode startups, the 
modified conf object during a NN startup is passed to the next NN startup.  The 
effect of the modified conf propagation and subsequent modifications is 
different depending on whether it is a single NN cluster, HA cluster or 
federation cluster.

It also depends on what test cases are doing with the config. For example, 
MiniDFSCluster#getConfiguration(int) returns the saved conf for the specified 
NN, but that is not actually the conf object used by the NN. It just contained 
the same content one time in the past and it is not guaranteed to be that way.

Restarting the same NN can also cause unexpected results. The new NN will 
switch to the conf that was cloned & saved AFTER the last startup.  The new NN 
will start with a changed config intentionally or unintentionally.  The config 
variables such as {{fs.defaultFs}}, {{dfs.namenode.rpc-address}} will be 
implicitly set differently than the initial condition.  Some test cases rely on 
this and others occasionally break because of this.

In summary,
* MiniDFSCluster does not properly isolate configs.
* Many test cases happen to work most of times. Correcting MiniDFSCluster 
causes mass breakages of test cases and requires fixing them.
* Many test cases rely on broken behavior and might pass when they should have 
actually failed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6359) WebHdfs NN servlet issues redirects in safemode or standby

[Daryn Sharp (JIRA)]

Daryn Sharp created HDFS-6359:
-

 Summary: WebHdfs NN servlet issues redirects in safemode or standby
 Key: HDFS-6359
 URL: https://issues.apache.org/jira/browse/HDFS-6359
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical


Webhdfs does not check for safemode or standby during issuing a redirect for 
open/create/checksum calls.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6358) WebHdfs DN's DFSClient should not use a retry policy

[Daryn Sharp (JIRA)]

Daryn Sharp created HDFS-6358:
-

 Summary: WebHdfs DN's DFSClient should not use a retry policy
 Key: HDFS-6358
 URL: https://issues.apache.org/jira/browse/HDFS-6358
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical


DFSClient retries on the DN are useless.  The webhdfs client is going to 
timeout before the retries complete.  The DFSClient will also continue to run 
until it timeouts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6363) Improve concurrency while checking inclusion and exclusion of datanodes

[Benoy Antony (JIRA)]

Benoy Antony created HDFS-6363:
--

 Summary: Improve concurrency while checking inclusion and 
exclusion of datanodes
 Key: HDFS-6363
 URL: https://issues.apache.org/jira/browse/HDFS-6363
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Benoy Antony
Assignee: Benoy Antony
 Attachments: HDFS-6363.patch

HostFileManager holds two effectively immutable objects - includes and 
excludes. These two objects can be safely published together using a volatile 
container instead of synchronizing for all mutators and accessors.
This improves the concurrency while using HostFileManager.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6364) Incorrect check for unknown datanode in Balancer

[Benoy Antony (JIRA)]

Benoy Antony created HDFS-6364:
--

 Summary: Incorrect check for unknown datanode in Balancer
 Key: HDFS-6364
 URL: https://issues.apache.org/jira/browse/HDFS-6364
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: balancer
Affects Versions: 2.4.0
Reporter: Benoy Antony
Assignee: Benoy Antony
 Attachments: HDFS-6364.patch

The Balancer makes a check to see if a block's location is known datanode. But 
the variable it uses to check is wrong.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6366) FsImage loading failed with RemoveXattr op

[Uma Maheswara Rao G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uma Maheswara Rao G resolved HDFS-6366.
---

   Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
 Hadoop Flags: Reviewed

Thanks for the review Yi. I have just committed this to branch!

> FsImage loading failed with RemoveXattr op
> --
>
> Key: HDFS-6366
> URL: https://issues.apache.org/jira/browse/HDFS-6366
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: HDFS XAttrs (HDFS-2006)
>
> Attachments: HDFS-6366.patch
>
>
> Should break after the removeXattr op loading.
> Otherwise fsimage loading will fail:
> java.io.IOException: Invalid operation read OP_REMOVE_XATTR
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:816)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:227)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:136)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:902)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:651)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:479)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:535)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:694)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:679)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1332)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Jenkins build is back to normal : Hadoop-Hdfs-trunk #1752

[Apache Jenkins Server]

See 

Hadoop-Hdfs-trunk - Build # 1751 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1751/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 12874 lines...]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ hadoop-hdfs-project 
---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE 
[2:05:08.365s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [4.308s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 2:05:14.035s
[INFO] Finished at: Thu May 08 14:34:37 UTC 2014
[INFO] Final Memory: 31M/310M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Updating MAPREDUCE-5636
Updating HADOOP-10562
Updating HADOOP-10541
Updating MAPREDUCE-5402
Updating HADOOP-10467
Updating HADOOP-10566
Updating HADOOP-10556
Updating HADOOP-10433
Updating HADOOP-10471
Updating YARN-1864
Updating YARN-2018
Updating HDFS-6337
Updating HADOOP-10517
Updating MAPREDUCE-5637
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality

Error Message:
expected:<1800> but was:<1810>

Stack Trace:
java.lang.AssertionError: expected:<1800> but was:<1810>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)

[jira] [Created] (HDFS-6370) Web UI fails to display in intranet under IE

[Haohui Mai (JIRA)]

Haohui Mai created HDFS-6370:


 Summary: Web UI fails to display in intranet under IE
 Key: HDFS-6370
 URL: https://issues.apache.org/jira/browse/HDFS-6370
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai


When IE renders the web UI of a cluster than runs in the intranet, it forces 
the compatibility mode to be turned on which causes the UI fails to render 
correctly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()

[Ted Yu (JIRA)]

Ted Yu created HDFS-6369:


 Summary: RemoteBlockReader#available() should call 
FSInputChecker.available()
 Key: HDFS-6369
 URL: https://issues.apache.org/jira/browse/HDFS-6369
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Trivial


Currently DFSClient.TCP_WINDOW_SIZE is directly returned.
However, FSInputChecker.available(), in the superclass, may return value lower 
than the constant.



--
This message was sent by Atlassian JIRA
(v6.2#6252)