[jira] [Created] (HDDS-2205) checkstyle.sh reports wrong failure count
Attila Doroszlai created HDDS-2205:
--
Summary: checkstyle.sh reports wrong failure count
Key: HDDS-2205
URL: https://issues.apache.org/jira/browse/HDDS-2205
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: test
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
{{checkstyle.sh}} outputs files with checkstyle violations and the violations
themselves on separate lines. It then reports line count as number of failures.
{code:title=target/checkstyle/summary.txt}
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
49: Unused import - org.apache.hadoop.ozone.om.OMMetadataManager.
{code}
{code:title=target/checkstyle/failures}
2
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [DISCUSS] Release Docs pointers Hadoop site
Bumping up this thread again for feedback. @Zhankun Tang is now waiting for a confirmation to complete 3.1.3 release publish activities. - Sunil On Fri, Sep 27, 2019 at 11:03 AM Sunil Govindan wrote: > Hi Folks, > > At present, > http://hadoop.apache.org/docs/stable/ points to *Apache Hadoop 3.2.1* > http://hadoop.apache.org/docs/current/ points to *Apache Hadoop 3.2.1* > http://hadoop.apache.org/docs/stable2/ points to *Apache Hadoop 2.9.2* > http://hadoop.apache.org/docs/current2/ points to *Apache Hadoop 2.9.2* > > 3.2.1 is released last day. *Now 3.1.3 has completed voting* and it is in > the final stages of staging > As per me, > a) 3.2.1 will be still be pointing to > http://hadoop.apache.org/docs/stable/ ? > b) 3.1.3 should be pointing to http://hadoop.apache.org/docs/current/ ? > > Now my questions, > 1. But if the release manager of 3.1 line thinks 3.1.3 is stable, and 3.2 > line is also in stable state, which release should get precedence to be > called as *stable* in any release line (2.x or 3.x) ? > or do we need a vote or discuss thread to decide which release shall be > called as stable per release line? > 2. Given 3.2.1 is released and pointing to 3.2.1 as stable, then when > 3.1.3 is getting released now, could > http://hadoop.apache.org/docs/current/ shall be updated to 3.1.3 ? is it > the norms ? > > Thanks > Sunil >
[jira] [Created] (HDDS-2206) Separate handling for OMException and IOException in the Ozone Manager
Supratim Deka created HDDS-2206: --- Summary: Separate handling for OMException and IOException in the Ozone Manager Key: HDDS-2206 URL: https://issues.apache.org/jira/browse/HDDS-2206 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Supratim Deka Assignee: Supratim Deka As part of improving error propagation from the OM for ease of troubleshooting and diagnosis, the proposal is to handle IOExceptions separately from the business exceptions which are thrown as OMExceptions. Handling for OMExceptions will not be changed in this jira. Handling for IOExceptions will include logging the stacktrace on the server, and propagation to the client under the control of a config parameter. Similar handling is also proposed for SCMException. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2207) Update Ratis to latest snnapshot
Shashikant Banerjee created HDDS-2207: - Summary: Update Ratis to latest snnapshot Key: HDDS-2207 URL: https://issues.apache.org/jira/browse/HDDS-2207 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.5.0 This Jira aims to update ozone with latest ratis snapshot which has a crtical fix for retry behaviour on getting not leader exception in client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2208) OzoneManagerStateMachine does not track failures in applyTransaction
Supratim Deka created HDDS-2208: --- Summary: OzoneManagerStateMachine does not track failures in applyTransaction Key: HDDS-2208 URL: https://issues.apache.org/jira/browse/HDDS-2208 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Supratim Deka Assignee: Supratim Deka applyTransaction handling in the OzoneManagerStateMachine does not propagate exceptions/failures to the initiator. The future which is returned from applyTransaction simply tracks completion of the async executor represented by the "executorService" in OzoneManagerStateMachine.java -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14882) Consider DataNode load when #getBlockLocation
Xiaoqiao He created HDFS-14882: -- Summary: Consider DataNode load when #getBlockLocation Key: HDFS-14882 URL: https://issues.apache.org/jira/browse/HDFS-14882 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Xiaoqiao He Assignee: Xiaoqiao He Currently, we consider load of datanode when #chooseTarget for writer, however not consider it for reader. Thus, the process slot of datanode could be occupied by #BlockSender for reader, and disk/network will be busy workload, then meet some slow node exception. IIRC same case is reported times. Based on the fact, I propose to consider load for reader same as it did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/ No changes -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client Boxed value is unboxed and then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:then immediately reboxed in org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result, byte[], byte[], KeyConverter, ValueConverter, boolean) At ColumnRWHelper.java:[line 335] Failed junit tests : hadoop.util.TestReadWriteDiskValidator hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA hadoop.hdfs.tools.TestDFSAdminWithHA hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend hadoop.registry.secure.TestSecureLogins hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.yarn.client.api.impl.TestAMRMClient cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt [328K] cc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-compile-cc-root-jdk1.8.0_222.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-compile-javac-root-jdk1.8.0_222.txt [308K] checkstyle: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-checkstyle-root.txt [16M] hadolint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-patch-hadolint.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-patch-shellcheck.txt [72K] shelldocs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-patch-shelldocs.txt [8.0K] whitespace: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/whitespace-eol.txt [12M] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/whitespace-tabs.txt [1.3M] xml: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/xml.txt [12K] findbugs: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-client-warnings.html [8.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-javadoc-javadoc-root-jdk1.7.0_95.txt [16K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/diff-javadoc-javadoc-root-jdk1.8.0_222.txt [1.1M] unit: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [160K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [240K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [24K] https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/460/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-re
[jira] [Created] (HDFS-14883) NPE when the second SNN is starting
Ranith Sardar created HDFS-14883:
Summary: NPE when the second SNN is starting
Key: HDFS-14883
URL: https://issues.apache.org/jira/browse/HDFS-14883
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ranith Sardar
Assignee: Ranith Sardar
{{2019-09-25 22:41:31,889 | WARN | qtp79782883-47 | /imagetransfer |
ServletHandler.java:632
java.io.IOException: PutImage failed. java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.ImageServlet.validateRequest(ImageServlet.java:198)
at
org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:485)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2209) Checkstyle issue in OmUtils on trunk
Marton Elek created HDDS-2209:
-
Summary: Checkstyle issue in OmUtils on trunk
Key: HDDS-2209
URL: https://issues.apache.org/jira/browse/HDDS-2209
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Marton Elek
HDDS-2174 introduced a new checkstyle error:
{code:java}
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
49: Unused import - org.apache.hadoop.ozone.om.OMMetadataManager. {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2209) Checkstyle issue in OmUtils on trunk
[
https://issues.apache.org/jira/browse/HDDS-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia resolved HDDS-2209.
-
Resolution: Duplicate
> Checkstyle issue in OmUtils on trunk
> -
>
> Key: HDDS-2209
> URL: https://issues.apache.org/jira/browse/HDDS-2209
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Marton Elek
>Priority: Trivial
> Labels: newbie, newbie++
>
> HDDS-2174 introduced a new checkstyle error:
> {code:java}
> hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
> 49: Unused import - org.apache.hadoop.ozone.om.OMMetadataManager. {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2207) Update Ratis to latest snapshot
[ https://issues.apache.org/jira/browse/HDDS-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDDS-2207. - Resolution: Fixed Thanks for working on this [~shashikant]. I have committed this to trunk. > Update Ratis to latest snapshot > --- > > Key: HDDS-2207 > URL: https://issues.apache.org/jira/browse/HDDS-2207 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 40m > Remaining Estimate: 0h > > This Jira aims to update ozone with latest ratis snapshot which has a crtical > fix for retry behaviour on getting not leader exception in client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2210) ContainerStateMachine should not be marked unhealthy if applyTransaction fails with closed container exception
Shashikant Banerjee created HDDS-2210: - Summary: ContainerStateMachine should not be marked unhealthy if applyTransaction fails with closed container exception Key: HDDS-2210 URL: https://issues.apache.org/jira/browse/HDDS-2210 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: 0.5.0 Currently, if applyTransaction fails, the stateMachine is marked unhealthy and next snapshot creation will fail. As a result of which the the raftServer will close down leading to pipeline failure. ClosedContainer exception should be ignored while marking the stateMachine unhealthy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14884) Add sanity check that zone key equals feinfo key while setting Xattrs
Mukul Kumar Singh created HDFS-14884: Summary: Add sanity check that zone key equals feinfo key while setting Xattrs Key: HDFS-14884 URL: https://issues.apache.org/jira/browse/HDFS-14884 Project: Hadoop HDFS Issue Type: Bug Components: encryption, hdfs Reporter: Mukul Kumar Singh Currently, it is possible to set an external attribute where the zone key is not the same as feinfo key. This jira will add a precondition before setting this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2211) Collect docker logs if env fails to start
Attila Doroszlai created HDDS-2211: -- Summary: Collect docker logs if env fails to start Key: HDDS-2211 URL: https://issues.apache.org/jira/browse/HDDS-2211 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: test Reporter: Attila Doroszlai Assignee: Attila Doroszlai Occasionally some acceptance test docker environment fails to start up properly. We need docker logs for analysis, but they are not being collected. https://github.com/elek/ozone-ci-q4/blob/master/trunk/trunk-nightly-extra-20190930-74rp4/acceptance/output.log#L3765-L3768 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[INFO] Ozone/Ratis Community meeting notes
https://cwiki.apache.org/confluence/display/HADOOP/2019-09-30+Meeting+notes --Anu
[jira] [Created] (HDDS-2212) Genconf tool should generate config files for secure cluster setup
Dinesh Chitlangia created HDDS-2212: --- Summary: Genconf tool should generate config files for secure cluster setup Key: HDDS-2212 URL: https://issues.apache.org/jira/browse/HDDS-2212 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Dinesh Chitlangia Ozone Genconf tool currently generates a minimal ozone-site.xml file. [~raje2411] was trying out a secure ozone setup over existing HDP-2.x cluster and found the config set up was not as straight forward. This jira proposes to extend the Genconf tool so we can generate required template config files for a secure setup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-7134) Replication count for a block should not update till the blocks have settled on Datanodes
[ https://issues.apache.org/jira/browse/HDFS-7134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-7134: --- > Replication count for a block should not update till the blocks have settled > on Datanodes > - > > Key: HDFS-7134 > URL: https://issues.apache.org/jira/browse/HDFS-7134 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 1.2.1, 2.6.0, 2.7.3 > Environment: Linux nn1.cluster1.com 2.6.32-431.20.3.el6.x86_64 #1 SMP > Thu Jun 19 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > [hadoop@nn1 conf]$ cat /etc/redhat-release > CentOS release 6.5 (Final) >Reporter: gurmukh singh >Priority: Critical > Labels: HDFS > Fix For: 3.1.0 > > > The count for the number of replica's for a block should not change till the > blocks have settled on the datanodes. > Test Case: > Hadoop Cluster with 1 namenode and 3 datanodes. > nn1.cluster1.com(192.168.1.70) > dn1.cluster1.com(192.168.1.72) > dn2.cluster1.com(192.168.1.73) > dn3.cluster1.com(192.168.1.74) > Cluster up and running fine with replication set to "1" for parameter > "dfs.replication on all nodes" > > dfs.replication > 1 > > To reduce the wait time, have reduced the dfs.heartbeat and recheck > parameters. > on datanode2 (192.168.1.72) > [hadoop@dn2 ~]$ hadoop fs -Ddfs.replication=2 -put from_dn2 / > [hadoop@dn2 ~]$ hadoop fs -ls /from_dn2 > Found 1 items > -rw-r--r-- 2 hadoop supergroup 17 2014-09-23 13:33 /from_dn2 > On Namenode > === > As expected, copy was done from datanode2, one copy will go locally. > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 13:53:16 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=2 [192.168.1.74:50010, > 192.168.1.73:50010] > Can see the blocks on the data nodes disks as well under the "current" > directory. > Now, shutdown datanode2(192.168.1.73) and as expected block moves to another > datanode to maintain a replication of 2 > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 13:54:21 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=2 [192.168.1.74:50010, > 192.168.1.72:50010] > But, now if i bring back the datanode2, and although the namenode see that > this block is at 3 places now and fires a invalidate command for > datanode1(192.168.1.72) but the replication on the namenode is bumped to 3 > immediately. > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 13:56:12 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=3 [192.168.1.74:50010, > 192.168.1.72:50010, 192.168.1.73:50010] > on Datanode1 - The invalidate command has been fired immediately and the > block deleted. > = > 2014-09-23 13:54:17,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Receiving blk_8132629811771280764_1175 src: /192.168.1.74:38099 dest: > /192.168.1.72:50010 > 2014-09-23 13:54:17,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Received blk_8132629811771280764_1175 src: /192.168.1.74:38099 dest: > /192.168.1.72:50010 size 17 > 2014-09-23 13:55:28,720 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Scheduling blk_8132629811771280764_1175 file > /space/disk1/current/blk_8132629811771280764 for deletion > 2014-09-23 13:55:28,721 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Deleted blk_8132629811771280764_1175 at file > /space/disk1/current/blk_8132629811771280764 > The namenode still shows 3 replica's. even if one has been deleted, even > after more then 30 mins. > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 14:21:27 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=3 [192.168.1.74:50010, > 192.168.1.72:50010, 192.168.1.73:50010] > This could be a dangerous, if someone remove or other 2 datanodes fail. > On Datanode 1 > = > Before, the datanode1 is brought back > [hadoop@dn1 conf]$ ls -l /space/disk*/current > /space/disk1/current: > total 28 > -rw-rw-r-- 1 hadoop hadoop 13 Sep 21 09:09 blk_2278001646987517832 > -rw-rw-r-- 1 hadoop hadoop 11 Sep 21 09:09 blk_2278001646987517832_1171.meta > -rw-rw-r-- 1 hadoop hadoop 17 Sep 23 13:54 blk_8132629811771280764 > -rw-rw-r-- 1 hadoop hadoop 11 Sep 23 13:54 blk_8132629811771280764_1175.meta > -rw-rw-r--
[jira] [Resolved] (HDFS-7134) Replication count for a block should not update till the blocks have settled on Datanodes
[ https://issues.apache.org/jira/browse/HDFS-7134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-7134. --- Resolution: Cannot Reproduce Resolve as cannot reproduce. > Replication count for a block should not update till the blocks have settled > on Datanodes > - > > Key: HDFS-7134 > URL: https://issues.apache.org/jira/browse/HDFS-7134 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 1.2.1, 2.6.0, 2.7.3 > Environment: Linux nn1.cluster1.com 2.6.32-431.20.3.el6.x86_64 #1 SMP > Thu Jun 19 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > [hadoop@nn1 conf]$ cat /etc/redhat-release > CentOS release 6.5 (Final) >Reporter: gurmukh singh >Priority: Critical > Labels: HDFS > Fix For: 3.1.0 > > > The count for the number of replica's for a block should not change till the > blocks have settled on the datanodes. > Test Case: > Hadoop Cluster with 1 namenode and 3 datanodes. > nn1.cluster1.com(192.168.1.70) > dn1.cluster1.com(192.168.1.72) > dn2.cluster1.com(192.168.1.73) > dn3.cluster1.com(192.168.1.74) > Cluster up and running fine with replication set to "1" for parameter > "dfs.replication on all nodes" > > dfs.replication > 1 > > To reduce the wait time, have reduced the dfs.heartbeat and recheck > parameters. > on datanode2 (192.168.1.72) > [hadoop@dn2 ~]$ hadoop fs -Ddfs.replication=2 -put from_dn2 / > [hadoop@dn2 ~]$ hadoop fs -ls /from_dn2 > Found 1 items > -rw-r--r-- 2 hadoop supergroup 17 2014-09-23 13:33 /from_dn2 > On Namenode > === > As expected, copy was done from datanode2, one copy will go locally. > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 13:53:16 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=2 [192.168.1.74:50010, > 192.168.1.73:50010] > Can see the blocks on the data nodes disks as well under the "current" > directory. > Now, shutdown datanode2(192.168.1.73) and as expected block moves to another > datanode to maintain a replication of 2 > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 13:54:21 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=2 [192.168.1.74:50010, > 192.168.1.72:50010] > But, now if i bring back the datanode2, and although the namenode see that > this block is at 3 places now and fires a invalidate command for > datanode1(192.168.1.72) but the replication on the namenode is bumped to 3 > immediately. > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 13:56:12 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=3 [192.168.1.74:50010, > 192.168.1.72:50010, 192.168.1.73:50010] > on Datanode1 - The invalidate command has been fired immediately and the > block deleted. > = > 2014-09-23 13:54:17,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Receiving blk_8132629811771280764_1175 src: /192.168.1.74:38099 dest: > /192.168.1.72:50010 > 2014-09-23 13:54:17,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Received blk_8132629811771280764_1175 src: /192.168.1.74:38099 dest: > /192.168.1.72:50010 size 17 > 2014-09-23 13:55:28,720 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Scheduling blk_8132629811771280764_1175 file > /space/disk1/current/blk_8132629811771280764 for deletion > 2014-09-23 13:55:28,721 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Deleted blk_8132629811771280764_1175 at file > /space/disk1/current/blk_8132629811771280764 > The namenode still shows 3 replica's. even if one has been deleted, even > after more then 30 mins. > [hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations > FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23 > 14:21:27 IST 2014 > /from_dn2 17 bytes, 1 block(s): OK > 0. blk_8132629811771280764_1175 len=17 repl=3 [192.168.1.74:50010, > 192.168.1.72:50010, 192.168.1.73:50010] > This could be a dangerous, if someone remove or other 2 datanodes fail. > On Datanode 1 > = > Before, the datanode1 is brought back > [hadoop@dn1 conf]$ ls -l /space/disk*/current > /space/disk1/current: > total 28 > -rw-rw-r-- 1 hadoop hadoop 13 Sep 21 09:09 blk_2278001646987517832 > -rw-rw-r-- 1 hadoop hadoop 11 Sep 21 09:09 blk_2278001646987517832_1171.meta > -rw-rw-r-- 1 hadoop hadoop 17 Sep 23 13:54 blk_8132629811771280764 > -rw-rw-r-- 1 hadoop hadoop
[jira] [Created] (HDDS-2213) Reduce key provider loading log level in OzoneFileSystem#getAdditionalTokenIssuers
Xiaoyu Yao created HDDS-2213:
Summary: Reduce key provider loading log level in
OzoneFileSystem#getAdditionalTokenIssuers
Key: HDDS-2213
URL: https://issues.apache.org/jira/browse/HDDS-2213
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Vivek Ratnavel Subramanian
OzoneFileSystem#getAdditionalTokenIssuers log an error when secure client tries
to collect ozone delegation token to run MR/Spark jobs but ozone file system
does not have a kms provider configured. In this case, we simply return null
provider here in the code below. This is a benign error and we should reduce
the log level to debug level.
\{code}
KeyProvider keyProvider;
try {
keyProvider = getKeyProvider();
} catch (IOException ioe) {
LOG.error("Error retrieving KeyProvider.", ioe);
return null;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/461/ [Sep 30, 2019 8:24:32 PM] (iwasakims) HADOOP-16544. update io.netty in branch-2. [Oct 1, 2019 1:04:16 AM] (shv) HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14885) UI: Fix a typo in
Xieming Li created HDFS-14885:
-
Summary: UI: Fix a typo in
Key: HDFS-14885
URL: https://issues.apache.org/jira/browse/HDFS-14885
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, ui
Reporter: Xieming Li
Assignee: Xieming Li
Attachments: Screen Shot 2019-10-01 at 12.40.29.png
A Period('.') should be added to the end of following sentence on WebUI of
DataNode.
"No nodes are decommissioning"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2001) Update Ratis version to 0.4.0
[ https://issues.apache.org/jira/browse/HDDS-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar resolved HDDS-2001. --- Resolution: Fixed > Update Ratis version to 0.4.0 > - > > Key: HDDS-2001 > URL: https://issues.apache.org/jira/browse/HDDS-2001 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Update Ratis version to 0.4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
