Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.yarn.server.timelineservice.reader.TestTimelineReaderServer 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.tools.TestDistCpSystem 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-compile-javac-root.txt
  [500K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-mvnsite-root.txt
  [1.2M]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-patch-pylint.txt
  [48K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-javadoc-root.txt
  [32K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [232K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [428K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [40K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [128K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/505/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-b

[jira] [Resolved] (HDFS-16345) Fix test cases fail in TestBlockStoragePolicy

[Ayush Saxena (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayush Saxena resolved HDFS-16345.
-
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Fix test cases fail in TestBlockStoragePolicy
> -
>
> Key: HDFS-16345
> URL: https://issues.apache.org/jira/browse/HDFS-16345
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.3.1
>Reporter: guophilipse
>Assignee: guophilipse
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> test class ``TestBlockStoragePolicy` ` fail frequently for the 
> `BindException`, it block all normal source code build. we can improve it.
> [ERROR] Tests run: 26, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 49.295 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestBlockStoragePolicy 
> [ERROR] 
> testChooseTargetWithTopology(org.apache.hadoop.hdfs.TestBlockStoragePolicy) 
> Time elapsed: 0.551 s <<< ERROR! java.net.BindException: Problem binding to 
> [localhost:43947] java.net.BindException: Address already in use; For more 
> details see: http://wiki.apache.org/hadoop/BindException at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at 
> org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:931) at 
> org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:827) at 
> org.apache.hadoop.ipc.Server.bind(Server.java:657) at 
> org.apache.hadoop.ipc.Server$Listener.(Server.java:1352) at 
> org.apache.hadoop.ipc.Server.(Server.java:3252) at 
> org.apache.hadoop.ipc.RPC$Server.(RPC.java:1062) at 
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server.(ProtobufRpcEngine2.java:468)
>  at 
> org.apache.hadoop.ipc.ProtobufRpcEngine2.getServer(ProtobufRpcEngine2.java:371)
>  at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:853) at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:466)
>  at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:860)
>  at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:766) 
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:1017) 
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) 
> at 
> org.apache.hadoop.hdfs.TestBlockStoragePolicy.testChooseTargetWithTopology(TestBlockStoragePolicy.java:1275)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>  at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>  at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>  at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>  at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>  at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>  at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>  at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at 
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at 
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at 
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at 
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at 
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
> org.junit.runners.ParentRunner.run(ParentRunner.java:413) at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>  at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>  at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>  at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>  at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>  at 
> org.apach

[jira] [Created] (HDFS-16374) Fix error log format in AbfsListStatusRemoteIterator

[guophilipse (Jira)]

guophilipse created HDFS-16374:
--

 Summary: Fix error log format in AbfsListStatusRemoteIterator
 Key: HDFS-16374
 URL: https://issues.apache.org/jira/browse/HDFS-16374
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs
Affects Versions: 3.3.1
Reporter: guophilipse


Fix error log format in AbfsListStatusRemoteIterator



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/712/

[Dec 7, 2021 7:42:57 AM] (Szilard Nemeth) YARN-11014. YARN incorrectly 
validates maximum capacity resources on the validation API. Contributed by 
Benjamin Teke
[Dec 7, 2021 7:51:03 AM] (Szilard Nemeth) YARN-11020. [UI2] No container is 
found for an application attempt with a single AM container. Contributed by 
Andras Gyori
[Dec 7, 2021 8:49:27 AM] (noreply) HDFS-16351. Add path exception information 
in FSNamesystem (#3713). Contributed by guophilipse.
[Dec 7, 2021 12:39:04 PM] (noreply) HDFS-16354. Add description of 
GETSNAPSHOTDIFFLISTING to WebHDFS doc. (#3740)




-1 overall


The following subsystems voted -1:
blanks pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.tools.TestDistCpWithRawXAttrs 
   hadoop.tools.TestDistCpWithXAttrs 
   hadoop.tools.contract.TestLocalContractDistCp 
   hadoop.tools.TestDistCpSync 
   hadoop.tools.TestIntegration 
   hadoop.tools.TestExternalCall 
   hadoop.tools.contract.TestHDFSContractDistCp 
   hadoop.tools.TestDistCpViewFs 
   hadoop.tools.TestDistCpWithAcls 
   hadoop.yarn.service.client.TestSystemServiceManagerImpl 
   hadoop.yarn.service.client.TestApiServiceClient 
   hadoop.yarn.service.TestCleanupAfterKill 
   hadoop.yarn.service.TestApiServer 
   hadoop.yarn.csi.client.TestCsiClient 
   hadoop.mapred.nativetask.kvtest.KVTest 
   hadoop.mapred.nativetask.kvtest.LargeKVTest 
   hadoop.mapred.nativetask.compresstest.CompressTest 
   hadoop.mapred.nativetask.combinertest.OldAPICombinerTest 
   hadoop.mapred.nativetask.nonsorttest.NonSortTest 
   hadoop.mapred.nativetask.combinertest.CombinerTest 
   hadoop.mapred.nativetask.combinertest.LargeKVCombinerTest 
   hadoop.streaming.TestStreaming 
   hadoop.streaming.TestStreamingStderr 
   hadoop.streaming.TestStreamAggregate 
   hadoop.streaming.mapreduce.TestStreamXmlRecordReader 
   hadoop.streaming.TestStreamingOutputKeyValueTypes 
   hadoop.streaming.TestStreamXmlMultipleRecords 
   hadoop.streaming.TestMultipleArchiveFiles 
   hadoop.streaming.TestStreamingCounters 
   hadoop.streaming.TestStreamingBackground 
   hadoop.streaming.TestStreamingKeyValue 
   hadoop.streaming.TestStreamReduceNone 
   hadoop.streaming.TestFileArgs 
   hadoop.streaming.TestTypedBytesStreaming 
   hadoop.streaming.TestStreamingCombiner 
   hadoop.streaming.TestGzipInput 
   hadoop.streaming.TestStreamDataProtocol 
   hadoop.streaming.TestRawBytesStreaming 
   hadoop.streaming.TestStreamingExitStatus 
   hadoop.streaming.TestStreamingSeparator 
   hadoop.streaming.TestStreamXmlRecordReader 
   hadoop.streaming.TestSymLink 
   hadoop.streaming.TestUnconsumedInput 
   hadoop.streaming.TestStreamingFailure 
   hadoop.streaming.TestStreamingBadRecords 
   hadoop.streaming.TestMultipleCachefiles 
   hadoop.streaming.TestStreamingOutputOnlyKeys 
   hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.blockgenerator.TestBlockGen 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.blockgenerator.TestBlockGen 
   hadoop.tools.dynamometer.workloadgenerator.TestWorkloadGenerator 
   hadoop.tools.TestHadoopArchives 
   hadoop.tools.TestHadoopArchiveLogsRunner 
   hadoop.contrib.utils.join.TestDataJoin 
   hadoop.tools.TestDistCh 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/712/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apach

[jira] [Resolved] (HDFS-16324) Fix error log in BlockManagerSafeMode

[Akira Ajisaka (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka resolved HDFS-16324.
--
Fix Version/s: 3.4.0
   Resolution: Fixed

Merged the PR into trunk.

> Fix error log in BlockManagerSafeMode
> -
>
> Key: HDFS-16324
> URL: https://issues.apache.org/jira/browse/HDFS-16324
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: guophilipse
>Assignee: guophilipse
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> if `recheckInterval` was set as invalid value, there will be warning log 
> output, but the message seems not that proper ,we can improve it.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16374) Fix error log format in AbfsListStatusRemoteIterator

[guophilipse (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

guophilipse resolved HDFS-16374.

Resolution: Not A Bug

> Fix error log format in AbfsListStatusRemoteIterator
> 
>
> Key: HDFS-16374
> URL: https://issues.apache.org/jira/browse/HDFS-16374
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs
>Affects Versions: 3.3.1
>Reporter: guophilipse
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Fix error log format in AbfsListStatusRemoteIterator



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16375) The FBR lease ID should be exposed to the log

[tomscut (Jira)]

tomscut created HDFS-16375:
--

 Summary: The FBR lease ID should be exposed to the log
 Key: HDFS-16375
 URL: https://issues.apache.org/jira/browse/HDFS-16375
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: tomscut
Assignee: tomscut


Our Hadoop version is 3.1.0. We encountered HDFS-12914 and HDFS-14314 in the 
production environment.

When locating the problem, the *fullBrLeaseId* was not exposed in the log, 
which caused some difficulties. We should expose it to the log.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Backport of Router-Based Federation to 2.10.

[Owen O'Malley]

All,
   I've started backporting Router-Based Federation work from trunk back
into Hadoop 2.10. The goal is to backport everything that is relevant in
Hadoop 2. Would anyone outside of LinkedIn be interested in it?

Thanks,
Owen

Re: Next Hadoop community online meetup (Mandarin)

[Takanobu Asanuma]

What is the timezone?

2021年11月29日(月) 14:50 Wei-Chiu Chuang :

> Hello HDFS devs
>
> After a long hiatus, I'm happy to share that we'll have Yiyang from Shopee
> who is going to talk about their experience of using HDFS at Shopee.
>
> This is a Mandarin talk.
>
> 题目：《HDFS在Shopee的演进》
>
> 概要：HDFS作为最底层的存储架构，在Shopee的业务发展中起到了重要支撑作用。随着Shopee业务的扩展，HDFS存储服务也面临了越来越多的挑战。本次分享主要介绍在面临这些挑战时，如何借鉴社区成果，并结合内部具体需求，解决性能和稳定性上的问题。
> 关键词： Observer, RBF, slownode
> 演讲者：Yiyang Zhou (Apache id: Symious)
>
> 12月9日周四下午2点
>
> Zoom Meeting
>
> https://cloudera.zoom.us/j/880548968
>
> One tap mobile
>
> +16465588656,,880548968# US (New York)
>
> +17207072699,,880548968# US
>
> Download Center 
>
> Dial by your location
>
> +1 646 558 8656 US (New York)
>
> +1 720 707 2699 US
>
> 877 853 5257 US Toll-free
>
> 888 475 4499 US Toll-free
>
> Meeting ID: 880 548 968
>
> Find your local number: https://zoom.us/u/acaGRDfMVl
>

[jira] [Resolved] (HDFS-16333) fix balancer bug when transfer an EC block

[Takanobu Asanuma (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma resolved HDFS-16333.
-
Fix Version/s: 3.4.0
   3.2.4
   3.3.3
   Resolution: Fixed

> fix balancer bug when transfer an EC block
> --
>
> Key: HDFS-16333
> URL: https://issues.apache.org/jira/browse/HDFS-16333
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: balancer & mover
>Reporter: qinyuren
>Assignee: qinyuren
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.3
>
> Attachments: image-2021-11-18-17-25-13-089.png, 
> image-2021-11-18-17-25-50-556.png, image-2021-11-18-17-28-03-155.png
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> We set the EC policy to (6+3) and we also have nodes that were 
> decommissioning when we executed balancer.
> With the balancer running, we find many error logs as follow.
> !image-2021-11-18-17-25-13-089.png|width=858,height=135!
> Node A wants to transfer an EC block to node B, but we found that the block 
> is not on node A. The FSCK command to show the block status as follow
> !image-2021-11-18-17-25-50-556.png|width=607,height=189!
> In the dispatcher. getBlockList function
> !image-2021-11-18-17-28-03-155.png!
>  
> Assume that the location of the an EC block in storageGroupMap look like this
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> node:[a, b, c, d, e, f, g, h, i]
> after decommission operation, the internal block on indices[1] were 
> decommission to another node.
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> node:[a, {color:#FF}j{color}, c, d, e, f, g, h, i]
> the location of indices[1] change from node {color:#FF}b{color} to node 
> {color:#FF}j{color}.
>  
> When the balancer get the block location and check it with the location in 
> storageGroupMap.
> If a node is not found in storageGroupMap, it will not be add to block 
> locations.
> In this case, node {color:#FF}j {color}will not be added to the block 
> locations, while the indices is not updated.
> Finally, the block location may look like this, 
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> {color:#FF}block.location:[a, c, d, e, f, g, h, i]{color}
> the location of the nodes does not match their indices
>  
> Solution:
> we should update the indices and match with the nodes
> {color:#FF}indices:[0, 2, 3, 4, 5, 6, 7, 8]{color}
> {color:#FF}block.location:[a, c, d, e, f, g, h, i]{color}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16285) Make HDFS ownership tools cross platform

[Gautham Banasandra (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gautham Banasandra resolved HDFS-16285.
---
Resolution: Fixed

Merged [HDFS-16285. Make HDFS ownership tools cross platform by 
GauthamBanasandra · Pull Request #3588 · apache/hadoop 
(github.com)|https://github.com/apache/hadoop/pull/3588] to trunk.

> Make HDFS ownership tools cross platform
> 
>
> Key: HDFS-16285
> URL: https://issues.apache.org/jira/browse/HDFS-16285
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client, libhdfs++, tools
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
>  Labels: libhdfscpp, pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The source files for *hdfs_chown*, *hdfs_chmod* and *hdfs_chgrp* uses getopt 
> for parsing the command line arguments. getopt is available only on Linux and 
> thus, isn't cross platform. We need to replace getopt with 
> boost::program_options to make these tools cross platform.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Next Hadoop community online meetup (Mandarin)

[Wei-Chiu Chuang]

Hello, this talk will start in about an hour.

(I'm sorry forgot to add this is 2PM Beijing time)

On Thu, Dec 9, 2021 at 11:00 AM Takanobu Asanuma 
wrote:

> What is the timezone?
>
> 2021年11月29日(月) 14:50 Wei-Chiu Chuang :
>
>> Hello HDFS devs
>>
>> After a long hiatus, I'm happy to share that we'll have Yiyang from Shopee
>> who is going to talk about their experience of using HDFS at Shopee.
>>
>> This is a Mandarin talk.
>>
>> 题目：《HDFS在Shopee的演进》
>>
>> 概要：HDFS作为最底层的存储架构，在Shopee的业务发展中起到了重要支撑作用。随着Shopee业务的扩展，HDFS存储服务也面临了越来越多的挑战。本次分享主要介绍在面临这些挑战时，如何借鉴社区成果，并结合内部具体需求，解决性能和稳定性上的问题。
>> 关键词： Observer, RBF, slownode
>> 演讲者：Yiyang Zhou (Apache id: Symious)
>>
>> 12月9日周四下午2点
>>
>> Zoom Meeting
>>
>> https://cloudera.zoom.us/j/880548968
>>
>> One tap mobile
>>
>> +16465588656,,880548968# US (New York)
>>
>> +17207072699,,880548968# US
>>
>> Download Center 
>>
>> Dial by your location
>>
>> +1 646 558 8656 US (New York)
>>
>> +1 720 707 2699 US
>>
>> 877 853 5257 US Toll-free
>>
>> 888 475 4499 US Toll-free
>>
>> Meeting ID: 880 548 968
>>
>> Find your local number: https://zoom.us/u/acaGRDfMVl
>>
>

Re: Next Hadoop community online meetup (Mandarin)

[Takanobu Asanuma]

Thanks for the information!

2021年12月9日(木) 13:52 Wei-Chiu Chuang :

> Hello, this talk will start in about an hour.
>
> (I'm sorry forgot to add this is 2PM Beijing time)
>
> On Thu, Dec 9, 2021 at 11:00 AM Takanobu Asanuma 
> wrote:
>
>> What is the timezone?
>>
>> 2021年11月29日(月) 14:50 Wei-Chiu Chuang :
>>
>>> Hello HDFS devs
>>>
>>> After a long hiatus, I'm happy to share that we'll have Yiyang from
>>> Shopee
>>> who is going to talk about their experience of using HDFS at Shopee.
>>>
>>> This is a Mandarin talk.
>>>
>>> 题目：《HDFS在Shopee的演进》
>>>
>>> 概要：HDFS作为最底层的存储架构，在Shopee的业务发展中起到了重要支撑作用。随着Shopee业务的扩展，HDFS存储服务也面临了越来越多的挑战。本次分享主要介绍在面临这些挑战时，如何借鉴社区成果，并结合内部具体需求，解决性能和稳定性上的问题。
>>> 关键词： Observer, RBF, slownode
>>> 演讲者：Yiyang Zhou (Apache id: Symious)
>>>
>>> 12月9日周四下午2点
>>>
>>> Zoom Meeting
>>>
>>> https://cloudera.zoom.us/j/880548968
>>>
>>> One tap mobile
>>>
>>> +16465588656,,880548968# US (New York)
>>>
>>> +17207072699,,880548968# US
>>>
>>> Download Center 
>>>
>>> Dial by your location
>>>
>>> +1 646 558 8656 US (New York)
>>>
>>> +1 720 707 2699 US
>>>
>>> 877 853 5257 US Toll-free
>>>
>>> 888 475 4499 US Toll-free
>>>
>>> Meeting ID: 880 548 968
>>>
>>> Find your local number: https://zoom.us/u/acaGRDfMVl
>>>
>>

Re: Next Hadoop community online meetup (Mandarin)

[Wei-Chiu Chuang]

We are having a little technical difficulties.
For those who are late, we are using this link instead:
https://cloudera.zoom.us/j/93901387554

On Thu, Dec 9, 2021 at 12:55 PM Takanobu Asanuma 
wrote:

> Thanks for the information!
>
> 2021年12月9日(木) 13:52 Wei-Chiu Chuang :
>
>> Hello, this talk will start in about an hour.
>>
>> (I'm sorry forgot to add this is 2PM Beijing time)
>>
>> On Thu, Dec 9, 2021 at 11:00 AM Takanobu Asanuma 
>> wrote:
>>
>>> What is the timezone?
>>>
>>> 2021年11月29日(月) 14:50 Wei-Chiu Chuang :
>>>
 Hello HDFS devs

 After a long hiatus, I'm happy to share that we'll have Yiyang from
 Shopee
 who is going to talk about their experience of using HDFS at Shopee.

 This is a Mandarin talk.

 题目：《HDFS在Shopee的演进》

 概要：HDFS作为最底层的存储架构，在Shopee的业务发展中起到了重要支撑作用。随着Shopee业务的扩展，HDFS存储服务也面临了越来越多的挑战。本次分享主要介绍在面临这些挑战时，如何借鉴社区成果，并结合内部具体需求，解决性能和稳定性上的问题。
 关键词： Observer, RBF, slownode
 演讲者：Yiyang Zhou (Apache id: Symious)

 12月9日周四下午2点

 Zoom Meeting

 https://cloudera.zoom.us/j/880548968

 One tap mobile

 +16465588656,,880548968# US (New York)

 +17207072699,,880548968# US

 Download Center 

 Dial by your location

 +1 646 558 8656 US (New York)

 +1 720 707 2699 US

 877 853 5257 US Toll-free

 888 475 4499 US Toll-free

 Meeting ID: 880 548 968

 Find your local number: https://zoom.us/u/acaGRDfMVl

>>>