date:20250405

[jira] [Created] (HDFS-17765) Provide a mechanism for reordering datanode priority when reading

2025-04-05 Thread Istvan Toth (Jira)

Istvan Toth created HDFS-17765:
--

 Summary: Provide a mechanism for reordering datanode priority  
when reading
 Key: HDFS-17765
 URL: https://issues.apache.org/jira/browse/HDFS-17765
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Reporter: Istvan Toth


HBASE-6435 has added an ugly hack to avoid long waits for HDFS timeouts when 
reading data whoose primary DataNode is known to be down.

With JEP-416 that solution is no longer working.

Provide some solution for HBASE-6435 that does require horrible reflection 
based hacks.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-04-05 Thread Chris Nauroth

-1

Signatures look good, but I found a few spots that need an update for the
new protobuf version:

https://issues.apache.org/jira/browse/HADOOP-19510

https://github.com/apache/hadoop-thirdparty/pull/44

Chris Nauroth


On Tue, Mar 18, 2025 at 8:47 AM Wei-Chiu Chuang  wrote:

> I just did
> gpg --verify .tar.gz.asc .tar.gz
>
> Ok after importing hadoop KEYS
> , this is the output
> which
> looks good to me:
>
> gpg: Signature made Fri Mar 14 08:56:01 2025 PDT
> gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0
> gpg: Good signature from "Steve Loughran (ASF code sign key  - 2018) <
> ste...@apache.org>" [unknown]
> gpg: aka "[jpeg image of size 8070]" [unknown]
> gpg: aka "Steve Loughran " [unknown]
> gpg: aka "Steve Loughran "
> [unknown]
> gpg: aka "Steve Loughran "
> [unknown]
> gpg: WARNING: This key is not certified with a trusted signature!
> gpg:  There is no indication that the signature belongs to the
> owner.
> Primary key fingerprint: 3823 7EE4 2505 0285 077D  B57A D22C F846 DBB1 62A0
>
> Sorry. False alarm!
>
> On Mon, Mar 17, 2025 at 12:31 PM Steve Loughran
> 
> wrote:
>
> > can you give me what command you ran to check the signature, and see if
> you
> > can update all your keys from the central servers?
> >
> > My key thinks it is not expired, so maybe its the KEYS file i need to
> > update or you need to refresh your keys. I will have a look at the keys
> > file tomorrow
> >
> > On Fri, 14 Mar 2025 at 21:46, Wei-Chiu Chuang 
> wrote:
> >
> > > Looks like your GPG key expired?
> > >
> > > gpg: Signature made Fri Mar 14 08:56:00 2025 PDT
> > > gpg:using RSA key
> > 38237EE425050285077DB57AD22CF846DBB162A0
> > > gpg: Good signature from "Steve Loughran (ASF code sign key  - 2018) <
> > > ste...@apache.org>" [expired]
> > > gpg: aka "[jpeg image of size 8070]" [expired]
> > > gpg: Note: This key has expired!
> > > Primary key fingerprint: 3823 7EE4 2505 0285 077D  B57A D22C F846 DBB1
> > 62A0
> > >
> > > On Fri, Mar 14, 2025 at 11:23 AM Steve Loughran
> > > 
> > > wrote:
> > >
> > > > I have built a release candidate (RC0) for Hadoop-Thirdparty 1.4.0.
> > > >
> > > > The RC is available at:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.4.0-RC0/
> > > >
> > > > The git tag is release-1.4.0-RC0,
> > > > and commit 5595f1357eaad02e1cfb660bcce7fd34515197ff
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1433
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > I will vote once I've done more of the testing myself
> > > >
> > > > steve
> > > >
> > >
> >
>

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2025-04-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/839/

[Mar 31, 2025, 2:27:48 AM] (github) YARN-11802. Avoid reflection in 
TestAsyncDispatcher (#7541) Contributed by Istvan Toth.




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:[line 695] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:[line 138] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:[line 75] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:[line 85] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:[line 130] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
 doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$

[VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-04-05 Thread Steve Loughran

Apache Hadoop-Thirdparty 1.4.0

I have built a release candidate (RC1) for Hadoop-Thirdparty 1.4.0.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/thirdparty-1.4.0-RC1/

The git tag is release-1.4.0-RC1,
and commit a1eda71bbfa630138b97ecac1a977c18d05f0003

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1434

Please try the release and vote. The vote will run for 5 days.

Here is my vote:

+1 (binding)

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2025-04-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/840/

[Mar 31, 2025, 6:47:28 AM] (github) HDFS-17762. [ARR] Reset CallerContext 
information when async handler thread done. (#7539). Contributed by 
hfutatzhanghb.
[Apr 1, 2025, 1:18:02 AM] (github) HADOOP-19438. [JDK17] Upgrade JUnit from 4 
to 5 in hadoop-resourceestimator. (#7548)
[Apr 1, 2025, 4:55:51 AM] (github) HADOOP-19471. [ABFS] Support Fixed SAS Token 
at Container Level (#7461)




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:[line 695] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:[line 138] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:[line 75] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:[line 85] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:[line 130] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
 doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTr

[jira] [Created] (HDFS-17766) [ARR] Avoid initializing unused threadPool in RouterAsyncRpcClient

2025-04-05 Thread farmmamba (Jira)

farmmamba created HDFS-17766:


 Summary: [ARR] Avoid initializing unused threadPool in 
RouterAsyncRpcClient
 Key: HDFS-17766
 URL: https://issues.apache.org/jira/browse/HDFS-17766
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: rbf
Reporter: farmmamba
Assignee: farmmamba


Avoid initializing unused threadPool in RouterAsyncRpcClient.

This can save thread resources.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2025-04-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/836/

[Mar 24, 2025, 12:53:13 AM] (github) HADOOP-18991 Remove commons-beanutils 
dependency from Hadoop 3 (#7524) Contributed by Istvan Toth.
[Mar 24, 2025, 12:54:37 AM] (github) YARN-11795. Skip 
BaseFederationPoliciesTest.testReinitilializeBad3() on JVMs where Mockito 
cannot mock ByteBuffers (#7510) Contributed by Istvan Toth.
[Mar 24, 2025, 12:58:32 AM] (github) HADOOP-19508. Set charsetEncoder in 
HadoopArchiveLogs (#7520) Contributed by Istvan Toth.
[Mar 25, 2025, 12:00:57 AM] (github) YARN-11262. Upgrade JUnit from 4 to 5 in 
hadoop-yarn-server-resourcemanager Part4. (#7488)
[Mar 25, 2025, 2:23:55 AM] (github) HDFS-17749. [ARR] Fix wrong results due to 
the wrong usage of asyncComplete in getListing. (#7466). Contributed by 
hfutatzhanghb.

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Fw: Data copy from HDFS to MinIO regularly

2025-04-05 Thread Steve Loughran

distcp can upload a directory tree of changed files, for cloud storage it
looks for the different in file timestamps

Otherwise, the HDFS namenode has an log4j audit logger:
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit

This prints out all namenode filesystem operations in a structured format
(which is consistent across releases)...you can read that and use whatever
copy command you want to use to upload it.
https://stackoverflow.com/questions/44533589/hdfs-audit-logs-format-and-explanation

AFAIK There's no tool which takes a list of changed files and uploads them;
I've played with using spark for this but it's not easy.

the cloud store app we use for diagnostics has a 'cloudup` command to
upload a directory tree to cloud storage (or any hadoop filesystem to
filesystem)
https://github.com/steveloughran/cloudstore
https://github.com/steveloughran/cloudstore/blob/main/src/main/site/cloudup.md

That's a single process but it is aggressive, optimised for cloud storage
with rate limiting by shard
https://github.com/steveloughran/cloudstore/blob/main/src/main/site/cloudup.md

you could probably modify that to take a list of source + destination paths
to build its upload list and let it do the work.

You would have to do that on your own you, I'm afraid -through a PR of your
changes would be welcome, especially with tests.

steve

On Wed, 12 Mar 2025 at 04:35, Arif Kamil Yilmaz
 wrote:

>
> Subject: Data copy from HDFS to MinIO regularly
>
> Hello Team,
>
> There is an application that was developed a long time ago, and this
> application processes 10GB of binary data per hour using MapReduce and
> generates 100GB of data, which is then written to the HDFS file system.
>
> My goal is to move a portion of the processed data (approximately 25%) to
> a MinIO cluster that I plan to use as new object storage. I want this
> operation to be repeated every time new data is added to the HDFS cluster.
>
> What kind of solution would you suggest to complete this task?
> Additionally, I would like to remind you that I have requirements related
> to monitoring the pipeline I am developing.
>
> Thank you.
>
>

[jira] [Resolved] (HDFS-17713) [ARR] Throtting asynchronous calls for each nameservice

2025-04-05 Thread farmmamba (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-17713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

farmmamba resolved HDFS-17713.
--
Resolution: Fixed

> [ARR] Throtting asynchronous calls for each nameservice
> ---
>
> Key: HDFS-17713
> URL: https://issues.apache.org/jira/browse/HDFS-17713
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
>  Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-17720) Upgrade JUnit from 4 to 5 in hadoop-hdfs-rbf.

2025-04-05 Thread Chris Nauroth (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-17720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Nauroth resolved HDFS-17720.
--
Fix Version/s: 3.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Upgrade JUnit from 4 to 5 in hadoop-hdfs-rbf.
> -
>
> Key: HDFS-17720
> URL: https://issues.apache.org/jira/browse/HDFS-17720
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Shilun Fan
>Assignee: Hualong Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.5.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2025-04-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.TestFileLengthOnClusterRestart 
   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.v2.app.TestRuntimeEstimators 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.TestReservationSystemInvariants 
   hadoop.yarn.sls.TestSLSRunner 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/diff-compile-javac-root.txt
  [488K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-mvnsite-root.txt
  [572K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-javadoc-root.txt
  [36K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [220K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [452K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [16K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  [44K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1717/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
  [20K]
   
https://ci-hadoop.apache.org/job/hadoo

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2025-04-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/

[Apr 4, 2025, 10:57:40 AM] (github) HADOOP-19515. ABFS: Updating Documentations 
of Hadoop Drivers for Azure (#7540)
[Apr 4, 2025, 11:49:06 AM] (github) HADOOP-19494: [ABFS][FnsOverBlob] Fix Case 
Sensitivity Issue for hdi_isfolder metadata (#7496)




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
shadedclient unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerMultiNodes
 
   hadoop.hdfs.server.federation.router.TestRouterMountTableCacheRefresh 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-compile-javac-root.txt
 [8.0K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/blanks-eol.txt
 [15M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-hadolint.txt
 [24K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-pathlen.txt
 [12K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/results-javadoc-javadoc-root.txt
 [240K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 [148K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1902/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
 [696K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2025-04-05 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/842/

[Apr 3, 2025, 5:06:04 PM] (github) HADOOP-19440. [JDK17] Upgrade JUnit from 4 
to 5 in hadoop-sls. (#7553)
[Apr 4, 2025, 10:57:40 AM] (github) HADOOP-19515. ABFS: Updating Documentations 
of Hadoop Drivers for Azure (#7540)
[Apr 4, 2025, 11:49:06 AM] (github) HADOOP-19494: [ABFS][FnsOverBlob] Fix Case 
Sensitivity Issue for hdi_isfolder metadata (#7496)




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Redundant nullcheck of oldLock, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.isPreUpgradableLayout(Storage$StorageDirectory)
 Redundant null check at DataStorage.java:[line 695] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MappableBlockLoader.verifyChecksum(long,
 FileInputStream, FileChannel, String) Redundant null check at 
MappableBlockLoader.java:[line 138] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at MemoryMappableBlockLoader.java:[line 75] 
   Redundant nullcheck of blockChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.load(long,
 FileInputStream, FileInputStream, String, ExtendedBlockId) Redundant null 
check at NativePmemMappableBlockLoader.java:[line 85] 
   Redundant nullcheck of metaChannel, which is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:is known to be non-null in 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.NativePmemMappableBlockLoader.verifyChecksumAndMapBlock(NativeIO$POSIX$PmemMappedRegion,
 long, FileInputStream, FileChannel, String) Redundant null check at 
NativePmemMappableBlockLoader.java:[line 130] 
   
org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager$UserCounts
 doesn't override java.util.ArrayList.equals(Object) At 
RollingWindowManager.java:At RollingWindowManager.java:[line 1] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check

[jira] [Created] (HDFS-17765) Provide a mechanism for reordering datanode priority when reading

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[VOTE] Release Apache Hadoop Thirdparty 1.4.0

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

[jira] [Created] (HDFS-17766) [ARR] Avoid initializing unused threadPool in RouterAsyncRpcClient

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

Re: Fw: Data copy from HDFS to MinIO regularly

[jira] [Resolved] (HDFS-17713) [ARR] Throtting asynchronous calls for each nameservice

[jira] [Resolved] (HDFS-17720) Upgrade JUnit from 4 to 5 in hadoop-hdfs-rbf.

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

13 matches

Site Navigation

Mail list logo

Footer information