[jira] [Created] (HADOOP-18568) Magic Committer optional clean up
André F. created HADOOP-18568:
-
Summary: Magic Committer optional clean up
Key: HADOOP-18568
URL: https://issues.apache.org/jira/browse/HADOOP-18568
Project: Hadoop Common
Issue Type: Wish
Components: fs/s3
Affects Versions: 3.3.3
Reporter: André F.
It seems that deleting the `__magic` folder, depending on the number of
tasks/partitions used on a given spark job, can take really long time. I'm
having the following behavior on a given Spark job (processing ~30TB, with
~420k tasks) using the magic committer:
{code:java}
2022-12-10T21:25:19.629Z pool-3-thread-32 INFO MagicS3GuardCommitter: Starting:
Deleting magic directory s3a://my-bucket/random_hash/__magic
2022-12-10T21:52:03.250Z pool-3-thread-32 INFO MagicS3GuardCommitter: Deleting
magic directory s3a://my-bucket/random_hash/__magic: duration 26:43.620s {code}
I don't see a way out of it since the deletion of s3 objects needs to list all
objects under a prefix and this is what may be taking too much time. Could we
somehow make this cleanup optional? (the idea would be to delegate it through
s3 lifecycle policies in order to not create this overhead on the commit phase).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.fs.viewfs.TestViewFileSystemHdfs hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap hadoop.hdfs.TestLeaseRecovery2 hadoop.hdfs.TestDFSInotifyEventInputStream hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator hadoop.tools.TestDistCpSystem hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.solver.impl.TestLpSolver hadoop.resourceestimator.service.TestResourceEstimatorService cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/diff-compile-javac-root.txt [488K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-mvnsite-root.txt [568K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [220K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [1.8M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [16K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [72K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/873/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-se
[jira] [Created] (HADOOP-18569) NFS Gateway may release buffer too early
Attila Doroszlai created HADOOP-18569: - Summary: NFS Gateway may release buffer too early Key: HADOOP-18569 URL: https://issues.apache.org/jira/browse/HADOOP-18569 Project: Hadoop Common Issue Type: Bug Components: nfs Affects Versions: 3.4.0, 3.3.5, 3.2.5, 3.3.9 Reporter: Attila Doroszlai Assignee: Attila Doroszlai After upgrading Netty from 4.1.68 to 4.1.77 (HADOOP-18079), NFS Gateway started crashing when writing data (can be easily reproduced by a few 10MB+ files). The problem was triggered by [reduced default chunk size in PooledByteBufAllocator|https://github.com/netty/netty/commit/f650303911] (in 4.1.75), but it turned out to be caused by a buffer released too early in NFS Gateway. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18570) Update region logic
Ahmar Suhail created HADOOP-18570: - Summary: Update region logic Key: HADOOP-18570 URL: https://issues.apache.org/jira/browse/HADOOP-18570 Project: Hadoop Common Issue Type: Sub-task Reporter: Ahmar Suhail SDK V2 will no longer resolve a buckets region if it is not set when initialising the client. Current logic will always make a head bucket call on FS initialisation. We should review this. Possible solution: * Warn if region is not set. * If no region, try and resolve. If resolution fails, throw an exception. Cache the region to optimise for short lived FS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18571) Qualify the upgrade.
Ahmar Suhail created HADOOP-18571: - Summary: Qualify the upgrade. Key: HADOOP-18571 URL: https://issues.apache.org/jira/browse/HADOOP-18571 Project: Hadoop Common Issue Type: Sub-task Reporter: Ahmar Suhail Run tests as per [qualifying aws ask update|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md#-qualifying-an-aws-sdk-update] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
yetus reporting javadoc errors on @InterfaceAudience attributes
yetus is now reporting errors on our @InterfaceAudience tags in java8 and java11 javadoc generation https://github.com/apache/hadoop/pull/5205#issuecomment-1344664692 https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5205/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt it looks a bit like the javadocs are both being done in the java11 version, and is is unhappy. any suggestions as to a fix?
Re: yetus reporting javadoc errors on @InterfaceAudience attributes
Try to fix in the same way it was done here and couple of similar PRs: https://github.com/apache/hadoop/pull/5179 There are a bunch of PRs in yarn getting the similar error fixed module wise, the problem would be there in many other modules as well... The daily JDK-11 build also shows that failure here: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/410/artifact/out/patch-javadoc-root.txt BTW. the daily build is also broken with some whooping 150+ failures https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/ Mostly some Junit upgrade patch being the reason. -Ayush On Mon, 12 Dec 2022 at 18:46, Steve Loughran wrote: > yetus is now reporting errors on our @InterfaceAudience tags in java8 and > java11 javadoc generation > https://github.com/apache/hadoop/pull/5205#issuecomment-1344664692 > > https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5205/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt > > it looks a bit like the javadocs are both being done in the java11 version, > and is is unhappy. > > any suggestions as to a fix? >
Re: yetus reporting javadoc errors on @InterfaceAudience attributes
Thanks Ayush for pointing out the failures related to the Junit 5 upgrade. As I have closely worked in upgrading Junit 4 to Junit 5 throughout the hadoop project. I will create a JIRA for these failures and fix them on priority. -Ashutosh On Mon, Dec 12, 2022 at 1:59 PM Ayush Saxena wrote: > Try to fix in the same way it was done here and couple of similar PRs: > https://github.com/apache/hadoop/pull/5179 > > There are a bunch of PRs in yarn getting the similar error fixed module > wise, the problem would be there in many other modules as well... > > The daily JDK-11 build also shows that failure here: > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/410/artifact/out/patch-javadoc-root.txt > > BTW. the daily build is also broken with some whooping 150+ failures > > https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/ > > Mostly some Junit upgrade patch being the reason. > > -Ayush > > On Mon, 12 Dec 2022 at 18:46, Steve Loughran > wrote: > > > yetus is now reporting errors on our @InterfaceAudience tags in java8 and > > java11 javadoc generation > > https://github.com/apache/hadoop/pull/5205#issuecomment-1344664692 > > > > > https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5205/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt > > > > it looks a bit like the javadocs are both being done in the java11 > version, > > and is is unhappy. > > > > any suggestions as to a fix? > > >
[jira] [Resolved] (HADOOP-18329) Add support for IBM Semeru OE JRE 11.0.15.0 and greater
[
https://issues.apache.org/jira/browse/HADOOP-18329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18329.
-
Fix Version/s: 3.4.0
3.3.5
Resolution: Fixed
fixed in 3.3.5+
if you need it in branch-3.2 reopen this and submit a new PR
> Add support for IBM Semeru OE JRE 11.0.15.0 and greater
> ---
>
> Key: HADOOP-18329
> URL: https://issues.apache.org/jira/browse/HADOOP-18329
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth, common
>Affects Versions: 3.0.0, 3.1.0, 3.0.1, 3.2.0, 3.0.2, 3.1.1, 3.0.3, 3.3.0,
> 3.1.2, 3.2.1, 3.1.3, 3.1.4, 3.2.2, 3.3.1, 3.2.3, 3.3.2, 3.3.3
> Environment: Running Hadoop (or Apache Spark 3.2.1 or above) on IBM
> Semeru runtimes open edition 11.0.15.0 or greater.
>Reporter: Jack
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Original Estimate: 1h
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> There are checks within the PlatformName class that use the Vendor property
> of the provided runtime JVM specifically looking for `IBM` within the name.
> Whilst this check worked for IBM's [java technology
> edition|https://www.ibm.com/docs/en/sdk-java-technology] it fails to work on
> [Semeru|https://developer.ibm.com/languages/java/semeru-runtimes/] since
> 11.0.15.0 due to the following change:
> h4. java.vendor system property
> In this release, the {{java.vendor}} system property has been changed from
> "International Business Machines Corporation" to "IBM Corporation".
> Modules such as the below are not provided in these runtimes.
> com.ibm.security.auth.module.JAASLoginModule
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1072/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:root Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] Failed junit tests : hadoop.hdfs.TestLeaseRecovery2 hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf hadoop.mapreduce.v2.app.webapp.TestAMWebServices hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobConf hadoop.mapreduce.v2.hs.webapp.TestHsWebServices hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesLogs hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesTasks hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesAttempts cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1072/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1072/artifact/out/results-compile-javac-root.txt [528K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt
