Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1076/ [Dec 15, 2022, 4:49:18 AM] (noreply) HADOOP-18574. Changing log level of IOStatistics increment to make the DEBUG logs less noisy (#5223) [Dec 15, 2022, 11:42:36 AM] (noreply) HADOOP-18573. Improve error reporting on non-standard kerberos names (#5221) [Dec 15, 2022, 4:45:05 PM] (noreply) HADOOP-18561. Update commons-net to 3.9.0 (#5214) [Dec 15, 2022, 5:08:25 PM] (noreply) HADOOP-18577. ABFS: Add probes of readahead fix (#5205) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16852) Register the shutdown hook only when not in shutdown for KeyProviderCache constructor
[
https://issues.apache.org/jira/browse/HDFS-16852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen resolved HDFS-16852.
Resolution: Fixed
> Register the shutdown hook only when not in shutdown for KeyProviderCache
> constructor
> -
>
> Key: HDFS-16852
> URL: https://issues.apache.org/jira/browse/HDFS-16852
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.3, 3.3.6
>
>
> When an HDFS client is created, it will register a shutdownhook to
> shutdownHookManager. ShutdownHookManager doesn't allow adding a new
> shutdownHook when the process is already in shutdown and throws an
> IllegalStateException.
> This behavior is not ideal, when a spark program failed during pre-launch. In
> that case, during shutdown, spark would call cleanStagingDir() to clean the
> staging dir. In cleanStagingDir(), it will create a FileSystem object to talk
> to HDFS. However, since this would be the first time to use a filesystem
> object in that process, it will need to create an hdfs client and register
> the shutdownHook. Then, we will hit the IllegalStateException. This
> illegalStateException will mask the actual exception which causes the spark
> program to fail during pre-launch.
> We propose to swallow IllegalStateException in KeyProviderCache and log a
> warning. The TCP connection between the client and NameNode should be closed
> by the OS when the process is shutdown.
> Example stacktrace
> {code:java}
> 13-09-2022 14:39:42 PDT INFO - 22/09/13 21:39:41 ERROR util.Utils: Uncaught
> exception in thread shutdown-hook-0
> 13-09-2022 14:39:42 PDT INFO - java.lang.IllegalStateException: Shutdown in
> progress, cannot add a shutdownHook
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:299)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.hdfs.KeyProviderCache.(KeyProviderCache.java:71)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.hdfs.ClientContext.(ClientContext.java:130)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.hdfs.ClientContext.get(ClientContext.java:167)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:383)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:287)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3261)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3310)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3278)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:475)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.deploy.yarn.ApplicationMaster.cleanupStagingDir(ApplicationMaster.scala:675)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.deploy.yarn.ApplicationMaster.$anonfun$run$2(ApplicationMaster.scala:259)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
>
> 13-09-2022 14:39:42 PDT INFO - at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2023)
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188)
>
> 13-09-2022 14:39:42 PDT INFO - at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
>
> 13-09-2022 14:39:42 PDT INFO - at scala.util.Try$.apply(Try.scala:213)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
>
> 13-09-2022 14:39:42 PDT INFO - at
> org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHook
[jira] [Created] (HDFS-16869) DelegationTokenRenewer timeout feature may cause high utilization of CPU and object leak
Daniel Ma created HDFS-16869: Summary: DelegationTokenRenewer timeout feature may cause high utilization of CPU and object leak Key: HDFS-16869 URL: https://issues.apache.org/jira/browse/HDFS-16869 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.3.4, 3.3.3, 3.3.2, 3.3.1 Reporter: Daniel Ma Assignee: Daniel Ma DelegationTokenRenewer timeout feature may cause high utilization of CPU and object leak。 1-If yarn cluster is in idle state, that is almost no token renewer event triggered, the DelegationTokenRenewerPoolTracker thread will do nothing but dead loop, it will cause high CPU utilization. 2-The renewer event is hold in a map named futures, will has no remove logic , that is the map will become increasingly great with time going by. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
