Piotr Nowojski created FLINK-37807:
--------------------------------------
Summary: Report s5cmd cancellation in a nicer way
Key: FLINK-37807
URL: https://issues.apache.org/jira/browse/FLINK-37807
Project: Flink
Issue Type: Improvement
Components: Connectors / FileSystem
Affects Versions: 2.0.0
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
Fix For: 2.1.0
If s5cmd is cancelled it logs for example:
{noformat}
java.io.IOException: Failed to cast s5cmd spell [/opt/s5cmd -r 3 --numworkers 5
run] [exit code = 1] [cfg: S5CmdConfiguration{path='/opt/s5cmd', args=[-r, 3,
--numworkers, 5], accessArtifact='null', secretArtifact='null',
endpoint='null'}] maybe due to:
at
org.apache.flink.fs.s3.common.FlinkS3FileSystem.castSpell(FlinkS3FileSystem.java:379)
at
org.apache.flink.fs.s3.common.FlinkS3FileSystem.copyFiles(FlinkS3FileSystem.java:296)
at
org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.copyFiles(PluginFileSystemFactory.java:157)
at
org.apache.flink.core.fs.monitoring.MonitoringFileSystem.copyFiles(MonitoringFileSystem.java:86)
at
org.apache.flink.core.fs.SafetyNetWrapperFileSystem.copyFiles(SafetyNetWrapperFileSystem.java:56)
at
org.apache.flink.core.fs.bandwidth.BandwidthLimitingFileSystem.copyFiles(BandwidthLimitingFileSystem.java:202)
at
org.apache.flink.contrib.streaming.state.RocksDBStateDownloader.lambda$createDownloadRunnableUsingCopyFiles$3(RocksDBStateDownloader.java:247)
at
org.apache.flink.util.function.ThrowingRunnable.lambda$unchecked$0(ThrowingRunnable.java:49)
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Copy process destroyed by CloseableRegistry.
at
org.apache.flink.fs.s3.common.FlinkS3FileSystem.lambda$castSpell$0(FlinkS3FileSystem.java:362)
at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:295)
at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:282)
at
org.apache.flink.core.fs.CloseableRegistry.doClose(CloseableRegistry.java:65)
at
org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:295)
at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:282)
at
org.apache.flink.core.fs.CloseableRegistry.doClose(CloseableRegistry.java:65)
at
org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:295)
at org.apache.flink.util.IOUtils.closeAllQuietly(IOUtils.java:282)
at
org.apache.flink.core.fs.CloseableRegistry.doClose(CloseableRegistry.java:65)
at
org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$cancel$13(StreamTask.java:1081)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
at
java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2257)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.cancel(StreamTask.java:1074)
at
org.apache.flink.runtime.taskmanager.Task$TaskCanceler.run(Task.java:1773)
... 1 more
{noformat}
Which can be miss leading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
