Robert Metzger created FLINK-5345:
-------------------------------------
Summary: IOManager failed to properly clean up temp file directory
Key: FLINK-5345
URL: https://issues.apache.org/jira/browse/FLINK-5345
Project: Flink
Issue Type: Bug
Affects Versions: 1.1.3
Reporter: Robert Metzger
While testing 1.1.3 RC3, I have the following message in my log:
{code}
2016-12-15 14:46:05,450 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - Timer service
is shutting down.
2016-12-15 14:46:05,452 INFO org.apache.flink.runtime.taskmanager.Task
- Source: control events generator (29/40)
(73915a232ba09e642f9dff92f8c8773a) switched from CANCELING to CANCELED.
2016-12-15 14:46:05,452 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Source: control events generator
(29/40) (73915a232ba09e642f9dff92f8c8773a).
2016-12-15 14:46:05,454 INFO org.apache.flink.yarn.YarnTaskManager
- Un-registering task and sending final execution state CANCELED to
JobManager for task Source: control events genera
tor (73915a232ba09e642f9dff92f8c8773a)
2016-12-15 14:46:40,609 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2016-12-15 14:46:40,611 INFO org.apache.flink.runtime.blob.BlobCache
- Shutting down BlobCache
2016-12-15 14:46:40,724 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://flink@10.240.0.34:33635] has failed, address is now gated for
[5000] ms.
Reason is: [Disassociated].
2016-12-15 14:46:40,808 ERROR
org.apache.flink.runtime.io.disk.iomanager.IOManager - IOManager
failed to properly clean up temp file directory:
/yarn/nm/usercache/robert/appcache/application_148129128
9979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5
java.lang.IllegalArgumentException:
/yarn/nm/usercache/robert/appcache/application_1481291289979_0024/flink-io-f0ff3f66-b9e2-4560-881f-2ab43bc448b5/62e14e1891fe1e334c921dfd19a32a84/StreamMap_11_24/dummy_state
does not exist
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1637)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
at
org.apache.flink.runtime.io.disk.iomanager.IOManager.shutdown(IOManager.java:109)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync.shutdown(IOManagerAsync.java:185)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$1.run(IOManagerAsync.java:105)
{code}
This was the last message logged from that machine. I suspect two threads are
trying to clean up the directories during shutdown?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
