Thanks Tison for looking into it - what I tried to say is that Flink keeps the files locked (hence, the space is still occupied) - this is visible during a lsof command
>From my investigations, after the job finishes, the local (and HA) blob store is deleted - the operation succeed in both case, but on our Linux machine, the space remains occupied until the process is restarted. I saw there were some improvement around this feature ( https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture) , but from my debugging, the responsible code is in org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(File file) which uses the new java class java.nio.file.deleteIfExists(Path path) where there is a comment * <p> On some operating systems it may not be possible to remove a > file when > * it is open and in use by this Java virtual machine or other > programs. > * There are a lot of changes compared with Flink 1.4, but i think the correspondent is below (pretty sure there are more things involved) try { Files.delete(directory.toPath()); } catch (NoSuchFileException ignored) { // if someone else deleted this concurrently, we don't mind // the result is the same for us, after all } Thanks again for your time Dan On Wed, Jun 12, 2019 at 4:36 AM Zili Chen <wander4...@gmail.com> wrote: > Hi Dan, > > Said "The files are removed after a restart of the process", it sounds > Flink > cleaned up blobs properly. From your description I don't understand clearly > in which case/situation you expected Flink deleted blobs but it doesn't. > > Could you describe the difference between 1.4.2 and 1.7.2/1.8.0 in detail? > Especially what moment exactly you want blobs to be deleted and it was not. > > Best, > tison. > > > Dede <rod...@gmail.com> 于2019年6月11日周二 下午11:06写道： > >> Hi Team, >> >> I'm struggling for a while with a strange issue: the local blob store >> files are not actually deleted from the job manager/task manager in >> versions 1.7.2 and 1.8.0 : lsof commands shows them like this: >> java 6528 root 63r REG 202,16 162458786 1802248 >> /mnt/tmp1/blobStore-542fc202-b263-482f-87d7-11b5ad70cc32/job_b3446b24474ac3e107bbde27ff24df98/blob_p-96a29e8796d15ce6359edb4ab80ff8661f8b1fd0-73395221d4ffcfd603dbd1d25961aee3 >> (deleted) >> >> The files are removed after a restart of the process, so I guess the >> flink itself is responsible for keeping a handle to the deleted file. >> >> The same process works fine on 1.4.2, deleting all the files properly. >> >> Is there something that I'm missing? I played around with the blob server >> configuration, but with no luck. >> >> I can provide more logs/debug if needed. >> >> Thanks, >> Dan >> >> >>
