Re: close file on job crash

2020-05-26 Thread Laurent Exsteens
Thanks! On Tue, May 26, 2020, 08:13 Piotr Nowojski wrote: > Hi, > > One clarification. `RichFunction#close` is of course called always, not > only after internal failure. It’s called after internal failure, external > failure or clean shutdown. > > `SourceFunction#cancel` is intended to inform t

Re: close file on job crash

2020-05-25 Thread Piotr Nowojski
Hi, One clarification. `RichFunction#close` is of course called always, not only after internal failure. It’s called after internal failure, external failure or clean shutdown. `SourceFunction#cancel` is intended to inform the `SourceFunction` to cleanly exit it’s `#run` method/loop (note SIGI

Re: close file on job crash

2020-05-25 Thread Piotr Nowojski
Hi, Cancel method is being invoked only when SourceTask is being cancelled from the outside, by JobManager - for example after detecting a failure of a different Task. > What is the proper way to handle this issue? Is there some kind of closable > source interface we should implement? Have yo

close file on job crash

2020-05-25 Thread Laurent Exsteens
Hello, we had to implement a specific source to read files in a certain way. The files we read are a NAS mounted through NFS. If an error occurs in a map after this specific source when the file is still being read, the file is never closed, resulting in the task manager keeping the file open (ap