Yes, in the same thread.
File list is not empty, because after samza job shutdown, all files in writing
are not in complete state.
————————
Qi Shu
> 在 2017年8月24日，15:00，Jagadish Venkatraman <jagadish1...@gmail.com> 写道：
>
>>> Now the problem is not program hang, but the container quit
> before the “closeFiles” method executed completely.
>
> It's unlikely that the container quit without returning from *close*. Are
> you sure there were filesList is not empty? I'd suggest adding more logging.
>
> I'm assuming you are calling *closeFiles* in the same thread?
>
> On Wed, Aug 23, 2017 at 11:53 PM, 舒琦 <sh...@eefung.com> wrote:
>
>> Hi Jagadish,
>>
>> When samza job is shutdown, “Begin to stop” is printed out in the
>> log file, but not the log in closeFiles.
>> Now the problem is not program hang, but the container quit before
>> the “closeFiles” method executed completely.
>>
>> Thanks.
>>
>> ————————
>> Qi Shu
>>
>>> 在 2017年8月24日，14:44，Jagadish Venkatraman <jagadish1...@gmail.com> 写道：
>>>
>>> Is "Begin to close file" printed? Where exactly is your application
>> stalled
>>> ? I'd suggest you take a stack dump.
>>>
>>> On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
>>>
>>>> Hi Jagadish,
>>>>
>>>> Below is part of the related code, log “"Begin to stop” is
>> printed
>>>> out.
>>>>
>>>> Thank you!
>>>>
>>>> public abstract class MyProducer implements SystemProducer {
>>>>
>>>> @Override
>>>> public void stop() {
>>>> LOGGER.info("Begin to stop");
>>>>
>>>> closeFiles();
>>>>
>>>> LOGGER.info("End to stop");
>>>> }
>>>>
>>>> @Override
>>>> protected void closeFiles() throws IOException {
>>>> closeFiles(statusFiles);
>>>> closeFiles(interactionFiles);
>>>> }
>>>>
>>>> private void closeFiles(Map<String, FileInWriting> files) throws
>>>> IOException {
>>>> for (FileInWriting statusFile : files.values()) {
>>>> LOGGER.info("Begin to close file: {}",
>>>> statusFile.getFilePath());
>>>> statusFile.getWriter().hflush();
>>>> statusFile.getWriter().close();
>>>> renameFile(statusFile.getFilePath());
>>>> LOGGER.info("Successfully close file: {}",
>>>> statusFile.getFilePath());
>>>> }
>>>>
>>>> files.clear();
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> ————————
>>>> Qi Shu
>>>>
>>>>> 在 2017年8月24日，14:21，Jagadish Venkatraman <jagadish1...@gmail.com> 写道：
>>>>>
>>>>> Hi Qi,
>>>>>
>>>>>>> the stop method in SystemProducer is called, but the close files
>>>>> operation(may need some time, there may be cache data to be flushed) in
>>>>> stop method is not executed completely
>>>>>
>>>>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
>>>>> synchronous operation, and will block for the *close* method to finish.
>>>>>
>>>>> Best,
>>>>> Jagadish
>>>>>
>>>>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I write a SystemProducer for HDFS and everything is fine. When
>>>>>> samza job is shutdown, the stop method in SystemProducer is called,
>> but
>>>> the
>>>>>> close files operation(may need some time, there may be cache data to
>> be
>>>>>> flushed) in stop method is not executed completely.
>>>>>>
>>>>>> How can I resolve this problem?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> ————————
>>>>>> Qi Shu
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jagadish V,
>>>>> Graduate Student,
>>>>> Department of Computer Science,
>>>>> Stanford University
>>>>
>>>>
>>>
>>>
>>> --
>>> Jagadish V,
>>> Graduate Student,
>>> Department of Computer Science,
>>> Stanford University
>>
>>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
