Hi Shuqi,
Glad that you figured out the problem. In general, we recommend Filesystem
instances should always be scoped to their corresponding objects and
discourage sharing.
Please do let us know should you need more help!
Thanks
---------- Forwarded message ---------
From: 舒琦 <sh...@eefung.com>
Date: Thu, Aug 24, 2017 at 11:43 PM
Subject: Re: Question about stop in SystemProducer
To: <dev@samza.apache.org>
Hi Jagadish,
I resolved the problem. Samza must close the FileSystem somewhere
when “shutdown” is triggered. Now I don’t get FileSystem instance from
cache and everything works fine.
Thanks!
————————
Qi Shu
> 在 2017年8月24日，15:06，舒琦 <sh...@eefung.com> 写道：
>
> Yes, in the same thread.
>
> File list is not empty, because after samza job shutdown, all files in
writing are not in complete state.
>
> ————————
> Qi Shu
>
>> 在 2017年8月24日，15:00，Jagadish Venkatraman <jagadish1...@gmail.com> 写道：
>>
>>>> Now the problem is not program hang, but the container quit
>> before the “closeFiles” method executed completely.
>>
>> It's unlikely that the container quit without returning from *close*. Are
>> you sure there were filesList is not empty? I'd suggest adding more
logging.
>>
>> I'm assuming you are calling *closeFiles* in the same thread?
>>
>> On Wed, Aug 23, 2017 at 11:53 PM, 舒琦 <sh...@eefung.com> wrote:
>>
>>> Hi Jagadish,
>>>
>>> When samza job is shutdown, “Begin to stop” is printed out in the
>>> log file, but not the log in closeFiles.
>>> Now the problem is not program hang, but the container quit before
>>> the “closeFiles” method executed completely.
>>>
>>> Thanks.
>>>
>>> ————————
>>> Qi Shu
>>>
>>>> 在 2017年8月24日，14:44，Jagadish Venkatraman <jagadish1...@gmail.com> 写道：
>>>>
>>>> Is "Begin to close file" printed? Where exactly is your application
>>> stalled
>>>> ? I'd suggest you take a stack dump.
>>>>
>>>> On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>
>>>>> Hi Jagadish,
>>>>>
>>>>> Below is part of the related code, log “"Begin to stop” is
>>> printed
>>>>> out.
>>>>>
>>>>> Thank you!
>>>>>
>>>>> public abstract class MyProducer implements SystemProducer {
>>>>>
>>>>> @Override
>>>>> public void stop() {
>>>>> LOGGER.info("Begin to stop");
>>>>>
>>>>> closeFiles();
>>>>>
>>>>> LOGGER.info("End to stop");
>>>>> }
>>>>>
>>>>> @Override
>>>>> protected void closeFiles() throws IOException {
>>>>> closeFiles(statusFiles);
>>>>> closeFiles(interactionFiles);
>>>>> }
>>>>>
>>>>> private void closeFiles(Map<String, FileInWriting> files) throws
>>>>> IOException {
>>>>> for (FileInWriting statusFile : files.values()) {
>>>>> LOGGER.info("Begin to close file: {}",
>>>>> statusFile.getFilePath());
>>>>> statusFile.getWriter().hflush();
>>>>> statusFile.getWriter().close();
>>>>> renameFile(statusFile.getFilePath());
>>>>> LOGGER.info("Successfully close file: {}",
>>>>> statusFile.getFilePath());
>>>>> }
>>>>>
>>>>> files.clear();
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> ————————
>>>>> Qi Shu
>>>>>
>>>>>> 在 2017年8月24日，14:21，Jagadish Venkatraman <jagadish1...@gmail.com> 写道：
>>>>>>
>>>>>> Hi Qi,
>>>>>>
>>>>>>>> the stop method in SystemProducer is called, but the close files
>>>>>> operation(may need some time, there may be cache data to be flushed)
in
>>>>>> stop method is not executed completely
>>>>>>
>>>>>> Are you seeing the *close()* method hang? SystemProducer.*close* is a
>>>>>> synchronous operation, and will block for the *close* method to
finish.
>>>>>>
>>>>>> Best,
>>>>>> Jagadish
>>>>>>
>>>>>> On Wed, Aug 23, 2017 at 11:17 PM, 舒琦 <sh...@eefung.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I write a SystemProducer for HDFS and everything is fine. When
>>>>>>> samza job is shutdown, the stop method in SystemProducer is called,
>>> but
>>>>> the
>>>>>>> close files operation(may need some time, there may be cache data to
>>> be
>>>>>>> flushed) in stop method is not executed completely.
>>>>>>>
>>>>>>> How can I resolve this problem?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> ————————
>>>>>>> Qi Shu
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jagadish V,
>>>>>> Graduate Student,
>>>>>> Department of Computer Science,
>>>>>> Stanford University
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Jagadish V,
>>>> Graduate Student,
>>>> Department of Computer Science,
>>>> Stanford University
>>>
>>>
>>
>>
>> --
>> Jagadish V,
>> Graduate Student,
>> Department of Computer Science,
>> Stanford University
>
--
Sent from my iphone.
