24, 2017 at 11:43 PM
Subject: Re: Question about stop in SystemProducer
To:
Hi Jagadish,
I resolved the problem. Samza must close the FileSystem somewhere
when “shutdown” is triggered. Now I don’t get FileSystem instance from
cache and everything works fine.
Thanks
Hi Jagadish,
I resolved the problem. Samza must close the FileSystem somewhere when
“shutdown” is triggered. Now I don’t get FileSystem instance from cache and
everything works fine.
Thanks!
Qi Shu
> 在 2017年8月24日，15:06，舒琦 写道：
>
> Yes, in the same thread.
>
> File l
Yes, in the same thread.
File list is not empty, because after samza job shutdown, all files in writing
are not in complete state.
Qi Shu
> 在 2017年8月24日，15:00，Jagadish Venkatraman 写道：
>
>>>Now the problem is not program hang, but the container quit
> before the “closeFiles” m
>> Now the problem is not program hang, but the container quit
before the “closeFiles” method executed completely.
It's unlikely that the container quit without returning from *close*. Are
you sure there were filesList is not empty? I'd suggest adding more logging.
I'm assuming you are ca
Hi Jagadish,
When samza job is shutdown, “Begin to stop” is printed out in the log
file, but not the log in closeFiles.
Now the problem is not program hang, but the container quit before the
“closeFiles” method executed completely.
Thanks.
Qi Shu
> 在 2017年8月24
Is "Begin to close file" printed? Where exactly is your application stalled
? I'd suggest you take a stack dump.
On Wed, Aug 23, 2017 at 11:32 PM, 舒琦 wrote:
> Hi Jagadish,
>
> Below is part of the related code, log “"Begin to stop” is printed
> out.
>
> Thank you!
>
> public abst
Hi Jagadish,
Below is part of the related code, log “"Begin to stop” is printed out.
Thank you!
public abstract class MyProducer implements SystemProducer {
@Override
public void stop() {
LOGGER.info("Begin to stop");
closeFiles();
LOGGER.info("
Hi Qi,
>> the stop method in SystemProducer is called, but the close files
operation(may need some time, there may be cache data to be flushed) in
stop method is not executed completely
Are you seeing the *close()* method hang? SystemProducer.*close* is a
synchronous operation, and will block for
Hi,
I write a SystemProducer for HDFS and everything is fine. When samza
job is shutdown, the stop method in SystemProducer is called, but the close
files operation(may need some time, there may be cache data to be flushed) in
stop method is not executed completely.
How can I r
