Hi
I don't think you can view samza container logs in the web as mr job
history, try to check the dump folder at HDFS.
There should be one aggregated log file per machine in the folder named
according to the job_id
On 16 January 2017 at 15:30, 舒琦 <sh...@eefung.com> wrote:
> Hi,
>
> Thanks for your help.
>
> Here are 2 questions:
>
> 1. I have defined my own HDFS producer which implemented SystemProducer
> and overwrite stop method(I log something in the first line of stop
> method), but when I kill the app, the log are not printed out. The tricky
> thing is the logic defined in stop method sometimes can be executed and
> sometimes not.
>
> Below is stop method:
>
> @Override
> public void stop() {
> try {
> LOGGER.info("Begin to close files");
> closeFiles();
> } catch (IOException e) {
> LOGGER.error("Error when close Files", e);
> }
>
> if (fs != null) {
> try {
> fs.close();
> } catch (IOException e) {
> //do nothing
> }
> }
> }
>
>
> Below is the log:
>
> 2017-01-16 15:13:35.273 [Thread-9] SamzaContainer [INFO] Shutting down,
> will wait up to 5000 ms
> 2017-01-16 15:13:35.284 [main] SamzaContainer [INFO] Shutting down.
> 2017-01-16 15:13:35.285 [main] SamzaContainer [INFO] Shutting down
> consumer multiplexer.
> 2017-01-16 15:13:35.287 [main] BrokerProxy [INFO] Shutting down
> BrokerProxy for 172.19.105.22:9096
> 2017-01-16 15:13:35.288 [main] BrokerProxy [INFO] closing simple
> consumer...
> 2017-01-16 15:13:35.340 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at
> 172.19.105.22:9096 for client samza_consumer-canal_status_persistent_hstore-1]
> BrokerProxy [INFO] Got interrupt exception in broker proxy thread.
> 2017-01-16 15:13:35.340 [main] BrokerProxy [INFO] Shutting down
> BrokerProxy for 172.19.105.21:9096
> 2017-01-16 15:13:35.341 [main] BrokerProxy [INFO] closing simple consumer…
>
> You can see the log “Begin to close files” are not printed out and of
> course the logic is not executed.
>
> 2. The hadoop cluster I use is “HDP-2.5.0”,the log aggregation is also
> enabled, but logs of containers can not be collected, only the log of am
> can be seen.
>
>
>
> ————————
> ShuQi
>
> 在 2017年1月16日，10:39，Liu Bo <diabl...@gmail.com> 写道：
>
> Hi,
>
> *container log will be removed automatically,*
>
>
> you can turn on yarn log aggregation, so that terminated yarn jobs' log
> will be dumped to HDFS
>
> On 14 January 2017 at 07:44, Yi Pan <nickpa...@gmail.com> wrote:
>
> Hi, Qi,
>
> Sorry to reply late. I am curious on your comment that the close and stop
> methods are not called. When user initiated a kill request, the graceful
> shutdown sequence is triggered by the shutdown hook added to
> SamzaContainer. The shutdown sequence is the following in the code:
> {code}
> info("Shutting down.")
>
> shutdownConsumers
> shutdownTask
> shutdownStores
> shutdownDiskSpaceMonitor
> shutdownHostStatisticsMonitor
> shutdownProducers
> shutdownLocalityManager
> shutdownOffsetManager
> shutdownMetrics
> shutdownSecurityManger
>
> info("Shutdown complete.")
> {code}
>
> in which, MessageChooser.stop() is invoked in shutdownConsumers, and
> SystemProducer.close() is invoked in shutdownProducers.
>
> Could you explain why you are not able to shutdown a Samza job gracefully?
>
> Thanks!
>
> -Yi
>
> On Mon, Dec 12, 2016 at 6:33 PM, 舒琦 <sh...@eefung.com> wrote:
>
> Hi Guys,
>
> How can I stop running samza job gracefully except killing it?
>
> Because when samza job was killed, the close and stop method in
> BaseMessageChooser and SystemProducer will not be called and the
>
> container
>
> log will be removed automatically, how can resolve this?
>
> Thanks.
>
> ————————
> ShuQi
>
>
>
>
>
> --
> All the best
>
> Liu Bo
>
>
>
--
All the best
Liu Bo
