shanthoosh opened a new pull request #935: SAMZA-2118: Improve the shutdown 
sequence of AsyncRunLoop.
URL: https://github.com/apache/samza/pull/935
 
 
   A samza container is comprised of multiple tasks and event-loop. Event loop 
is the main orchestration layer between the different stages of a task(process, 
commit, window) and delivers messages to task by polling them from different 
consumers. 
   
   Currently in samza, the event loop dispatches messages to the tasks as per 
user-defined configuration `task.max.concurrency`. After dispatching messages 
with count equal to `task.max.concurrency` to the tasks,  the event loop is 
blocked until the processing of at-least a single dispatched message 
finishes(aka the task implementation invokes `taskCallBack.complete()` for 
at-least a dispatched message). 
   
   When the event loop is blocked by this, the user thread would sometimes try 
to shutdown the event loop. This shutdown intent marked by the user-thread will 
take effect only when the processing of a single message completes. In 
some-cases, when the task implementation does retry on failure for a message, 
this could prolong the shutdown of the samza event loop.
   
   This introduces tight coupling between the task processing time and the 
event loop shutdown time.
   
   This behavior was observed predominantly in standalone applications where 
the samza container is shutdown as a part of the re-balancing phase and task 
processing time was significantly larger.
   
   To alleviate this problem, as a part of this patch, we wake-up the samza 
event-loop thread when the user-thread marks the intent to shutdown the 
event-loop.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to