GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/87
SAMZA-1146: TaskCallbackManager commit fix.
Each task callback in samza belongs to different SystemStreamPartition.
When multiple callbacks in contagious order are available for commit, callback
with highest sequence number is chosen for commit. This will prevent
checkpointing of completed callbacks that has commit request and doesn't have
highest sequence number. Upon task restart this will lead to duplicate
reprocessing of already processed messages (since completed callbacks for some
SystemStreamPartition's aren't committed earlier).
This PR fixes it and commits all completed callbacks that has commit
request defined. Added a test to verify the behavior.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza Fixing_CallBackManager_Commit
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/87.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #87
----
commit 7afbbcef414bb2d3e64117a8e1880072e0a99471
Author: Shanthoosh Venkataraman <svenkatara...@linkedin.com>
Date: 2017-03-09T19:54:20Z
SAMZA-1146 : TaskCallbackManager commit fix.
Changing TaskCallbackManager.update to return all completed callbacks that
has
commit request defined for commit(not just the callback with highest
sequence number).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
