GitHub user dnishimura opened a pull request:
https://github.com/apache/samza/pull/411
Move the changelog and checkpoint stream creation to the job coordinators.
**Overview**
The purpose of this PR is to consolidate the creation of the changelog and
checkpoint streams into the JobCoordinators. In the current state, the
changelog stream is created from the JobModelManager and the checkpoint stream
is created within the OffsetManager. The issue with creating the checkpoint in
the OffsetManager is that the first call happens from the first SamzaContainer
that runs and each subsequent SamzaContainer run will attempt to create the
checkpoint stream.
**Motivations**
There are three driving forces for this refactoring. The first motivation
is to assign the creation of the changelog and checkpoint streams to the
JobCoordinators where it is most appropriate. This was discussed in more detail
with @nickpan47 . The second motivation is to have any potential failure to
stream creation happen no later than during job coordination. The third
motivation is to accommodate future security work to provide a robust way to
set ACLs on streams.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dnishimura/samza
samza-1555-move-changelog-checkpoint-creation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/411.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #411
----
commit 971fa91afb7f57fc8c099a7fb17e7bc96bc826c2
Author: Daniel Nishimura <dnishimura@...>
Date: 2018-01-16T21:20:07Z
Move the responsibility of changelog and checkpoint stream creation to the
job coordinators.
----
---
