GitHub user vjagadish1989 opened a pull request:
https://github.com/apache/samza/pull/348
SAMZA-1479 : Refactor KafkaCheckpointManager, KafkaCheckpointLogKey and
their tests
Notable changes:
* Rewrite `KafkaCheckpointLogKey` into two classes - an immutable class,
and a SerDe
* Remove dependency on static setters in the `KafkaCheckpointLogKey`
* Change lifecycle of components in KafkaCheckpointManager
- It's safe to start producers and consumers during `start` as opposed to
lazy loading them during writes, and reads.
- Initialize systemProducer and systemConsumer during construction
* Simplify logic for ignoring checkpoint validations
* Re-write checkpointManager#readLog() to use a simpler API.
* Remove unnecessary complexity after the migration from 0.8
* Remove unnecessary locking in startup, and shut-down
* Remove dependencies on SimpleConsumer configs like bufferSize, fetchSize,
socketTimeout
* Refactor KafkaCheckpointManagerFactory and remove static
getCheckpointSystemNameAndFactory
* Bug-fix : Register the taskName correctly (instead of using a dummy
string for the taskName)
* Add unit tests to verify more checkpoint scenarios
* Consolidate unit tests into utils for creating producer, consumer and
admin instances
* Convert/consolidate most long-running integration tests into unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vjagadish1989/samza
kafka-checkpointmanager-refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/348.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #348
----
commit c8b225f1b44fc180a5d6bb385e917a42bc54d97f
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T01:37:09Z
Refactor checkpoint log key
commit f28d2e24a1d91cb5cefad6d5d0245c694d6d83ac
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T01:43:12Z
Merge branch 'master' of https://github.com/apache/samza
commit 289eaa58730965a51d4b32e788568f554d645eae
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T02:24:02Z
Fixes for KafkaCheckpointManager
commit 9e533283ff991dd227f81c0af363ff5a2c2580e0
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T03:11:54Z
Fixes for KafkaCheckpointManager
commit 9ffcf5391f3e78cb1e402bba8f8a2abdc4a4a729
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T05:13:09Z
checkpoint manager refactor
commit c10159d2cca756e65661307c7d9fa1f8ce7857ff
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T06:01:42Z
rewrite kcm
commit 7db01525084b2c91410323531c8716abcbfd4e77
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T06:03:59Z
Renamed LogKey class
commit 8db287198d9239729478d1631e6b535865123b34
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-20T06:33:16Z
Initial pass at getting tests to compile
commit fc7d30b2ecd03c5e10f3c9c8d2482b2e6a922a44
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-21T02:11:10Z
Fixed checkpoint serdes, and a bunch of unit tests
commit 904446173f920eefc676c561f7780768ef84b886
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-26T01:36:36Z
stage commits for KCM
commit 4f04fe204555759a59e89f9e07d4f9c63e8dd6bf
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-31T15:38:54Z
Cleanup tests, separate unit tests and integration tests
commit 34208f71c62b59e0bb060305f35c36c630248bdd
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-10-31T17:01:16Z
Merge with latest master
commit d5411c7d4ac793dd880b9b1550f29a8a153cace6
Author: Jagadish <jvenkatra...@linkedin.com>
Date: 2017-11-01T19:17:10Z
Rewrite KafkaCheckpointManagerFactory. Refactor static methods. Remove
duplicate getCheckpointTopicProperties
----
---
