GitHub user nickpan47 opened a pull request:
https://github.com/apache/samza/pull/642
Samza 1814: consolidate JobNode and JobGraph configuration generation for
high and low-level API applications
High-level changes:
- Move configuration generation to JobNodeConfigurationGenerator
- Move the intermediate partition calculation to
IntermediationStreamPartitionPlanner
- Consolidate the code in JobPlanner and ExecutionPlanner for high and
low-level API plan/configuration generation
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickpan47/samza SAMZA-1814
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/642.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #642
----
commit 8e6fc2dac5af11f0748fdcd32175c2ae95418fc7
Author: prateekm <prateekm@...>
Date: 2018-06-09T22:01:10Z
Remove all usages of StreamSpec and ApplicationRunner from the operator
spec and impl layers.
commit 7d7aa50882d656143ea6fbc8064aca390024dd31
Author: Prateek Maheshwari <pmaheshwari@...>
Date: 2018-06-13T19:56:51Z
Updated with Cameron and Daniel's feedback.
commit 8561930133a0da52061bceee3596d02c6c7f9370
Author: Prateek Maheshwari <prateekm@...>
Date: 2018-06-15T22:25:50Z
Merge branch 'master' into stream-spec-cleanup
commit 7e71dc7e04c5e1a0357663a76068d3dead4ebfd8
Author: Yi Pan (Data Infrastructure) <yipan@...>
Date: 2018-06-27T22:22:17Z
Merge with master
commit f1cb8f0eb29feefe7e01c4a159f8792b5df0ce39
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-07-10T17:55:20Z
Merge branch 'master' into single-app-api-May-21-18
commit d43e923192b8b78a9eb6ba4ba15a05151ec457f2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-07-11T04:30:12Z
WIP: proto-type with ApplicationRunnable and no ApplicationRunner exposed
to user
commit 42782d815133ee75757c831f60a67f6980069e18
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-07-17T21:54:06Z
Merge branch 'prateek-remove-app-runner-stream-spec' into
app-spec-with-app-runtime-Jul-16-18
commit 95577b74c7330f11ee055c363b944c745f034ee0
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-07-18T10:09:52Z
WIP: trying to figure out the two interface classes for spec: a) spec
builder in init(); b) spec reader in all other lifecycle methods
commit 30a4e5f0ae6ff1b99fc8a6d08e742d8bff06e0cb
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-07-23T09:02:06Z
WIP: application runner refactor - proto-type for SEP-13
commit fb1bc49e04c5a420c22c3589c39eb08a2964d01d
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-07-30T20:55:59Z
Merge branch 'master' into app-spec-with-app-runtime-Jul-16-18
commit 973eb5261432da50a3540c96a101136f9c854f1d
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-03T19:08:30Z
WIP: compiles, still working on LocalContainerRunner refactor
commit f20cdcda68ebf96924ece4ad16fbe25a9b370323
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-06T08:32:03Z
WIP: adding unit tests. Pending update on StreamProcessorLifecycleListener,
LocalContainerRunner, and SamzaContainerListener
commit c4bb0dce6f812d19a5dcf02f19b7c2e455120bfd
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-06T17:46:11Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit a82708bb0f509affba19a81a0b2a10b2d8191df8
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-13T06:52:45Z
SAMZA-1789: unify ApplicationDescriptor and ApplicationRunner for high- and
low-level APIs in YARN and standalone environment
commit 014337170811527f7d3dcbe5af9e6ec353dc3efd
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-13T07:48:21Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit db96da830a11d001eec7c068bf8a7d03a9a11799
Author: Yi Pan (Data Infrastructure) <yipan@...>
Date: 2018-08-17T00:15:48Z
SAMZA-1789: WIP - revision to address review feedbacks.
commit 3b2f04d5434fba4a3793f85e7e57d09802d5d627
Author: Yi Pan (Data Infrastructure) <yipan@...>
Date: 2018-08-19T16:18:23Z
SAMZA-1789: moved all impl classes from samza-api to samza-core.
commit 4382d45dbdb2a62070d424661f2459e3d2af9942
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-19T16:51:40Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit 6e446fe6d201ab8b1e30ceeeb546a990ae2b77b2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-23T01:36:33Z
SAMZA-1789: address Cameron's review comments.
commit 1621c4d00a185cf0905a7500420955cefa56938c
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-23T04:55:01Z
SAMZA-1789: a few more fixes to address Cameron's reviews
commit 247dcff4cdbf3c417290742d632001898524a8fd
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-23T05:16:14Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit 055bd91e475770a785b8a1e1d4aedb188561b5d2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-23T05:41:06Z
SAMZA-1789: fix unit test with ThreadJobFactory
commit 8d4d3ffda2650fa41757d8ded94a7fb0d878feb0
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-24T20:37:39Z
Merge with master
commit e7af6932d520168dfcda3b06acb7e587f2e5f8a2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-24T21:10:26Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit a072118d09ea4050589b75483366cea2c5d6fa0d
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-27T04:44:23Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit 12c09af06430a9b59412f369a4e86a70ee686418
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-27T06:09:11Z
SAMZA-1789: Fix a merging error (with SAMZA-1813)
commit 33753f72d14b6fa8a413b15bee0062d003f27059
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-29T02:03:11Z
Merge branch 'master' into app-runtime-with-processor-callbacks
commit f04404cc2e13e2b41c0ddd656fca7bc4fec721d2
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-29T03:51:20Z
SAMZA-1789: move createStreams out of the loop in prepareJobs
commit f2969f8dfa1bbe19bf823836daa03c55cb064051
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-31T01:56:17Z
SAMZA-1789: fixed ApplicationDescriptor to use InputDescriptor and
OutputDescriptor; addressed Prateek's comments.
commit f4b3d43a478ab9300d4b3e15257288323ff62348
Author: Yi Pan (Data Infrastructure) <nickpan47@...>
Date: 2018-08-31T06:05:11Z
SAMZA-1789: Fxing TaskApplication examples and some checkstyle errors
----
---
