----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36006/ -----------------------------------------------------------
Review request for samza. Repository: samza Description ------- After a job is submitted, it might need some configuration change, specifically it might need more containers. In SAMZA-704 a tool is being added to write to the coordinator stream (CoordinatorStreamWriter). This tool can be used to write new configurations to the coordinator stream. However, another tool (ConfigManager) is needed to read the config changes and react to them, which is the goal of this task. This tool should be brought up after the job is submitted and read any config changes added to the coordinator stream, and react to each accordingly. This tool, called the Config Manager, is focusing on handling container changs by reacting to cset-config massages with key "yarn.container.count". If the config manager is enabled, it will come up after the JobRunner submits the job, and listen to config messages added there after. To enable the ConfigManager use the flag "--config-manager-is-active=true" when submitting the job. The config manger will periodically poll the coordinator stream to see if there are any new messages. This period is set to 100 ms by deafualt. However, it can be configured using "--config-manager-polling-interval=200" when submitting the job. Thus, overal the command to run the config manager along with the job would be: <path to samza deployment>/bin/run-job.sh --config-factory=<config factory> --config-path=<path to config file of a job> --config-manager-is-active=<false,true> --config-manager-polling-interval=<polling-interval> And this command should be used to runa a job instead. Diffs ----- checkstyle/import-control.xml 3374f0c432e61ac4cda275377604cfd481f0cddf samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamMessage.java 6c1e488d00d8593d59c89b57e673e0b6b90fd7d2 samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java PRE-CREATION samza-core/src/main/java/org/apache/samza/job/ConfigManager.java PRE-CREATION samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 samza-core/src/main/scala/org/apache/samza/util/CommandLine.scala f26501b2820b99d1ad2964c6f7833ef7eaddba97 samza-core/src/test/java/org/apache/samza/coordinator/stream/MockCoordinatorStreamSystemFactory.java 647cadb3a4e51bec8204197d77ad35a6b29afcec samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamSystemProducer.java 68e32554c18f443565284b807f43f4a5b05afbce samza-core/src/test/java/org/apache/samza/coordinator/stream/TestCoordinatorStreamWriter.java PRE-CREATION samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala ea702a919348305ff95ce0b4ca1996a13aff04ec samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala 20aa3736a842c462449682f3fc54c7f66c356d7d samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMaster.scala 7b7d86a43c69e72c47eaa91f68be24e0f4022891 Diff: https://reviews.apache.org/r/36006/diff/ Testing ------- Thanks, Shadi A. Noghabi
