> On July 22, 2015, 6:25 p.m., Yi Pan (Data Infrastructure) wrote: > > samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMaster.scala, > > line 130 > > <https://reviews.apache.org/r/36006/diff/2/?file=1015156#file1015156line130> > > > > Question: can we move this code into the JobCoordinator.start()? I > > don't see any reason why this has to be outside JobCoordinator?
It cannot be moved to there since the server url is not set yet in JobCoordinator.start(). I have moved this to SamzaAppMasterService.onInit() On July 22, 2015, 6:25 p.m., Shadi A. Noghabi wrote: > > Some high-level comments: I don't see a reason that these tools classes > > have to be in samza-core. Can we move it to a separate module, like > > samza-man? That will also allow us to keep the minimum dependencies in > > samza-core. I will continue w/ the implementation of ConfigManager. I moved the config manager to a separate module. What do you mean by "I will continue w/ the implementation of ConfigManager" ? - Shadi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36006/#review92616 ----------------------------------------------------------- On July 28, 2015, 2:39 a.m., Shadi A. Noghabi wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36006/ > ----------------------------------------------------------- > > (Updated July 28, 2015, 2:39 a.m.) > > > Review request for samza, Yi Pan (Data Infrastructure), Navina Ramesh, and > Naveen Somasundaram. > > > Repository: samza > > > Description > ------- > > After a job is submitted, it might need some configuration change, > specifically it might need more containers. In SAMZA-704 a tool is being > added to write to the coordinator stream (CoordinatorStreamWriter). This > tool can be used to write new configurations to the coordinator stream. > However, another tool (ConfigManager) is needed to read the config changes > and react to them, which is the goal of this task. This tool should be > brought up after the job is submitted and read any config changes added to > the coordinator stream, and react to each accordingly. > > This tool, called the Config Manager, is focusing on handling container > changs by reacting to set-config massages with key "yarn.container.count". > > The config manager is a separate standa alone module, that should be brought > up separately after the submission of a job. Therefore, you have to add two > configurations to the input config file: > 1. yarn.rm.address= <ip of resource manager in yarn. ex: localhost > > 2. yarn.rm.port= <the port of the resource manager http server. ex: 8088 > > > The config manger will periodically poll the coordinator stream to see if > there are any new messages. This period is set to 100 ms by deafualt. > However, it can be configured by adding > configManager.polling.interval=<polling interval> to the input config file. > Thus, overal the command to run the config manager along with the job would > be: > > > <path to samza deployment>/bin/run-config-manager.sh --config-factory=<config > factory> --config-path=<path to config file of a job> > > > Diffs > ----- > > build.gradle 0852adc4e8e0c2816afd1ebf433f1af6b44852f7 > checkstyle/import-control.xml 6654319392929857bb861d77763afd8a5ea7674c > gradle/dependency-versions.gradle fb06e8ed393d1a38abfa1a48fe5244fc7f6c7339 > > samza-autoscaling/src/main/java/org/apache/samza/autoScaling/deployer/ConfigManager.java > PRE-CREATION > > samza-autoscaling/src/main/java/org/apache/samza/autoScaling/utils/YarnUtil.java > PRE-CREATION > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamSystemConsumer.java > b1078bdf7bddd16c9ccc6559b9efd40ca5ae67bc > samza-core/src/main/scala/org/apache/samza/job/JobRunner.scala > 1c178a661e449c6bdfc4ce431aef9bb2d261a6c2 > samza-shell/src/main/bash/run-config-manager.sh PRE-CREATION > > samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala > ea702a919348305ff95ce0b4ca1996a13aff04ec > > samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterService.scala > ce88698c12c4bf6f4cf128f92d60b0b9496997d7 > settings.gradle 19bff971ad221084dac10d3f7f3facfa42b829a7 > > Diff: https://reviews.apache.org/r/36006/diff/ > > > Testing > ------- > > Tested with hello samza and works properly. > > > Thanks, > > Shadi A. Noghabi > >
