> On July 27, 2015, 7:36 a.m., Yi Pan (Data Infrastructure) wrote: > > The code LGTM. For testing, if we can verify this fix w/ a stateful > > StreamTask w/ changelog enabled with some partition numbers that are > > different from the default auto-creation partition number (i.e. 8) in > > Kafka, it would be good. The integration test suite in samza-test should be > > a good place to add the test there. Try following the steps in > > samza-test/src/main/config/join/README and run the integration test. The > > joiner task has a changelog configured with partition number of 2. You can > > verify the test passed w/ your fix. > > Robert Zuljevic wrote: > Hi Yi, sorry for bothering you so much with this task : ) I'll just write > down what I managed to do regarding integration tests: > > 1. I ran integration tests via Zopkio and they all finished successfully. > 2. I ran the integration per guide in > samza-test/src/main/config/join/README and I suspect they ran successfully, > since none of them had an abnormal final status. I also ran the failure tests > (albeit after some limited fidling with the python scripts involved). > 3. I ran "./gradlew clean build" (which runs TestStatefulTask). It > finished with a STANDARD_ERROR, which I assume is a good thing, but here is > the output, just in case: http://pastebin.com/aLT5jRdd > > What I suspect are the next (possible) steps: > > 1. Create integration tests to be used with Zopkio. Here I am uncertain > how I would kill/stop Samza task to verify that changelog stream is being > consumed properly. > 2. Create another set of tasks similar to Checker/Emitter/Joiner/Watcher. > I believe this is unnecessary since they have their changelogs and their > restartability is being tested. Of course, I might be wrong. > 3. Add another test similar to TestStatefulTask. > a. Or add num.partitions param to TestStatefulTask. > 4. None of the above : ) > > Again, I am very sorry for relying on you this much, but I'm really > unclear on how to proceed regarding this.
Hi, @Robert, sorry that I was not too specific in the comment before. If you have successfully ran the integration tests via the steps in samza-test/src/main/config/join/README. It should be good to go. Thanks! - Yi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36163/#review93093 ----------------------------------------------------------- On July 9, 2015, 2:39 p.m., Robert Zuljevic wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/36163/ > ----------------------------------------------------------- > > (Updated July 9, 2015, 2:39 p.m.) > > > Review request for samza. > > > Repository: samza > > > Description > ------- > > Removed trailing whitespaces > > > Diffs > ----- > > samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java > 7a588ebc99b5f07d533e48e10061a3075a63665a > > samza-api/src/main/java/org/apache/samza/util/SinglePartitionWithoutOffsetsSystemAdmin.java > 249b8ae3a904716ea51a2b27c7701ac30d13b854 > samza-core/src/main/scala/org/apache/samza/coordinator/JobCoordinator.scala > 8ee034a642a13af1b0fdb4ebbb3b2592bb8e2be1 > samza-core/src/main/scala/org/apache/samza/storage/TaskStorageManager.scala > aeba61a95371faaba23c97d896321b8d95467f87 > > samza-core/src/main/scala/org/apache/samza/system/filereader/FileReaderSystemAdmin.scala > 097f41062f3432ae9dc9a9737b48ed7b2f709f20 > > samza-core/src/test/scala/org/apache/samza/checkpoint/TestOffsetManager.scala > 8d54c4639fc226b34e64915935c1d90e5917af2e > > samza-core/src/test/scala/org/apache/samza/coordinator/TestJobCoordinator.scala > d9ae187c7707673fe15c8cb7ea854e02c4a89a54 > > samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala > 35086f54f526d5d88ad3bc312b71fce40260e7c6 > samza-test/src/main/java/org/apache/samza/system/mock/MockSystemAdmin.java > b063366f0f60e401765a000fa265c59dee4a461e > > samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala > 1e936b42a5b9a4bfb43766c17847b2947ebdb21d > > Diff: https://reviews.apache.org/r/36163/diff/ > > > Testing > ------- > > I wasn't really sure what kind of test (unit test / integration test) I > should make here, so any pointers would be greatly appreaciated! I tested the > change with the unit/integration tests already available. > > > Thanks, > > Robert Zuljevic > >
