Re: Questions about partitioning

Hi, Susan, Welcome to Samza! First I will try to answer your question about partition assignment in Samza. The assignment from stream partition to Samza tasks is determined by the SystemStreamPartitionGrouper. The default implementation include two assignment methods: 1 task per system stream par

Re: Questions about partitioning

Hey Susan- That volume of topics (or partitions) would be a significant burden on both the Kafka cluster and underlying YARN cluster (for the Samza job). A 'large number of partitions' even at places with huge Kafka clusters is on the order of 512 or so. It sounds like you're trying to use part

Re: Questions about partitioning

Hey Susan, As far as I know, there is very minimal differences between Partition vs Topic strategy in terms of performance - in terms of how they are allocated in the memory they should be very similar, but I'll get some Kafka experts to comment on that. >From Samza's perspective,

Questions about partitioning

Hi there, I'm new to Samza/Kafka and we're evaluating Samza to see whether it would be a good fit for our application. I just had a few questions about how partitioning works. I understand there is a limitation on the number of topics we can create [1], and I was wondering, if we need more than, s

Re: consistency between input, output and changelog streams

Hi Bart, Found the case you described in Samza state-management doc : *"For many of the stateful processing use cases discussed above, this is not a problem: if the effect of a message on state is idempotent, it is

Re: Review Request 33503: SAMZA-604 - Add support for using gradle to build/run hello-samza tasks.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33503/ --- (Updated April 24, 2015, 7:47 p.m.) Review request for samza. Changes ---

Re: Review Request 33503: SAMZA-604 - Add support for using gradle to build/run hello-samza tasks.

> On April 24, 2015, 6:54 p.m., Yan Fang wrote: > > README-gradle.txt, line 10 > > > > > > "build hello-samza job package" sounds better because actually > > "./gradlw build" is for "build hello-samza project", not "di

Re: Review Request 33409: SAMZA-433 - Rename "task" to "container" in AM and container

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33409/ --- (Updated April 24, 2015, 7:21 p.m.) Review request for samza. Changes ---

Re: Review Request 33409: SAMZA-433 - Rename "task" to "container" in AM and container

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33409/#review81527 --- samza-yarn/src/main/scala/org/apache/samza/job/yarn/SamzaAppMasterM

Re: Review Request 33409: SAMZA-433 - Rename "task" to "container" in AM and container

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33409/ --- (Updated April 24, 2015, 7:09 p.m.) Review request for samza. Changes ---

Re: Review Request 33503: SAMZA-604 - Add support for using gradle to build/run hello-samza tasks.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33503/#review81506 --- README-gradle.txt

Re: Review Request 33146: New KeyValueStore Features

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33146/#review81497 --- Ship it! Ship It! - Yi Pan (Data Infrastructure) On April 24, 20

Re: Review Request 33146: New KeyValueStore Features

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33146/ --- (Updated April 24, 2015, 4:59 p.m.) Review request for samza. Changes ---

Re: Review Request 33409: SAMZA-433 - Rename "task" to "container" in AM and container

> On April 24, 2015, 12:58 a.m., Yan Fang wrote: > > samza-yarn/src/main/resources/scalate/WEB-INF/views/index.scaml, lines > > 131-135 > > > > > > This one was correct. Only need to change the state.runningTasks to