Re: Thoughts and obesrvations on Samza

2015-07-03 Thread Gianmarco De Francisci Morales
Hi Jay, Thanks for your answer. > However a few things have changed since that original design: > 1. We now have the additional use cases of copycat and Samza > 2. We now realize that the assignment strategies don't actually necessarily > ensure each partition is assigned to only one consumer--t

Review Request 36163: SAMZA-690: changelog topic creation should not be in the container code

2015-07-03 Thread Robert Zuljevic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36163/ --- Review request for samza. Repository: samza Description --- Moved the cr

Re: Review Request 36163: SAMZA-690: changelog topic creation should not be in the container code

2015-07-03 Thread Robert Zuljevic
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36163/ --- (Updated July 3, 2015, 9:58 a.m.) Review request for samza. Repository: samza

Re: Samza and sliding window

2015-07-03 Thread Shekar Tippur
Any answer on how to get all the kv values and reinitialise the kv store? Had one more question on implementing sliding window. If i use a kv store like rocksdb, and I use yarn (say 3 node cluster), the job that it runs to aggregate gets distributed as well and I am guessing the aggregation numbe

Re: Thoughts and obesrvations on Samza

2015-07-03 Thread Jay Kreps
Hey Yi, Yeah let me try to be more concrete. Any partition assignment management requires two things G - a group of consumer instances f - a function f(G) that yields the mapping of partitions to consumer instances The hard thing is to determine G, this is what requires the heartbeats, coordi

Re: Thoughts and obesrvations on Samza

2015-07-03 Thread Jay Kreps
Hey Gianmarco, To your broader point, I agree that having a close alignment with Kafka would be a great thing in terms of adoption/discoverability/etc. There areas where I think this matters a lot are: 1. Website and docs: ideally when reading about Kafka you should be able to find out about Samza

Re: Review Request 35933: SAMZA-449 Expose RocksDB statistic

2015-07-03 Thread Gustavo Anatoly F . V . Solís
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35933/ --- (Updated Julho 3, 2015, 11:05 p.m.) Review request for samza. Repository: sam

Re: Review Request 35933: SAMZA-449 Expose RocksDB statistic

2015-07-03 Thread Yan Fang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35933/#review90373 --- samza-kv-rocksdb/src/main/scala/org/apache/samza/storage/kv/RocksDb

Re: Review Request 35933: SAMZA-449 Expose RocksDB statistic

2015-07-03 Thread Gustavo Anatoly F . V . Solís
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35933/ --- (Updated Julho 4, 2015, 12:13 a.m.) Review request for samza. Repository: sam