Re: Samza and sliding window

Hi Shekar, You can use Kafka's partitioning capabilities to partition your stream based on application. That will make sure events related to a application will always ended up in same partition. With this you will have multiple applications in same partition and each partition will be mapped to a

Re: Samza and sliding window

Milinda, This is a stream of events where I dont know how many applications are sending events. I need to dynamically create Kafka partitions. Can you please confirm the flow: 1. New event comes in 2. Check to see if a partition exists for the application. If not create one. 3. Implement public st

Re: Samza and sliding window

Shekar, You won't be creating a partition per application. By using the application name as the partitioning key you ensure all events for a given application are consistently mapped to the same partition. Multiple applications will be mapped to each partition without any need for a priori knowled

Re: Samza and sliding window

Benjamin, Thanks for the explanation. We dont have any specific partition scheme as yet. We just have 2 topics - raw and processed and we use default partitioning scheme. Can you share any code snippet so I can understand it better? - Shekar

Re: Samza and sliding window

Hi Shekar, Please have a look at [1]. Milinda [1] https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example On Mon, Jun 29, 2015 at 1:06 PM, Shekar Tippur wrote: > Benjamin, > > Thanks for the explanation. We dont have any specific partition scheme as > yet. We just have 2 topi

Re: Samza and sliding window

Hi, Shekar, First, I would like to clarify what you meant by sliding window: is it defined as windows with size N and advance step size of 1 (which means that windows overlap and each input message would contribute to multiple counts in different windows)? Or windows with size N and advance step s

Review Request 36006: refactoring the code for coordinator stream writer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36006/ --- Review request for samza. Repository: samza Description --- After a job

Re: Samza and sliding window

Yi, My use case is more of the latter. Your explanation makes sense now. I was also looking into Milinda's wiki. She has a section for Kafka partition SimplePartitioner, which is simple enough as well. Thanks for all the inputs. Let me see what I come up with while implementing it. - Shekar On

Re: Hopping and tumbling windows in streaming SQL

Hey, Julian, That's awesome! I read through all the examples and it is really easy to express most of our use cases now! Thanks a lot! I have just a few additional points here: Q5. Aligned tumbling window TUMBLE does not have an align argument, so you need to use HOP. SELECT STREAM START(rowti

Re: Review Request 35241: refactoring the code for coordinator stream writer

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35241/ --- (Updated June 29, 2015, 11:08 p.m.) Review request for samza, Yi Pan (Data Infr

Re: Review Request 35241: refactoring the code for coordinator stream writer

> On June 17, 2015, 12:25 a.m., Navina Ramesh wrote: > > samza-core/src/main/java/org/apache/samza/coordinator/stream/CoordinatorStreamWriter.java, > > line 41 > > > > > > Can you add a .sh wrapper using run-job to act

Re: [VOTE] Apache Samza 0.9.1 RC1

+1 Verified MD5, Signature. Tested locally. Thanks, Fang, Yan yanfang...@gmail.com On Sun, Jun 28, 2015 at 12:31 PM, Yi Pan wrote: > Hey all, > > This is a call for a vote on a release of Apache Samza 0.9.1. This is a > bug-fix release against 0.9.0. > > The release candidate can be download