Re: Kafka Streams vs Spark Streaming

On Sun, Feb 26, 2017 at 8:56 PM, Guozhang Wang wrote: > Hello Kohki, > > Given your data traffic and the state volume I cannot think of a better > solution but suggest using large number of partitioned local states. > > I'm wondering how would "per partition watermark&q

Re: Kafka Streams vs Spark Streaming

t; > > > > Regarding your question about maturity and users of Streams, you can > take a> > > look at a bunch of the blog posts written about their Streams usage in> > > production, for example:> > > > > http://engineering.skybettingandgaming.com/201

Re: Kafka Streams vs Spark Streaming

Guozhang, Let me explain what I'm trying to do. The message volume is large (TB per Day) and that is coming to a topic. Now I want to do per minute aggregation(Windowed) and send the output to the downstream (a topic) (Topic1 - Large Volume) -> [Stream App] -> (Topic2 - Large Volume) I assume th

Re: Immutable Record with Kafka Stream

to > >> wait for the late arrival of records whose timestamps fall within the > >> window interval. If a record arrives after the retention period has > passed, > >> the record cannot be processed and is dropped. > > > > > > And I believe I can set retention period by using 'until' > > > > TimeWindows.of(6).until(6) > > > > > > After receiving a data from (00:06:00), I don't know why it still > continue > > receiving data from time of 00:00:00, what is 'until' supposed to do ? > > > > Thanks > > -Kohki > > > > -- Kohki Nishio

Re: Kafka Streams vs Spark Streaming

ve Spark a try else you have to write a code which is optimized for your use case thanks -Kohki On Fri, Feb 24, 2017 at 6:22 PM, Tianji Li wrote: > Hi there, > > Can anyone give a good explanation in what cases Kafka Streams is > preferred, and in what cases Sparking Streaming is better? > > Thanks > Tianji > -- Kohki Nishio

Re: Immutable Record with Kafka Stream

Guozhang, thanks for the reply, but I'm having trouble understanding, here's the statement from the document Windowing operations are available in the Kafka Streams DSL > , > where users can specify a *retenti

Re: Immutable Record with Kafka Stream

dup cache work better (for documentation see > http://docs.confluent.io/3.1.2/streams/developer-guide. > html#memory-management <http://docs.confluent.io/3.1. > 2/streams/developer-guide.html#memory-management>). However, this does > not guarantee deduplicates do not happen. > &

Immutable Record with Kafka Stream

3) I want to drop the last one so that I don't have duplicate messages, Thanks -- Kohki Nishio