Re: Question about bootstrap processing in KafkaStreams.

2016-06-29 Thread Matthias J. Sax
Hi, there was a similar discussion on the list already "Kafka stream join scenario": http://search-hadoop.com/m/uyzND1WsAGW1vB5O91&subj=Kafka+stream+join+scenarios Long story short: there is no explicit support or guarantee. As Jay mentioned, some alignment is best effort. However, the main issu

Re: Question about bootstrap processing in KafkaStreams.

2016-06-28 Thread Gwen Shapira
Upgrade :) On Tue, Jun 28, 2016 at 6:49 PM, Rohit Valsakumar wrote: > Hi Jay, > > Thanks for the reply. > > Unfortunately in our case due to legacy reasons we are using > WallclockTimestampExtractor in the application for all the streams and the > existing messages in the stream probably won¹t ha

Re: Question about bootstrap processing in KafkaStreams.

2016-06-28 Thread Rohit Valsakumar
Hi Jay, Thanks for the reply. Unfortunately in our case due to legacy reasons we are using WallclockTimestampExtractor in the application for all the streams and the existing messages in the stream probably won¹t have timestamps as they are being produced by legacy clients. So the events are bein

Re: Question about bootstrap processing in KafkaStreams.

2016-06-28 Thread Jay Kreps
I think you may get this for free as Kafka Streams attempts to align consumption across different topics/partitions by the timestamp in the messages. So in a case where you are starting a job fresh and it has a database changelog to consume and a event stream to consume, it will attempt to keep the

Question about bootstrap processing in KafkaStreams.

2016-06-28 Thread Rohit Valsakumar
Hi all, Is there a way to consume all the contents of a kafka topic into a KTable before doing a left join with another Kstream? I am looking at something that simulates a bootstrap topic in a Samza job. Thanks, Rohit Valsakumar This email and any attachments