Re: "End of Batch" event

2017-02-01 Thread Matthias J. Sax
gt;>>> https://www.confluent.io/blog/unifying-stream-processing- >>>>>> and-interactive-queries-in-apache-kafka/ >>>>>> >>>>>> >>>>>> >>>>>> -Matthias >>>>>> >>>>>>

Re: "End of Batch" event

2017-02-01 Thread Gwen Shapira
ce's inventory. I need to delete all items from that source that >>>>>> doesn't exist in the latest CSV file. >>>>>> >>>>>> I was thinking of using "End of Batch" me

Re: "End of Batch" event

2017-02-01 Thread Matthias J. Sax
of doing that? >>>>> >>>>> Thanks, >>>>> Eric >>>>> >>>>> >>>>> >>>>> On Sun, Jan 29, 2017 at 4:37 PM, Matthias J. Sax < >> matth...@confluent.io> >>>>>

Re: "End of Batch" event

2017-01-31 Thread Eric Dain
> wrote: > >>> > >>>> Hi, > >>>> > >>>> currently, a Kafka Streams application is designed to "run forever" > and > >>>> there is no notion of "End of Batch" -- we have plans to add this > >>&g

Re: "End of Batch" event

2017-01-31 Thread Matthias J. Sax
ay/KAFKA/KIP- >>>> 95%3A+Incremental+Batch+Processing+for+Kafka+Streams) >>>> >>>> Thus, right now you need to stop your application manually. You would >>>> need to observe the application's committed offsets (and lag) using >>>

Re: "End of Batch" event

2017-01-31 Thread Eric Dain
o observe the application's committed offsets (and lag) using > >> bin/kafka-consumer-groups.sh (the application ID is user as group ID) to > >> monitor the app's progress to see when it is done. > >> > >> Cf. > >> https://cwiki.apache.org/co

Re: "End of Batch" event

2017-01-31 Thread Matthias J. Sax
29Processing+Scenarios >> >> >> -Matthias >> >> >> On 1/28/17 1:07 PM, Eric Dain wrote: >>> Hi, >>> >>> I'm pretty new to Kafka Streams. I am using Kafka Streams to ingest large >>> csv file. I need to run some clean-up code after all records in the file >>> are processed. Is there a way to send "End of Batch" event that is >>> guaranteed to be processed after all records? If not is there alternative >>> solution? >>> >>> Thanks, >>> Eric >>> >> >> > signature.asc Description: OpenPGP digital signature

Re: "End of Batch" event

2017-01-30 Thread Eric Dain
> Cf. > https://cwiki.apache.org/confluence/display/KAFKA/ > Kafka+Streams+Data+%28Re%29Processing+Scenarios > > > -Matthias > > > On 1/28/17 1:07 PM, Eric Dain wrote: > > Hi, > > > > I'm pretty new to Kafka Streams. I am using Kafka Streams to ingest l

Re: "End of Batch" event

2017-01-29 Thread Matthias J. Sax
ce/display/KAFKA/Kafka+Streams+Data+%28Re%29Processing+Scenarios -Matthias On 1/28/17 1:07 PM, Eric Dain wrote: > Hi, > > I'm pretty new to Kafka Streams. I am using Kafka Streams to ingest large > csv file. I need to run some clean-up code after all records in the file > a

"End of Batch" event

2017-01-28 Thread Eric Dain
Hi, I'm pretty new to Kafka Streams. I am using Kafka Streams to ingest large csv file. I need to run some clean-up code after all records in the file are processed. Is there a way to send "End of Batch" event that is guaranteed to be processed after all records? If not is th