2) I am not sure if I understand correctly * punctuation is independent from committing (ie, you cannot use it to flush) * if you need to align writes with commits you can either use a KStream/KTable or need to register a state (see StateStore.java)
5) The application goes down -- neither process() nor punctuate() should throw! -Matthias On 05/27/2016 03:52 AM, Tobias Adamson wrote: > Thank you. > Some more follow-up questions > > 1) great, will do some tests > > 2) if auto commit is used how do we prevent a commit happening when an error > happens in processing. Basically our scenario is that we build up aggregation > contexts for specific keys (these are a bit special so most probably can't > use KTables) and we would then on each punctuate call want to save these > contexts to our external systems. Once saved we would commit our offset. > However if the progress is committed before punctuate and we have an error > saving we could end up with the offset being ahead of our saved progress. > > 3) great > > 4) great > > 5) What happens if a process/punctuate call throws a RuntimeException? > > Regards > Toby > > > >> On 27 May 2016, at 1:32 AM, Matthias J. Sax <matth...@confluent.io> wrote: >> >> Hi Toby, >> >> 1) I am not an expert for RocksDB, but I don't see a problem with larger >> objects. >> >> 2) I assume, by "guaranteed" you mean that the commit is performed when >> the call return. In this case, no. It only sets a flag to commit at the >> next earliest point in time possible. Ie, you can trigger commits in >> between the regular commit interval that is configured via >> "commit.interval.ms" >> >> 3) Yes. >> >> 4) Yes. >> >> -Matthias >> >> >> On 05/26/2016 02:36 PM, Tobias Adamson wrote: >>> Hi >>> We have a scenario where we could benefit from the new API’s instead of our >>> in house ones. >>> However we have a couple of questions >>> >>> 1. Is it feasible to save 2-3MB size values in the RocksDBStorage? >>> 2. When is the offset committed as processed when using a custom Processor, >>> is it when you call commit on the context and is the commit guaranteed? >>> 3. Is it ok to put values in to the KVStore in the punctuate method? >>> 4. Is the punctuate method run by the same thread as the process method? >>> >>> >>> Regards >>> Toby >>> >> >
signature.asc
Description: OpenPGP digital signature
