Hi Guozhang, thanks for the answer. I meant that I was considering the idea to use a surrogate key X= (A,B) where A is the actual key and B is something let me understanding it belongs to the list B, which is the last list collected until the n-th emission. Anyway, 1) is far away from RocksDB optimization layer AFAIK, 2) comes little tricky, third approach sounds really interesting and I think I'll give a shot.
So AFAIU kafka streams does not provide any construct allowing this task, is this right? Thanks, Andrea 2018-07-24 23:54 GMT+02:00 Guozhang Wang <wangg...@gmail.com>: > Hello Andrea, > > I do not fully understand what does `nth-id-before-emission` mean here, but > I can think of a couple of options here: > > 1) Just use a key-value store, with the value encoding the list of events > for that key. Whenever a new event of the key gets in, you retrieve the > current list for that key, update the list, and put it back into the > key-value store. This is programmably most simple, but may not be ideal in > performance since the default persistent store RocksDB is a log-structured > store. > > 2) If your computation is commutative and associative, you can just update > your computed result for a key whenever a new event of that key is being > received. > > 3) It is more complicated: you can use two stores, where the first store is > just a plain persistent buffer of all the events not processed / emitted so > far, and second store is an index from key to the locations of the list of > events for this key. Its efficiency should be better than 1) but also more > complicated to program. > > > > Guozhang > > > > On Tue, Jul 24, 2018 at 3:53 AM, Andrea Spina <andrea.sp...@radicalbit.io> > wrote: > > > Dear community, > > I'd add to my topology a stateful operator - graced with a Store - > demanded > > to save some compuation A. > > > > I'd like to implement it so that it can store, by the same key, a list of > > values by appending A by events come in. Something similar e.g. in Apache > > Flink, this can be achieved by the ListState construct. When the semantic > > of the events change somehow, I'll trigger the emission of what I stored > in > > my list state. > > > > I went through the window store code because I thought the base concept > was > > quite simliar (appending events as they come and computing updates with > the > > related iterator) but I wasn't able to find any inspiration. > > > > By now I'm considering a key value store, with a surrogate key like key = > > (event_key, nth-id-before-emission), which allows me to retrieve the list > > of computations when the trigger is fired. > > > > Are there better approaches by which achieving this task, either are > there > > construct already making this possible? > > > > Thank you everybody. > > > > -- > > *Andrea Spina* > > Software Engineer @ Radicalbit Srl > > Via Borsieri 41, 20159, Milano - IT > > > > > > -- > -- Guozhang > -- *Andrea Spina* Software Engineer @ Radicalbit Srl Via Borsieri 41, 20159, Milano - IT
