You might try coordinating this activity outside of riak if at all possible. If there is a single point of origin of these events (ie, a dedicated master for each partition of writes) then you could maintain reasonable guarantees that you dont need sibling processing on the riak end since data is being aggregated upstream.
Not sure if I'm understanding everything completely, but dynamic creation and teardown of new secondary indexes with every writes sounds like disaster. On Tue, May 15, 2012 at 5:56 AM, Max Ivanov <ivanov.ma...@gmail.com> wrote: > Hi, > > what's the best approach to process batch of events in N seconds after > latest event in a group happen? Events are grouped by key. > > I am thinking about following scheme: > > 1) events are recorded in a way that every write creates new sibling > to avoid read/write multiple cycles per event > 2) with every write new secondary index is created with value = > "sweep_at_$current_time + N" > 3) every second process queries Riak for secondary keys with values <= > "sweep_at_$current_time" > 4) for every item returned it queries all it's siblings: > - if there are siblings, then merge them into 1 record, calculate and > write new secondary index "seep_at_$latest_sibling_time + N". Go to > next substep if newly calculated timeout value is <= current time. > - if there are no siblings, process them and remove key from Riak > > Therefore for every batch of N events on average (given that 99% of > event batches timespans are less than N) there will be: > N+1 writes and 2 secondary index seek and 2 reads > > Is it correct approach for Riak? It could be improved further by > carefully setting secondary index on stage 2 so that merge of all > sibling will be immediately followed by processing of events batch, > but right now I am more intrested wether it fit nicely to Riak. > > Thank you. > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
