Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-08 Thread Joshua Suskalo
On Mon, Nov 7, 2022 at 8:05 PM Matthias J. Sax wrote: > However, I don't think that there is any guarantee that you might "see" > concurrent modification (IIRC, RocksDB uses snapshot isolation for > iterators). But maybe that's good enough for you? Yes, thankfully that is exactly what I'm after,

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-07 Thread Matthias J. Sax
Good point about the docs... I guess it must support concurrency do to IQ, which might iterator over the store while `process()` modifies it. I was only reasoning about `process()` and punctuation and forgot IQ. So it seems we indeed have this contract -- what is good new for you. However, I d

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-07 Thread Joshua Suskalo
"Matthias J. Sax" writes: > In general, it's not safe to keep the iterator open, because when process() is > executed in-between two punctuator calls, it might modify the store and > invalidate the iterator. There is no guarantee that the returned iterator > supports concurrency. This makes sens

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-07 Thread Matthias J. Sax
In general, it's not safe to keep the iterator open, because when process() is executed in-between two punctuator calls, it might modify the store and invalidate the iterator. There is no guarantee that the returned iterator supports concurrency. Hence, even if it happens that the currently us

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-07 Thread Joshua Suskalo
Hello Matthias, thanks for the response! "Matthias J. Sax" writes: > Spanning your own thread and calling context.forward() is _not_ safe, and > there > is currently no way for you to make is safe. The runtime code makes certain > assumptions about being single threaded which would break if yo

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-04 Thread Matthias J. Sax
Your observation is correct. The Processor#process() and punctuation callback are executed on a single thread. It's by design to avoid the issue of concurrency (writing thread safe code is hard and we want to avoid putting this burden onto the user). There is currently no plans to make process(

Thread Safety of Punctuator Functions and Processor Contexts

2022-11-03 Thread Joshua Suskalo
I have data that I am storing in a state store which I would like to periodically run some code over, and the way I have decided to do this is via a punctuator from inside a Transformer, which gets an iterator over the state store, performs actions, and forwards events on. So far, everything works