Hello Matthias, thanks for the response!
"Matthias J. Sax" <mj...@apache.org> writes: > Spanning your own thread and calling context.forward() is _not_ safe, and > there > is currently no way for you to make is safe. The runtime code makes certain > assumptions about being single threaded which would break if you call > context.forward() from a different thread. (The runtime _always_ assume that > context.forward() is called inside process() or inside the punctuation > callback.) This is how I expected, so I'm not too concerned here. > The only way forward I can see, would be trying to make the punctuation call > shorter, eg, not scanning the full store but only a small part of it, such > that > the thread can go back to execute process() quicker (it's of course an > additional challenge to keep track where you stopped the scan and to resume > it...), and to make the punctuation interval shorter. This is where I have another question about safety. Is it safe to use a KVStoreIterator that was retrieved during a punctuator after that punctuator call has exited, perhaps using it to store offset information across multiple runs? This seems to me to be the most obvious way to handle this. A related question if that one has an affirmative answer would be asking if the KVStoreIterator can be safely sent to another thread for use (akin to Rust's Send trait, in that I do not wish to have concurrent access, only to use it from another thread), as if that were possible I could use the j.u.c.ConcurrentQueue to allow a thread created on the punctuator to compute the messages that need to be sent and enqueue them, and then in further punctuator runs I could then send messages which have been computed. > Hope this helps. What you've said already is quite helpful and I will begin pursuing methods based on what you've said, but I also hope that someone can answer my further questions since they will be helpful to my implementation as well. Thanks so much, Joshua
