Good point about the docs... I guess it must support concurrency do to
IQ, which might iterator over the store while `process()` modifies it. I
was only reasoning about `process()` and punctuation and forgot IQ.
So it seems we indeed have this contract -- what is good new for you.
However, I don't think that there is any guarantee that you might "see"
concurrent modification (IIRC, RocksDB uses snapshot isolation for
iterators). But maybe that's good enough for you?
-Matthias
On 11/7/22 11:13 AM, Joshua Suskalo wrote:
"Matthias J. Sax" <mj...@apache.org> writes:
In general, it's not safe to keep the iterator open, because when process() is
executed in-between two punctuator calls, it might modify the store and
invalidate the iterator. There is no guarantee that the returned iterator
supports concurrency.
This makes sense but unfortunately adds significant challenge to making these
operations happen concurrently, which is effectively a hard requirement for my
usecase due to the scale of the state stores' contained data.
Hence, even if it happens that the currently used iterator is concurrent, there
is no API contract about it.
This surprises me though, because it seems to contratict what's stated in the
ReadOnlyKeyValueStore documentation[1].
The returned iterator must be safe from ConcurrentModificationExceptions and
must not return null values. Order is not guaranteed as bytes lexicographical
ordering might not represent key order.
Maybe I'm reading this wrong, but it seems to imply that the returned iterator
must be safe in the face of concurrent modifications, which is required if you
are permitted to make a read-modify-write cycle while using the iterator, which
the existing version of my application has been doing correctly so far as I can
tell.
Am I vastly misunderstanding the intended usecase for punctuators and need to
determine a different mechanism for performing periodic operations on a data
store?
[1]:
<https://kafka.apache.org/28/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html#all()>
