Re: Queryable state client read guarantees

Hi, this helps, thanks! Basically, after each read, I'll check if the key is still supposed to be on the host. Doing the check after the read is necessary to handle the case when a rebalance happens in between the metadata lookup and the store get. When checking after the read, it may happen that

Re: Queryable state client read guarantees

Hi Mikael, Just adding to Damian's comment above, that the IllegalStateStoreException here is thrown to indicate a "transient" state where the state store hosting this key is being migrated and hence not available, where users implementing the REST APIs on top of it, for example, can choose to han

Re: Queryable state client read guarantees

Hi Mikael, During rebalance both instances should throw IllegalStateStoreException until the rebalance has completed. Once the rebalance has completed if the key is not found on the local store, then you would get a null value. You can always find the Kafka Streams instance that will have that key

Re: Queryable state client read guarantees

Hi Damian, thanks for fixing this so quickly, I re-ran the test and it works fine. The next test I tried was to read from two service instances implementing the same string count topology. First, the client is started sending two read requests, one per instance, every second. Next, I start the fi

Re: Queryable state client read guarantees

Hi Mikael, A fix for KAFKA-4123 (the issue you found with receiving null values) has now been committed to trunk. I've tried it with your github repo and it appears to be working. You will have to make a small change to your code as we now throw I

Re: Queryable state client read guarantees

Hi Damian, > > Failed to read key hello, org.mkhq.kafka.Topology$StoreUnavailable > > > Failed to read key hello, org.mkhq.kafka.Topology$KeyNotFound > > > hello -> 10 > > > > > The case where you get KeyNotFound looks like a bug to me. This shouldn't > happen. I can see why it might happen and we

Re: Queryable state client read guarantees

Hi Mikael, > > Failed to read key hello, org.mkhq.kafka.Topology$StoreUnavailable > > Failed to read key hello, org.mkhq.kafka.Topology$KeyNotFound > > hello -> 10 > > The case where you get KeyNotFound looks like a bug to me. This shouldn't happen. I can see why it might happen and we will creat

Re: Queryable state client read guarantees

Hi Eno, thanks for the response and sorry for not getting back earlier. I think it makes sense and the example is great! To make it possible to experiment with the guarantees/semantics, I've created a tool available at: https://github.com/mkhq/kafka-qs-verify. Basically it can be used to trace rea

Re: Queryable state client read guarantees

One more thing, there is an example of an end-to-end REST service that demonstrates one possible way to query at https://github.com/confluentinc/examples/tree/master/kafka-streams/src/main/java/io/confluent/examples/streams/queryablestate

Re: Queryable state client read guarantees

Hi Mikael, Very good question. You are correct about the desired semantics. The semantic of case (a) depends on the local store as you mention. For case (b), the final check is always performed again on get(), and if the store has disappeared between the lookup and get, the user will get an exc

Queryable state client read guarantees

Hi, I've tried to understand the implementation and APIs from KIP-67 and would like to know the possible semantics for read requests from a client perspective. As a developer of a queryable state client, the access semantics I would like to have (I think...) is one where subsequent reads always re