Re: [kafka streams] 'null' values in state stores

2017-08-09 Thread Bart Vercammen
Hi Guy, Indeed, I referenced the wrong source-code, sorry about that ;-) I created KAFKA-5717 for this. Thanks Bart On Tue, Aug 8, 2017 at 8:08 PM, Damian Guy wrote: > The change logger is not used during restoration of the in-memory-store. > Restoration is handled > https://github.com/apach

Re: [kafka streams] 'null' values in state stores

2017-08-08 Thread Damian Guy
The change logger is not used during restoration of the in-memory-store. Restoration is handled https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueStore.java#L79 But, even then it is just putting `null` when it should be delet

Re: [kafka streams] 'null' values in state stores

2017-08-08 Thread Bart Vercammen
That's RocksDB .. I'm using in-memory stores ... here: https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/ChangeLoggingKeyValueBytesStore.java#L56 the 'null' is not checked ... On Tue, Aug 8, 2017 at 12:52 PM, Damian Guy wrote: > Hi, > The

Re: [kafka streams] 'null' values in state stores

2017-08-08 Thread Damian Guy
Hi, The null values are treated as deletes when they are written to the store. You can see here: https://github.com/apache/kafka/blob/0.11.0/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L261 On Tue, 8 Aug 2017 at 11:22 Bart Vercammen wrote: > Hi, > > I noticed

[kafka streams] 'null' values in state stores

2017-08-08 Thread Bart Vercammen
Hi, I noticed the following: When a kafka streams application starts, it will restore its state in its state-stores (from the log-compacted kafka topic). All good so far, but I noticed that the 'deleted' entries are actually read in into the store as 'key' with value:`null` Is this expected beha