0.10.2.1 is compatible with Kafka 0.10.1.
Eno
> On 16 May 2017, at 20:45, Vincent Bernardi <vinc...@kameleoon.com> wrote:
>
> The LOG files stay small. The SST files are growing but not in size, in
> numbers. Old .sst files seem never written to anymore but are not deleted
> and new ones appear regularly.
> I can certainly try streams 0.10.2.1 if it's compatible with Kafka 0.10.1.
> I have not checked the compatibility matrix yet.
>
> Thanks for the help,
> V.
>
> On Tue, 16 May 2017 at 17:57, Eno Thereska <eno.there...@gmail.com> wrote:
>
>> Thanks. Which RocksDb files are growing indefinitely, the LOG or SST ones?
>> Also, any chance you could use the latest streams library 0.10.2.1 to
>> check if problem still exists?
>>
>>
>> Eno
>>
>>> On 16 May 2017, at 16:43, Vincent Bernardi <vinc...@kameleoon.com>
>> wrote:
>>>
>>> Just tried setting compaction threads to 5, but I have the exact same
>>> problem: the rocksdb files get bigger and bigger, while my application
>>> never stores more than 200k K/V pairs.
>>>
>>> V.
>>>
>>> On Tue, May 16, 2017 at 5:22 PM, Vincent Bernardi <vinc...@kameleoon.com
>>>
>>> wrote:
>>>
>>>> Hi Eno,
>>>> Thanks for your answer. I tried sending a followup email when I
>> realised I
>>>> forgot to tell you the version number but it must have fallen through.
>>>> I'm using 0.10.1.1 both for Kafka and for the streams library.
>>>> Currently my application works on 4 partitions and only uses about 100%
>> of
>>>> one core, so I don't see how it could be CPU starved. Still I will of
>>>> course try your suggestion.
>>>>
>>>> Thanks again,
>>>> V.
>>>>
>>>>
>>>> On Tue, May 16, 2017 at 5:15 PM, Eno Thereska <eno.there...@gmail.com>
>>>> wrote:
>>>>
>>>>> Which version of Kafka are you using? It might be that RocksDb doesn't
>>>>> get enough resources to compact the data fast enough. If that's the
>> case
>>>>> you can try increasing the number of background compaction threads for
>>>>> RocksDb through the RocksDbConfigSetter class (see
>>>>> http://docs.confluent.io/current/streams/developer-guide.
>>>>> html#streams-developer-guide-rocksdb-config <
>>>>> http://docs.confluent.io/current/streams/developer-guide.
>>>>> html#streams-developer-guide-rocksdb-config>) by calling
>>>>> "options.setIncreaseParallelism(/* number of threads for compaction,
>>>>> e.g., 5 */)"
>>>>>
>>>>> Eno
>>>>>
>>>>>> On 16 May 2017, at 14:58, Vincent Bernardi <vinc...@kameleoon.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I'm running an experimental Kafka Stream Processor which accumulates
>>>>> lots
>>>>>> of data in a StateStoreSupplier during transform() and forwards lots
>> of
>>>>>> data during punctuate (and deletes it form the StateStoreSupplier).
>> I'm
>>>>>> currently using a persistent StateStore, meaning that Kafka Streams
>>>>>> provides me with a RocksDB instance which writes everything on disk.
>> The
>>>>>> average amount of data that I keep in my StateStore at any time is at
>>>>> most
>>>>>> 1GB.
>>>>>>
>>>>>> My problem is that it seems that this data is never really deleted, as
>>>>> if
>>>>>> no compaction never happened: the directory size for my RocksDB
>> instance
>>>>>> goes ever up and eventually uses up all disk space at which point my
>>>>>> application crashes (I've seen it go up to 60GB before I stopped it).
>>>>>>
>>>>>> Does anyone know if this can be a normal behaviour for RocksDB? Is
>> there
>>>>>> any way that I can manually log or trigger RocksDB compactions to see
>> if
>>>>>> that is my problem?
>>>>>>
>>>>>> Thanks in advance for any pointer,
>>>>>> V.
>>>>>
>>>>>
>>>>
>>
>>
