Re: Long delay between incoming and outgoing messages using kafka streams

2019-10-22 Thread Petter Arvidsson
Hi Sophie, Thank you for your elaborate response. On Mon, Oct 14, 2019 at 10:13 PM Sophie Blee-Goldman wrote: > Honestly I can't say whether 256 partitions is enough to trigger the > performance issues > in 2.3.0 but I'd definitely recommend upgrading as soon as you can, just in > case. On a >

Re: Long delay between incoming and outgoing messages using kafka streams

2019-10-14 Thread Sophie Blee-Goldman
Honestly I can't say whether 256 partitions is enough to trigger the performance issues in 2.3.0 but I'd definitely recommend upgrading as soon as you can, just in case. On a related note, how many instances, with how many threads, are you running? 256 partitions with several subtopologies will res

Re: Long delay between incoming and outgoing messages using kafka streams

2019-10-10 Thread Petter Arvidsson
Hi Sophie, Thank you for your response. I tested the proposed setting for CACHE_MAX_BYTES_BUFFERING_CONFIG and it seem to not significantly change the behavior of the application. The latency remains very similar. The documentation states the following regarding CACHE_MAX_BYTES_BUFFERING_CONFIG a

Re: Long delay between incoming and outgoing messages using kafka streams

2019-10-09 Thread Sophie Blee-Goldman
Hi Petter, I'd recommend turning off caching by setting p.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING, 0); 2.3.0 also has some known performance issues that will be fixed in 2.3.1, but they shouldn't be noticeable if you turn caching off and aren't reading/writing to topics with a very high parti