Hi Nicolas, Please do file a JIRA.
Many thanks, Damian On Thu, 9 Mar 2017 at 15:54 Nicolas Fouché <nfou...@onfocus.io> wrote: > Hi, > > I just wanted to share how we misinterpreted logs from Streams at the INFO > level. Version 0.10.1.1, I think it's the same for 0.10.2.0. > > So, we configured `commit.interval.ms` and `cache.max.bytes.buffering`, > and > we expected to always reach the commit interval before the maximum bytes. > It was confirmed by looking at the logs the commit interval actually > triggered commits. No logs about the maximum bytes. > > But then I noticed in our monitoring graphs and by consuming output topics > that the rate of messages was way too high (thanks to pv[1]). > > I checked the code of Streams to write a kinda "sequence diagram" to see > what was logged and at what level. > > For `commit.interval.ms`: > - StreamThread#maybeCommit does an INFO recalling the configured duration > - then StreamThread#commitOne does one INFO per task. > - then NamedCache#flush announces the flushes at DEBUG level. > > For `cache.max.bytes.buffering`: > - ThreadCache#maybeEvict does a TRACE (?!?) to announce that the cache is > too big > - then NamedCache#flush announces the flushes at DEBUG level. > > So I think it makes the logs not exhaustive enough concerning the writes to > States + Changelog topics + Output topics (Processor forwards). Thus logs > are easily misinterpreted. > > Should I file a JIRA ? > > [1] https://linux.die.net/man/1/pv > > - Nicolas >
