DEBUG level for some classes could have dramatic performance impact. I tested DEBUG level for *kafka.authorizer.logger *for audit purposes, like a naive approach to have audit logs about actions taken by engineers when working with cluster directly. I don't have exact data right now, since it was around 2 years ago, but I remember we managed to kill the cluster completely by this change and had to rollback since it caused quite a wide incident.
On Sat, Nov 2, 2024 at 12:53 AM Brebner, Paul <paul.breb...@netapp.com.invalid> wrote: > And the flip side, if the cluster is heavily loaded then increasing log > levels is likely to have a detectable impact on performance! Paul > > From: Brebner, Paul <paul.breb...@netapp.com> > Date: Saturday, 2 November 2024 at 9:50 am > To: users@kafka.apache.org <users@kafka.apache.org>, om22sh...@gmail.com < > om22sh...@gmail.com> > Subject: Re: Q: Does Kafka log level affect performance and latency > Hi Om, I asked some of our techops people about this, and their general > advice is that increasing the log level from the default (INFO?) is likely > to increase the I/O (the amount depending on a variety of factors including > the cluster traffic etc) – and my take on this is that assuming there is > sufficient I/O headroom it shouldn’t impact the cluster throughput or > latency. However, in previous performance benchmarking I haven’t tried > changing log levels so I would suggest you do some more detailed > benchmarking with realistic workloads and increased log levels, good luck, > Paul Brebner > > From: Om Shree <om22sh...@gmail.com> > Date: Friday, 1 November 2024 at 2:47 am > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: Q: Does Kafka log level affect performance and latency > [You don't often get email from om22sh...@gmail.com. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > Hello community, > > We use Kafka extensively at our organisation. > Our SLAs are strict and require:- > — throughputs north of 1000TPS and > — 60ms latency per transaction > > We run:- > — 6 brokers and 6 zookeepers > — brokers and zookeepers are hosted on ec2 instances with sufficient iops, > throughput and network bandwidth to meet our requirements > > We were under the impression that using log levels (on brokers) such as > INFO or DEBUG would produce too many server logs and have an adverse impact > on performance. > > Can someone with experience please confirm that log levels won’t have an > adverse effect on latency and throughput of these clusters or vice-versa, > given that we come up with a strategy to clean out these logs on broker > servers after a defined unit of time ? > Does generation of system logs have potential to impact disk or compute > iops and slow us down ? > > Thanks! >
