Re: [DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-25 Thread Jonathan Ellis
I'd like to approach this from a different angle. At a very high level we can expect to see two types of users: 1. Early adopters generally want to understand Pulsar's design, how the pieces fit together, and why 2. Mainstream users just want it to work, preferably with as little attention from

Re: [DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-25 Thread Sijie Guo
I don't think we should change the log level by default. Certain log information has been very valuable for real production issues. They exist there from many years' experience. Instead of changing the log level in the code, a better way is to add a logger in log4j2.xml to change the log level for

Re: [DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-24 Thread Enrico Olivelli
Michael, I saw the same issue. Probably we are adding sometimes too much log during development and we forget about the impact in production. I would create an issue for your cases and address each case one at a time. If we go for a general review of all "info" level logs we could end up in removi

Re: [DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-23 Thread Michael Marshall
Thanks for your replies Chris and Joshua. I agree that it is challenging to find the right balance with log levels and that it can come down to preference. However, I think it's telling that you both have implemented work arounds because the logging is too verbose right now. It's unlikely we'll fi

Re: [DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-19 Thread Joshua Odmark
We have had the same issue Michael. In my experience log levels come down to opinion many times. It also comes down to use cases or implementations. Because of that, our solution was to create a module at our local collector that has the ability to rewrite log levels and in some cases filter the

Re: [DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-19 Thread Chris Bartholomew
Hi Michael, I agree that the current default logging levels are too verbose and would welcome a review of the logs. At some scale, the logging is sure to have a performance impact and puts a lot of strain on any centralized log collection system people are running. When using centralized logging t

[DISCUSS] Log Levels for Broker's ServerCnx Class

2021-02-18 Thread Michael Marshall
Hello Pulsar Community, I'm running a Pulsar cluster with thousands of topics where each topic has active producers and consumers that scale up and down dynamically depending on load. The brokers are producing a ton of logs. Many come from the "org.apache.pulsar.broker.service.ServerCnx" class. A