Hi John, In out case it wont help, since we are running instance per partition and even with summary only we get 32 warnings per rollout.
Hi Gwen, Thanks for you reply, I understand and share your concern, I also mentioned it earlier in the thread. Do you think it will work if we change DEBUG to INFO? Thanks, Artur On Thu, Feb 6, 2020 at 4:21 AM Gwen Shapira <g...@confluent.io> wrote: > > Sorry for late response. The reason that unused configs is in WARN is that if > you misspell a config, it means that it will not apply. In some cases > (default retention) you want know until too late. We wanted to warn admins > about possible misconfigurations. > > In the context of a company supporting Kafka - customers run logs at INFO > level normally, so if we suspect a misconfiguration, we don't want to ask the > customer to change level to DEBUG and bounce the broker. It is time consuming > and can be risky. > > *Gwen Shapira* > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter ( https://twitter.com/ConfluentInc ) | blog ( > http://www.confluent.io/blog ) > > Sent via Superhuman ( https://sprh.mn/?vip=g...@confluent.io ) > > On Mon, Jan 06, 2020 at 4:21 AM, Stanislav Kozlovski < stanis...@confluent.io > > wrote: > > > > > > > > > Hey Artur, > > > > > > > > Perhaps changing the log level to DEBUG is the simplest approach. > > > > > > > > I wonder if other people know what the motivation behind the WARN log was? > > I'm struggling to think up of a scenario where I'd like to see unused > > values printed in anything above DEBUG. > > > > > > > > Best, > > Stanislav > > > > > > > > On Mon, Dec 30, 2019 at 12:52 PM Artur Burtsev < artjock@ gmail. com ( > > artj...@gmail.com ) > wrote: > > > > > >> > >> > >> Hi, > >> > >> > >> > >> Indeed changing the log level for the whole AbstractConfig is not an > >> option, because logAll is extremely useful. > >> > >> > >> > >> Grouping warnings into 1 (with the count of unused only) will not be a > >> good option for us either. It will still be pretty noisy. Imagine we have > >> 32 partitions and scaled up the application to 32 instances then we still > >> have 32 warnings per application (instead of 96 now) while we would like > >> to have 0 warnings because we are perfectly aware of using > >> schema.registry.url and its totally fine, and we don't have to be warned > >> every time we start the application. Now imagine we use more than one > >> consumer per application, then it will add another multiplication factor > >> to these grouped warnings and we still have a lot of those. So I would say > >> grouping doesn't help much. > >> > >> > >> > >> I think adding extra logger like > >> "org.apache.kafka.clients.producer.ProducerConfig.unused" could be another > >> good option. That would leave the existing interface untouched and give > >> everyone an option to mute irrelevant warnings. > >> > >> > >> > >> To summarize, I still can see 3 options with its pros and cons discussed > >> in the thread: > >> 1) extra config with interface to handle unused > >> 2) change unused warn to debug > >> 3) add extra logger for unused > >> > >> > >> > >> Please let me know what do you think. > >> > >> > >> > >> Thanks, > >> Artur > >> > >> > >> > >> On Mon, Dec 30, 2019 at 11:07 AM Stanislav Kozlovski > >> < stanislav@ confluent. io ( stanis...@confluent.io ) > wrote: > >> > >> > >>> > >>> > >>> Hi all, > >>> > >>> > >>> > >>> Would printing all the unused configurations in one line, versus N lines, > >>> be more helpful? I know that it would greatly reduce the verbosity in log > >>> visualization tools like Kibana while still allowing us to see all the > >>> relevant information without the need for an explicit action (e.g changing > >>> the log level) > >>> > >>> > >>> > >>> Best, > >>> Stanislav > >>> > >>> > >>> > >>> On Sat, Dec 28, 2019 at 3:13 PM John Roesler < vvcephei@ apache. org ( > >>> vvcep...@apache.org ) > > >>> > >>> > >> > >> > >> > >> wrote: > >> > >> > >>> > >>>> > >>>> > >>>> Hi Artur, > >>>> > >>>> > >>>> > >>>> That’s a good point. > >>>> > >>>> > >>>> > >>>> One thing you can do is log a summary at WARN level, like “27 > >>>> configurations were ignored. Ignored configurations are logged at DEBUG > >>>> level.” > >>>> > >>>> > >>>> > >>>> I looked into the code a little, and these log messages are generated > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> in > >> > >> > >>> > >>>> > >>>> > >>>> AbstractConfig (logAll and logUnused). They both use the logger > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> associated > >> > >> > >>> > >>>> > >>>> > >>>> with the relevant config class (StreamsConfig, ProducerConfig, etc.). > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> The > >> > >> > >>> > >>>> > >>>> > >>>> list of all configs is logged at INFO level, and the list of unused > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> configs > >> > >> > >>> > >>>> > >>>> > >>>> is logged at WARN level. This means that it's not possible to silence > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> the > >> > >> > >>> > >>>> > >>>> > >>>> unused config messages while still logging the list of all configs. You > >>>> could only silence both by setting (for example) ProducerConfig logger > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> to > >> > >> > >>> > >>>> > >>>> > >>>> ERROR or OFF. > >>>> > >>>> > >>>> > >>>> If it's desirable to be able to toggle them independently, then you can > >>>> create a separate logger for unused configs, named something like > >>>> "org.apache.kafka.clients.producer.ProducerConfig.unused". Then, you > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> can > >> > >> > >>> > >>>> > >>>> > >>>> leave the log at WARN, so it would continue to be printed by default, > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> and > >> > >> > >>> > >>>> > >>>> > >>>> anyone could disable it by setting > >>>> "org.apache.kafka.clients.producer.ProducerConfig.unused" to ERROR or > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> OFF, > >> > >> > >>> > >>>> > >>>> > >>>> without disturbing the rest of the config log messages. > >>>> > >>>> > >>>> > >>>> It's simpler without the extra logger, but you also get less control. > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> Do > >> > >> > >>> > >>>> > >>>> > >>>> you think the extra control is necessary, versus printing a summary at > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> WARN > >> > >> > >>> > >>>> > >>>> > >>>> level? > >>>> -John > >>>> > >>>> > >>>> > >>>> On Fri, Dec 27, 2019, at 04:26, Artur Burtsev wrote: > >>>> > >>>> > >>>>> > >>>>> > >>>>> Hi, > >>>>> > >>>>> > >>>>> > >>>>> Indeed changing log level to debug would be the easiest and I think that > >>>>> would be a good solution. When no one object I'm ready to move forward > >>>>> with this approach and submit a MR. > >>>>> > >>>>> > >>>>> > >>>>> The only minor thing I have – having it at debug log level might make > >>>>> it a > >>>>> bit less friendly for developers, especially for those who just do the > >>>>> first steps in Kafka. For example, if you misspelled the property name > >>>>> and > >>>>> trying to understand why things don't do what you expect. Having a > >>>>> warning > >>>>> might save some time in this case. Other > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> than > >> > >> > >>> > >>>> > >>>>> > >>>>> > >>>>> that I cannot see any reasons to have warnings there. > >>>>> > >>>>> > >>>>> > >>>>> Thanks, > >>>>> Artur > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Dec 26, 2019 at 10:01 PM John Roesler < vvcephei@ apache. org ( > >>>>> vvcep...@apache.org ) > > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> wrote: > >>>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> Thanks for the KIP, Artur! > >>>>>> > >>>>>> > >>>>>> > >>>>>> For reference, here is the kip: > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ > >> KIP-552%3A+Add+interface+to+handle+unused+config > >> ( > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-552%3A+Add+interface+to+handle+unused+config > >> ) > >> > >> > >>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> I agree, these warnings are kind of a nuisance. Would it be > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> feasible > >> > >> > >>> > >>>> > >>>> > >>>> just to leverage log4j in some way to make it easy to filter these > >>>> messages? For example, we could move those warnings to debug level, or > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> even > >> > >> > >>> > >>>> > >>>> > >>>> use a separate logger for them. > >>>> > >>>> > >>>>> > >>>>>> > >>>>>> > >>>>>> Thanks for starting the discussion. > >>>>>> -John > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, Dec 24, 2019, at 07:23, Artur Burtsev wrote: > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> This KIP provides a way to deal with a warning "The > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> configuration {} > >> > >> > >>> > >>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> was supplied but isn't a known config." when it is not relevant. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Artur > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Best, > >>> Stanislav > >>> > >>> > >> > >> > > > > > > > > -- > > Best, > > Stanislav > > > > > >
