Hello all Can anyone say something about what CPU utilization we can expect for a producer/consumer process that is idle, i.e. not producing or consuming any messages? Should it be like 0%? What is your experience?
We have a small program with a few kafka producers and consumers and we are concerned about the CPU utilization when it is idle. Currently it uses on average 6% CPU when idle and about 12% when under a low load of 4 messages per second. In a scaled down test program with only one producer and one consumer the corresponding figures are 2.6% and 4.8%. We want to have a lot of different small processes each producing and consuming a number of topics so we really want to minimize the CPU utilization of each process, they will not all do heavy work at the same time so occasional high loads for a process is not a problem, but if all processes are using the CPU when not really doing anything useful we might get into problem. Tests are run with one local kafka broker, on a machine with Intel i7 2 Cores @ 3Ghz, 16 GB RAM, in a virtual environment with Ubuntu 14.04.4 LTS. Using java kafka client 0.10.0.0. Our only kafka configuration parameter that is not default is KAFKA_FETCH_MIN_BYTES that is 1 in order to reduce latency. So far we have not run so many processes at the same time, but we fear that if we try to run a couple of hundred of these processes we will get into problem. Would greatly appreciate any input. If not else, please tell me the CPU utilization of your producer/consumer processes when they are not really under load. Just so I can conclude if our program behaves as expected or if we have any configuration or environment issues Thanks Niklas Ström
