Hi, I have a 3 node kafka in production with 1gb in each JVM. Suddenly one day, two of them when down and the other tried to make all the work. The log errors were that they could not sincronize. We applied rebalance to them and it never ends. Unfortunately, we have not GC log, so we had to add it and then realized than each JVM was out of memory.
We changed specifically the line 29 of kafka-server-start.sh to add 4 gb in each one and everything went perfect again. So we think that could be a good idea add a comment in this script explaining that the memory could be neccesary incremented or/and add a variable to set the memory easily. Also, then we realized that in LinkedIn it is used setting 6gb in https://kafka.apache.org/documentation/#java. Perhaps it is something that could be a good idea to take into consideration for production environments and for new users... Regards, -- Fernando Bugni