Thanks Peter, I think we are using version 2.8.1 will verify this tomorrow. I can remember seeing the election message too in the server logs and will also confirm this in the morning too.
I will answer the rest of the questions tomorrow. Many Thanks Chris > On 21 Sep 2022, at 10:24 pm, Peter Bukowinski <pmb...@gmail.com> wrote: > > Hmmm. Let’s start with some low level troubleshooting. > > What kafka version are you using? > > Can reach the ip:port of all broker listener addresses? I like to use `nc -vz > ip port` to validate connectivity from my kafka client. > > Check the controller.log on all brokers. All but one of them should say > something like `DEBUG [Controller id=X] Broker Y has been elected as the > controller, so stopping the election process. > (kafka.controller.KafkaController)`. This will tell you that all brokers are > in agreement that only one of them is the controller. This will confirm > zookeeper is configured and working correctly. > > Can you share the command you’re using to create your topics? > > When you run `kafka-broker-api-versions.sh --bootstrap-server host:port | > grep ‘>’ | sort -n -k3`, do you get a list of all your brokers? > > — > Peter > >> On Sep 21, 2022, at 10:26 AM, Chris Peart <ch...@peart.me.uk> wrote: >> >> Hi Peter, >> Kafka brokers are staying up and show when I query zookeeper brokers/ids >> Is the any other files that require deletion as it doesn’t make sense I have >> 0 brokers, when trying to create a topic. >> >> I get no errors when listing topics, but obviously don’t see any topics as I >> cannot create any. >> >> I have rebooted the Kafka brokers and restarted the all the zookeeper >> services on our production zookeeper cluster. >> >> Not sure what other steps I can take to resolve this, any thoughts would be >> appreciated. >> >> Many Thanks >> Chris >> >> >>>> On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com> wrote: >>> >>> Hi Chris, >>> >>> Are you sure kafka is starting up (and remains up) successfully? From the >>> error, it seems like none of the brokers are online. >>> >>> — >>> Peter >>> >>> >>>> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk> wrote: >>>> >>>> >>>> >>>> Hi Peter, >>>> >>>> I have made the changes i suggested: >>>> >>>> Stopping Kafka >>>> >>>> Deleting all files and folders in /Kafka/data >>>> >>>> Changing the zookeeper setting to point to a different path in the >>>> zookeeper cluster >>>> >>>> Start Kafka >>>> >>>> I see the usual files in /kafka/data/ meta.properties and the offset files >>>> in all 6 disks. >>>> >>>> I see all 4 brokers in zookeeper using the new path i specified in >>>> server.properties. >>>> >>>> When i try to create a topic now i receive: Replication factor: 3 larger >>>> than available brokers: 0 >>>> >>>> I see no errors in server.log & controller.log. >>>> >>>> Any advice would be great please as i've exhausted all my options. >>>> >>>> Many Thanks, >>>> >>>> Chris >>>> >>>>> On 2022-09-20 21:43, Chris Peart wrote: >>>>> >>>>> Thanks Peter, >>>>> I'll give this a go tomorrow and let you know how I get on. >>>>> Many Thanks, >>>>> Chris >>>>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com> wrote: >>>>> Hi Chris, >>>>> If the configs are correct and the permissions on all the >>>>> /data/X/kafka/data directories are correct, then kafka should use all of >>>>> the log dirs when creating topics. Remember that kafka will not >>>>> automatically move any existing topic data when the cluster configs >>>>> change. I'd test by creating a topic with more partitions than storage >>>>> locations. >>>>> If you'd rather start fresh, you have the steps correct. An alternative >>>>> to changing the zk path is to use zkCli to remove the paths. If you use a >>>>> zookeeper chroot, just delete everything from that chroot down from >>>>> zkCli, e.g. `rmr /[kafka-chroot]` >>>>> -- >>>>> Peter >>>>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk> wrote: >>>>> Thinking about this, as this is not in production it might be easier just >>>>> reset everything. >>>>> Would it be something like: >>>>> Stopping Kafka >>>>> Deleting all files and folders in /Kafka/data >>>>> Changing the zookeeper setting to point to a different path in the >>>>> zookeeper cluster >>>>> Start Kafka >>>>> Some help on resetting Kafka would be great if ok please. >>>>> Many Thanks >>>>> Chris >>>>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk> wrote: >>>>> Hi Peter, >>>>> I have checked the logs on all 4 brokers and could only see >>>>> /data/1/data/kafka being used, log.dirs config in the logs showed all the >>>>> disks but no errors. >>>>> I managed to get the other 5 disks working by adding the path >>>>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting >>>>> kafka. >>>>> So now when i create topics i see disks 2-6 being used but not disk 1. >>>>> I have stopped all the brokers deleted all files /data/1/kafka/data on >>>>> all brokers and started them, but i still don't see disk 1 being used >>>>> when creating topics, it's as if there is some dirty configuration >>>>> somewhere, maybe in zookeeper? >>>>> Any help here would be much appreciated :) >>>>> Many Thanks, >>>>> Chris >>>>>> On 2022-09-16 08:22, Chris Peart wrote: >>>>> Thanks Peter, >>>>> I'll check the logs next week and let you know my findings. >>>>> Many Thanks >>>>> Chris >>>>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote: >>>>> The next thing I'd check is the broker logs. The parameters read from the >>>>> config should appear in the logs when kafka starts up. Search the logs >>>>> for 'log.dirs' and ensure the correct configs are loaded. >>>>> -- >>>>> Peter >>>>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote: >>>>> Hi Peter, >>>>> Thanks for your response, we have the following configuration: >>>>> Partition count=4 >>>>> Replication factor=3 >>>>> All four brokers have topics-partitions in /data/1/kafka/data and are >>>>> receiving data. >>>>> Each server has 6 x 2TB disks for kaka data. >>>>> Many Thanks, >>>>> Chris >>>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote: >>>>> Hi Chris, >>>>> Can you share the partition count and replication factor of your >>>>> partitions? Also, do all four brokers contain topic-partition directories >>>>> in /data/1/kafka/data or just a single broker? Depending on your topic >>>>> config, it may be entirely normal that his has happened. >>>>> -- >>>>> Peter Bukowinski >>>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote: >>>>> Hi All, >>>>> I have a 4 node kafka cluster running version 2.8.1, we have started >>>>> pushing data to the cluster but can only see one disk being used. >>>>> We had 6 disk configured as non-raid and 1 partition per disk, we have >>>>> the following in fstab: >>>>> /dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail 1 >>>>> 2 >>>>> /dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1 2 >>>>> /dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail 1 >>>>> 2 >>>>> /dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail 1 >>>>> 2 >>>>> /dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail 1 >>>>> 2 >>>>> /dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail 1 >>>>> 2 >>>>> We configured server.properties to be: >>>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data >>>>> i can see all our topics in /data/1/kafka/data but don't see anything in >>>>> /data/2-5 >>>>> Any help would be appreciated as this is going to production next week? >>>>> Many Thanks, >>>>> Chris >>> >> >
