Hi Peter, Kafka brokers are staying up and show when I query zookeeper brokers/ids Is the any other files that require deletion as it doesn’t make sense I have 0 brokers, when trying to create a topic.
I get no errors when listing topics, but obviously don’t see any topics as I cannot create any. I have rebooted the Kafka brokers and restarted the all the zookeeper services on our production zookeeper cluster. Not sure what other steps I can take to resolve this, any thoughts would be appreciated. Many Thanks Chris > On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com> wrote: > > Hi Chris, > > Are you sure kafka is starting up (and remains up) successfully? From the > error, it seems like none of the brokers are online. > > — > Peter > > >> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk> wrote: >> >> >> >> Hi Peter, >> >> I have made the changes i suggested: >> >> Stopping Kafka >> >> Deleting all files and folders in /Kafka/data >> >> Changing the zookeeper setting to point to a different path in the zookeeper >> cluster >> >> Start Kafka >> >> I see the usual files in /kafka/data/ meta.properties and the offset files >> in all 6 disks. >> >> I see all 4 brokers in zookeeper using the new path i specified in >> server.properties. >> >> When i try to create a topic now i receive: Replication factor: 3 larger >> than available brokers: 0 >> >> I see no errors in server.log & controller.log. >> >> Any advice would be great please as i've exhausted all my options. >> >> Many Thanks, >> >> Chris >> >>> On 2022-09-20 21:43, Chris Peart wrote: >>> >>> Thanks Peter, >>> I'll give this a go tomorrow and let you know how I get on. >>> Many Thanks, >>> Chris >>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com> wrote: >>> Hi Chris, >>> If the configs are correct and the permissions on all the >>> /data/X/kafka/data directories are correct, then kafka should use all of >>> the log dirs when creating topics. Remember that kafka will not >>> automatically move any existing topic data when the cluster configs change. >>> I'd test by creating a topic with more partitions than storage locations. >>> If you'd rather start fresh, you have the steps correct. An alternative to >>> changing the zk path is to use zkCli to remove the paths. If you use a >>> zookeeper chroot, just delete everything from that chroot down from zkCli, >>> e.g. `rmr /[kafka-chroot]` >>> -- >>> Peter >>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk> wrote: >>> Thinking about this, as this is not in production it might be easier just >>> reset everything. >>> Would it be something like: >>> Stopping Kafka >>> Deleting all files and folders in /Kafka/data >>> Changing the zookeeper setting to point to a different path in the >>> zookeeper cluster >>> Start Kafka >>> Some help on resetting Kafka would be great if ok please. >>> Many Thanks >>> Chris >>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk> wrote: >>> Hi Peter, >>> I have checked the logs on all 4 brokers and could only see >>> /data/1/data/kafka being used, log.dirs config in the logs showed all the >>> disks but no errors. >>> I managed to get the other 5 disks working by adding the path >>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting kafka. >>> So now when i create topics i see disks 2-6 being used but not disk 1. >>> I have stopped all the brokers deleted all files /data/1/kafka/data on all >>> brokers and started them, but i still don't see disk 1 being used when >>> creating topics, it's as if there is some dirty configuration somewhere, >>> maybe in zookeeper? >>> Any help here would be much appreciated :) >>> Many Thanks, >>> Chris >>>> On 2022-09-16 08:22, Chris Peart wrote: >>> Thanks Peter, >>> I'll check the logs next week and let you know my findings. >>> Many Thanks >>> Chris >>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote: >>> The next thing I'd check is the broker logs. The parameters read from the >>> config should appear in the logs when kafka starts up. Search the logs for >>> 'log.dirs' and ensure the correct configs are loaded. >>> -- >>> Peter >>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote: >>> Hi Peter, >>> Thanks for your response, we have the following configuration: >>> Partition count=4 >>> Replication factor=3 >>> All four brokers have topics-partitions in /data/1/kafka/data and are >>> receiving data. >>> Each server has 6 x 2TB disks for kaka data. >>> Many Thanks, >>> Chris >>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote: >>> Hi Chris, >>> Can you share the partition count and replication factor of your >>> partitions? Also, do all four brokers contain topic-partition directories >>> in /data/1/kafka/data or just a single broker? Depending on your topic >>> config, it may be entirely normal that his has happened. >>> -- >>> Peter Bukowinski >>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote: >>> Hi All, >>> I have a 4 node kafka cluster running version 2.8.1, we have started >>> pushing data to the cluster but can only see one disk being used. >>> We had 6 disk configured as non-raid and 1 partition per disk, we have the >>> following in fstab: >>> /dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail 1 2 >>> /dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1 2 >>> /dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail 1 2 >>> /dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail 1 2 >>> /dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail 1 2 >>> /dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail 1 2 >>> We configured server.properties to be: >>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data >>> i can see all our topics in /data/1/kafka/data but don't see anything in >>> /data/2-5 >>> Any help would be appreciated as this is going to production next week? >>> Many Thanks, >>> Chris >
