Hi Chris, Are you sure kafka is starting up (and remains up) successfully? From the error, it seems like none of the brokers are online.
— Peter > On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk> wrote: > > > > Hi Peter, > > I have made the changes i suggested: > > Stopping Kafka > > Deleting all files and folders in /Kafka/data > > Changing the zookeeper setting to point to a different path in the zookeeper > cluster > > Start Kafka > > I see the usual files in /kafka/data/ meta.properties and the offset files in > all 6 disks. > > I see all 4 brokers in zookeeper using the new path i specified in > server.properties. > > When i try to create a topic now i receive: Replication factor: 3 larger than > available brokers: 0 > > I see no errors in server.log & controller.log. > > Any advice would be great please as i've exhausted all my options. > > Many Thanks, > > Chris > > On 2022-09-20 21:43, Chris Peart wrote: > >> Thanks Peter, >> I'll give this a go tomorrow and let you know how I get on. >> Many Thanks, >> Chris >> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com> wrote: >> Hi Chris, >> If the configs are correct and the permissions on all the /data/X/kafka/data >> directories are correct, then kafka should use all of the log dirs when >> creating topics. Remember that kafka will not automatically move any >> existing topic data when the cluster configs change. I'd test by creating a >> topic with more partitions than storage locations. >> If you'd rather start fresh, you have the steps correct. An alternative to >> changing the zk path is to use zkCli to remove the paths. If you use a >> zookeeper chroot, just delete everything from that chroot down from zkCli, >> e.g. `rmr /[kafka-chroot]` >> -- >> Peter >> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk> wrote: >> Thinking about this, as this is not in production it might be easier just >> reset everything. >> Would it be something like: >> Stopping Kafka >> Deleting all files and folders in /Kafka/data >> Changing the zookeeper setting to point to a different path in the zookeeper >> cluster >> Start Kafka >> Some help on resetting Kafka would be great if ok please. >> Many Thanks >> Chris >> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk> wrote: >> Hi Peter, >> I have checked the logs on all 4 brokers and could only see >> /data/1/data/kafka being used, log.dirs config in the logs showed all the >> disks but no errors. >> I managed to get the other 5 disks working by adding the path >> /data/[2-6]/data/kafka and setting the owner as kafka and restarting kafka. >> So now when i create topics i see disks 2-6 being used but not disk 1. >> I have stopped all the brokers deleted all files /data/1/kafka/data on all >> brokers and started them, but i still don't see disk 1 being used when >> creating topics, it's as if there is some dirty configuration somewhere, >> maybe in zookeeper? >> Any help here would be much appreciated :) >> Many Thanks, >> Chris >> On 2022-09-16 08:22, Chris Peart wrote: >> Thanks Peter, >> I'll check the logs next week and let you know my findings. >> Many Thanks >> Chris >> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote: >> The next thing I'd check is the broker logs. The parameters read from the >> config should appear in the logs when kafka starts up. Search the logs for >> 'log.dirs' and ensure the correct configs are loaded. >> -- >> Peter >> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote: >> Hi Peter, >> Thanks for your response, we have the following configuration: >> Partition count=4 >> Replication factor=3 >> All four brokers have topics-partitions in /data/1/kafka/data and are >> receiving data. >> Each server has 6 x 2TB disks for kaka data. >> Many Thanks, >> Chris >> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote: >> Hi Chris, >> Can you share the partition count and replication factor of your partitions? >> Also, do all four brokers contain topic-partition directories in >> /data/1/kafka/data or just a single broker? Depending on your topic config, >> it may be entirely normal that his has happened. >> -- >> Peter Bukowinski >> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote: >> Hi All, >> I have a 4 node kafka cluster running version 2.8.1, we have started pushing >> data to the cluster but can only see one disk being used. >> We had 6 disk configured as non-raid and 1 partition per disk, we have the >> following in fstab: >> /dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail 1 2 >> /dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1 2 >> /dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail 1 2 >> /dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail 1 2 >> /dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail 1 2 >> /dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail 1 2 >> We configured server.properties to be: >> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data >> i can see all our topics in /data/1/kafka/data but don't see anything in >> /data/2-5 >> Any help would be appreciated as this is going to production next week? >> Many Thanks, >> Chris
