Thinking about this, as this is not in production it might be easier just reset everything.
Would it be something like: Stopping Kafka Deleting all files and folders in /Kafka/data Changing the zookeeper setting to point to a different path in the zookeeper cluster Start Kafka Some help on resetting Kafka would be great if ok please. Many Thanks Chris > On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk> wrote: > > > Hi Peter, > > I have checked the logs on all 4 brokers and could only see > /data/1/data/kafka being used, log.dirs config in the logs showed all the > disks but no errors. > > I managed to get the other 5 disks working by adding the path > /data/[2-6]/data/kafka and setting the owner as kafka and restarting kafka. > > So now when i create topics i see disks 2-6 being used but not disk 1. > > I have stopped all the brokers deleted all files /data/1/kafka/data on all > brokers and started them, but i still don't see disk 1 being used when > creating topics, it's as if there is some dirty configuration somewhere, > maybe in zookeeper? > > Any help here would be much appreciated :) > > > > Many Thanks, > > Chris > > > > > >> On 2022-09-16 08:22, Chris Peart wrote: >> >> Thanks Peter, >> I’ll check the logs next week and let you know my findings. >> Many Thanks >> Chris >> >>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote: >>> >>> The next thing I’d check is the broker logs. The parameters read from the >>> config should appear in the logs when kafka starts up. Search the logs for >>> ‘log.dirs’ and ensure the correct configs are loaded. >>> >>> -- >>> Peter >>> >>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote: >>>> >>>> Hi Peter, >>>> >>>> Thanks for your response, we have the following configuration: >>>> >>>> Partition count=4 >>>> Replication factor=3 >>>> >>>> All four brokers have topics-partitions in /data/1/kafka/data and are >>>> receiving data. >>>> >>>> Each server has 6 x 2TB disks for kaka data. >>>> >>>> Many Thanks, >>>> Chris >>>> >>>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote: >>>>> >>>>> Hi Chris, >>>>> >>>>> Can you share the partition count and replication factor of your >>>>> partitions? Also, do all four brokers contain topic-partition directories >>>>> in /data/1/kafka/data or just a single broker? Depending on your topic >>>>> config, it may be entirely normal that his has happened. >>>>> >>>>> >>>>> — >>>>> Peter Bukowinski >>>>> >>>>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote: >>>>>> >>>>>> >>>>>> >>>>>> Hi All, >>>>>> >>>>>> I have a 4 node kafka cluster running version 2.8.1, we have started >>>>>> pushing data to the cluster but can only see one disk being used. >>>>>> >>>>>> We had 6 disk configured as non-raid and 1 partition per disk, we have >>>>>> the following in fstab: >>>>>> >>>>>> /dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> >>>>>> /dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> >>>>>> /dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> >>>>>> /dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> >>>>>> /dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> >>>>>> /dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail 1 >>>>>> 2 >>>>>> >>>>>> We configured server.properties to be: >>>>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data >>>>>> >>>>>> i can see all our topics in /data/1/kafka/data but don't see anything in >>>>>> /data/2-5 >>>>>> >>>>>> Any help would be appreciated as this is going to production next week? >>>>>> >>>>>> Many Thanks, >>>>>> >>>>>> Chris
