Hi Peter,
Kafka brokers are staying up and show when I query zookeeper brokers/ids
Is the any other files that require deletion as it doesn’t make sense I have 0 
brokers, when trying to create a topic. 

I get no errors when listing topics, but obviously don’t see any topics as I 
cannot create any. 

I have rebooted the Kafka brokers and restarted the all the zookeeper services 
on our production zookeeper cluster. 

Not sure what other steps I can take to resolve this, any thoughts would be 
appreciated. 

Many Thanks 
Chris


> On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
> 
> Hi Chris,
> 
> Are you sure kafka is starting up (and remains up) successfully? From the 
> error, it seems like none of the brokers are online.
> 
> —
> Peter
> 
> 
>> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk> wrote:
>> 
>> 
>> 
>> Hi Peter,
>> 
>> I have made the changes i suggested:
>> 
>> Stopping Kafka
>> 
>> Deleting all files and folders in /Kafka/data
>> 
>> Changing the zookeeper setting to point to a different path in the zookeeper 
>> cluster
>> 
>> Start Kafka
>> 
>> I see the usual files in /kafka/data/ meta.properties and the offset files 
>> in all 6 disks.
>> 
>> I see all 4 brokers in zookeeper using the new path i specified in 
>> server.properties.
>> 
>> When i try to create a topic now i receive: Replication factor: 3 larger 
>> than available brokers: 0
>> 
>> I see no errors in server.log & controller.log.
>> 
>> Any advice would be great please as i've exhausted all my options.
>> 
>> Many Thanks,
>> 
>> Chris
>> 
>>> On 2022-09-20 21:43, Chris Peart wrote:
>>> 
>>> Thanks Peter,
>>> I'll give this a go tomorrow and let you know how I get on.
>>> Many Thanks,
>>> Chris
>>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
>>> Hi Chris,
>>> If the configs are correct and the permissions on all the 
>>> /data/X/kafka/data directories are correct, then kafka should use all of 
>>> the log dirs when creating topics. Remember that kafka will not 
>>> automatically move any existing topic data when the cluster configs change. 
>>> I'd test by creating a topic with more partitions than storage locations.
>>> If you'd rather start fresh, you have the steps correct. An alternative to 
>>> changing the zk path is to use zkCli to remove the paths. If you use a 
>>> zookeeper chroot, just delete everything from that chroot down from zkCli, 
>>> e.g. `rmr /[kafka-chroot]`
>>> --
>>> Peter
>>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk> wrote:
>>> Thinking about this, as this is not in production it might be easier just 
>>> reset everything.
>>> Would it be something like:
>>> Stopping Kafka
>>> Deleting all files and folders in /Kafka/data
>>> Changing the zookeeper setting to point to a different path in the 
>>> zookeeper cluster
>>> Start Kafka
>>> Some help on resetting Kafka would be great if ok please.
>>> Many Thanks
>>> Chris
>>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk> wrote:
>>> Hi Peter,
>>> I have checked the logs on all 4 brokers and could only see 
>>> /data/1/data/kafka being used, log.dirs config in the logs showed all the 
>>> disks but no errors.
>>> I managed to get the other 5 disks working by adding the path 
>>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting kafka.
>>> So now when i create topics i see disks 2-6 being used but not disk 1.
>>> I have stopped all the brokers deleted all files /data/1/kafka/data on all 
>>> brokers and started them, but i still don't see disk 1 being used when 
>>> creating topics, it's as if there is some dirty configuration somewhere, 
>>> maybe in zookeeper?
>>> Any help here would be much appreciated :)
>>> Many Thanks,
>>> Chris
>>>> On 2022-09-16 08:22, Chris Peart wrote:
>>> Thanks Peter,
>>> I'll check the logs next week and let you know my findings.
>>> Many Thanks
>>> Chris
>>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote:
>>> The next thing I'd check is the broker logs. The parameters read from the 
>>> config should appear in the logs when kafka starts up. Search the logs for 
>>> 'log.dirs' and ensure the correct configs are loaded.
>>> --
>>> Peter
>>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote:
>>> Hi Peter,
>>> Thanks for your response, we have the following configuration:
>>> Partition count=4
>>> Replication factor=3
>>> All four brokers have topics-partitions in /data/1/kafka/data and are 
>>> receiving data.
>>> Each server has 6 x 2TB disks for kaka data.
>>> Many Thanks,
>>> Chris
>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote:
>>> Hi Chris,
>>> Can you share the partition count and replication factor of your 
>>> partitions? Also, do all four brokers contain topic-partition directories 
>>> in /data/1/kafka/data or just a single broker? Depending on your topic 
>>> config, it may be entirely normal that his has happened.
>>> --
>>> Peter Bukowinski
>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote:
>>> Hi All,
>>> I have a 4 node kafka cluster running version 2.8.1, we have started 
>>> pushing data to the cluster but can only see one disk being used.
>>> We had 6 disk configured as non-raid and 1 partition per disk, we have the 
>>> following in fstab:
>>> /dev/mapper/VolGroup01-data    /data/1    xfs    nodev,noatime,nofail  1  2
>>> /dev/mapper/VolGroup02-data    /data/2   xfs    nodev,noatime,nofail  1  2
>>> /dev/mapper/VolGroup03-data    /data/3    xfs    nodev,noatime,nofail  1  2
>>> /dev/mapper/VolGroup04-data    /data/4    xfs    nodev,noatime,nofail  1  2
>>> /dev/mapper/VolGroup05-data    /data/5    xfs    nodev,noatime,nofail  1  2
>>> /dev/mapper/VolGroup06-data    /data/6    xfs    nodev,noatime,nofail  1  2
>>> We configured server.properties to be: 
>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data
>>> i can see all our topics in /data/1/kafka/data but don't see anything in 
>>> /data/2-5
>>> Any help would be appreciated as this is going to production next week?
>>> Many Thanks,
>>> Chris
> 

Reply via email to