Thanks Peter,

I think we are using version 2.8.1 will verify this tomorrow. 
I can remember seeing the election message too in the server logs and will also 
confirm this in the morning too. 

I will answer the rest of the questions tomorrow. 

Many Thanks 
Chris 

> On 21 Sep 2022, at 10:24 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
> 
> Hmmm. Let’s start with some low level troubleshooting.
> 
> What kafka version are you using?
> 
> Can reach the ip:port of all broker listener addresses? I like to use `nc -vz 
> ip port` to validate connectivity from my kafka client.
> 
> Check the controller.log on all brokers. All but one of them should say 
> something like `DEBUG [Controller id=X] Broker Y has been elected as the 
> controller, so stopping the election process. 
> (kafka.controller.KafkaController)`. This will tell you that all brokers are 
> in agreement that only one of them is the controller. This will confirm 
> zookeeper is configured and working correctly.
> 
> Can you share the command you’re using to create your topics?
> 
> When you run `kafka-broker-api-versions.sh --bootstrap-server host:port | 
> grep ‘>’ | sort -n -k3`, do you get a list of all your brokers?
> 
> —
> Peter
> 
>> On Sep 21, 2022, at 10:26 AM, Chris Peart <ch...@peart.me.uk> wrote:
>> 
>> Hi Peter,
>> Kafka brokers are staying up and show when I query zookeeper brokers/ids
>> Is the any other files that require deletion as it doesn’t make sense I have 
>> 0 brokers, when trying to create a topic. 
>> 
>> I get no errors when listing topics, but obviously don’t see any topics as I 
>> cannot create any. 
>> 
>> I have rebooted the Kafka brokers and restarted the all the zookeeper 
>> services on our production zookeeper cluster. 
>> 
>> Not sure what other steps I can take to resolve this, any thoughts would be 
>> appreciated. 
>> 
>> Many Thanks 
>> Chris
>> 
>> 
>>>> On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
>>> 
>>> Hi Chris,
>>> 
>>> Are you sure kafka is starting up (and remains up) successfully? From the 
>>> error, it seems like none of the brokers are online.
>>> 
>>> —
>>> Peter
>>> 
>>> 
>>>> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk> wrote:
>>>> 
>>>> 
>>>> 
>>>> Hi Peter,
>>>> 
>>>> I have made the changes i suggested:
>>>> 
>>>> Stopping Kafka
>>>> 
>>>> Deleting all files and folders in /Kafka/data
>>>> 
>>>> Changing the zookeeper setting to point to a different path in the 
>>>> zookeeper cluster
>>>> 
>>>> Start Kafka
>>>> 
>>>> I see the usual files in /kafka/data/ meta.properties and the offset files 
>>>> in all 6 disks.
>>>> 
>>>> I see all 4 brokers in zookeeper using the new path i specified in 
>>>> server.properties.
>>>> 
>>>> When i try to create a topic now i receive: Replication factor: 3 larger 
>>>> than available brokers: 0
>>>> 
>>>> I see no errors in server.log & controller.log.
>>>> 
>>>> Any advice would be great please as i've exhausted all my options.
>>>> 
>>>> Many Thanks,
>>>> 
>>>> Chris
>>>> 
>>>>> On 2022-09-20 21:43, Chris Peart wrote:
>>>>> 
>>>>> Thanks Peter,
>>>>> I'll give this a go tomorrow and let you know how I get on.
>>>>> Many Thanks,
>>>>> Chris
>>>>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
>>>>> Hi Chris,
>>>>> If the configs are correct and the permissions on all the 
>>>>> /data/X/kafka/data directories are correct, then kafka should use all of 
>>>>> the log dirs when creating topics. Remember that kafka will not 
>>>>> automatically move any existing topic data when the cluster configs 
>>>>> change. I'd test by creating a topic with more partitions than storage 
>>>>> locations.
>>>>> If you'd rather start fresh, you have the steps correct. An alternative 
>>>>> to changing the zk path is to use zkCli to remove the paths. If you use a 
>>>>> zookeeper chroot, just delete everything from that chroot down from 
>>>>> zkCli, e.g. `rmr /[kafka-chroot]`
>>>>> --
>>>>> Peter
>>>>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk> wrote:
>>>>> Thinking about this, as this is not in production it might be easier just 
>>>>> reset everything.
>>>>> Would it be something like:
>>>>> Stopping Kafka
>>>>> Deleting all files and folders in /Kafka/data
>>>>> Changing the zookeeper setting to point to a different path in the 
>>>>> zookeeper cluster
>>>>> Start Kafka
>>>>> Some help on resetting Kafka would be great if ok please.
>>>>> Many Thanks
>>>>> Chris
>>>>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk> wrote:
>>>>> Hi Peter,
>>>>> I have checked the logs on all 4 brokers and could only see 
>>>>> /data/1/data/kafka being used, log.dirs config in the logs showed all the 
>>>>> disks but no errors.
>>>>> I managed to get the other 5 disks working by adding the path 
>>>>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting 
>>>>> kafka.
>>>>> So now when i create topics i see disks 2-6 being used but not disk 1.
>>>>> I have stopped all the brokers deleted all files /data/1/kafka/data on 
>>>>> all brokers and started them, but i still don't see disk 1 being used 
>>>>> when creating topics, it's as if there is some dirty configuration 
>>>>> somewhere, maybe in zookeeper?
>>>>> Any help here would be much appreciated :)
>>>>> Many Thanks,
>>>>> Chris
>>>>>> On 2022-09-16 08:22, Chris Peart wrote:
>>>>> Thanks Peter,
>>>>> I'll check the logs next week and let you know my findings.
>>>>> Many Thanks
>>>>> Chris
>>>>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com> wrote:
>>>>> The next thing I'd check is the broker logs. The parameters read from the 
>>>>> config should appear in the logs when kafka starts up. Search the logs 
>>>>> for 'log.dirs' and ensure the correct configs are loaded.
>>>>> --
>>>>> Peter
>>>>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk> wrote:
>>>>> Hi Peter,
>>>>> Thanks for your response, we have the following configuration:
>>>>> Partition count=4
>>>>> Replication factor=3
>>>>> All four brokers have topics-partitions in /data/1/kafka/data and are 
>>>>> receiving data.
>>>>> Each server has 6 x 2TB disks for kaka data.
>>>>> Many Thanks,
>>>>> Chris
>>>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com> wrote:
>>>>> Hi Chris,
>>>>> Can you share the partition count and replication factor of your 
>>>>> partitions? Also, do all four brokers contain topic-partition directories 
>>>>> in /data/1/kafka/data or just a single broker? Depending on your topic 
>>>>> config, it may be entirely normal that his has happened.
>>>>> --
>>>>> Peter Bukowinski
>>>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk> wrote:
>>>>> Hi All,
>>>>> I have a 4 node kafka cluster running version 2.8.1, we have started 
>>>>> pushing data to the cluster but can only see one disk being used.
>>>>> We had 6 disk configured as non-raid and 1 partition per disk, we have 
>>>>> the following in fstab:
>>>>> /dev/mapper/VolGroup01-data    /data/1    xfs    nodev,noatime,nofail  1  
>>>>> 2
>>>>> /dev/mapper/VolGroup02-data    /data/2   xfs    nodev,noatime,nofail  1  2
>>>>> /dev/mapper/VolGroup03-data    /data/3    xfs    nodev,noatime,nofail  1  
>>>>> 2
>>>>> /dev/mapper/VolGroup04-data    /data/4    xfs    nodev,noatime,nofail  1  
>>>>> 2
>>>>> /dev/mapper/VolGroup05-data    /data/5    xfs    nodev,noatime,nofail  1  
>>>>> 2
>>>>> /dev/mapper/VolGroup06-data    /data/6    xfs    nodev,noatime,nofail  1  
>>>>> 2
>>>>> We configured server.properties to be: 
>>>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data
>>>>> i can see all our topics in /data/1/kafka/data but don't see anything in 
>>>>> /data/2-5
>>>>> Any help would be appreciated as this is going to production next week?
>>>>> Many Thanks,
>>>>> Chris
>>> 
>> 
> 

Reply via email to