Thanks Peter. 
Chris

> On 23 Sep 2022, at 6:05 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
> 
> Great to hear, Chris!
> 
> I would recommend not using zookeeper addresses in any of your kafka commands 
> as that has been deprecated for some time. Switch to using 
> `--bootstrap-server`.
> 
> —
> Peter
> 
>> On Sep 22, 2022, at 1:08 AM, Chris Peart <ch...@peart.me.uk> wrote:
>> 
>> Hi Peter,
>> 
>> 
>> 
>> Thanks for getting back to me on this, i have done the following checks as 
>> suggested:
>> 
>> Kafka version: 2.8.1
>> 
>> I can connect to all 4 brokers on port 9092 from my client system
>> 
>> I can see this on 3 of the 4 brokers have the following message:
>> 
>> DEBUG [Controller id=X] Broker Y has been elected as the controller, so 
>> stopping the election process. (kafka.controller.KafkaController)
>> 
>> I was just going to send you the command for my topic creation and realised 
>> i was not using the new zookeeper path in my ZK connection string, so i was 
>> using /kafka instead of /kafkaprod   MyBad :-(
>> 
>> 
>> 
>> So all was working after all, once i removed the data/files from /data/kafka 
>> and changing the zookeeper.connect string to use /kafkaprod.
>> 
>> 
>> 
>> We now have all six disk working on all 4 brokers.
>> 
>> 
>> 
>> Thanks again for your help on this and sorry for not spotting my mistake 
>> sooner :-)
>> 
>> 
>> 
>> Many Thanks
>> 
>> Chris
>> 
>> 
>> 
>>> On 2022-09-21 22:24, Peter Bukowinski wrote:
>>> 
>>> Hmmm. Let’s start with some low level troubleshooting.
>>> 
>>> What kafka version are you using?
>>> 
>>> Can reach the ip:port of all broker listener addresses? I like to use `nc 
>>> -vz ip port` to validate connectivity from my kafka client.
>>> 
>>> Check the controller.log on all brokers. All but one of them should say 
>>> something like `DEBUG [Controller id=X] Broker Y has been elected as the 
>>> controller, so stopping the election process. 
>>> (kafka.controller.KafkaController)`. This will tell you that all brokers 
>>> are in agreement that only one of them is the controller. This will confirm 
>>> zookeeper is configured and working correctly.
>>> 
>>> Can you share the command you’re using to create your topics?
>>> 
>>> When you run `kafka-broker-api-versions.sh --bootstrap-server host:port | 
>>> grep ‘>’ | sort -n -k3`, do you get a list of all your brokers?
>>> 
>>> —
>>> Peter
>>> 
>>>> On Sep 21, 2022, at 10:26 AM, Chris Peart <ch...@peart.me.uk 
>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>> 
>>>> Hi Peter,
>>>> Kafka brokers are staying up and show when I query zookeeper brokers/ids
>>>> Is the any other files that require deletion as it doesn’t make sense I 
>>>> have 0 brokers, when trying to create a topic. 
>>>> 
>>>> I get no errors when listing topics, but obviously don’t see any topics as 
>>>> I cannot create any. 
>>>> 
>>>> I have rebooted the Kafka brokers and restarted the all the zookeeper 
>>>> services on our production zookeeper cluster. 
>>>> 
>>>> Not sure what other steps I can take to resolve this, any thoughts would 
>>>> be appreciated. 
>>>> 
>>>> Many Thanks 
>>>> Chris
>>>> 
>>>> 
>>>>> On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com 
>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>> 
>>>>> Hi Chris,
>>>>> 
>>>>> Are you sure kafka is starting up (and remains up) successfully? From the 
>>>>> error, it seems like none of the brokers are online.
>>>>> 
>>>>> —
>>>>> Peter
>>>>> 
>>>>> 
>>>>>> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk 
>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Hi Peter,
>>>>>> 
>>>>>> I have made the changes i suggested:
>>>>>> 
>>>>>> Stopping Kafka
>>>>>> 
>>>>>> Deleting all files and folders in /Kafka/data
>>>>>> 
>>>>>> Changing the zookeeper setting to point to a different path in the 
>>>>>> zookeeper cluster
>>>>>> 
>>>>>> Start Kafka
>>>>>> 
>>>>>> I see the usual files in /kafka/data/ meta.properties and the offset 
>>>>>> files in all 6 disks.
>>>>>> 
>>>>>> I see all 4 brokers in zookeeper using the new path i specified in 
>>>>>> server.properties.
>>>>>> 
>>>>>> When i try to create a topic now i receive: Replication factor: 3 larger 
>>>>>> than available brokers: 0
>>>>>> 
>>>>>> I see no errors in server.log & controller.log.
>>>>>> 
>>>>>> Any advice would be great please as i've exhausted all my options.
>>>>>> 
>>>>>> Many Thanks,
>>>>>> 
>>>>>> Chris
>>>>>> 
>>>>>>> On 2022-09-20 21:43, Chris Peart wrote:
>>>>>>> 
>>>>>>> Thanks Peter,
>>>>>>> I'll give this a go tomorrow and let you know how I get on.
>>>>>>> Many Thanks,
>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com 
>>>>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>>> Hi Chris,
>>>>>>> If the configs are correct and the permissions on all the 
>>>>>>> /data/X/kafka/data directories are correct, then kafka should use all 
>>>>>>> of the log dirs when creating topics. Remember that kafka will not 
>>>>>>> automatically move any existing topic data when the cluster configs 
>>>>>>> change. I'd test by creating a topic with more partitions than storage 
>>>>>>> locations.
>>>>>>> If you'd rather start fresh, you have the steps correct. An alternative 
>>>>>>> to changing the zk path is to use zkCli to remove the paths. If you use 
>>>>>>> a zookeeper chroot, just delete everything from that chroot down from 
>>>>>>> zkCli, e.g. `rmr /[kafka-chroot]`
>>>>>>> --
>>>>>>> Peter
>>>>>>>> 
>>>>>>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk 
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Thinking about this, as this is not in production it might be easier 
>>>>>>> just reset everything.
>>>>>>> Would it be something like:
>>>>>>> Stopping Kafka
>>>>>>> Deleting all files and folders in /Kafka/data
>>>>>>> Changing the zookeeper setting to point to a different path in the 
>>>>>>> zookeeper cluster
>>>>>>> Start Kafka
>>>>>>> Some help on resetting Kafka would be great if ok please.
>>>>>>> Many Thanks
>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk 
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Hi Peter,
>>>>>>> I have checked the logs on all 4 brokers and could only see 
>>>>>>> /data/1/data/kafka being used, log.dirs config in the logs showed all 
>>>>>>> the disks but no errors.
>>>>>>> I managed to get the other 5 disks working by adding the path 
>>>>>>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting 
>>>>>>> kafka.
>>>>>>> So now when i create topics i see disks 2-6 being used but not disk 1.
>>>>>>> I have stopped all the brokers deleted all files /data/1/kafka/data on 
>>>>>>> all brokers and started them, but i still don't see disk 1 being used 
>>>>>>> when creating topics, it's as if there is some dirty configuration 
>>>>>>> somewhere, maybe in zookeeper?
>>>>>>> Any help here would be much appreciated :)
>>>>>>> Many Thanks,
>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On 2022-09-16 08:22, Chris Peart wrote:
>>>>>>> Thanks Peter,
>>>>>>> I'll check the logs next week and let you know my findings.
>>>>>>> Many Thanks
>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com 
>>>>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>>> The next thing I'd check is the broker logs. The parameters read from 
>>>>>>> the config should appear in the logs when kafka starts up. Search the 
>>>>>>> logs for 'log.dirs' and ensure the correct configs are loaded.
>>>>>>> --
>>>>>>> Peter
>>>>>>>> 
>>>>>>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk 
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Hi Peter,
>>>>>>> Thanks for your response, we have the following configuration:
>>>>>>> Partition count=4
>>>>>>> Replication factor=3
>>>>>>> All four brokers have topics-partitions in /data/1/kafka/data and are 
>>>>>>> receiving data.
>>>>>>> Each server has 6 x 2TB disks for kaka data.
>>>>>>> Many Thanks,
>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com 
>>>>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>>> Hi Chris,
>>>>>>> Can you share the partition count and replication factor of your 
>>>>>>> partitions? Also, do all four brokers contain topic-partition 
>>>>>>> directories in /data/1/kafka/data or just a single broker? Depending on 
>>>>>>> your topic config, it may be entirely normal that his has happened.
>>>>>>> --
>>>>>>> Peter Bukowinski
>>>>>>>> 
>>>>>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk 
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Hi All,
>>>>>>> I have a 4 node kafka cluster running version 2.8.1, we have started 
>>>>>>> pushing data to the cluster but can only see one disk being used.
>>>>>>> We had 6 disk configured as non-raid and 1 partition per disk, we have 
>>>>>>> the following in fstab:
>>>>>>> /dev/mapper/VolGroup01-data    /data/1    xfs    nodev,noatime,nofail  
>>>>>>> 1  2
>>>>>>> /dev/mapper/VolGroup02-data    /data/2   xfs    nodev,noatime,nofail  1 
>>>>>>>  2
>>>>>>> /dev/mapper/VolGroup03-data    /data/3    xfs    nodev,noatime,nofail  
>>>>>>> 1  2
>>>>>>> /dev/mapper/VolGroup04-data    /data/4    xfs    nodev,noatime,nofail  
>>>>>>> 1  2
>>>>>>> /dev/mapper/VolGroup05-data    /data/5    xfs    nodev,noatime,nofail  
>>>>>>> 1  2
>>>>>>> /dev/mapper/VolGroup06-data    /data/6    xfs    nodev,noatime,nofail  
>>>>>>> 1  2
>>>>>>> We configured server.properties to be: 
>>>>>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data
>>>>>>> i can see all our topics in /data/1/kafka/data but don't see anything 
>>>>>>> in /data/2-5
>>>>>>> Any help would be appreciated as this is going to production next week?
>>>>>>> Many Thanks,
>>>>>>> Chris
> 

Reply via email to