Thanks Peter.
Chris
> On 23 Sep 2022, at 6:05 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
>
> Great to hear, Chris!
>
> I would recommend not using zookeeper addresses in any of your kafka commands
> as that has been deprecated for some time. Switch to using
> `--bootstrap-server`.
>
> —
> Peter
>
>> On Sep 22, 2022, at 1:08 AM, Chris Peart <ch...@peart.me.uk> wrote:
>>
>> Hi Peter,
>>
>>
>>
>> Thanks for getting back to me on this, i have done the following checks as
>> suggested:
>>
>> Kafka version: 2.8.1
>>
>> I can connect to all 4 brokers on port 9092 from my client system
>>
>> I can see this on 3 of the 4 brokers have the following message:
>>
>> DEBUG [Controller id=X] Broker Y has been elected as the controller, so
>> stopping the election process. (kafka.controller.KafkaController)
>>
>> I was just going to send you the command for my topic creation and realised
>> i was not using the new zookeeper path in my ZK connection string, so i was
>> using /kafka instead of /kafkaprod MyBad :-(
>>
>>
>>
>> So all was working after all, once i removed the data/files from /data/kafka
>> and changing the zookeeper.connect string to use /kafkaprod.
>>
>>
>>
>> We now have all six disk working on all 4 brokers.
>>
>>
>>
>> Thanks again for your help on this and sorry for not spotting my mistake
>> sooner :-)
>>
>>
>>
>> Many Thanks
>>
>> Chris
>>
>>
>>
>>> On 2022-09-21 22:24, Peter Bukowinski wrote:
>>>
>>> Hmmm. Let’s start with some low level troubleshooting.
>>>
>>> What kafka version are you using?
>>>
>>> Can reach the ip:port of all broker listener addresses? I like to use `nc
>>> -vz ip port` to validate connectivity from my kafka client.
>>>
>>> Check the controller.log on all brokers. All but one of them should say
>>> something like `DEBUG [Controller id=X] Broker Y has been elected as the
>>> controller, so stopping the election process.
>>> (kafka.controller.KafkaController)`. This will tell you that all brokers
>>> are in agreement that only one of them is the controller. This will confirm
>>> zookeeper is configured and working correctly.
>>>
>>> Can you share the command you’re using to create your topics?
>>>
>>> When you run `kafka-broker-api-versions.sh --bootstrap-server host:port |
>>> grep ‘>’ | sort -n -k3`, do you get a list of all your brokers?
>>>
>>> —
>>> Peter
>>>
>>>> On Sep 21, 2022, at 10:26 AM, Chris Peart <ch...@peart.me.uk
>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>
>>>> Hi Peter,
>>>> Kafka brokers are staying up and show when I query zookeeper brokers/ids
>>>> Is the any other files that require deletion as it doesn’t make sense I
>>>> have 0 brokers, when trying to create a topic.
>>>>
>>>> I get no errors when listing topics, but obviously don’t see any topics as
>>>> I cannot create any.
>>>>
>>>> I have rebooted the Kafka brokers and restarted the all the zookeeper
>>>> services on our production zookeeper cluster.
>>>>
>>>> Not sure what other steps I can take to resolve this, any thoughts would
>>>> be appreciated.
>>>>
>>>> Many Thanks
>>>> Chris
>>>>
>>>>
>>>>> On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com
>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Are you sure kafka is starting up (and remains up) successfully? From the
>>>>> error, it seems like none of the brokers are online.
>>>>>
>>>>> —
>>>>> Peter
>>>>>
>>>>>
>>>>>> On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk
>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Peter,
>>>>>>
>>>>>> I have made the changes i suggested:
>>>>>>
>>>>>> Stopping Kafka
>>>>>>
>>>>>> Deleting all files and folders in /Kafka/data
>>>>>>
>>>>>> Changing the zookeeper setting to point to a different path in the
>>>>>> zookeeper cluster
>>>>>>
>>>>>> Start Kafka
>>>>>>
>>>>>> I see the usual files in /kafka/data/ meta.properties and the offset
>>>>>> files in all 6 disks.
>>>>>>
>>>>>> I see all 4 brokers in zookeeper using the new path i specified in
>>>>>> server.properties.
>>>>>>
>>>>>> When i try to create a topic now i receive: Replication factor: 3 larger
>>>>>> than available brokers: 0
>>>>>>
>>>>>> I see no errors in server.log & controller.log.
>>>>>>
>>>>>> Any advice would be great please as i've exhausted all my options.
>>>>>>
>>>>>> Many Thanks,
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>> On 2022-09-20 21:43, Chris Peart wrote:
>>>>>>>
>>>>>>> Thanks Peter,
>>>>>>> I'll give this a go tomorrow and let you know how I get on.
>>>>>>> Many Thanks,
>>>>>>> Chris
>>>>>>>>
>>>>>>>> On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com
>>>>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>>> Hi Chris,
>>>>>>> If the configs are correct and the permissions on all the
>>>>>>> /data/X/kafka/data directories are correct, then kafka should use all
>>>>>>> of the log dirs when creating topics. Remember that kafka will not
>>>>>>> automatically move any existing topic data when the cluster configs
>>>>>>> change. I'd test by creating a topic with more partitions than storage
>>>>>>> locations.
>>>>>>> If you'd rather start fresh, you have the steps correct. An alternative
>>>>>>> to changing the zk path is to use zkCli to remove the paths. If you use
>>>>>>> a zookeeper chroot, just delete everything from that chroot down from
>>>>>>> zkCli, e.g. `rmr /[kafka-chroot]`
>>>>>>> --
>>>>>>> Peter
>>>>>>>>
>>>>>>>> On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Thinking about this, as this is not in production it might be easier
>>>>>>> just reset everything.
>>>>>>> Would it be something like:
>>>>>>> Stopping Kafka
>>>>>>> Deleting all files and folders in /Kafka/data
>>>>>>> Changing the zookeeper setting to point to a different path in the
>>>>>>> zookeeper cluster
>>>>>>> Start Kafka
>>>>>>> Some help on resetting Kafka would be great if ok please.
>>>>>>> Many Thanks
>>>>>>> Chris
>>>>>>>>
>>>>>>>> On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Hi Peter,
>>>>>>> I have checked the logs on all 4 brokers and could only see
>>>>>>> /data/1/data/kafka being used, log.dirs config in the logs showed all
>>>>>>> the disks but no errors.
>>>>>>> I managed to get the other 5 disks working by adding the path
>>>>>>> /data/[2-6]/data/kafka and setting the owner as kafka and restarting
>>>>>>> kafka.
>>>>>>> So now when i create topics i see disks 2-6 being used but not disk 1.
>>>>>>> I have stopped all the brokers deleted all files /data/1/kafka/data on
>>>>>>> all brokers and started them, but i still don't see disk 1 being used
>>>>>>> when creating topics, it's as if there is some dirty configuration
>>>>>>> somewhere, maybe in zookeeper?
>>>>>>> Any help here would be much appreciated :)
>>>>>>> Many Thanks,
>>>>>>> Chris
>>>>>>>>
>>>>>>>> On 2022-09-16 08:22, Chris Peart wrote:
>>>>>>> Thanks Peter,
>>>>>>> I'll check the logs next week and let you know my findings.
>>>>>>> Many Thanks
>>>>>>> Chris
>>>>>>>>
>>>>>>>> On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com
>>>>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>>> The next thing I'd check is the broker logs. The parameters read from
>>>>>>> the config should appear in the logs when kafka starts up. Search the
>>>>>>> logs for 'log.dirs' and ensure the correct configs are loaded.
>>>>>>> --
>>>>>>> Peter
>>>>>>>>
>>>>>>>> On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Hi Peter,
>>>>>>> Thanks for your response, we have the following configuration:
>>>>>>> Partition count=4
>>>>>>> Replication factor=3
>>>>>>> All four brokers have topics-partitions in /data/1/kafka/data and are
>>>>>>> receiving data.
>>>>>>> Each server has 6 x 2TB disks for kaka data.
>>>>>>> Many Thanks,
>>>>>>> Chris
>>>>>>>>
>>>>>>>> On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com
>>>>>>>> <mailto:pmb...@gmail.com>> wrote:
>>>>>>> Hi Chris,
>>>>>>> Can you share the partition count and replication factor of your
>>>>>>> partitions? Also, do all four brokers contain topic-partition
>>>>>>> directories in /data/1/kafka/data or just a single broker? Depending on
>>>>>>> your topic config, it may be entirely normal that his has happened.
>>>>>>> --
>>>>>>> Peter Bukowinski
>>>>>>>>
>>>>>>>> On Sep 15, 2022, at 3:35 AM, Chris Peart <ch...@peart.me.uk
>>>>>>>> <mailto:ch...@peart.me.uk>> wrote:
>>>>>>> Hi All,
>>>>>>> I have a 4 node kafka cluster running version 2.8.1, we have started
>>>>>>> pushing data to the cluster but can only see one disk being used.
>>>>>>> We had 6 disk configured as non-raid and 1 partition per disk, we have
>>>>>>> the following in fstab:
>>>>>>> /dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail
>>>>>>> 1 2
>>>>>>> /dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1
>>>>>>> 2
>>>>>>> /dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail
>>>>>>> 1 2
>>>>>>> /dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail
>>>>>>> 1 2
>>>>>>> /dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail
>>>>>>> 1 2
>>>>>>> /dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail
>>>>>>> 1 2
>>>>>>> We configured server.properties to be:
>>>>>>> log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data
>>>>>>> i can see all our topics in /data/1/kafka/data but don't see anything
>>>>>>> in /data/2-5
>>>>>>> Any help would be appreciated as this is going to production next week?
>>>>>>> Many Thanks,
>>>>>>> Chris
>
