Hi Peter,
Thanks for getting back to me on this, i have done the following checks
as suggested:
Kafka version: 2.8.1
I can connect to all 4 brokers on port 9092 from my client system
I can see this on 3 of the 4 brokers have the following message:
DEBUG [Controller id=X] Broker Y has been elected as the controller, so
stopping the election process. (kafka.controller.KafkaController)
I was just going to send you the command for my topic creation and
realised i was not using the new zookeeper path in my ZK connection
string, so i was using /kafka instead of /kafkaprod MyBad :-(
So all was working after all, once i removed the data/files from
/data/kafka and changing the zookeeper.connect string to use /kafkaprod.
We now have all six disk working on all 4 brokers.
Thanks again for your help on this and sorry for not spotting my mistake
sooner :-)
Many Thanks
Chris
On 2022-09-21 22:24, Peter Bukowinski wrote:
Hmmm. Let's start with some low level troubleshooting.
What kafka version are you using?
Can reach the ip:port of all broker listener addresses? I like to use
`nc -vz ip port` to validate connectivity from my kafka client.
Check the controller.log on all brokers. All but one of them should say
something like `DEBUG [Controller id=X] Broker Y has been elected as
the controller, so stopping the election process.
(kafka.controller.KafkaController)`. This will tell you that all
brokers are in agreement that only one of them is the controller. This
will confirm zookeeper is configured and working correctly.
Can you share the command you're using to create your topics?
When you run `kafka-broker-api-versions.sh --bootstrap-server host:port
| grep '>' | sort -n -k3`, do you get a list of all your brokers?
--
Peter
On Sep 21, 2022, at 10:26 AM, Chris Peart <ch...@peart.me.uk> wrote:
Hi Peter,
Kafka brokers are staying up and show when I query zookeeper
brokers/ids
Is the any other files that require deletion as it doesn't make sense I
have 0 brokers, when trying to create a topic.
I get no errors when listing topics, but obviously don't see any topics
as I cannot create any.
I have rebooted the Kafka brokers and restarted the all the zookeeper
services on our production zookeeper cluster.
Not sure what other steps I can take to resolve this, any thoughts
would be appreciated.
Many Thanks
Chris
On 21 Sep 2022, at 6:03 pm, Peter Bukowinski <pmb...@gmail.com> wrote:
Hi Chris,
Are you sure kafka is starting up (and remains up) successfully? From
the error, it seems like none of the brokers are online.
--
Peter
On Sep 21, 2022, at 4:02 AM, Chris Peart <ch...@peart.me.uk> wrote:
Hi Peter,
I have made the changes i suggested:
Stopping Kafka
Deleting all files and folders in /Kafka/data
Changing the zookeeper setting to point to a different path in the
zookeeper cluster
Start Kafka
I see the usual files in /kafka/data/ meta.properties and the offset
files in all 6 disks.
I see all 4 brokers in zookeeper using the new path i specified in
server.properties.
When i try to create a topic now i receive: Replication factor: 3
larger than available brokers: 0
I see no errors in server.log & controller.log.
Any advice would be great please as i've exhausted all my options.
Many Thanks,
Chris
On 2022-09-20 21:43, Chris Peart wrote:
Thanks Peter,
I'll give this a go tomorrow and let you know how I get on.
Many Thanks,
Chris On 20 Sep 2022, at 9:32 pm, Peter Bukowinski <pmb...@gmail.com>
wrote: Hi Chris,
If the configs are correct and the permissions on all the
/data/X/kafka/data directories are correct, then kafka should use all
of the log dirs when creating topics. Remember that kafka will not
automatically move any existing topic data when the cluster configs
change. I'd test by creating a topic with more partitions than storage
locations.
If you'd rather start fresh, you have the steps correct. An alternative
to changing the zk path is to use zkCli to remove the paths. If you use
a zookeeper chroot, just delete everything from that chroot down from
zkCli, e.g. `rmr /[kafka-chroot]`
--
Peter On Sep 20, 2022, at 11:56 AM, Chris Peart <ch...@peart.me.uk>
wrote: Thinking about this, as this is not in production it might be
easier just reset everything.
Would it be something like:
Stopping Kafka
Deleting all files and folders in /Kafka/data
Changing the zookeeper setting to point to a different path in the
zookeeper cluster
Start Kafka
Some help on resetting Kafka would be great if ok please.
Many Thanks
Chris On 20 Sep 2022, at 3:37 pm, Chris Peart <ch...@peart.me.uk>
wrote: Hi Peter,
I have checked the logs on all 4 brokers and could only see
/data/1/data/kafka being used, log.dirs config in the logs showed all
the disks but no errors.
I managed to get the other 5 disks working by adding the path
/data/[2-6]/data/kafka and setting the owner as kafka and restarting
kafka.
So now when i create topics i see disks 2-6 being used but not disk 1.
I have stopped all the brokers deleted all files /data/1/kafka/data on
all brokers and started them, but i still don't see disk 1 being used
when creating topics, it's as if there is some dirty configuration
somewhere, maybe in zookeeper?
Any help here would be much appreciated :)
Many Thanks,
Chris On 2022-09-16 08:22, Chris Peart wrote: Thanks Peter,
I'll check the logs next week and let you know my findings.
Many Thanks
Chris On 16 Sep 2022, at 7:45 am, Peter Bukowinski <pmb...@gmail.com>
wrote: The next thing I'd check is the broker logs. The parameters read
from the config should appear in the logs when kafka starts up. Search
the logs for 'log.dirs' and ensure the correct configs are loaded.
--
Peter On Sep 15, 2022, at 11:10 PM, Chris Peart <ch...@peart.me.uk>
wrote: Hi Peter,
Thanks for your response, we have the following configuration:
Partition count=4
Replication factor=3
All four brokers have topics-partitions in /data/1/kafka/data and are
receiving data.
Each server has 6 x 2TB disks for kaka data.
Many Thanks,
Chris On 16 Sep 2022, at 1:56 am, Peter Bukowinski <pmb...@gmail.com>
wrote: Hi Chris,
Can you share the partition count and replication factor of your
partitions? Also, do all four brokers contain topic-partition
directories in /data/1/kafka/data or just a single broker? Depending on
your topic config, it may be entirely normal that his has happened.
--
Peter Bukowinski On Sep 15, 2022, at 3:35 AM, Chris Peart
<ch...@peart.me.uk> wrote: Hi All,
I have a 4 node kafka cluster running version 2.8.1, we have started
pushing data to the cluster but can only see one disk being used.
We had 6 disk configured as non-raid and 1 partition per disk, we have
the following in fstab:
/dev/mapper/VolGroup01-data /data/1 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup02-data /data/2 xfs nodev,noatime,nofail 1
2
/dev/mapper/VolGroup03-data /data/3 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup04-data /data/4 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup05-data /data/5 xfs nodev,noatime,nofail
1 2
/dev/mapper/VolGroup06-data /data/6 xfs nodev,noatime,nofail
1 2
We configured server.properties to be:
log.dirs=/data/1/kafka/data,/data/2/kafka/data,/data/3/kafka/data,/data/4/kafka/data,/data/5/kafka/data,/data/6/kafka/data
i can see all our topics in /data/1/kafka/data but don't see anything
in /data/2-5
Any help would be appreciated as this is going to production next week?
Many Thanks,
Chris
