Sorry forgot to mention they'll usually be under $KAFKA_DIR/logs.
On Wed, 10 Nov 2021, 1:53 pm Liam Clarke-Hutchinson, <lclar...@redhat.com>
wrote:
> Thanks :)
>
> If you grep for "broker epochs cache" in the controller.log.* files, are
> you seeing all of your brokers listed?
> Should see log messages like "Initialized|Updated broker epochs cache:
> HashMap(<broker_id> -> epoch, <broker_id_2> -> epoch...)"
>
> This is to check if the controller knows that all of your brokers are live
> at the time of topic creation. If their id is in that hashmap, they're
> alive.
>
> Cheers,
>
> Liam
>
> On Wed, Nov 10, 2021 at 1:21 PM David Ballano Fernandez <
> dfernan...@demonware.net> wrote:
>
>> We are using Kafka with zookeeper
>>
>> On Tue, Nov 9, 2021 at 4:12 PM Liam Clarke-Hutchinson <
>> lclar...@redhat.com>
>> wrote:
>>
>> > Yeah, it's broker side, just wanted to eliminate the obscure edge case.
>> >
>> > Oh, and are you using Zookeeper or KRaft?
>> >
>> > Cheers,
>> >
>> > Liam
>> >
>> > On Wed, Nov 10, 2021 at 1:00 PM David Ballano Fernandez <
>> > dfernan...@demonware.net> wrote:
>> >
>> > > I don't seem to have that config in any of our clusters. Is that
>> broker
>> > > config?
>> > >
>> > >
>> > > On Tue, Nov 9, 2021 at 3:50 PM Liam Clarke-Hutchinson <
>> > lclar...@redhat.com
>> > > >
>> > > wrote:
>> > >
>> > > > Thanks David,
>> > > >
>> > > > Hmm, is the property create.topic.policy.class.name set in
>> > > > server.properties at all?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Liam
>> > > >
>> > > > On Wed, Nov 10, 2021 at 12:21 PM David Ballano Fernandez <
>> > > > dfernan...@demonware.net> wrote:
>> > > >
>> > > > > Hi Liam,
>> > > > >
>> > > > > I did a test creating topics with kafka-topics.sh and admin API
>> from
>> > > > > confluent kafka python.
>> > > > > The same happened for both.
>> > > > >
>> > > > > thanks!
>> > > > >
>> > > > > On Tue, Nov 9, 2021 at 2:58 PM Liam Clarke-Hutchinson <
>> > > > lclar...@redhat.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi David,
>> > > > > >
>> > > > > > What tool(s) are you using to create new topics? Is it the
>> > > > > kafka-topics.sh
>> > > > > > that ships with Apache Kafka?
>> > > > > >
>> > > > > > Cheers,
>> > > > > >
>> > > > > > Liam Clarke-Hutchinson
>> > > > > >
>> > > > > > On Wed, Nov 10, 2021 at 11:41 AM David Ballano Fernandez <
>> > > > > > dfernan...@demonware.net> wrote:
>> > > > > >
>> > > > > > > Hi All,
>> > > > > > > Trying to figure out why my brokers have some disk imbalance I
>> > have
>> > > > > found
>> > > > > > > that Kafka (maybe this is the way it is supposed to work?) is
>> not
>> > > > > > spreading
>> > > > > > > all replicas to all available brokers.
>> > > > > > >
>> > > > > > > I have been trying to figure out how a topic with 5 partitions
>> > with
>> > > > > > > replication_factor=3 (15 replicas) could endup having all
>> > replicas
>> > > > > > spread
>> > > > > > > over 9 brokers instead of 15, especially when there are more
>> > > brokers
>> > > > > than
>> > > > > > > the total replicas for that specific topic.
>> > > > > > >
>> > > > > > > cluster has 48 brokers.
>> > > > > > >
>> > > > > > > # topics.py describe -topic topic1
>> > > > > > > {145: 1, 148: 2, *101: 3*, 146: 1, 102: 2, 147: 1, 103: 2,
>> 104:
>> > 2,
>> > > > 105:
>> > > > > > 1}
>> > > > > > > the keys are the brokerid and the values is how many replicas
>> > they
>> > > > > have.
>> > > > > > >
>> > > > > > > As you can see brokerid 101 has 3 replicas. which make the
>> disk
>> > > > > > unbalanced
>> > > > > > > compared to other brokers.
>> > > > > > >
>> > > > > > > I created a brand new topic in a test cluster with 24 brokers.
>> > > topic
>> > > > > has
>> > > > > > 5
>> > > > > > > partitions with replication factor 3
>> > > > > > > topics.py describe -topic test
>> > > > > > > {119: 1, 103: 1, 106: 2, 109: 1, 101: 2, 114: 1, 116: 2, 118:
>> 1,
>> > > 111:
>> > > > > 2,
>> > > > > > > 104: 1, 121: 1}
>> > > > > > >
>> > > > > > > This time kafka decided to spread the replicas over 11 brokers
>> > > > instead
>> > > > > of
>> > > > > > > 15.
>> > > > > > > just for fun i ran a partition reassignment for topic test,
>> > > > spreading
>> > > > > > all
>> > > > > > > replicas to all brokers, result:
>> > > > > > >
>> > > > > > > # topics.py describe -topic test
>> > > > > > > {110: 1, 111: 1, 109: 1, 108: 1, 112: 1, 103: 1, 107: 1, 105:
>> 1,
>> > > 104:
>> > > > > 1,
>> > > > > > > 106: 1, 102: 1, 118: 1, 116: 1, 113: 1, 117: 1}
>> > > > > > >
>> > > > > > > Now all replicas are spread across 15 brokers.
>> > > > > > >
>> > > > > > > Is there something I am missing? Maybe the reason is to keep
>> > > network
>> > > > > > > chatter down?. By the way, I don't have any rack awareness
>> > > > configured.
>> > > > > > > Thanks!
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
