Hi Liam,
I tried set all loggers to DEBUG on the controller
this are the only messages that i can see when i create a topic, couldn't
find the logs you mention but got this:
==> controller.log <==
[2021-11-10 05:06:19,042] INFO [Controller id=103] New topics:
[HashSet(davidballano20)], deleted topics: [HashSet()], new partition
replica assignment [HashMap(davidballano20-3 ->
ReplicaAssignment(replicas=112,111,121, addingReplicas=,
removingReplicas=), davidballano20-1 ->
ReplicaAssignment(replicas=107,101,116, addingReplicas=,
removingReplicas=), davidballano20-2 ->
ReplicaAssignment(replicas=113,116,111, addingReplicas=,
removingReplicas=), davidballano20-4 ->
ReplicaAssignment(replicas=120,121,122, addingReplicas=,
removingReplicas=), davidballano20-0 ->
ReplicaAssignment(replicas=100,106,101, addingReplicas=,
removingReplicas=))] (kafka.controller.KafkaController)
[2021-11-10 05:06:19,042] INFO [Controller id=103] New partition creation
callback for
davidballano20-3,davidballano20-1,davidballano20-2,davidballano20-4,davidballano20-0
(kafka.controller.KafkaController)
...
...
==> state-change.log <==
[2021-11-10 05:06:19,054] INFO [Controller id=103 epoch=11] Sending
LeaderAndIsr request to broker 122 with 0 become-leader and 1
become-follower partitions (state.change.logger)
[2021-11-10 05:06:19,054] INFO [Controller id=103 epoch=11] Sending
UpdateMetadata request to brokers HashSet(100, 101, 102, 103, 104, 105,
106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120,
121, 122, 123) for 5 partitions (state.change.logger)
...
...
thanks!
On Tue, Nov 9, 2021 at 5:04 PM Liam Clarke-Hutchinson <lclar...@redhat.com>
wrote:
> Sorry forgot to mention they'll usually be under $KAFKA_DIR/logs.
>
> On Wed, 10 Nov 2021, 1:53 pm Liam Clarke-Hutchinson, <lclar...@redhat.com>
> wrote:
>
> > Thanks :)
> >
> > If you grep for "broker epochs cache" in the controller.log.* files, are
> > you seeing all of your brokers listed?
> > Should see log messages like "Initialized|Updated broker epochs cache:
> > HashMap(<broker_id> -> epoch, <broker_id_2> -> epoch...)"
> >
> > This is to check if the controller knows that all of your brokers are
> live
> > at the time of topic creation. If their id is in that hashmap, they're
> > alive.
> >
> > Cheers,
> >
> > Liam
> >
> > On Wed, Nov 10, 2021 at 1:21 PM David Ballano Fernandez <
> > dfernan...@demonware.net> wrote:
> >
> >> We are using Kafka with zookeeper
> >>
> >> On Tue, Nov 9, 2021 at 4:12 PM Liam Clarke-Hutchinson <
> >> lclar...@redhat.com>
> >> wrote:
> >>
> >> > Yeah, it's broker side, just wanted to eliminate the obscure edge
> case.
> >> >
> >> > Oh, and are you using Zookeeper or KRaft?
> >> >
> >> > Cheers,
> >> >
> >> > Liam
> >> >
> >> > On Wed, Nov 10, 2021 at 1:00 PM David Ballano Fernandez <
> >> > dfernan...@demonware.net> wrote:
> >> >
> >> > > I don't seem to have that config in any of our clusters. Is that
> >> broker
> >> > > config?
> >> > >
> >> > >
> >> > > On Tue, Nov 9, 2021 at 3:50 PM Liam Clarke-Hutchinson <
> >> > lclar...@redhat.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Thanks David,
> >> > > >
> >> > > > Hmm, is the property create.topic.policy.class.name set in
> >> > > > server.properties at all?
> >> > > >
> >> > > > Cheers,
> >> > > >
> >> > > > Liam
> >> > > >
> >> > > > On Wed, Nov 10, 2021 at 12:21 PM David Ballano Fernandez <
> >> > > > dfernan...@demonware.net> wrote:
> >> > > >
> >> > > > > Hi Liam,
> >> > > > >
> >> > > > > I did a test creating topics with kafka-topics.sh and admin API
> >> from
> >> > > > > confluent kafka python.
> >> > > > > The same happened for both.
> >> > > > >
> >> > > > > thanks!
> >> > > > >
> >> > > > > On Tue, Nov 9, 2021 at 2:58 PM Liam Clarke-Hutchinson <
> >> > > > lclar...@redhat.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi David,
> >> > > > > >
> >> > > > > > What tool(s) are you using to create new topics? Is it the
> >> > > > > kafka-topics.sh
> >> > > > > > that ships with Apache Kafka?
> >> > > > > >
> >> > > > > > Cheers,
> >> > > > > >
> >> > > > > > Liam Clarke-Hutchinson
> >> > > > > >
> >> > > > > > On Wed, Nov 10, 2021 at 11:41 AM David Ballano Fernandez <
> >> > > > > > dfernan...@demonware.net> wrote:
> >> > > > > >
> >> > > > > > > Hi All,
> >> > > > > > > Trying to figure out why my brokers have some disk
> imbalance I
> >> > have
> >> > > > > found
> >> > > > > > > that Kafka (maybe this is the way it is supposed to work?)
> is
> >> not
> >> > > > > > spreading
> >> > > > > > > all replicas to all available brokers.
> >> > > > > > >
> >> > > > > > > I have been trying to figure out how a topic with 5
> partitions
> >> > with
> >> > > > > > > replication_factor=3 (15 replicas) could endup having all
> >> > replicas
> >> > > > > > spread
> >> > > > > > > over 9 brokers instead of 15, especially when there are more
> >> > > brokers
> >> > > > > than
> >> > > > > > > the total replicas for that specific topic.
> >> > > > > > >
> >> > > > > > > cluster has 48 brokers.
> >> > > > > > >
> >> > > > > > > # topics.py describe -topic topic1
> >> > > > > > > {145: 1, 148: 2, *101: 3*, 146: 1, 102: 2, 147: 1, 103: 2,
> >> 104:
> >> > 2,
> >> > > > 105:
> >> > > > > > 1}
> >> > > > > > > the keys are the brokerid and the values is how many
> replicas
> >> > they
> >> > > > > have.
> >> > > > > > >
> >> > > > > > > As you can see brokerid 101 has 3 replicas. which make the
> >> disk
> >> > > > > > unbalanced
> >> > > > > > > compared to other brokers.
> >> > > > > > >
> >> > > > > > > I created a brand new topic in a test cluster with 24
> brokers.
> >> > > topic
> >> > > > > has
> >> > > > > > 5
> >> > > > > > > partitions with replication factor 3
> >> > > > > > > topics.py describe -topic test
> >> > > > > > > {119: 1, 103: 1, 106: 2, 109: 1, 101: 2, 114: 1, 116: 2,
> 118:
> >> 1,
> >> > > 111:
> >> > > > > 2,
> >> > > > > > > 104: 1, 121: 1}
> >> > > > > > >
> >> > > > > > > This time kafka decided to spread the replicas over 11
> brokers
> >> > > > instead
> >> > > > > of
> >> > > > > > > 15.
> >> > > > > > > just for fun i ran a partition reassignment for topic
> test,
> >> > > > spreading
> >> > > > > > all
> >> > > > > > > replicas to all brokers, result:
> >> > > > > > >
> >> > > > > > > # topics.py describe -topic test
> >> > > > > > > {110: 1, 111: 1, 109: 1, 108: 1, 112: 1, 103: 1, 107: 1,
> 105:
> >> 1,
> >> > > 104:
> >> > > > > 1,
> >> > > > > > > 106: 1, 102: 1, 118: 1, 116: 1, 113: 1, 117: 1}
> >> > > > > > >
> >> > > > > > > Now all replicas are spread across 15 brokers.
> >> > > > > > >
> >> > > > > > > Is there something I am missing? Maybe the reason is to keep
> >> > > network
> >> > > > > > > chatter down?. By the way, I don't have any rack awareness
> >> > > > configured.
> >> > > > > > > Thanks!
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
