Hi, Mickael,
Thanks for the updated KIP. LGTM
Jun
On Thu, Jul 31, 2025 at 7:03 AM Mickael Maison
wrote:
> Hi Jun,
>
> There wasn't a technical reason why this is the case.
>
> BrokerHeartbeatRequest already used a tagged field for OfflineLogDirs.
> Since CordonedLogDirs works the same way as O
Hi Jun,
There wasn't a technical reason why this is the case.
BrokerHeartbeatRequest already used a tagged field for OfflineLogDirs.
Since CordonedLogDirs works the same way as OfflineLogDirs I think it
made sense to use a tagged field too.
On the other hand BrokerRegistrationRequest does not hav
Hi, Mickael,
Thanks for the reply. Just one more comment.
Why is CordonedLogDirs a tagged field in BrokerHeartbeatRequest, but not in
BrokerRegistrationRequest?
Jun
On Fri, Jul 18, 2025 at 9:46 AM Mickael Maison
wrote:
> Hi,
>
> Not sure what happened, I thought I had saved my changes but upo
Hi,
Not sure what happened, I thought I had saved my changes but upon
reopening the page on the wiki, my changes appeared as draft.
I saved again and verified the page has updated now.
Thanks,
Mickael
On Fri, Jul 18, 2025 at 5:52 PM Jun Rao wrote:
>
> Hi, Michael,
>
> Thanks for the reply.
>
>
Hi, Michael,
Thanks for the reply.
It seems that the migration section hasn't changed. Have you updated the
wiki?
Jun
On Fri, Jul 18, 2025 at 6:43 AM Mickael Maison
wrote:
> Hi Jun,
>
> Thanks for taking another look.
>
> JR10: We already have anoother metrics
> kafka.log:type=LogManager,name
Hi Jun,
Thanks for taking another look.
JR10: We already have anoother metrics
kafka.log:type=LogManager,name=LogDirectoryOffline,logDirectory=""
that contains the path in the MBean name.
The path is between quotes which should avoid issues. So I kept the
same naming for consistency.
JR11.1: I u
Hi, Mickael,
Thanks for the updated KIP. A few more comments.
JR10. kafka.log:type=LogManager,name=LogDirectoryCordoned,logDirectory=:
Path contains /, which has special meaning in mbean. Could we use
kafka.log:type=LogManager,name=LogDirectoryCordoned and show the cordoned
paths as the value?
J
Bump
On Mon, Apr 7, 2025 at 4:29 PM Mickael Maison wrote:
>
> Hi David,
>
> DA1: Done
>
> DA2.1: Yes because we only have the directory ids in the metadata log.
> At the moment only brokers have the path -> id mappings.
>
> DA2.2: I proposed this mechanism because of DA2.1. Also this is
> consist
Hi David,
DA1: Done
DA2.1: Yes because we only have the directory ids in the metadata log.
At the moment only brokers have the path -> id mappings.
DA2.2: I proposed this mechanism because of DA2.1. Also this is
consistent with other operations like updating log levels. If people
would prefer to
Hi Jun,
10: Yes that's good idea. I've added that to the KIP.
11: Yes that was my plan. We need to check the metadata version and
reject updates to the configuration is the metadata version is too
low. I clarified that in the KIP.
Did you have anything else in mind (if you can remember what you
t
DA1: I like Jun's suggestion of using a wildcard, this would also help with
the case I mentioned (cordon a whole broker, regardless of how many log
dirs).
DA2.1: Re: log dir names to UUID mapping -- do you mean a new CordonLogDirs
RPC would need to send the UUID instead of the log dir name?
DA2.2
Hi, Mickael,
Thanks for the KIP.
10. A common case is only one log dir per broker. Could we support sth
like --add-config cordoned.log.dirs=* to make it more convenient for this
case?
11. Since we changed the metadata record format, should we gate the new
configuration based on a new metadata ve
Hi Kamal,
Good spot, yes this is a typo. The flexibleVersions stays as "0+". Fixed
Thanks,
Mickael
On Wed, Jul 17, 2024 at 6:14 PM Mickael Maison wrote:
>
> Hi David,
>
> DA1: It's a point I considered, especially being able to cordon a
> whole broker. With the current proposal you just need to
Hi David,
DA1: It's a point I considered, especially being able to cordon a
whole broker. With the current proposal you just need to set
cordoned.log.dirs to the same value as log.dirs. That does not seem
excessively difficult.
DA2: I did consider using a new metadata record like we do for
fence/
Hi Mickael,
In the BrokerHearbeatRequest.json, the flexibleVersions are bumped from
"0+" to "1+". Is it a typo?
On Fri, Jul 12, 2024 at 11:42 PM David Arthur wrote:
> Mickael, thanks for the KIP! I think this could be quite a useful feature.
>
> DA1: Having to know each of the log dirs for a b
Mickael, thanks for the KIP! I think this could be quite a useful feature.
DA1: Having to know each of the log dirs for a broker seems a bit
inconvenient for cases where we want to cordon off a whole broker. I do
think having the ability to cordon off a specific log dir is useful for
JBOD, but I i
Hi Kamal,
Thanks for taking a look at the KIP!
I briefly considered that option initially but I found it not very
practical once you have more than a few cordoned log directories.
I find your example is already not very easy to read, and it only has
2 entries. Also if the configuration is at the
Hi Mickael,
Thanks for the KIP!
This is a useful feature which helps to decommission the nodes by
essentially
creating a new replica exclude broker list.
To cordon a list of brokers, we have to apply the config on each of the
broker nodes
and similarly to see the list of cordoned brokers, we hav
Hi Luke,
4. You're right this scenario can happen. In this case I think the
broker should enforce its new state and not create the replica as all
its log directories are now cordoned. The replica will be offline and
an administrator would need to reassign it to another broker. I expect
most users
Hi Chia-Ping,
Question 1) Yes that's a good summary. I'd also add that managing
cordoned log directories is intended to be done by cluster
administrators who also know about operations in-progress or planned
such as scaling or adding/removing log directories. In practice you
can't expect users to
Hi Mickael,
Thanks for the response.
> 4. Cordoned log directories are persisted to the metadata log via the
RegisterBrokerRecord, BrokerRegistrationChangeRecord records. If a
broker is offline, the controller will use the latest known state of
the broker to determine the broker's cordoned log di
hi Michael
Thanks for sharing. I have read those two discussion threads.
The replicas assignment is a interesting topic to me, so the following
questions are used for my education (and learn something from this KIP)
Question: 1
When adding brokers to a cluster, Kafka currently does not always p
Hi Chia-Ping,
As mentioned in the rejected alternatives section, the community was
against KIP-660. The consensus was to not let users build optimized
policy for specific scenarios. The main points were:
- the difficulty to expose an extensive view of the cluster state
(with KRaft, the cluster met
Hi Luke,
2. isCordoned() is a new method on LogDirDescription. It does not take
any arguments. It just returns true if this log directory the
LogDirDescription represents is cordoned.
3. Sorry that was a typo. This method will only return a log directory
that is not cordoned. Fixed
4. Cordoned l
hi Mickael
That is totally a good idea, but I have a question about the implementation
Do we consider making pluggable ReplicaPlacer (KIP-660) first and then add
another impl of ReplicaPlacer to offer cordon mechanism? Noted that
`ReplicaPlacer` can implement Reconfigurable to get updated at runt
Hi Mickael,
Thanks for the KIP!
This is a long waiting feature for many users!
Questions:
1. I think piggyback the "BrokerHeartbeatRequest" to forward the corden log
dir to controller makes sense to me.
We already did similar things for fence, controller shutdown, failed log
dir...etc.
2. In the
Hi,
Thanks for taking a look.
- Yes you're right, I meant AlterPartitionReassignments. Fixed.
- That's a good idea. I was expecting users to discover cordoned log
directories by describing broker configurations. But being able to
also get this information when describing log directories makes sen
Hi,
Thank you for the KIP.
The motivation sounds make sense to me.
I have a few questions:
- [nits] "AlterPartitions request" in Error handling section is
"AlterPartitionReassignments request" actually, right?
- Don't we need to include cordoned information in DescribeLogDirs response
too? Some
28 matches
Mail list logo
