Thanks for the updates. The latest KIP LGTM.
On Fri, Sep 20, 2024 at 8:38 AM Gantigmaa Selenge wrote:
>
> Hi everyone,
>
> Thanks for reviewing the KIP again! I have updated it to address the latest
> comments from Colin and Jose.
>
> Going back to Jose's earlier comment here:
> > Did you conside
Hi everyone,
Thanks for reviewing the KIP again! I have updated it to address the latest
comments from Colin and Jose.
Going back to Jose's earlier comment here:
> Did you consider making "includeFencedBroker" field value implicitly
based on the DescribeClusterRequest version and the EndpointType
Hi Gantigmaa Selenge,
I think we should have a boolean in the request like "includeFenced" which
defaults to false. Then if we can't include the fenced brokers, we throw an
UnsupportedVersionException and fall back to the old behavior.
There are two main reasons for this:
1. if we don't do this
Thanks Gantigmaa. See comments below. The KIP LGTM after this.
On Thu, Sep 12, 2024 at 4:50 AM Gantigmaa Selenge wrote:
> Will this make it more confusing for combined nodes? Users might expect to
> see both controller and broker under ROLES if it is a combined node. Since
> we would be able to o
Thank you Jose.
> I think that we should always output the RACK column even if all of
the nodes do not contain rack information. This would make it easier
for Kafka operators that develop scripts which use and parse CLI
outputs. What do you think?
That makes sense to me, so I updated the KIP and
Gantigmaa, thanks for the changes, they look good to me in general, I
have some UX questions.
The KIP mentions the following:
> Also if there is no rack information for any of the nodes, the RACK column
> will be omitted from the output.
What will the tool output for nodes that don't have a rack
Thank you Luke for taking a look again.
If there are no further comments, I would like to start the vote for this
KIP tomorrow.
Regards,
Gantigmaa
On Thu, Sep 5, 2024 at 8:30 AM Luke Chen wrote:
> Hi Tina,
>
> Thanks for updating the KIP.
> This change makes sense to me.
>
> Thanks.
> Luke
>
>
Thank you both for reviewing the KIP again.
Addressing Colin's comments:
> The admin client changes don't seem to include a way to find out if a
given broker is fenced, by examining the result. Seems like we need that as
well?
I think admins usually send describeCluster requests to find out abo
Thanks for the KIP Gantigmaa,
> { "name": "IncludeFencedBrokers", "type": "bool", "versions": "2+", "about":
> "Whether to include fenced brokers." }
Did you consider making "includeFencedBroker" field value implicitly
based on the DescribeClusterRequest version and the EndpointType? For
example
> Thanks for the KIP Gantigmaa!
>>>
>>> I agree with the motivation but it is not clear to me that this should be
>>> solved in the KRaft layer. The KRaft leader only keeps track, in-memory,
>>> of
>>> observers that have fetched. It is possible,
solved in the KRaft layer. The KRaft leader only keeps track, in-memory,
>> of
>> observers that have fetched. It is possible, after a kraft leader change,
>> for this state to get lost. If the shutdown brokers never fetch they will
>> not show up in the kraft leader's
o get lost. If the shutdown brokers never fetch they will
> not show up in the kraft leader's observer list.
>
> As Colin has pointed out it is better to solve this in the metadata module
> and the KRaft/Quorum controller. The QuorumController knows of all of the
> registered broke
brokers never fetch they will
not show up in the kraft leader's observer list.
As Colin has pointed out it is better to solve this in the metadata module
and the KRaft/Quorum controller. The QuorumController knows of all of the
registered brokers and their state: fence, unfence and eve
s. The
> broker is one example of a Raft observer, but there may be others. For
> example, the metadata shell is supposed to be able to connect to the
> quorum and read the current metadata. (I think this feature was
> disabled for now because we didn't have enough time to finish
Hi Gantigmaa,
I agree with you that we need a way to list fenced brokers.
However, I don't think this KIP is the right way to do that. This KIP, if I
understand correctly, is about listing all Raft observers. The broker is one
example of a Raft observer, but there may be others. For ex
eful
for finding out how long an observer node has been inactive.
Regards,
Gantigmaa
On Thu, Aug 8, 2024 at 9:31 AM Luke Chen wrote:
> Hi Gantigmaa,
>
> Thanks for the KIP!
> The motivation and change looks good to me.
>
> Some comments:
> 1. typo: When a KRaft broker node
uorum.sh describe --replication"?
I don't think we must have it, but maybe it's useful to let users know the
offset lag for the inactive observers?
Thank you.
Luke
On Thu, Jul 25, 2024 at 9:21 PM Gantigmaa Selenge
wrote:
> Hi everyone,
>
> I would like to start a discus
Hi everyone,
I would like to start a discussion on KIP-1073 that includes inactive
observer nodes in the response for describeQuorum request.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1073%3A+Return+inactive+observer+nodes+in+DescribeQuorum+response
The initial discussion on this
Hi Sanaa,
yes it's enough NOT having the "controller" role to be just an "observer".
As soon as your node has "controller" role configured (in addition to
"broker" or alone) it acts as a "voter" and then being a leader or
"follower"
Hi Paolo,
If I'm understanding correctly, a kafka broker (aside from the controller
quorum) can be the observer? Do we need to set the process role to anything
in this case to indicate that we want a broker to be an observer?
On Sun, Mar 24, 2024 at 3:01 PM Paolo Patierno
wrote:
>
Hi Sanaa,
in KRaft mode there is the role of "observer" which is typically taken by
brokers as part of the data plane. The broker/observer can discover which
node is the leader/active controller and fetches the metadata from it (as
the "followers" in the KRaft quorum), but ca
Hello,
Currently in Zookeeper, there is the concept of an observer/dummy node that
can be used when scaling up a cluster (
https://zookeeper.apache.org/doc/r3.4.13/zookeeperObservers.html).
Is there an equivalent of an observer node in KRaft mode? Asking because we
use the observer node for our
[
https://issues.apache.org/jira/browse/KAFKA-14883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Chen resolved KAFKA-14883.
---
Fix Version/s: 3.5.0
Resolution: Fixed
> Broker state should be "observer" in
Paolo Patierno created KAFKA-14883:
--
Summary: Broker state should be "observer" in KRaft quorum
Key: KAFKA-14883
URL: https://issues.apache.org/jira/browse/KAFKA-14883
Project: Kafka
lay/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response>
> and
> proposed a different way of recording each request and response. Here is
> a
> summary of the change.
>
> Instead of having interfaces as wrapper on AbstractRequest and
> AbstractResponse, I
Hi everyone,
Thanks for all feedback on this KIP. I have had some lengthy offline
discussions with Dong, Joel and other insightful developers. I updated KIP
388
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response>
and
prop
Apache Kafka source
code in order to implement more advanced observer plugin, which would
considerably reduces the extensibility and customizability of observer
plugins:
Here are two use-cases that can be made possible if we can provide the raw
request/response to the observer interface:
- Get the
e implementations of "RequestInfo"
and "ResponseInfo" respectively. Users can figure out what information they
can use in their implementation of the observer whereas how to extract
those information is provided so that no internal interface is provided. In
the future, if im
Hi Lincong,
Thanks for the KIP.
If I understand correctly, the reason for having RequestInfo and ResponseInfo
is that you want to pre-define what information can be exposed to the observer.
Users cannot implement RequestAdapter/ResponseAdapter and they are just
internal helper classes to
t; Hi Wesley,
>
> Thank you very much for your feedback. The concern on memory pressure is
> definitely valid. However it should be the user's job to keep this concern
> in mind and implement the observer in the most reasonable way for their use
> case. In other words, implemen
Hi Wesley,
Thank you very much for your feedback. The concern on memory pressure is
definitely valid. However it should be the user's job to keep this concern
in mind and implement the observer in the most reasonable way for their use
case. In other words, implement it at their own risks.
Lincong,
Thanks for the KIP.
I have a question about the lifecycle of request and response.
With the current (requestAdapter, responseAdapter) implementation,
the observer can potentially extend the lifecycle of request and response
through adapter.
Anyone can implement own observer, and some
and response in their own implementation(s) of the
observer interface and they would not constraint future implementation
changes in neither RequestChannel.Request nor AbstractResponse. There could
be more getters defined in these two interfaces. The implementation of
these two interfaces will be
using a consumer group, there is no direct way to
> > query
> > > > the state of the consumer without looking at raw network connections
> to
> > > > determine the extent of the traffic generated by that particular
> > consumer.
> > > >
> > >
o be a shared service across a potentially very
> > > large surface area of clients, central observation of client activity
> is in
> > > my opinion an essential feature.
> > >
> > > Peter
> > >
> > > On Thu, Nov 8, 2018 at 12:13 PM ra
client activity is in
> > my opinion an essential feature.
> >
> > Peter
> >
> > On Thu, Nov 8, 2018 at 12:13 PM radai wrote:
> >
> > > bump.
> > >
> > > I think the proposed API (Observer) is useful for any sort of
> > > multi-tenant
ce area of clients, central observation of client activity is in
> my opinion an essential feature.
>
> Peter
>
> On Thu, Nov 8, 2018 at 12:13 PM radai wrote:
>
> > bump.
> >
> > I think the proposed API (Observer) is useful for any sort of
> > multi-tenant en
radai wrote:
> bump.
>
> I think the proposed API (Observer) is useful for any sort of
> multi-tenant environment for chargeback and reporting purposes.
>
> if no one wants to comment, can we initiate a vote?
> On Mon, Nov 5, 2018 at 6:31 PM Lincong Li wrote:
> >
>
bump.
I think the proposed API (Observer) is useful for any sort of
multi-tenant environment for chargeback and reporting purposes.
if no one wants to comment, can we initiate a vote?
On Mon, Nov 5, 2018 at 6:31 PM Lincong Li wrote:
>
> Hi everyone. Here
> <https://cwiki.apache.or
Addition to my previous e-mail ...
I think, that there should be Observer pattern. I think, that Consumer
interface is Observer interface with method update (i do not know how this
method is called in Consumer). I have two ideas, which classes could be
concrete observers. First idea is, that
Hi everyone. Here
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response>
is
my KIP. Any feedback is appreciated.
Thanks,
Lincong Li
Lincong Li created KAFKA-7596:
-
Summary: Add observer interface to record request and response
Key: KAFKA-7596
URL: https://issues.apache.org/jira/browse/KAFKA-7596
Project: Kafka
Issue Type
42 matches
Mail list logo
