Hello,
I'm requesting permission to the Kafka Wiki, specifically to create a KIP.
Wiki ID is 'bbyrne'.
Thanks,
Brian
Dev team,
Requesting discussion for improvement to the producer when dealing with a
large number of topics.
KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics
JIRA: https://issues.apache.org/jira/browse/KAFKA-8904
Though
I've updated the 'Proposed Changes' to include two new producer
configuration variables: topic.expiry.ms and topic.refresh.ms. Please take
a look.
Thanks,
Brian
On Tue, Sep 17, 2019 at 12:59 PM Brian Byrne wrote:
> Dev team,
>
> Requesting discussion for improvem
occur. I see no real
> > advantages to this approach compared to the async method you’ve proposed,
> > but maybe we could add it to the rejected alternatives section?
> >
> > Thanks,
> >
> > Lucas
> >
> > On Fri, 20 Sep 2019 at 11:46, Brian Byrne wrote
Hello all,
I wrote a KIP about expanding the ConfigCommand's functionality when not
accessing ZooKeeper directly, i.e. when --bootstrap-servers is specified
instead of --zookeeper. This should bring the two into parity.
There's also a bit of bikeshedding about additional flags for shortening
enti
for the operations proposed for later, it
> might be nice to have a different color, like yellow, to indicate that
> we'll need a follow-on KIP to do them.
>
> best,
> Colin
>
>
> On Tue, Oct 22, 2019, at 12:42, Brian Byrne wrote:
> > Hello all,
> >
> > I w
for a --describe on all the broker-loggers
> resource at once. The output is very verbose - I count more than 150 lines.
> Although since I imagine more options to describe internal state wouldn't
> hurt, I don't feel strongly about this.
>
> Thanks,
> Stanislav
>
Hello all,
I'd like to call a vote on KIP-543: Expand ConfigCommand's non-ZK
functionality, linked here: https://cwiki.apache.org/confluence/x/ww-3Bw
Thanks,
Brian
etter, but as of now, you're the
first to voice it.
Thanks,
Brian
On Wed, Oct 30, 2019 at 5:01 AM Viktor Somogyi-Vass
wrote:
> Hi Brian,
>
> Have you thought about having one letter abbreviations like '-b' for
> --brokers etc.?
>
> Thanks,
> Viktor
>
> On Tu
Hello all,
Ping. Any further votes or opinions?
Thanks,
Brian
On Tue, Oct 29, 2019 at 9:39 AM Brian Byrne wrote:
> Hello all,
>
> I'd like to call a vote on KIP-543: Expand ConfigCommand's non-ZK
> functionality, linked here: https://cwiki.apache.org/confluence/x/ww-3Bw
>
> Thanks,
> Brian
>
Hello all,
I'd like to propose a vote for a producer change to improve producer
behavior when dealing with a large number of topics, in part by reducing
the amount of metadata fetching performed.
The full KIP is provided here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Pr
re not considered a public API contract (i.e. how we
> maintain the topic metadata in producer cache is never committed to users),
> I wonder if we still need a KIP for the proposed change any more?
>
>
> Guozhang
>
> On Thu, Nov 7, 2019 at 12:43 PM Brian Byrne wrote:
>
7, 2019 at 7:00 PM Manikumar
> wrote:
>
> > +1 (binding), Thanks for the KIP.
> >
> >
> >
> > On Fri, Nov 8, 2019 at 8:14 AM Gwen Shapira wrote:
> >
> > > +1 (binding)
> > >
> > > Thank you.
> > >
> > > On Thu, No
ng some
> kind of jitter on the client side. There are definitely cases where people
> start up a lot of clients at the same time in parallel and there is a
> thundering herd problem with metadata updates. Adding jitter would spread
> the load across time rather than creating a spike
Hello all,
With 5 binding +1 votes from Colin McCabe, Gwen Shapira, Manikumar Reddy,
and Jason Gustafson, and 2 non-binding +1 votes from Viktor Somogyi-Vass
and Jose Armando Garcia Sancio, the vote passes.
Thanks,
Brian
On Fri, Nov 8, 2019 at 2:23 PM Brian Byrne wrote:
>
> Thanks ev
Excuse me, that should read "4 binding +1 votes".
Brian
On Wed, Nov 13, 2019 at 9:26 AM Brian Byrne wrote:
> Hello all,
>
> With 5 binding +1 votes from Colin McCabe, Gwen Shapira, Manikumar Reddy,
> and Jason Gustafson, and 2 non-binding +1 votes from Viktor Somogyi-V
Hi Nag,
To address (2) first, the --entity-default flag requests the default
configuration that all brokers inherit. An individual broker may override
any of the default config's entries, which is done by specifying the broker
ID to the --entity-name flag.
The reason you're getting blank output f
were overridden . Is there any command if i wanted to query
> only the overridden values and the rest will be default values
>
> On Fri, Jun 12, 2020 at 7:58 PM Brian Byrne wrote:
>
> > Hi Nag,
> >
> > To address (2) first, the --entity-default flag requests the def
My apologies, the command mentioned should include --all:
kafka-configs ... --entity-type brokers --entity-name 1 --describe --all
| grep DYNAMIC
Brian
On Fri, Jun 12, 2020 at 8:45 AM Brian Byrne wrote:
> Hi Nag,
>
> Correct, --all will include both. Removing --all should give you
the protocol. Rather than referencing java.lang.Double, it would be more
> helpful to say something equivalent, like "the float is serialized as a
> 64-bit IEEE 754 value, ordered as big-endian." That way people who are
> developing clients in other languages can more easily unde
mittently.
> > >
> > > Can you clarify this part a bit? It seems like we have a metadata
> > > expiration policy now for topics, and we will have one after this KIP,
> but
> > > they will be somewhat different. But it's not clear to me what the
&g
metadata RPCs may still
need to be improved, but this would be a large improvement on the current
situation.
Thanks,
Brian
> I think the above 2 ways are enough to solve the current problem.
>
> On Tue, Nov 19, 2019 at 3:20 AM Colin McCabe wrote:
>
> > On Mon, Nov 18, 2019, at
look
when possible.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-526
%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics
Thanks,
Brian
On Fri, Oct 4, 2019 at 10:04 AM Brian Byrne wrote:
> Lucas, Guozhang,
>
> Thank you for the comments. Good point on METADATA_
le RPC. Like you say, however,
if a topic transitions to urgent and there's several non-urgent ones, we'll
piggyback the non-urgent updates up to the target size.
Thanks,
Brian
On Wed, Nov 20, 2019 at 7:00 PM deng ziming
> wrote:
>
> > I think it's ok, and you can add an
xpiration, but should only be added if there are sending data pending.
>
>
> Guozhang
>
> On Mon, Dec 9, 2019 at 10:57 AM Brian Byrne wrote:
>
> > Hi Guozhang,
> >
> > Thanks for the feedback!
> >
> > On Sun, Dec 8, 2019 at 6:25 PM Guozhang Wang wro
Hello all,
I'm reviving the discussion for adding a quotas API to the admin client by
submitting a new proposal. There are some notable changes from previous
attempts, namely a way to deduce the effective quota for a client (entity),
a way to query for configured quotas, and the concept of "units"
difference in practice.
>
> On Mon, Dec 9, 2019 at 2:07 PM Brian Byrne wrote:
>
> > Hi Guozhang,
> >
> > I see, we agree on the topic threshold not applying to urgent topics, but
> > differ slightly on what should be considered urgent. I would argue th
is the behavior of `metadata.max.age.ms` after this KIP? Are we adding a
> new, more proactive metadata fetch for topics in |U|?
>
> Thanks,
> Stanislav
>
> On Thu, Dec 19, 2019 at 11:37 PM Brian Byrne wrote:
>
> > Hello everyone,
> >
> > For all interested, plea
r
> > NetworkClient
> > metadataUpdater.handleCompletedMetadataResponse
> >
> > so
> > 1. we should also add a metadataUpdater to KafkaProducer?
> > 2. if the topic really does not exists? the intermediate record queue
> will
> > become
ce for that topic), and
> eviction period is P2, and the metadata.max.age is P3. Then the optimal
> scenario in terms of metadata update frequency seems to be P1 = P2 < P3. So
> I feel making the default value for metadata.evict.ms smaller is fine,
> e.g.
> say 1minute or 30second
that for a round 2, if necessary.
I've updated the KIP to change the default eviction time to 2 minutes.
Thanks,
Brian
On Thu, Jan 2, 2020 at 1:28 PM Guozhang Wang wrote:
> On Thu, Jan 2, 2020 at 12:42 PM Brian Byrne wrote:
>
> > Hi Guozhang,
> >
> > You
counter the above, the batching/piggybacking logic isn't
necessarily about reducing the total number of RPCs, but rather to
distribute the load more evenly over time. For example, if a large number
of topics need to be refreshed at the approximate same time (common for
startups cases tha
ce,
but worth sanitizing.
Brian
On Mon, Jan 6, 2020 at 1:31 PM Colin McCabe wrote:
> On Mon, Jan 6, 2020, at 13:07, Brian Byrne wrote:
> > Hi Colin,
> >
> > Thanks again for the feedback!
> >
> > On Mon, Jan 6, 2020 at 12:07 PM Colin McCabe wrote:
> >
&
Hello all,
Does anyone else have opinions on the issue of RPC frequency? Would it be
better to remove the fetching of non-urgent topics altogether, so that the
refreshes are contained in a larger batch?
Thanks,
Brian
On Mon, Jan 6, 2020 at 2:40 PM Brian Byrne wrote:
>
> So the performa
The votes have been cleared. My apologies for continually interrupting and
making changes to the KIP, but hopefully this is an agreeable minimum
solution to move forward.
Thanks,
Brian
On Mon, Jan 6, 2020 at 5:23 PM Colin McCabe wrote:
> On Mon, Jan 6, 2020, at 14:40, Brian Byrne wrote:
>
are you suggesting an interface where the user might
provide {type=user, name=x} and it would return all entities that match,
with their resulting quota values? Should I scrap pattern matching for now,
since it's a simple extension that can be done at a future time?
Thanks,
Brian
> On Wed
ever Value
is actually defined to have two parts: the entry, and a list of overridden
entries (where an entry is the value, along with the source). Perhaps the
Value is poorly named, or maybe there's a simpler structure to be had?
Thanks,
Brian
> On Tue, Jan 14, 2020, at 13:32, Brian Byrn
urgent ones but it is
> more complicated to reason about the trade-off so a simpler approach is
> fine too.
>
> c. fixing 3) with a new config, which is relatively orthogonal to a) and
> b).
>
>
>
> Guozhang
>
>
>
>
> On Tue, Jan 14, 2020 at 10:39 AM Brian
se, I'm thinking about very extreme cases
> here, assuming that's was what we've seen in 8904). Since these two steps
> are very light-weighted, doing that in a synchronized block would not hurt
> the concurrency too much.
>
>
> Guozhang
>
>
> On Tue, Jan 21, 2
s still not very
> clear to me why we need both options. Basically I'm finding the
> "config-centric" mode not very intuitive.
>
> Thanks,
> Jason
>
>
> On Fri, Jan 17, 2020 at 2:14 PM Brian Byrne wrote:
>
> > Thanks Colin, I've updated the KIP
I propose we do the synchronous mini batching still but obviously
> > it is already there :) I'm +1 on the current proposal scope.
> >
> > Guozhang
> >
> > On Tue, Jan 21, 2020 at 10:16 AM Brian Byrne
> wrote:
> >
> > > Hi Guozhang,
> >
keep these consistent? Are we
> planning to deprecate some of those?
>
>
> Regards,
>
> Rajini
>
>
> On Wed, Jan 22, 2020 at 7:28 PM Brian Byrne wrote:
>
> > Hi Jason,
> >
> > I agree on (1). It was Colin's original suggestion, too, but he had
>
pret the
> value as cluster-wide-bytes-per-second. We could update callbacks to work
> with units, but as you said, it may be better to leave it out initially and
> address later.
>
>
>
> On Thu, Jan 23, 2020 at 6:29 PM Brian Byrne wrote:
>
> > Thanks Rajini,
> >
&
deliberately chose Longs for this API. if so, we should mention why under
> Rejected Alternatives. We actually use request quotas < 1 in integration
> tests to ensure we can throttle easily.
>
>
>
> On Fri, Jan 24, 2020 at 5:28 PM Brian Byrne wrote:
>
> > Thanks again, Raj
return the quota
> setting specifically for /config/users/”user-1”, which is 200 in this case.
>
> Is my understanding correct?
>
> Thanks,
>
> Anna
>
>
> On Fri, Jan 24, 2020 at 11:32 AM Brian Byrne wrote:
>
> > My apologies, Rajini. My hasty edits omitt
Hello all,
I'd like to start a vote on KIP-456: Add quota-specific APIs to the Admin
Client, redux
The KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-546%3A+Add+quota-specific+APIs+to+the+Admin+Client%2C+redux
The discussion thread is here:
https://lists.apache.org/thread.htm
r the KIP, Brian!
> >
> > Regards,
> >
> > Rajini
> >
> > On Thu, Jan 23, 2020 at 7:34 PM Jason Gustafson
> > wrote:
> >
> > > Sounds good. +1 from me.
> > >
> > > On Thu, Jan 23, 2020 at 9:00 AM Brian Byrne
> wrote:
> > &g
wrote:
>
> > +1 (binding)
> >
> > Thanks for the KIP, Brian!
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, Jan 27, 2020 at 10:35 PM Colin McCabe
> wrote:
> >
> > > Thanks, Brian.
> > >
> > > +1 (bi
+1 (non-binding) - thanks!
My only suggestion would be to make the enum-to-int conversion explicit for
the new ConfigType, with a surrounding comment, to ensure that no
accidental reordering and for easier readability should the response
message message be read.
Brian
On Fri, Mar 20, 2020 at 8:1
Brian Byrne created KAFKA-8904:
--
Summary: Reduce metadata lookups when producting to a large number
of topics
Key: KAFKA-8904
URL: https://issues.apache.org/jira/browse/KAFKA-8904
Project: Kafka
Brian Byrne created KAFKA-9082:
--
Summary: Move ConfigCommand to use KafkaAdminClient APIs
Key: KAFKA-9082
URL: https://issues.apache.org/jira/browse/KAFKA-9082
Project: Kafka
Issue Type
Brian Byrne created KAFKA-9139:
--
Summary: Dynamic broker config types aren't being discovered
Key: KAFKA-9139
URL: https://issues.apache.org/jira/browse/KAFKA-9139
Project: Kafka
Issue
Brian Byrne created KAFKA-9164:
--
Summary: Don't evict active topics' metadata from the producer's
cache
Key: KAFKA-9164
URL: https://issues.apache.org/jira/browse/KAFKA-9164
[
https://issues.apache.org/jira/browse/KAFKA-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-10278.
-
Resolution: Won't Fix
Hi Kaushik,
The command is set to only return modified (non-de
[
https://issues.apache.org/jira/browse/KAFKA-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-10158.
-
Resolution: Fixed
> Fix flaky
> kafka.admin.TopicCommandWithAdminClie
[
https://issues.apache.org/jira/browse/KAFKA-9713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-9713.
Resolution: Duplicate
> Remove BufferExhausedExcept
[
https://issues.apache.org/jira/browse/KAFKA-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-9139.
Resolution: Duplicate
> Dynamic broker config types aren't being di
[
https://issues.apache.org/jira/browse/KAFKA-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-9164.
Resolution: Invalid
Marking issue invalid since the producer logic was actually handling this
Brian Byrne created KAFKA-9395:
--
Summary: Improve Kafka scheduler's periodic maybeShrinkIsr()
Key: KAFKA-9395
URL: https://issues.apache.org/jira/browse/KAFKA-9395
Project: Kafka
Issue
[
https://issues.apache.org/jira/browse/KAFKA-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-9395.
Assignee: Rajini Sivaram (was: Brian Byrne)
Resolution: Done
> Improve Kafka schedule
Brian Byrne created KAFKA-9449:
--
Summary: Producer's BufferPool may block the producer from closing.
Key: KAFKA-9449
URL: https://issues.apache.org/jira/browse/KAFKA-9449
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-9082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-9082.
Resolution: Duplicate
The outstanding work to be completed is now identical to KAFKA-7740. Marking
Brian Byrne created KAFKA-9474:
--
Summary: Kafka RPC protocol should support type 'double'
Key: KAFKA-9474
URL: https://issues.apache.org/jira/browse/KAFKA-9474
Project: Kafka
Brian Byrne created KAFKA-9510:
--
Summary: Quotas may resolve to incorrect value if user is empty
Key: KAFKA-9510
URL: https://issues.apache.org/jira/browse/KAFKA-9510
Project: Kafka
Issue Type
[
https://issues.apache.org/jira/browse/KAFKA-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-8623.
Fix Version/s: 2.3.0
Resolution: Fixed
This appears to be due to an issue concerning the
[
https://issues.apache.org/jira/browse/KAFKA-8904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brian Byrne resolved KAFKA-8904.
Fix Version/s: 2.5.0
Reviewer: Rajini Sivaram
Resolution: Fixed
> Reduce metad
Brian Byrne created KAFKA-9713:
--
Summary: Remove BufferExhausedException
Key: KAFKA-9713
URL: https://issues.apache.org/jira/browse/KAFKA-9713
Project: Kafka
Issue Type: Task
67 matches
Mail list logo
