Supporting 1 Kafka record -> many rows in sink connectors would be really
useful to properly support some data formats without incurring additional
overhead.
As an example, when consuming OpenTelmetry OTLP protobuf data, each record
represents multiple data points, which significantly reduces both
>
> Are there cases where the metrics plugin developers would want to forward
> the compressed payload without decompressing?
The only interoperable use-case I can think of would be to forward the
payloads directly to an OpenTelemetry collector backend.
Today OTLP only mandates gzip/none compress
>
> 28. On the broker, we typically use Yammer metrics. Only for metrics that
> depend on Kafka metric features (e.g., quota), we use the Kafka metric.
> Yammer metrics have 4 types: gauge, meter, histogram and timer. meter
> calculates a rate, but also exposes an accumulated value.
>
I don't see
24.2 Does delta only apply to Counter type?
> 24.3 In the delta representation, the first request needs to send the full
> value, how does the broker plugin know whether a value is full or delta?
>
The temporarily semantics are defined by the OpenTelemetry data model.
Deferring to OpenTelemetry
>
> 1. Did you consider using a `default ClientTelemetryReceiver
> clientReceiver() { return null; }` method on the existing MetricsReporter
> interface, avoiding the need for the ClientTelemetry trait?
I did. Part of the motivation was to separate more clearly the
MetricsReporter methods which a
I had one quick question in the discussion thread. Any chance we can
provide some thought there?
On Thu, Jun 10, 2021 at 9:46 AM Ryan Dielhenn
wrote:
> Hello,
>
> I would like to start a vote on KIP-748: Add Broker Count Metrics.
>
> Here is the KIP:
>
> https://cwiki.apache.org/confluence/displ
Any reason we need two different metrics for ZK an Quorum based controllers?
Wouldn't it make sense to have one metric that abstracts the implementation
detail?
On Mon, Jun 7, 2021 at 2:29 PM Ryan Dielhenn
wrote:
> Hey Colin and David,
>
> I added another table for the ZK version of RegisteredBr
gt; >>> +1 (non-binding)
> > > > >>>
> > > > >>> On Fri, Jun 26, 2020 at 3:41 AM Jay Kreps
> > > wrote:
> > > > >>>>
> > > > >>>> +1
> > > > >>&
Hi Everyone,
I would like to initiate the voting process for KIP-629.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-629%3A+Use+racially+neutral+terms+in+our+codebase
Thank you,
Xavier
> Probably obvious but is documentation and website considered as well as
> part of the KIP?
>
Documentation and website changes don't require a KIP to my knowledge,
however we should also update them as needed (beyond the obvious
documentation updates for the configuration names).
Please check the list for updated config / argument names.
I also added a proposal to replace the term "blackout" with "backoff",
which is used internally in the replication protocol.
On Mon, Jun 22, 2020 at 3:10 PM Xavier Léauté wrote:
> I agree we could improve on s
n Sat, Jun 20, 2020, at 09:31, Ron Dagostino wrote:
> > > Yes. Thank you.
> > >
> > > > On Jun 20, 2020, at 12:20 AM, Gwen Shapira
> wrote:
> > > >
> > > > Thank you so much for this initiative. Small change, but it makes our
> > > &
Hi Everyone,
There are a number of places in our codebase that use racially charged
terms. I am proposing we update them to use more neutral terms.
The KIP lists the ones I have found and proposes alternatives. If you see
any I missed or did not consider, please reply and I'll add them.
https://
On Sat, May 9, 2020 at 7:21 AM Randall Hauch wrote:
> Thanks, Xavier. Looks great.
>
> On Fri, May 8, 2020 at 7:31 PM Xavier Léauté wrote:
>
> > > This does seem very useful. A minor request would be to mention the new
> > > configs for Connect, Streams and clie
gt; > On Wed, May 13, 2020 at 4:10 PM Gwen Shapira
> wrote:
> > >
> > > > +1 (binding)
> > > > Thanks for the proposal, Xavier.
> > > >
> > > > On Wed, May 13, 2020 at 11:54 AM Xavier Léauté
> > wrote:
> > > >
>
Hi everyone,
Folks seem happy with the state of the KIP, so I'd like to start the vote
for KIP-606
https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter
- Xavier
> This does seem very useful. A minor request would be to mention the new
> configs for Connect, Streams and clients, specifically that because they
> are optional they will not hinder upgrades, and because they are namespaced
> are unlikely to clash or conflict with other configs from extensions.
Hi Everyone,
I've published a KIP to address some shortcoming of our current metrics
reporter interface. Would appreciate feedback.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter
Thank you,
Xavier
ps://github.com/prometheus/jmx_exporter
> (*) On a 4-cores Xeon 8175M broker, hosting 1,000 replicas, the time to
> fetch all beans dropped from 13 seconds to ~400 ms.
>
> Le ven. 8 nov. 2019 à 17:29, Guozhang Wang a écrit :
>
> > Sounds good, thanks.
> >
> > Gu
Thanks everyone,
KIP-544 has been adopted with 4 binding, and 2 non-binding votes.
A PR for the corresponding changes has been submitted
https://github.com/apache/kafka/pull/7674
Xavier
On Mon, Nov 11, 2019 at 7:56 AM Stanislav Kozlovski
wrote:
> +1 (non-binding). Thanks Xavier
>
>
> On Sat,
>
> 1. I do feel there're similar needs for clients make JMX configurable. Some
> context: in modules like Connect and Streams we have added / refactored a
> large number of metrics so far [0, 1], and although we've added a reporting
> level config [2] to clients, this is statically defined at code
Hi everyone,
I'd like to open up the vote for KIP-544 on making exposed JMX metrics more
configurable.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-544%3A+Make+metrics+exposed+via+JMX+configurable
Thank you!
Xavier
>
> Since these configs will work with Kafka's own metrics library, will the
> configs be part of the clients' configurations? It would be good to point
> that out explicitly in the KIP.
>
Those configs are currently only at the broker level. If we feel this is
useful on the client as well, we cou
>
> A follow-up question, maybe to list in the future work section as it's
> somewhat parallel to this KIP: have you thought about implementing a REST
> reporter for metrics?
That's certainly an option, however it does not solve the problem for our
users that still rely on JMX integration to coll
>
> How would the practical application look like if this was implemented?
>
One useful application is to hide partition-level metrics, some of which
may only be needed for debugging purposes.
> Would monitoring agents switch between the whitelist and blacklist
> periodically if they wanted to m
Hi All,
I wrote a short KIP to make the set of metrics exposed via JMX configurable.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-544%3A+Make+metrics+exposed+via+JMX+configurable
Let me know what you think.
Thanks,
Xavier
>
> This kind of change will be problematic to us as the total RequestsPerSec
> will be double counted in our metric system as we do automatic aggregation.
> It has been a headache for us as we always have to do some special handling
> when querying such metrics.
>
I agree with Allen, most of the
Hi Steven, do you think you'll get a chance to address the points Ismael
made? It'd be great to get this change into 1.1.
Thanks!
Xavier
On Tue, Dec 19, 2017 at 12:20 PM Ismael Juma wrote:
> Hi Steven,
>
> As a general rule, we don't freeze KIPs after the vote passes. It's
> reasonably common f
Hi Damian, I believe the list should also include KAFKA-5886 (KIP-91) which
was voted for 1.0 but wasn't ready to be merged in time.
On Tue, Jan 16, 2018 at 5:13 AM Damian Guy wrote:
> Hi,
>
> This is a reminder that we have one week left until the KIP deadline of Jan
> 23. There are still some
> On Mon, Dec 11, 2017, at 13:14, Ewen Cheslack-Postava wrote:
> > > +1 (binding)
> > >
> > > -Ewen
> > >
> > > On Mon, Dec 11, 2017 at 12:40 PM, Gwen Shapira
> > wrote:
> > >
> > > > +1 (binding) - nice API improvement, thanks for drivin
ns are thrown from these functions.
> > >
> > > I would suggest maybe we should just implement whenComplete, rather
> than
> > > exposing addWaiter. addWaiter was never intended as a public API, and
> > > it's a little weird. whenComplete is nice becaus
Thanks
>
> Op za 9 dec. 2017 om 01:29 schreef Xavier Léauté :
>
> > Hi Steve, I just posted in the discussion thread, there's just one tiny
> fix
> > I think would be useful to add while we're making changes to this API.
> > Do you mind having a look?
>
Hi Steve, I just posted in the discussion thread, there's just one tiny fix
I think would be useful to add while we're making changes to this API.
Do you mind having a look?
On Fri, Dec 8, 2017 at 11:37 AM Mickael Maison
wrote:
> +1 (non binding)
> Thanks for the KIP
>
> On Fri, Dec 8, 2017 at 6
Hi Steven,
I noticed you are making KafkaFuture.addWaiter(...) public as part of your
PR. This is a very useful method to add – and you should mention it in the
KIP – however addWaiter currently doesn't guard against exceptions thrown
inside of the BiConsumer function, which is something we shoul
I agree with Matthias that keeping -1 special will be prone to errors. We
should accept this is mistake resulting from lack of foresight on our part
when adding timestamps in the first place and correct it.
Using deltas will probably cause lots of headaches. It means we have to
figure out the impl
Is there anything that would prevents us from moving directly to log4j2 as
the default log backend – independently of our efforts to move remaining
pieces to sl4fj? Unless we have some very custom code, it should be
possible to rely on log4j-1.2-api.jar to avoid having to migrate everything
at once
chard Yu
> wrote:
>
> > I think we can come up with this compromise: range(long timeFrom, long
> > timeTo) will be changed to getKeys(long timeFrom, long timeTo). Sounds
> fair?
> >
> >
> > On Tue, Oct 24, 2017 at 10:44 AM, Xavier Léauté
> > wrote:
> &g
>
> Generally I think having `all / range` is better in terms of consistency
> with key-value windows. I.e. queries with key are named as `get / fetch`
> for kv / window stores, and queries without key are named as `range / all`.
>
For kv stores, range takes a range of keys, and with this proposal
larifications, Xavier.
> I have removed most of the methods except for keys() and all() which has
> been renamed to Guozhang Wang's suggestions.
>
> Hope this helps.
>
> On Fri, Oct 13, 2017 at 3:28 PM, Xavier Léauté
> wrote:
>
> > Thanks for the KIP Richard, this is a
Thanks for the KIP Richard, this is a very useful addition!
As far as the API changes, I just have a few comments on the methods that
don't seem directly related to the KIP title, and naming of course :).
On the implementation, see my notes further down that will hopefully
clarify a few things.
R
A few comments on the KIP:
- I'm a bit confused about the BytesStoreSupplier interface. Nothing in its
definition is really specific to Bytes, and
when I see return types like BytesStoreSupplier>, it seems redundant to have "Bytes" in the supplier name.
Why can't we reuse the existing StateStoreSup
providing an
> >> Aggregator
> >> > >>> implementation.
> >> > >>>
> >> > >>>
> >> > >>> If they are really co-aggregated, why don't we turn this around:
> >> > >>> KGroupedStream grouped1
>>
>> Michał
>>
>> On 08/06/17 22:44, Guozhang Wang wrote:
>>
> Note that although the internal `AbstractStoreSupplier` does maintain the
>> key-value serdes, we do not enforce the interface of `StateStoreSupplier`
>> to always retain that information, and
ive than adding
> > it to the `KStreamBuilder` though.
> >
> > On Thu, Jun 8, 2017 at 10:58 AM, Xavier Léauté
> > wrote:
> >
> >> I have a minor suggestion to make the API a little bit more symmetric.
> >> I feel it would make more sense to move t
; >>>>>> seems you are allowing users to specify a (potentially different)
> > >>> window
> > >>>>>> spec in each co-grouped input stream. So if these window specs are
> > >>>>>> different how should we "al
>
> ...
>
>
> I'm wondering if it makes more sense to only start sending the update if
> the corresponding agg-key has seen at least one input from each of the
> input stream? Maybe it is out of the scope of this KIP and we can make it a
> more general discussi
Hi Kyle, I left a few more comments in the discussion thread, if you
wouldn't mind taking a look
On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman
wrote:
> Hello all,
>
> I would like to start the vote on KIP-150.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup
Sorry to jump on this thread so late. I agree this is a very useful
addition and wanted to provide an additional use-case and some more
comments.
This is actually a very common analytics use-case in the ad-tech industry.
The typical setup will have an auction stream, an impression stream, and a
cl
ey field would always be the same, as it
> is more consistent with the added APIs. But we need to do it in a backward
> compatible way, which we can discuss in a follow-up KIP.
>
>
> Guozhang
>
> On Thu, May 11, 2017 at 10:38 AM, Xavier Léauté
> wrote:
>
> > Thank
o you think?
>
> Thanks,
>
> Michal
>
> On 11/05/17 07:51, Michal Borowiecki wrote:
> > Well, another concern, apart from potential confusion, is that you
> > won't be able to peek the actual next key, just the timestamp. So the
> > tradeoff is between havin
onto this only once the voting has already
> begun.
> Thanks,
> Michał
>
> On 10/05/17 20:08, Sriram Subramanian wrote:
> > +1
> >
> > On Wed, May 10, 2017 at 11:42 AM, Bill Bejeck wrote:
> >
> >> +1
> >>
> >> Thanks,
> >
Hi everyone,
Since there aren't any objections to this addition, I would like to start
the voting on KIP-155 so we can hopefully get this into 0.11.
https://cwiki.apache.org/confluence/display/KAFKA/KIP+155+-+Add+range+scan+for+windowed+state+stores
Voting will stay active for at least 72 hours.
I'll be driving the implementation as well. Unless there are specific
concerns about the implementation, I mainly wrote the KIP to agree on the
additional public interfaces.
Current window store fetch operations already rely on range scanning
& filtering the relevant keys in the underlying state s
+1 (non-binding)
On Tue, May 9, 2017 at 11:00 AM Dong Lin wrote:
> +1
>
> On Sun, May 7, 2017 at 7:40 PM, Jun Rao wrote:
>
> > Hi, Everyone,
> >
> > Since this is a relatively simple change, I would like to start the
> voting
> > process for KIP-153 : Include only client traffic in BytesOutPerS
I updated the KIP to reflect Michal's comment. Unless there are any further
comments, I will initiate a vote this evening.
On Mon, May 8, 2017 at 11:01 AM Xavier Léauté wrote:
> That sounds reasonable, Michal. Given the underlying implementation uses
> the same segmented bytes st
IP. Do you think the same should apply to session stores?
>
> IMHO, all three should be on par wrt the ability to query key ranges
> (although I understand the implementation concerns for windowed stores are
> more involved than for normal key value stores).
>
> Thanks,
>
> Micha
Hi everyone,
I am proposing to add the ability to range scan windowed state stores in
Kafka Streams. The required interface changes are relatively minimal and
follow our existing conventions for state stores. Let me know if that
sounds reasonable.
https://cwiki.apache.org/confluence/display/KAFKA
Any reason we are not keeping the per-topic breakdown for inter-broker
traffic?
On Fri, May 5, 2017 at 4:52 PM Onur Karaman
wrote:
> Looks good. Thanks!
>
> On Fri, May 5, 2017 at 4:44 PM, Roger Hoover
> wrote:
>
> > Very helpful. Thank you, Jun.
> >
> > On Fri, May 5, 2017 at 4:42 PM, Guozha
One current benefit of having those classes extensible is the ability to
write simple wrappers around KStreamBuilder to add functionality that
currently doesn't exist.
In my case, I extend KStreamBuilder mainly to provide syntactic sugar and
make it easier to work with. For instance, I overload im
the
decision was made to go with invariant result types in those cases (for
now).
Let me know if you think everything makes sense.
Xavier
On Thu, Dec 8, 2016 at 10:35 AM Damian Guy wrote:
Hi Xavier,
The KIP looks good - thanks!
Damian
On Thu, 8 Dec 2016 at 18:12 Xavier Léauté wrote:
>
However, it seems that only the first case is covered by your proposal.
> This is an improvement, but is there any reason not to do the latter as
> well? It would be good to get it completely right this time.
>
> Ismael
>
> [1] http://openjdk.java.net/jeps/300
>
> O
Hi everyone,
I would like to start the vote for KIP-100 unless there are any more
comments.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API
corresponding PR here https://github.com/apache/kafka/pull/2205
Thanks,
Xavier
I recently filed https://issues.apache.org/jira/browse/KAFKA-4481, which
Guozhang agreed was a bug. However fixing this will require some minor
changes to the existing Kafka Streams APIs.
I've outlined the required changes and binary / source compatibility
implications in the KIP
https://cwiki.apa
gt; > +1. Thanks for the KIP!
> > > >
> > > > On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira
> > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Wed, Nov 30, 2016 at 1:34 PM, Xavier Léauté <
> xav.
Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to
initiate a vote on it.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count
Xavier
for the KIP. Sounds good to me.
> >
> > Ismael
> >
> > On Tue, Nov 29, 2016 at 12:40 AM, Xavier Léauté
> > wrote:
> >
> > > Hi,
> > >
> > > I created KIP-96 to propose per partition in-sync / assigned replica
> > > metrics.
Hi,
I created KIP-96 to propose per partition in-sync / assigned replica
metrics. Should be straightforward, but submitting it for proposal since we
require it for metrics changes.
Here's the link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for
>
> I kind of agree with James that it is a bit questionable how valuable any
> data in a delete marker can be since it will be deleted somewhat
> nondeterministically.
>
One could argue that even in normal topics, assuming a time-based log
retention policy is in place, any message will be deleted
Does this mean that starting with V4 requests we would allow storing null
messages in compacted topics? The KIP should probably clarify the behavior
for null messages where the tombstone flag is not net.
On Wed, Oct 26, 2016 at 1:32 AM Magnus Edenhill wrote:
> 2016-10-25 21:36 GMT+02:00 Nacho So
69 matches
Mail list logo
