Re: [DISCUSS] KIP-885: Expose Broker's Name and Version to Clients Skip to end of metadata

2022-11-11 Thread Magnus Edenhill
Hi Travis and thanks for the KIP, two comments below: Den fre 11 nov. 2022 kl 13:37 skrev David Jacot : > 02: I am a bit concerned by clients that could misuse these information. > For instance, one may be tempted to rely on the version to decide whether a > feature is enabled or not. The api ve

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-10-24 Thread Magnus Edenhill
Hi, one minor comment on the latest update: Den mån 24 okt. 2022 kl 16:26 skrev David Jacot : > * Jason pointed out that the member id handling is a tad weird. The > group coordinator generates the member id and then trusts the member > when it rejoins the group. This also implies that the clien

Re: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-10-19 Thread Magnus Edenhill
Great work on the KIP, David. +1 (nonbinding) Den fre 14 okt. 2022 kl 11:50 skrev Luke Chen : > Hi David, > > I made a final pass and LGTM now. > +1 from me. > > Luke > > On Wed, Oct 5, 2022 at 12:32 AM Guozhang Wang wrote: > > > Hello David, > > > > I've made my final pass on the doc and I thi

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

2022-10-17 Thread Magnus Edenhill
Hi David, kudos on a very well designed KIP, this will make a ton of difference for the maintainability of client implementations and operational aspects of consumer groups! Some comments: 1. Maybe I missed this, but where does max.poll.interval.ms fit into the new protocol? It seems like th

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-06-21 Thread Magnus Edenhill
sh vs > > dot) > > > different from other existing metrics also seems a bit confusing. It > > seems > > > that the main benefit of having standard metric names across clients is > > for > > > better server side monitoring. Could we do the standardization in the >

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-06-07 Thread Magnus Edenhill
ubtle differences across clients seem to cause more confusion during > > troubleshooting. > > > > Thanks, > > > > Jun > > > > On Tue, May 31, 2022 at 5:02 AM Magnus Edenhill > > wrote: > > > >> Den fre 20 maj 2022 kl 01:23 skrev Jun Rao : &g

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-05-31 Thread Magnus Edenhill
o suitable standard metric a client can still provide what special metrics it has. Thanks, Magnus On Thu, May 19, 2022 at 10:39 AM Magnus Edenhill wrote: > > > Den ons 18 maj 2022 kl 19:57 skrev Jun Rao : > > > > > Hi, Magnus, > > > > > > > Hi J

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-05-19 Thread Magnus Edenhill
Den ons 18 maj 2022 kl 19:57 skrev Jun Rao : > Hi, Magnus, > Hi Jun > > Thanks for the updated KIP. Just a couple of more comments. > > 50. To troubleshoot a particular client issue, I imagine that the client > needs to identify its client_instance_id. How does the client find this > out? Do we

Re: [VOTE] KIP-714: Client Metrics and Observability

2022-05-17 Thread Magnus Edenhill
atic for large deployments. Still, I think it is worth > > refactoring. > > > > IMHO, it is a fair trade-off. > > > > — Ricardo > > > > > On Jan 26, 2022, at 9:34 AM, Magnus Edenhill > wrote: > > > > > > Hi all, > > > &g

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-03-31 Thread Magnus Edenhill
Hey Ismael, > > The PushTelemetryRequest handler decompresses the payload before passing > it > > to the metrics plugin. > > This was done to avoid having to expose a public decompression interface > to > > metrics plugin developers. > > > > Are there cases where the metrics plugin developers wou

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-03-30 Thread Magnus Edenhill
ull metric name > >> including groups, tags, etc? We typically don't add the broker_id > >> label for > >> broker metrics. Also, brokers use Yammer metrics, which doesn't have > >> type > >> Sum. > >> > >> 29.

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-03-07 Thread Magnus Edenhill
permitting, and only secondarily as a Gauge average. While we might not want to maintain a bunch of histograms for each partition, since that could be quite space consuming, this client.io.wait.time is a single metric per client instance and can thus afford a Histogram representation. Thank

Re: [VOTE] KIP-714: Client Metrics and Observability

2022-01-26 Thread Magnus Edenhill
and outside the borders of the enterprise. Our clients like the performance > and data safeties related to the kafka connection. The observability has > been a problem... > > Jonathan Rivers > jrivers...@gmail.com > > > > > On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dol

Re: [DISCUSS] KIP-714: Client metrics and observability

2022-01-26 Thread Magnus Edenhill
e proposal listed idempotence=true. This is more of a > configuration > > > than a metric. Are we including that as a metric? What other > > configurations > > > are we including? Should we separate the configurations from the > metrics? > > > > > &

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-11-29 Thread Magnus Edenhill
what about adding metrics about the subscription cache itself? That I > think > > would improve its usability and debuggability as we'd be able to see its > > performance, hit/miss rates, eviction counts and others. > > > > Best, > > Viktor > > > &

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-11-29 Thread Magnus Edenhill
sed it in the KIP but > what about adding metrics about the subscription cache itself? That I think > would improve its usability and debuggability as we'd be able to see its > performance, hit/miss rates, eviction counts and others. > > Best, > Viktor > > On Thu, Nov 18, 2

Re: [DISCUSS] KIP-784: Add top-level error code field to DescribeLogDirsResponse

2021-11-22 Thread Magnus Edenhill
Hey Mickael and thanks for the KIP. I'm a bit late to the game, but one small nit for consistency: The global response error codes are typically at the start of the response, usually right after the ThrottleTime. This also makes it easier for the response parser to assign the global error for each

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-11-18 Thread Magnus Edenhill
I'd prefer to be > slightly defensive and see how it works in practice before enabling it > everywhere. > Right, and I agree on being defensive, but since this feature still requires manual enabling on the brokers before actually being used, I think that gives enough control to opt-in

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-11-07 Thread Magnus Edenhill
> the previous client instance ID proposal was to construct the client > instance ID using clientId as a prefix. Now that the client instance ID is > a UUID, would we want to add clientId as a supported selector? > Let me know what you think. > > David > > On Tue, Oct 19,

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-10-19 Thread Magnus Edenhill
number of partition/topics/etc the client is producing to/consuming from. I'll add some sizes to the KIP for some typical use-cases. Thanks, Magnus > Thanks > > On Tue, Oct 19, 2021 at 5:06 PM Magnus Edenhill > wrote: > > > > Den tis 19 okt. 2021 kl 13:22 skrev Tom Be

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-10-19 Thread Magnus Edenhill
plication instance id as an optional future nice-to-have > that may be included as a metrics label if it has been set by the user", so > I'm confused whether non-Kafka Streams clients should set an application_id > or not. > I'll clarify this in the KIP, but basicall

[VOTE] KIP-714: Client Metrics and Observability

2021-10-18 Thread Magnus Edenhill
Hi all, I'd like to start a vote on KIP-714. https://cwiki.apache.org/confluence/x/2xRRCg Discussion thread: https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html Thanks, Magnus

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-10-07 Thread Magnus Edenhill
day or two. Regards, Magnus Den mån 4 okt. 2021 kl 20:57 skrev Magnus Edenhill : > Hi Gwen, > > I'm finishing up the KIP based on the last couple of discussion points in > this thread > and will call the Vote later this week. > > Best, > Magnus > > De

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-10-04 Thread Magnus Edenhill
ouldn't > find the vote thread. Is there one that I'm missing? > > Gwen > > On Wed, Sep 22, 2021 at 4:58 AM Magnus Edenhill > wrote: > > > Den tis 21 sep. 2021 kl 06:58 skrev Colin McCabe : > > > > > On Mon, Sep 20, 2021, at 17:35

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-09-22 Thread Magnus Edenhill
Den tis 21 sep. 2021 kl 06:58 skrev Colin McCabe : > On Mon, Sep 20, 2021, at 17:35, Feng Min wrote: > > Thanks Magnus & Colin for the discussion. > > > > Based on KIP-714's stateless design, Client can pretty much use any > > connection to any broker to send metrics. We are not associating > conn

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-09-22 Thread Magnus Edenhill
Den mån 20 sep. 2021 kl 20:41 skrev Colin McCabe : > On Tue, Sep 14, 2021, at 00:47, Magnus Edenhill wrote: > > Thanks for your feedback Colin, see my updated proposal below. > > ... > > Hi Magnus, > > Thanks for the update. > > > > > Splitting up the A

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-09-14 Thread Magnus Edenhill
Thanks for your feedback Colin, see my updated proposal below. Den tors 22 juli 2021 kl 03:17 skrev Colin McCabe : > On Tue, Jun 29, 2021, at 07:22, Magnus Edenhill wrote: > > Den tors 17 juni 2021 kl 00:52 skrev Colin McCabe : > > > A few critiques: > > > > > &

Re: [VOTE] 3.0.0 RC1

2021-09-01 Thread Magnus Edenhill
Hi Konstantine, Some findings from running 3.0.0-RC1 with the librdkafka test suite: * Compaction seems to take slightly longer to kick in when segment sizes exceed their threshold. (Used to take less than 20 seconds, now takes 20..30 seconds.) * CreateTopic seems to take slightly longer to pr

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Hey Tom, Den mån 21 juni 2021 kl 21:08 skrev Tom Bentley : > > 1. Did you consider using a `default ClientTelemetryReceiver > clientReceiver() { return null; }` method on the existing MetricsReporter > interface, avoiding the need for the ClientTelemetry trait? > I'll let Xavier answer this one

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
ply "use a supported client". Even if this > > > (potentially unlikely) scenario is true, then hooks would still be a > great > > > alternative, because then Confluent could provide drop-in hooks for > any client > > > and the end result of e

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Den tors 17 juni 2021 kl 00:52 skrev Colin McCabe : > Hi Magnus, > > Thanks for the KIP. This is certainly something I've been wishing for for > a while. > > Maybe we should emphasize more that the metrics that are being gathered > here are Kafka metrics, not general application business logic met

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Thanks for your feedback, Colin, see response below. Den tors 17 juni 2021 kl 00:28 skrev Colin McCabe : > On Sun, Jun 13, 2021, at 21:51, Travis Bischel wrote: > ... > > Another downside is that by dictating the important metrics, this KIP > either > > has two choices: try to choose what is

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Magnus Edenhill
tor", so the opt-out appears to be the only motivation. Am I missing > something? > > Ryanne > > On Wed, Jun 2, 2021 at 7:46 AM Magnus Edenhill wrote: > > > Hey all, > > > > I'm proposing KIP-714 to add remote Client metrics and observability. > > This

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Magnus Edenhill
e, because then Confluent could provide drop-in hooks for any > client > and the end result of easy-panels would be the same. > > In summary, > > - Metrics are more of an organizational concern, not specifically a broker > operator concern. > > - The proposal seems to hija

[DISCUSS] KIP-714: Client metrics and observability

2021-06-02 Thread Magnus Edenhill
Hey all, I'm proposing KIP-714 to add remote Client metrics and observability. This functionality will allow centralized monitoring and troubleshooting of clients and their internals. Please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability Looking

[jira] [Created] (KAFKA-12712) KRaft: Missing controller.quorom.voters config not properly handled

2021-04-23 Thread Magnus Edenhill (Jira)
Magnus Edenhill created KAFKA-12712: --- Summary: KRaft: Missing controller.quorom.voters config not properly handled Key: KAFKA-12712 URL: https://issues.apache.org/jira/browse/KAFKA-12712 Project

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

2021-01-27 Thread Magnus Edenhill
Hey Thomas, I'm late to the game. It looks like the "top level" ErrorCode moved from the top-level to the Group array, which makes sense, but it would be good if it was marked as // MOVED in the KIP and also a note that top level errors that are unrelated to the group will be returned as per-grou

Re: [DISCUSSION] Upgrade system tests to python 3

2020-10-07 Thread Magnus Edenhill
3] > >>> > https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39 > >>> > >>>> 16 сент. 2020 г., в 07:32, Guozhang Wang > >>> написал(а): > >>>> > >>>> Hello Nikolay, > >>>>

Re: [DISCUSS] KIP-363

2020-07-08 Thread Magnus Edenhill
Hi Tom, I think it would be useful with some real world (or made up!) numbers on how much relative/% space is saved for the most error-dense protocol requests. E.g., an OffsetCommitResponse with 10 topics and 100 failing partitions would reduce the overall size by % bytes. Thanks, Magnus Den ti

Re: [DISCUSS] KIP-551: Expose disk read and write metrics

2019-12-10 Thread Magnus Edenhill
Hi Colin, aren't those counters (ever increasing), rather than gauges (fluctuating)? You also mention CPU usage as a side note, you could use getrusage(2)'s ru_utime (user) and ru_stime (sys) to allow the broker to monitor its own CPU usage. /Magnus Den tis 10 dec. 2019 kl 19:33 skrev Colin McC

[jira] [Resolved] (KAFKA-9180) Broker won't start with empty log dir

2019-11-13 Thread Magnus Edenhill (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Edenhill resolved KAFKA-9180. Resolution: Invalid Turned out to be old client jars making a mess.   > Broker wo

[jira] [Created] (KAFKA-9180) Broker won't start with empty log dir

2019-11-13 Thread Magnus Edenhill (Jira)
Magnus Edenhill created KAFKA-9180: -- Summary: Broker won't start with empty log dir Key: KAFKA-9180 URL: https://issues.apache.org/jira/browse/KAFKA-9180 Project: Kafka Issue Type

Re: [VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-28 Thread Magnus Edenhill
Den mån 23 sep. 2019 kl 14:42 skrev Colin McCabe : > On Fri, Sep 20, 2019, at 18:05, Jun Rao wrote: > > 101. We already use varInt in the message format. I assume that the > > protocol uses the same varInt representation? > > It uses a slightly different varint representation. Basically, the > di

Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-18 Thread Magnus Edenhill
> I think we should force client software names and versions to follow a regular expression and disconnect if they do not. Disconnecting is not really a great error propagation method since it leaves the client oblivious to what went wrong. Instead suggest we return an ApiVersionResponse with an e

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

2019-09-09 Thread Magnus Edenhill
. 2019 kl 08:49 skrev Magnus Edenhill : > Hey Guozhang, > > I'm late to the game, again, but if it is not too late I'd recommend that > instead > of having a single error code for all failed records, instead provide > per-record > error codes by having an array

Re: [DISCUSS] KIP-467: Augment ProduceResponse error messaging

2019-09-08 Thread Magnus Edenhill
Hey Guozhang, I'm late to the game, again, but if it is not too late I'd recommend that instead of having a single error code for all failed records, instead provide per-record error codes by having an array of [relative_offset, error_code], where they're both varint-encoded for space-efficiency.

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-26 Thread Magnus Edenhill
Great KIP as always, Colin! Some comments: > If the flexible versions are not specified, it is assumed that all versions are flexible. This is ambiguous, if a protocol-generator is pointed to an older Kafka protocol specification it can't know if the lack of flexibleVersions field means they're

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-23 Thread Magnus Edenhill
Great proposal, this feature is well overdue! 1) >From an operator's perspective I don't think the kafka client implementation name and version are sufficient, I also believe the application name and version are of interest. You could have all applications in your cluster run the same kafka client

Re: [VOTE] KIP-360: Improve handling of unknown producer when using EOS

2019-04-10 Thread Magnus Edenhill
+1 (non-binding) Den ons 10 apr. 2019 kl 02:38 skrev Guozhang Wang : > +1 (binding). Thanks for the written KIP! The approach lgtm. > > One minor thing: the name of "last epoch" maybe a bit misleading (although > it is for internal usage only and will not be exposed to users) for future > develo

Re: [DISCUSS] KIP-430 - Return Authorized Operations in Describe Responses

2019-02-13 Thread Magnus Edenhill
Hey Rajini, good work on the KIP! I'm personally not thrilled with piggy-backing the authorized_operations on existing APIs, it seems like a mix of concerns. Since there is already the notion of generic ResourceType,ResourceName tuples in the Admin API, I believe it would make more sense to add a

Re: [VOTE] 2.1.1 RC2

2019-02-08 Thread Magnus Edenhill
+1 Passes librdkafka test suite. Den fre 8 feb. 2019 kl 21:02 skrev Colin McCabe : > Hi all, > > This is the third candidate for release of Apache Kafka 2.1.1. This > release includes many bug fixes for Apache Kafka 2.1. > > Compared to rc1, this release includes the following changes: > * MINO

Re: [VOTE] KIP-228 Negative record timestamp support

2018-12-06 Thread Magnus Edenhill
Sorry for getting in the game this late, and on the wrong thread! I think negative timestamps makes sense and is a good addition, but I have a couple of concerns with the proposal: 1. I believe any change to the protocol format or semantics require a protocol bump, in this case for ProduceReques

Re: [VOTE] 2.0.1 RC0

2018-10-30 Thread Magnus Edenhill
+1 (non-binding) Passes librdkafka integration test suite Den mån 29 okt. 2018 kl 18:08 skrev Manikumar : > Hi Eno, > > This looks like an existing issue occuring only on source artifacts. We > are able to generate aggregate docs on cloned repo. > I am getting similar error on previous release a

Re: [VOTE] 2.1.0 RC0

2018-10-29 Thread Magnus Edenhill
+1 (non-binding) passes librdkafka integration test suite Den fre 26 okt. 2018 kl 15:58 skrev Manikumar : > minor observation: config sections are empty in the documentation page. > http://kafka.apache.org/21/documentation.html#producerconfigs > > On Wed, Oct 24, 2018 at 10:49 PM Ted Yu wrote:

[jira] [Created] (KAFKA-7549) Old ProduceRequest with zstd compression does not return error to client

2018-10-25 Thread Magnus Edenhill (JIRA)
Magnus Edenhill created KAFKA-7549: -- Summary: Old ProduceRequest with zstd compression does not return error to client Key: KAFKA-7549 URL: https://issues.apache.org/jira/browse/KAFKA-7549 Project

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

2018-09-06 Thread Magnus Edenhill
> Ismael wrote: > Jason, that's an interesting point regarding the Java client. Do we know > what clients in other languages do in these cases? librdkafka (and its bindings) passes unknown/future errors through to the application, the error code remains intact while the error string will be set to

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-16 Thread Magnus Edenhill
Thanks for driving this KIP, Colin. I agree with Dong that a new similar modifyConfigs API (and protocol API) is confusing and that we should try to extend the current alterConfigs interface to support the incremental mode instead, deprecating the non-incremental mode in the process. Another thin

Re: [VOTE] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

2018-07-04 Thread Magnus Edenhill
+1 (non-binding) 2018-07-04 13:40 GMT+02:00 Satish Duggana : > +1 > > Thanks, > Satish. > > On Wed, Jul 4, 2018 at 4:11 PM, Daniele Ascione > wrote: > > > +1 > > > > Thanks, > > Daniele > > > > Il giorno mar 3 lug 2018 alle ore 23:55 Harsha ha > > scritto: > > > > > +1. > > > > > > Thanks, > >

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-07-04 Thread Magnus Edenhill
to the authorizer > > we could have > > >>> >>> > > >> >> Zookeeper> >> >> as a default though and a client that > > manages SCRAM credentials > > >>> >>> > > >> >> in ZK.> >> >&

Re: TLS settings in librdkafka

2018-05-28 Thread Magnus Edenhill
wing us to avoid even offering insecure algorithms > during the TLS handshake. I.e: avoid the usage of certificate > signature algorithms involving SHA1. > > Magnus Edenhill mentioned in PR[3] some ongoing work/discussion about > keeping librdkafka capabilities inline with the corre

[jira] [Resolved] (KAFKA-6885) DescribeConfigs synonyms are are identical to parent entry for BROKER resources

2018-05-09 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Edenhill resolved KAFKA-6885. Resolution: Invalid > DescribeConfigs synonyms are are identical to parent entry for BRO

[jira] [Created] (KAFKA-6885) DescribeConfigs synonyms are are identical to parent entry for BROKER resources

2018-05-08 Thread Magnus Edenhill (JIRA)
Magnus Edenhill created KAFKA-6885: -- Summary: DescribeConfigs synonyms are are identical to parent entry for BROKER resources Key: KAFKA-6885 URL: https://issues.apache.org/jira/browse/KAFKA-6885

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-04-26 Thread Magnus Edenhill
alizer > > > > writes eight bytes containing the given double value, in Big Endian > > byte > > > > order, into this buffer at the current position, and then increments > > the > > > > position by eight. > > > > > > > > The implementa

[jira] [Created] (KAFKA-6778) DescribeConfigs does not return error for non-existent topic

2018-04-11 Thread Magnus Edenhill (JIRA)
Magnus Edenhill created KAFKA-6778: -- Summary: DescribeConfigs does not return error for non-existent topic Key: KAFKA-6778 URL: https://issues.apache.org/jira/browse/KAFKA-6778 Project: Kafka

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-04-09 Thread Magnus Edenhill
Hi Viktor, since serialization of floats isn't as straight forward as integers, please specify the exact serialization format of DOUBLE in the protocol docs (e.g., IEEE 754), including endianness (big-endian please). This will help the non-java client ecosystem. Thanks, Magnus 2018-04-09 15:16

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-08-30 Thread Magnus Edenhill
For what it is worth: librdkafka-based clients will treat any subscribed topic string that is prefixed with the regexp start-of-line anchor ("^") as a regexp, thus providing regex and actual topic subscriptions with the same API, also allowing a mix of actuals and regexps: E.g.: consumer.subscribe

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

2017-05-27 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16027366#comment-16027366 ] Magnus Edenhill commented on KAFKA-4340: Since this is a change to the prot

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

2017-05-24 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023668#comment-16023668 ] Magnus Edenhill commented on KAFKA-4340: Generally I would agree, but in

[jira] [Commented] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

2017-05-24 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022945#comment-16022945 ] Magnus Edenhill commented on KAFKA-4340: While the idea behind this JIRA is

[jira] [Reopened] (KAFKA-4340) Change the default value of log.message.timestamp.difference.max.ms to the same as log.retention.ms

2017-05-24 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Edenhill reopened KAFKA-4340: > Change the default value of log.message.timestamp.difference.max.ms to the > s

Re: [DISCUSS] KIP-152 - Improve diagnostics for SASL authentication failures

2017-05-04 Thread Magnus Edenhill
Hi Rajini, great KIP! This solution was proposed on the original KIP-43 thread but voted down, so let's hope it does better this time :) /Magnus 2017-05-04 13:37 GMT+02:00 Rajini Sivaram : > Hi all, > > I have created a KIP to improve diagnostics for SASL authentication > failures and reduce re

Re: [DISCUSS] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

2017-05-04 Thread Magnus Edenhill
Hey Colin, good KIP! Some comments: 1a. For operation, permission_type and resource_type: is there any reason for having the any and unknown enums as negative values? Since neither of these fields has an integer significance (unlike for example offsets which use negative offsets for logical offs

[jira] [Issue Comment Deleted] (KAFKA-4842) Streams integration tests occasionally fail with connection error

2017-04-18 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Edenhill updated KAFKA-4842: --- Comment: was deleted (was: Happened again on a trunk PR, https://github.com/apache/kafka

Re: [VOTE] 0.10.2.1 RC2

2017-04-18 Thread Magnus Edenhill
+1 (non-binding) - passes librdkafka regression tests 2017-04-18 17:38 GMT+02:00 Edoardo Comar : > Thanks Gwen > KAFKA-5075 is not included in the > http://home.apache.org/~gwenshap/kafka-0.10.2.1-rc2/RELEASE_NOTES.html > > -- > Edoardo Comar > IBM

[jira] [Updated] (KAFKA-4983) Test failure: kafka.api.ConsumerBounceTest.testSubscribeWhenTopicUnavailable

2017-03-30 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Edenhill updated KAFKA-4983: --- Description: The PR builder encountered this test failure

[jira] [Created] (KAFKA-4983) Test failure: kafka.api.ConsumerBounceTest.testSubscribeWhenTopicUnavailable

2017-03-30 Thread Magnus Edenhill (JIRA)
Magnus Edenhill created KAFKA-4983: -- Summary: Test failure: kafka.api.ConsumerBounceTest.testSubscribeWhenTopicUnavailable Key: KAFKA-4983 URL: https://issues.apache.org/jira/browse/KAFKA-4983

[jira] [Commented] (KAFKA-4476) Kafka Streams gets stuck if metadata is missing

2017-03-30 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15949128#comment-15949128 ] Magnus Edenhill commented on KAFKA-4476: Directed here from KAFKA-4482. Happ

[jira] [Commented] (KAFKA-4842) Streams integration tests occasionally fail with connection error

2017-03-30 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948898#comment-15948898 ] Magnus Edenhill commented on KAFKA-4842: Happened again on a trunk PR, h

[jira] [Updated] (KAFKA-4974) System test failure in 0.8.2.2 upgrade tests

2017-03-29 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Magnus Edenhill updated KAFKA-4974: --- Description: The 0.10.2 system test failed in one of the upgrade tests from 0.8.2.2: http

[jira] [Created] (KAFKA-4974) System test failure in 0.8.2.2 upgrade tests

2017-03-29 Thread Magnus Edenhill (JIRA)
Magnus Edenhill created KAFKA-4974: -- Summary: System test failure in 0.8.2.2 upgrade tests Key: KAFKA-4974 URL: https://issues.apache.org/jira/browse/KAFKA-4974 Project: Kafka Issue Type

Re: [DISCUSS] KIP-82 - Add Record Headers

2017-02-17 Thread Magnus Edenhill
Big +1 on VarInts. CPUs are fast, memory is slow. I agree with Jason that we'll want to continue verifying messages, including their headers, so while I appreciate the idea of the opaque header blob it won't be useful in practice. /Magnus 2017-02-17 10:41 GMT-08:00 Jason Gustafson : > Sorry, my

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression

2017-01-31 Thread Magnus Edenhill
Hi Dongjin and good work on the KIP, I understand that ZStandard is generally considered an improvement over LZ4, but the benchmark you provided on the KIP-110 wiki doesn't really reflect that, and even makes a note that they are comparable: *> As you can see above, ZStandard shows outstanding per

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-09 Thread Magnus Edenhill
5 micro seconds to process > a > >> >> >>> message. > >> >> >>>>> > >> >> >>>>> > >> >> >>>>> > >> >> >>>>> I love strings as much as the nex

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

2016-11-08 Thread Magnus Edenhill
Hey, I'm probably going to argue against bumping the SaslHandshake version, it is redundant to the existing SaslHandshakeResponse.enabled_mechanisms field. With SaslHandshake version bump: * Client performs ApiVersionRequest * If SaslHandshake >= v1: use SCRAM * If SaslHandshake < v1: use

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-11-07 Thread Magnus Edenhill
Hi, I'm +1 for adding generic message headers, but I do share the concerns previously aired on this thread and during the KIP meeting. So let me propose a slimmer alternative that does not require any sort of global header registry, does not affect broker performance or operations, and adds as li

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

2016-10-26 Thread Magnus Edenhill
to 1 for this change. > > But since we are not adding an actual field, we can do away with bumping > up > > the magic byte. > > > > If we really want to go the standard route of bumping up the magic byte > for > > any change to message format we should actually add a n

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

2016-10-26 Thread Magnus Edenhill
dont believe this is needed since it is already attributed through the request's API version. Producer: * if a client sends ProduceRequest V4 then attributes.bit5 indicates a tombstone * if a clients sends ProduceRequest On Tue, Oct 25, 2016 at 10:17 AM, Magnus Edenhill > wrote: > &

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

2016-10-25 Thread Magnus Edenhill
it would be flagged up in this discussion > thread ☺ > > Cheers > Mike > > On 10/25/16, 5:36 PM, "Magnus Edenhill" wrote: > > Hi Michael, > > With the version bumps for Produce and Fetch requests, do you really > need > to bump MagicByte too

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

2016-10-25 Thread Magnus Edenhill
Hi Michael, With the version bumps for Produce and Fetch requests, do you really need to bump MagicByte too? Regards, Magnus 2016-10-25 18:09 GMT+02:00 Michael Pearce : > Hi All, > > I would like to discuss the following KIP proposal: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Re: [VOTE] 0.10.1.0 RC3

2016-10-19 Thread Magnus Edenhill
+1 (non-binding) passes librdkafka test suites 2016-10-19 15:55 GMT+02:00 Ismael Juma : > +1 (non-binding). > > Verified source and Scala 2.11 binary artifacts, ran ./gradlew test with > JDK 7u80, quick start on source artifact and Scala 2.11 binary artifacts. > > Thanks for managing the release!

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-09-05 Thread Magnus Edenhill
Good write-up Qin, the API looks promising. I have one comment: 2016-09-03 5:20 GMT+02:00 Becket Qin : > The currently offsetsForTimes() API obviously does not support querying > multiple timestamps for the same partition. It doesn't seems a feature for > ListOffsetRequest v0 either (sounds more

[jira] [Commented] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

2016-08-10 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415000#comment-15415000 ] Magnus Edenhill commented on KAFKA-1588: I see now that this behaviour is in

[jira] [Commented] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

2016-08-10 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414995#comment-15414995 ] Magnus Edenhill commented on KAFKA-1588: Thanks [~ijuma]. But I dont see how

[jira] [Commented] (KAFKA-1588) Offset response does not support two requests for the same topic/partition combo

2016-08-10 Thread Magnus Edenhill (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414868#comment-15414868 ] Magnus Edenhill commented on KAFKA-1588: [~guozhang] Why would this fix requi

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

2016-06-03 Thread Magnus Edenhill
Bumping this thread so Wes can reply to it. Ignore this mail. 2016-02-24 0:36 GMT+01:00 Joel Koshy : > Great - thanks for clarifying. > > Joel > > On Tue, Feb 23, 2016 at 1:47 PM, Bill Warshaw wrote: > > > Sorry that I didn't see this comment before the meeting Joel. I'll try > to > > clarify w

[jira] [Created] (KAFKA-3743) kafka-server-start.sh: Unhelpful error message

2016-05-22 Thread Magnus Edenhill (JIRA)
Magnus Edenhill created KAFKA-3743: -- Summary: kafka-server-start.sh: Unhelpful error message Key: KAFKA-3743 URL: https://issues.apache.org/jira/browse/KAFKA-3743 Project: Kafka Issue Type

Re: [VOTE] KIP-57: Interoperable LZ4 Framing

2016-05-09 Thread Magnus Edenhill
I have verified this functionality on kafka trunk using librdkafka lz4 branch and it works as intended. 2016-05-07 18:07 GMT+02:00 Dana Powers : > Vote Passed! I will update the wiki. > > -Dana > On May 7, 2016 3:48 AM, "Ismael Juma" wrote: > > > Dana, a long time has passed since the vote sta

Re: [VOTE] KIP-57: Interoperable LZ4 Framing

2016-05-03 Thread Magnus Edenhill
Good work on this Dana. I'll test it with librdkafka (which uses the official liblz4) and report back. 2016-05-03 20:02 GMT+02:00 Dana Powers : > Yes, great point. The intent of adding "naive" support for the > remaining LZ4 header flags (contentsize and contentchecksum) is to > avoid rejecting

Re: [DISCUSS] KIP-43: Kafka SASL enhancements

2016-05-02 Thread Magnus Edenhill
with the handshake > request/response format. Have also added some more text to distinguish the > actual authentication flow from the Kafka handshake/request flow. > > Thank you, > > Rajini > > > On Mon, Apr 25, 2016 at 3:41 AM, Magnus Edenhill > wrote: > > &g

Re: [VOTE] KIP-57: Interoperable LZ4 Framing

2016-04-25 Thread Magnus Edenhill
+1 (non-binding) 2016-04-25 14:26 GMT-07:00 Dana Powers : > Hi all, > > Initiating a vote thread because the KIP-57 proposal is specific to > the 0.10 release. > > KIP-57 can be accessed here: > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-57+-+Interoperable+LZ4+Framing > >. > > The

  1   2   >