Re: [DISCUSS] Json libraries for Kafka

Currently the clients/server mismatch wouldn't be an issue since there are no client-side uses of JSON, right? That said, if Copycat ends up included in Kafka we'll need to provide at least one serializer which would be written in Java and I suspect some people would like JSON to be a candidate for

Re: [VOTE] KIP-26 Add Copycat connector framework for data import/export

> Joel > > On Tue, Jul 14, 2015 at 2:09 PM, Ewen Cheslack-Postava > wrote: > > Hi all, > > > > Let's start a vote on KIP-26: Add Copycat connector framework for data > > import/export > > > > For reference, here's the wiki: > > > http

Re: Review Request 36578: Patch for KAFKA-2338

to send data of different sizes to test the output. - Ewen Cheslack-Postava On July 18, 2015, 3:37 a.m., Edward Ribeiro wrote: > > --- > This is an automatically

Re: Review Request 36578: Patch for KAFKA-2338

lp *anywhere* is much better than the current situation of just silently stalling. - Ewen Cheslack-Postava On July 21, 2015, 4:21 p.m., Edward Ribeiro wrote: > > --- > This is an automatically generated e-mail.

Re: Round Robin Among Consumers

Gwen, I'm curious about this use case. Given the Kafka -> HDFS flow, it obviously relates to Copycat. More generally, this could be a problem even when streaming data if your processing takes too long such that your consumer simply can't keep up with the rate at which messages are produced. The "

Re: [VOTE] Switch to GitHub pull requests for new contributions

+1 (non-binding) One other thing that might be worth mentioning in "The Review Process" section is the hub tool: https://hub.github.com/ I only ever use one feature, but it's immensely useful if you're reviewing PRs on Github. You can just copy the URL into a command to checkout the PR branch: hu

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

Just some notes on the KIP doc itself: * It'd be useful to clarify at what point the plain consumer + custom code + producer breaks down. I think trivial filtering and aggregation on a single stream usually work fine with this model. Anything where you need more complex joins, windowing, etc. are

Re: [DISCUSS] KIP-27 - Conditional Publish

@Becket - for compressed batches, I think this just works out given the KIP as described. Without the change you're referring to, it still only makes sense to batch messages with this KIP if all the expected offsets are sequential (else some messages are guaranteed to fail). I think that probably j

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

ned. > > On Thu, Jul 23, 2015 at 10:25 PM, Ewen Cheslack-Postava > > wrote: > > > Just some notes on the KIP doc itself: > > > > * It'd be useful to clarify at what point the plain consumer + custom > code > > + producer breaks down. I think trivial

Re: Query about LogConfigs

This also came up recently when Geoff and I were discussing KAFKA-2257, which needs to add a timeout option to a tool that KAFKA-2276 is introducing. For that timeout, the question is whether we should strive for consistency (use ms), the unit that is most likely convenient (use s), or finest granu

Re: New Producer and "acks" configuration

It's different because it changes whether the client waits for the response from the broker at all. Take a look at NetworkClient.handleCompletedSends, which fills in dummy responses when a response is not expected (and that flag gets set via Sender.produceRequest using acks != 0 as a flag to Client

Re: Best practices - Using kafka (with http server) as source-of-truth

Hi Prabhjot, Confluent has a REST proxy with docs that may give some guidance: http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer that it uses is very efficient, so you should be able to get pretty good throughput. You take a bit of a hit due to the overhead of sending data t

Re: New Producer and "acks" configuration

since it will be handled in callbacks. So the use > of > >> acks = 0 sounds very rare to me and I am not able to think of an usecase > >> around it. > >> > >> Thanks, > >> > >> Mayuresh > >> > >> On Sun, Jul 26, 2015 at 2:40 PM

Re: Kafka Consumer thoughts

Kartik, on your second point about timeouts with poll() and heartbeats, the consumer now handles this properly. KAFKA-2123 introduced a DelayedTaskQueue and that is used internally to handle processing events at the right time even if poll() is called with a large timeout. The same mechanism is use

Re: [DISCUSS] Partitioning in Kafka

Gwen - this is really like two steps of map reduce though, right? The first step does the partial shuffle to two partitions per key, second step does partial reduce + final full shuffle, final step does the final reduce. This strikes me as similar to partition assignment strategies in the consumer

Re: Kafka Consumer thoughts

On Tue, Jul 28, 2015 at 5:18 PM, Guozhang Wang wrote: > I think Ewen has proposed these APIs for using callbacks along with > returning future in the commit calls, i.e. something similar to: > > public Future commit(ConsumerCommitCallback callback); > > public Future commit(Map offsets, > Consume

Re: Review Request 35421: Patch for KAFKA-2026

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35421/#review93636 --- Ship it! Ship It! - Ewen Cheslack-Postava On June 13, 2015, 10

Copycat data API & serializers

Hi all, With the initial patch of of the Copycat APIs under review, I want to get started on the data API. I wrote up some notes with a rough proposal for what it should cover and a few requirements here: https://cwiki.apache.org/confluence/display/KAFKA/Copycat+Data+API Since they are very relat

Re: New (0.8.2.1) Sync Producer with Batch ?

Sunil, Batching is performed for you automatically. There are a few settings you can use to tweak it's behavior (e.g. linger.ms and batch.size), but the defaults are good for common workloads and can achieve very high throughput. If you want to send some messages and then wait for them to be ackn

Re: Review Request 35437: Patch for KAFKA-2202

Cheslack-Postava On June 14, 2015, 11:27 a.m., Manikumar Reddy O wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Re: Copycat data API & serializers

ul 31, 2015 at 6:21 PM, Ewen Cheslack-Postava wrote: > Hi all, > > With the initial patch of of the Copycat APIs under review, I want to get > started on the data API. I wrote up some notes with a rough proposal for > what it should cover and a few requirements here: > https://cwik

Re: [DISCUSS] Client-side Assignment for New Consumer

On Tue, Aug 11, 2015 at 10:03 PM, Onur Karaman wrote: > Just to make the conversation a bit easier (I don't think we have really > established names for these modes yet), basically with the new > KafkaConsumer today there's: > - "external management", where the application figures out the group >

Re: [DISCUSS] Client-side Assignment for New Consumer

On Tue, Aug 11, 2015 at 10:15 PM, Jiangjie Qin wrote: > Hi Jason, > > Thanks for writing this up. It would be useful to generalize the group > concept. I have a few questions below. > > 1. In old consumer actually the partition assignment are done by consumers > themselves. We used zookeeper to g

Re: [DISCUSS] Client-side Assignment for New Consumer

ser to a true single-writer, but there are definitely still failure modes that this does not address. -Ewen > > Thanks, > > Jiangjie (Becket) Qin > > > On Tue, Aug 11, 2015 at 11:06 PM, Ewen Cheslack-Postava > > wrote: > > > On Tue, Aug 11, 2015 at 10:

Re: NEW: API Stability annotations!

We could potentially add more annotations to cover things like public/private APIs, deprecation, etc. I lifted this initial implementation from Hadoop (thanks for the pointer Gwen!). Hadoop also has a public/private annotation that we could lift, and they use this to filter javadocs. These are usef

Re: [DISCUSS] Client-side Assignment for New Consumer

; JoinGroupResponse will be 140KB * 100 = 14MB. And the > JoinGroupResponse > > >> will need to be sent to 100 different consumers, that means 14MB * > 100 = > > >> 1.4GB need to be sent by the consumer coordinator for one rebalance. > How > > >> would

Re: [copycat] How to upgrade connectors

Yes, I think that makes sense. As I see it, the tradeoffs are: 1. Complexity - adding these APIs increases the number of things connector developers need to implement just to get started. In a lot of cases, the first version of a connector might literally only have some connection string as a sett

Re: [copycat] support for "batch" jobs

The JDBC connector I started implementing just handles this manually, and isn't much code (and could be made into a simple utility): https://github.com/confluentinc/copycat-jdbc/blob/master/src/main/java/io/confluent/copycat/jdbc/JdbcSourceTask.java#L152 Given the current APIs, sources can just ha

Re: Copycat (KIP-26. PR-99) - plan on moving forward

For obvious reasons, I definitely won't object, but I will probably quickly have a follow-up patch to add a very basic system test for Copycat. I was trying to get it in with this patch since I think it is a good idea to encourage people to write system tests with major additions like this, but I t

Re: [Copycat] How will copycat serialize its metadata

I'm not sure the existing discussion is clear about how the format of offset data is decided. One possibility is that we choose one fixed format and that is what we use internally to store offsets no matter what serializer you choose. This would be similar to how the __offsets topic is currently ha

Re: [DISCUSS] Client-side Assignment for New Consumer

onsumer, we > must > >> > fetch > >> > > > > metadata for all topics. Rather than having all regex > subscribers > >> > embed > >> > > > all > >> > > > > of this metadata in their join group requests, the

Re: [copycat] support for "batch" jobs

On Fri, Aug 14, 2015 at 10:57 AM, Jay Kreps wrote: > I thought batch was dead? :-) > > Yeah I think this would be really useful. Kafka kind of allows you to unify > batch and streams since you produce or consume your stream on your own > schedule so you would want the ingress/egress to work the s

Re: [DISCUSS] Client-side Assignment for New Consumer

p. -Ewen > Thanks, > > Jiangjie (Becket) Qin > > On Fri, Aug 14, 2015 at 2:03 AM, Ewen Cheslack-Postava > wrote: > > > On Thu, Aug 13, 2015 at 11:07 PM, Neha Narkhede > wrote: > > > > > Becket, > > > > > > As you say, the

Re: [Copycat] How will copycat serialize its metadata

t read the topic and parse using their favorite JSON library) vs not being accessible (they won't pull in whatever serializer we use internally, or don't want to write a custom parser for our custom serialization format). -Ewen > > Gwen > > On Fri, Aug 14, 2015 at 1:29 AM,

Re: [DISCUSS] Client-side Assignment for New Consumer

y difference is that we don't need metadata hash > > anymore because the metadata is guaranteed to be the same. Replacing > > metadata hash with actual metadata will not have too much overhead for > > small subscription groups. There will be some overhead for r

Re: [DISCUSS] Client-side Assignment for New Consumer

s a snapshot of current metadata and the group enters > > > > > prepare-rebalance > > > > > > state. > > > > > > 4. The metadata snapshot will be used for this round of > rebalance. > > > i.e. > > > > > the > > > &g

Re: [Copycat] How will copycat serialize its metadata

still feel fairly comfortable using REST APIs... most > of them are :) > > On Fri, Aug 14, 2015 at 8:14 PM, Ewen Cheslack-Postava > wrote: > > > On Fri, Aug 14, 2015 at 6:35 PM, Gwen Shapira wrote: > > > > > Yeah, I missed the option to match seriali

Re: Review Request 36427: Patch for KAFKA-2330

73cf0390bc4c76be310c09ddedf91de5ddf1b473 Diff: https://reviews.apache.org/r/36427/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Re: Kafka New Consumer Performance Test ThroughPut Degradation

I don't think the commands are really equivalent despite just adding the --new-consumer flag. ConsumerPerformance uses a single thread when using the new consumer (it literally just allocates the consumer, loops until it's consumed enough, then exits), whereas the old consumer uses a bunch of addit

Re: Kafka New Consumer Performance Test ThroughPut Degradation

; > > > On my laptop the new consumer perf seems to have dropped from about > > ~200MB/sec to about 2MB/sec. > > > > -Jay > > > > > > On Thu, Aug 27, 2015 at 4:21 PM, Ewen Cheslack-Postava < > e...@confluent.io> > > wrote: > > > > &g

Re: [DISCUSS] Client-side Assignment for New Consumer

@Onur - I think you reviewed the KAFKA-2464 patch in some detail. wrt the specific problem you raised about using largest # of partitions, you're right, with the updated proposal this isn't possible because we only send a hash of the metadata in order to avoid the overhead of per-member metadata be

Re: How can I start writing a Copycat connector?

I'd highly recommend keeping your connector in a separate repo and following the process Gwen suggested. That's how I'm working on connectors (and convertalizers, as Gwen likes to call them). Keep in mind that copycat-hdfs and copycat-jdbc haven't been updated to the Kafka Copycat code yet, althou

[DISCUSS] KIP-75 - Add per-connector Converters

Hi all, I've added KIP-75 - Add per-connector Converters https://cwiki.apache.org/confluence/display/KAFKA/KIP-75+-+Add+per-connector+Converters This is a very small completely backwards compatible change, a patch is already available https://github.com/apache/kafka/pull/1721, and I think probab

Re: [DISCUSS] KIP-4 ACL Admin Schema

On Mon, Aug 8, 2016 at 2:44 PM, Grant Henke wrote: > Thank you for the feedback everyone. Below I respond to the last batch of > emails: > > You mention that "delete" actions > > will get processed before "add" actions, which makes sense to me. An > > alternative to avoid the confusion in the fir

[VOTE] KIP-75 - Add per-connector Converters

I would like to initiate the voting process for KIP-75: https://cwiki.apache.org/confluence/display/KAFKA/KIP-75+-+Add+per-connector+Converters I'll kick things off with a +1 (binding). -- Thanks, Ewen

Re: [VOTE] KIP:71 Enable log compaction and deletion to co-exist

+1 (binding) Thanks, -Ewen On Mon, Aug 15, 2016 at 4:26 PM, Jun Rao wrote: > Thanks for the proposal. +1 > > Jun > > On Mon, Aug 15, 2016 at 6:20 AM, Damian Guy wrote: > > > I would like to initiate the voting process for KIP-71 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >

Re: [VOTE] KIP-75 - Add per-connector Converters

Gwen Shapira wrote: > > > > > +1 (binding) > > > > > > On Mon, Aug 15, 2016 at 11:21 AM, Ewen Cheslack-Postava > > > wrote: > > > > I would like to initiate the voting process for KIP-75: > > > > https://cwiki.apache.org/confluence/dis

Re: [VOTE] KIP-77: Improve Kafka Streams Join Semantics

+1 (binding) I think the major gap I notice in the PR is a lack of docs updates to notify people of the change. Given these are improvements and streams is still new, I wouldn't necessarily call them out as anything critical, but the changes should be noted somewhere. -Ewen On Tue, Aug 30, 2016

Re: [VOTE] 0.10.1 Release Plan

+1 binding -Ewen On Tue, Sep 13, 2016 at 2:53 PM, Ismael Juma wrote: > +1 (binding) > > On Tue, Sep 13, 2016 at 10:52 PM, Gwen Shapira wrote: > > > +1 (binding) - yay for a new release! > > > > On Tue, Sep 13, 2016 at 2:40 PM, Jason Gustafson > > wrote: > > > > > Hi All, > > > > > > I'd like

Re: [DISCUSS] KIP-80: Kafka REST Server

Sorry that I've stayed quiet on this for a bit. My reason for doing so is well summed up by Jason's notes. First, I do want to say thanks for referencing the Confluent REST Proxy in the proposal! It makes me proud that it's being used as an important reference point. Second, Nacho, I want to say

Re: [VOTE] Add REST Server to Apache Kafka

Perhaps in the future we can use a [POLL] or similar? This confused a lot of folks because they thought a VOTE was being called on KIP-80 when it wasn't. (Even if we say shame on them for not reading the body in full, the title caught me as well.) -Ewen On Tue, Oct 25, 2016 at 7:11 PM, Harsha Chi

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

I think the bump isn't strictly required, but if the client is KIP-35 aware, it can proactively choose a compatible SASL mechanism based on its initial ApiVersionRequest and avoid an extra connection round trip when there are client/broker version differences. Without this, a newer client would hav

Re: [VOTE] KIP-507: Securing Internal Connect REST Endpoints

+1 (binding) A couple of comments on the details: - I think this might be the first time we're adding a new message type to the config topic. Might be worth noting that (somewhat unfortunately) we currently log unknown keys as an error. Kinda unclear whether it should be an error or warn (since i

Re: [VOTE] KIP-610: Error Reporting in Sink Connectors

+1 (binding) This will be a nice improvement. From the discussion thread it's clear this is tricky to get right, nice work! On Tue, May 19, 2020 at 8:16 AM Andrew Schofield wrote: > +1 (non-binding) > > This is now looking very nice. > > Andrew Schofield > > On 19/05/2020, 16:11, "Randall Hauc

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

+1 (binding), though there is definitely still some compatibility risk -- regardless of updates, people are going to have apps that may need other updates but upgrading between scala versions is still too painful for too long to warrant the effort to update. But in that case they probably aren't up

Re: [VOTE] 2.6.2 RC0

Hey Justin, Can you clarify a few things? - On the first point, the only thing I see is zstd, which we do not, in fact, ship the library itself, just the jni bindings. Was there anything else you saw? - On the second point, I checked the src download from the RC and afaict we use this header and

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

A couple of thoughts: First, agreed that we definitely want to expose header functionality. Thank you Mike for starting the conversation! Even if Connect doesn't do anything special with it, there's value in being able to access/set headers. On motivation -- I think there are much broader use cas

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

Thanks for the KIP. A few responses inline, followed by additional comments. On Mon, May 1, 2017 at 9:50 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > Gwen, Randall thank you for your very insightful observations. I'm glad you > find this first draft to be an adequate platform

Re: [DISCUSS] KIP 151 - Expose Connector type in REST API

Both the proposal and PR look good to me. -Ewen On Tue, May 2, 2017 at 1:48 PM, dan wrote: > hello. > > in an attempt to make the connect rest endpoints more useful i'd like to > add the Connector type (Sink/Source) in our rest endpoints to make them > more self descriptive. > > KIP here: > htt

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

es? And > undefined headers just either default to String or byte[] > > We could also pre define known headers like the jms ones mentioned above. > > E.g > > AwesomeHeader1=boolean > AwesomeHeader2=long > JMSCorrelationId=String > JMSXGroupId=String > > > Wha

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

stantine has mentioned these in the KIP -- if it could be clearer in the proposed changes, perhaps you could suggest where to add details? -Ewen > > > On 3/5/17, 4:24 am, "Ewen Cheslack-Postava" wrote: > > Thanks for the KIP. > > A few responses inline, follow

Re: [DISCUSS] KIP-137: Enhance TopicCommand --describe to show topics marked for deletion

Since everything is whitespace delimited anyway, I don't think we should worry about the compatibility issue. We don't guarantee this unstructured output format. I think it is fine to say that any parser that doesn't do something straightforward and reliable like splitting the line by whitespace th

Re: [VOTE] KIP-137: Enhance TopicCommand --describe to show topics marked for deletion

+1 (binding) Thanks for helping improve the CLI tools! -Ewen On Tue, May 2, 2017 at 8:25 AM, Vahid S Hashemian wrote: > +1 (non-binding) > > Thanks Mickael. > > --Vahid > > > > > From: Ismael Juma > To: dev@kafka.apache.org > Date: 05/02/2017 03:46 AM > Subject:Re: [VOTE] KIP-

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

I understand the convenience of pointing at a JIRA/PR, but can we put the concrete changes proposed in the JIRA (under "Proposed Changes"). I don't think voting on the KIP would be reasonable otherwise since the changes under vote could change arbitrarily... I'm increasingly skeptical of adding mo

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

s > set people up for success in the future. > Thoughts / pre vote? > > On 3/5/17, 4:20 pm, "Ewen Cheslack-Postava" wrote: > > I understand the convenience of pointing at a JIRA/PR, but can we put > the > concrete changes proposed in the JIRA (under "

Re: Permission request to create a KIP

Umesh, I've given you permissions on the wiki. Let me know if you run into any issues. -Ewen On Thu, May 4, 2017 at 12:04 AM, UMESH CHAUDHARY wrote: > Hello Mates, > I need to start a KIP for KAFKA-5057 > under this wiki page >

Re: [VOTE] KIP-151: Expose Connector type in REST API (first attempt :)

+1 binding Thanks, Ewen On Mon, May 8, 2017 at 4:54 PM, BigData dev wrote: > +1 (non-binding) > > Thanks, > Bharat > > > On Mon, May 8, 2017 at 4:39 PM, Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > +1 (non-binding) > > > > On Mon, May 8, 2017 at 3:39 PM, dan wrote: > > > >

Re: [VOTE] KIP-154 Add Kafka Connect configuration properties for creating internal topics

+1 binding -Ewen On Wed, May 10, 2017 at 9:34 AM, Gwen Shapira wrote: > +1 > Thanks for doing this, Randall. It will be a huge usability improvement. > > On Mon, May 8, 2017 at 8:43 PM BigData dev > wrote: > > > +1 (non-binding) > > > > On Mon, May 8, 2017 at 3:25 PM, Dongjin Lee wrote: > > >

Re: [VOTE] KIP-146: Isolation of dependencies and classes in Kafka Connect (restarted voting thread)

+1 binding, and I'm flexible on the config name. Somehow I am guessing no matter what terminology we use there somebody will find a way to be confused. -Ewen On Wed, May 10, 2017 at 9:27 AM, Gwen Shapira wrote: > +1 and proposing 'plugin.path' as we use the term connector plugins when > referri

Re: [VOTE] KIP-138: Change punctuate semantics

+1 (binding) -Ewen On Thu, May 11, 2017 at 7:12 AM, Ismael Juma wrote: > Thanks for the KIP, Michal. +1(binding) from me. > > Ismael > > On Sat, May 6, 2017 at 6:18 PM, Michal Borowiecki < > michal.borowie...@openbet.com> wrote: > >> Hi all, >> >> Given I'm not seeing any contentious issues rem

Re: [VOTE] KIP-154 Add Kafka Connect configuration properties for creating internal topics

Randall is out atm and given the KIP deadline and 72h passed w/ only +1s, I'm going to close this vote out. The KIP passes: 4 binding +1s 5 non-binding +1s 0 -1s Thanks everyone for voting! -Ewen On Thu, May 11, 2017 at 7:03 AM, Ismael Juma wrote: > Thanks for the KIP, Randall. +1 (binding)

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

+1 (binding) I mentioned this in the PR that triggered this: > KIP is accurate, though this is one of those things that we should probably get a KIP for a standard set of config options across all tools so additions like this can just fall under the umbrella of that KIP... I think it would be gr

Re: Kafka broker startup issue

Version 2 of UpdateMetadataRequest does not exist in version 0.9.0.1. This suggests that you have a broker with a newer version of Kafka running against the same ZK broker. Do you have any other versions running? Or is it possible this is a shared ZK cluster and you're not using a namespace within

Re: [DISCUSS] KIP-31 - Message format change proposal

A few questions: 1. If we update the producers to only support V1, doesn't that mean people get stuck on the current version of the producer until they can be sure all their consumers have been upgraded? Is that going to be a problem for anyone, and does it potentially keep important fixes/enhance

Re: [DISCUSS] KIP-31 - Message format change proposal

useful to say which release you're hoping to get this into. -Ewen On Thu, Sep 3, 2015 at 9:35 PM, Ewen Cheslack-Postava wrote: > A few questions: > > 1. If we update the producers to only support V1, doesn't that mean people > get stuck on the current version of the pro

Re: [DISCUSS] KIP-31 - Message format change proposal

On Thu, Sep 3, 2015 at 11:45 PM, Jiangjie Qin wrote: > Hey Ewen, > > Thanks for the comments and they are really good questions. Please inline > replies. > > On Thu, Sep 3, 2015 at 9:35 PM, Ewen Cheslack-Postava > wrote: > > > A few questions: > > >

Re: [DISCUSS] KIP-31 - Message format change proposal

On Sun, Sep 6, 2015 at 4:57 PM, Jay Kreps wrote: > > 2. Nobody cares what time it is on the server. > This is a good way of summarizing the issue I was trying to get at, from an app's perspective. Of the 3 stated goals of the KIP, #2 (lot retention) is reasonably handled by a server-side timesta

Re: [DISCUSS] KIP-31 - Message format change proposal

tractions if we can do > 2 > > >> and 3 > > >> >> > > > together > > >> >> > > > > instead of separately. Maybe after (1) is done it will be > > >> easier to > > >> >> > do > >

Re: Kafka releases - can they be more often please?

Just FYI, 0.8.2.2 is effectively released -- the jars are in Central and if you modify the 0.8.2.1 download URLs for 0.8.2.2 they should be there. I think the reason there hasn't been an official announcement is because the website transition may be holding things up. On Fri, Sep 18, 2015 at 12:2

Re: [VOTE] KIP-28 - Add a processor client for stream data processing

+1 (non-binding) On Mon, Sep 21, 2015 at 6:57 PM, Jay Kreps wrote: > +1 > > -Jay > > On Mon, Sep 21, 2015 at 6:28 PM, Guozhang Wang wrote: > > > Hello all, > > > > I would like to start the voting process on the following KIP: add a > > processor client > > < > > > https://cwiki.apache.org/conf

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

Congrats, Harsha! Glad to have another reviewer to target with review requests :P -Ewen On Mon, Sep 21, 2015 at 11:03 PM, Kartik Paramasivam < kparamasi...@linkedin.com.invalid> wrote: > Congratulations Sriharsha !! > > Special thanks for all the awesome work on SSL. We are looking forward to

Re: [VOTE] KIP-31 - Move to relative offsets in compressed message sets.

+1 -Ewen On Fri, Sep 25, 2015 at 11:15 AM, Jun Rao wrote: > +1. I agree that it's worth thinking through the migration plan a bit more. > > Thanks, > > Jun > > On Thu, Sep 24, 2015 at 6:14 PM, Joel Koshy wrote: > > > +1 on everything but the upgrade plan, which is a bit scary - will > > commen

Re: [DISCUSS] KIP-35 - Retrieve protocol version

The basic functionality is definitely useful here. I'm generally in favor of exposing some info about broker versions to clients. I'd prefer to keep the key/values explicit. Making them extensible string/string pairs is good for avoiding unnecessary version changes in the protocol, but I think we

Re: Should the KafkaProducer callback get the record batch?

Anything from the context of the initial send request that is included in the callback is for convenience -- the only things you *need* to have in the callback are the result values (exception/metadata). You can always capture that information when you construct the callback. A similar issue came u

Re: Kafka dotnet SDK suggestion

The project doesn't make official recommendations about third party clients. The only client directly supported is the Java/Scala library included in the project's source repository. However, the community does maintain a list of third-party clients here: https://cwiki.apache.org/confluence/displa

Re: Confluent Clarification

Dinesh, As Grant mentioned, Confluent's mailing list is a better place to ask questions about the Confluent Platform. The version of Kafka (and Zookeeper) match the Apache versions -- currently the CP 1.0.1 release packages the Apache Kafka 0.8.2.2 release directly. In the future, there may be ca

Re: [kafka-clients] Kafka 0.9.0 release branch

Not sure if I'd call it a blocker, but if we can get it in I would *really* like to see some solution to https://issues.apache.org/jira/browse/KAFKA-2397 committed. Without an explicit leave group, even normal operation of groups can leave some partitions unprocessed for 30-60s at a time under *nor

Re: Slow request log in Kafka

Kafka doesn't have the same type of queries that RDBMS systems have. What "slow queries" would we be trying to capture info about? -Ewen On Wed, Oct 14, 2015 at 4:27 PM, Gwen Shapira wrote: > I had some experience with the feature in MySQL. > > Its main good use is to identify queries that are

Re: [VOTE] KIP-40 ListGroups and DescribeGroup

+1 (non-binding) -Ewen On Thu, Oct 29, 2015 at 2:34 PM, Jun Rao wrote: > Just a minor suggestion. Perhaps DescribeGroupRequest can take an array of > groupIds instead of a single one. This allows the clients to batch > description on multiple groups if needed. > > Otherwise, +1 from me too. > >

Re: [VOTE] 0.9.0.0 Candiate 1

Jun, not sure if this is just because of the RC vs being published on the site, but the links in the release notes aren't pointing to issues.apache.org. They're relative URLs instead of absolute. -Ewen On Tue, Nov 10, 2015 at 3:38 AM, Flavio Junqueira wrote: > -1 (non-binding) > > I'm getting a

Re: [gradle build] The wrapper should be in the repository

Hi, Those instructions simply fell out of date -- you can see in the README in the repository that the first step after checkout is to bootstrap the gradle wrapper. The wrapper is not included due to licensing issues when creating packages from the repository. I've updated the wiki to explain how

Re: [VOTE] 0.9.0.0 Candiate 3

FYI, I found a blocker for Kafka Connect https://issues.apache.org/jira/browse/KAFKA-2859. The fix is already in, but we'll need to do one more round of RC. -Ewen On Thu, Nov 19, 2015 at 11:45 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > +1 (non-binding) > > We integrated this (so

Re: 0.9.0.0 RC4

+1 non-binding. Built in a fresh VM, ran some smoke tests for both the brokers and Kafka Connect. -Ewen On Mon, Nov 23, 2015 at 3:11 PM, Flavio Junqueira wrote: > +1 (non-binding). Built it, ran tests, ran the rat tool separately, > checked license and notice, checked digests, ran a few local s

Review Request 26563: Patch for KAFKA-1692

/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Review Request 26564: Patch for KAFKA-1471

/tools/ConsoleProducer.scala 8e9ba0b284671989f87d9c421bc98f5c4384c260 core/src/test/scala/integration/kafka/api/ProducerCompressionTest.scala 17e2c6e9dfd789acb4b6db37c780c862667e4e11 Diff: https://reviews.apache.org/r/26564/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Review Request 26566: Patch for KAFKA-1680

utils/ToolsUtils.scala fef93929ea03e181f87fe294c06d9bc9fc823e9e core/src/test/scala/other/kafka/TestLinearWriteSpeed.scala 7211c2529c1db76100432737da7a1d1d221dfba0 Diff: https://reviews.apache.org/r/26566/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Re: Review Request 26564: Patch for KAFKA-1471

/ProducerCompressionTest.scala 17e2c6e9dfd789acb4b6db37c780c862667e4e11 core/src/test/scala/unit/kafka/message/MessageTest.scala 4837585d03535043a6f25938368988128df9b94a Diff: https://reviews.apache.org/r/26564/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Review Request 26663: Patch for KAFKA-979

://reviews.apache.org/r/26663/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Review Request 26666: Patch for KAFKA-1653

core/src/main/scala/kafka/admin/TopicCommand.scala 7672c5aab4fba8c23b1bb5cd4785c332d300a3fa core/src/main/scala/kafka/tools/StateChangeLogMerger.scala d298e7e81acc7427c6cf4796b445966267ca54eb Diff: https://reviews.apache.org/r/2/diff/ Testing --- Thanks, Ewen Cheslack-Postava

Review Request 26667: Patch for KAFKA-1698

/apache/kafka/common/config/ConfigDefTest.java 09a82feeb7cae95209e54d3554224915a1498ebd Diff: https://reviews.apache.org/r/26667/diff/ Testing --- Thanks, Ewen Cheslack-Postava

<    1   2   3   4   5   6   7   8   9   10   >