Re: [ANNOUNCE] Apache Kafka 2.1.0

Thanks Dong for running the release, and congrats to everyone in the community! -James Sent from my iPhone > On Nov 21, 2018, at 10:09 AM, Dong Lin wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 2.1.0 > > > This is a major release and includes sig

Re: [ANNOUNCE] New Committer: Vahid Hashemian

Congrats, Vahid!! -James > On Jan 15, 2019, at 2:44 PM, Jason Gustafson wrote: > > Hi All, > > The PMC for Apache Kafka has invited Vahid Hashemian as a project committer > and > we are > pleased to announce that he has accepted! > > Vahid has made numerous contributions to the Kafka communi

Re: [DISCUSS] Kafka 2.2.0 in February 2018

I think KIP-291 is supposed to go into 2.2.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-291%3A+Separating+controller+connections+and+requests+from+the+data+plane At least, according to the associated JIRA, the pull request seems to have been merged just a couple days ago. https://i

Re: kafka ack=all and min-isr

I wrote a blog post on min.isr that explains it in more detail: https://logallthethings.com/2016/07/11/min-insync-replicas-what-does-it-actually-do/ The post is 2 years old, but I think it's still correct. -James > On Jun 7, 2018, at 10:31 PM, Carl Samuelson wrote: > > Hi > > Hopefully this i

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

The “Motivation” section of the KIP says that the starting offset will be useful but doesn’t say why. Can you add a use-case or two to describe how it will be useful? In our case (see https://github.com/wushujames/kafka-utilities/blob/master/ConsumerGroupLag/README.md), we found the starting o

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

Hi Stanislav! Thanks for this KIP! I agree that it would be good if the LogCleaner were more tolerant of errors. Currently, as you said, once it dies, it stays dead. Things are better now than they used to be. We have the metric kafka.log:type=LogCleanerManager,name=time-since-last-run-

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

> On Jul 26, 2018, at 11:09 PM, Guozhang Wang wrote: > > Hi Boyang, > > Thanks for the proposed KIP. I made a pass over the wiki and here are some > comments / questions: > > 1. In order to preserve broker compatibility, we need to make sure the > broker version discovery logic can be integra

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

the > tolerance of client side error is not unlimited. > > > Thank you! > > > > From: James Cheng > Sent: Saturday, July 28, 2018 1:26 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consum

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats and great job, everyone! Thanks Rajini for driving the release! -James Sent from my iPhone > On Jul 30, 2018, at 3:25 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > > This is a major release and i

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

>>>> itself, both of which I think are not true for the log cleaner. >>>>> >>>>> On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino >>> wrote: >>>>> >>>>>> <<>> in >>>>> an >>>>>> infi

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

egment? >>>>>> >>>>>> On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah >>>>> wrote: >>>>>> >>>>>>> For the cleaner thread specifically, I do not think respawning will >>>>> help at >>>>>>

Re: [ANNOUNCE] New committer: Matthias J. Sax

Congrats, Matthias!! Well deserved! -James > On Jan 12, 2018, at 2:59 PM, Guozhang Wang wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our > newest Kafka committer. > > Matthias has made tremendous contributions to Kafka Streams API since earl

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

+1 (non-binding) -James Sent from my iPhone > On Jan 17, 2018, at 6:09 PM, Matthias J. Sax wrote: > > Hi, > > I would like to start the vote for KIP-247: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > > -Matthias >

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congrats Rajini! -James Sent from my iPhone > On Jan 17, 2018, at 10:48 AM, Gwen Shapira wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the community and contributed major patches, reviews and KIP >

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Vahid, Under rejected alternatives, we had decided that we did NOT want to do per-partition expiration, and instead we wait until the entire group is empty and then (after the right time has passed) expire the entire group at once. I thought of one scenario that might benefit from per-partition

Re: [ANNOUNCE] Apache Kafka 1.0.1 Released

Congrats, everyone! Thanks for driving the release, Ewen! -James > On Mar 6, 2018, at 1:22 PM, Guozhang Wang wrote: > > Ewen, thanks for driving the release!! > > > Guozhang > > On Tue, Mar 6, 2018 at 1:14 PM, Ewen Cheslack-Postava wrote: > >> The Apache Kafka community is pleased to annou

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

+1 (non-binding) -James > On Mar 7, 2018, at 1:20 PM, Jay Kreps wrote: > > +1 > > I think we can improve this in the future, but this simple change will > avoid a lot of pain. Thanks for reviving it Ewen. > > -Jay > > On Mon, Mar 5, 2018 at 11:35 AM, Ewen Cheslack-Postava > wrote: > >> I'd

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Matthias, For all the upgrade paths, is it possible to get rid of the 2nd rolling bounce? For the in-place upgrade, it seems like primary difference between the 1st rolling bounce and the 2nd rolling bounce is to decide whether to send Subscription Version 2 or Subscription Version 3. (Actuall

Re: Log Retention Period of Kafka Messages

> On Mar 12, 2018, at 10:13 AM, Kyle Tinker > wrote: > > You have a couple options: > 1) You can adjust log.segment.bytes to make the segments smaller so that > individual segments can be trimmed > 2) You can set log.roll.hours, which will roll to a new log segment even if > the size hasn't

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

Sorry, I see that the VOTE started already, but I have a late question on this KIP. In the "version probing" protocol: > Detailed upgrade protocol from metadata version X to Y (with X >= 1.2): > On startup/rolling-bounce, an instance does not know what version the leader > understands and (optim

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

Manikumar brings up a good point. This is a breaking change to the existing metric. Do we want to break compatibility, or do we want to add a new metric and (optionally) deprecate the existing metric? For reference, in KIP-153 [1], we changed an existing metric without doing proper deprecation.

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

on-disk schema. In the case of an updated RocksDB state store but with a downgraded rebalance metadata... that should work, right? Because we still have updated code (which understands the on-disk format) but that it simply gets its partition assignments via the downgraded rebalance metadata?

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

>> Allen >> >> >> >>> On Wed, Mar 21, 2018 at 2:01 AM, Ted Yu wrote: >>> >>> Creating new metric and deprecating existing one seems better from >>> compatibility point of view. >>> Original message From: J

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

rics,name=RequestsPerSec,request=Produce > > And the KIP is proposing to change it to: > > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=1.0.0 I think that would make the KIP easier to understand. Thanks, -James Sent from my iPhone > On Mar 21, 2018,

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

her KIP (KIP-258). However, this KIP-268 looks fine. It gives us the mechanism to properly detect and automatically upgrade/downgrade the topology and allows the new/old code to co-exist within a topology, which is something we didn't have before. KIP-268 looks good to me. Thanks for all

Re: [VOTE] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

+1 (non-binding) -James > On Mar 21, 2018, at 2:28 AM, Damian Guy wrote: > > +1 > > On Wed, 21 Mar 2018 at 01:44 abel-sun wrote: > >> >> Thanks you of your offer ,agree with you! >> >> On 2018/03/21 00:56:11, Richard Yu wrote: >>> Hi Matthias, >>> Thanks for setting up the upgrade path.

Re: [ANNOUNCE] New Committer: Dong Lin

Congrats, Dong! -James > On Mar 28, 2018, at 10:58 AM, Becket Qin wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce that Dong Lin has accepted > our invitation to be a new Kafka committer. > > Dong started working on Kafka about four years ago, since which he has >

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

+1 (non-binding) Thanks for all the hard work on this, Vahid! -James > On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian > wrote: > > Hi all, > > As I believe the feedback and suggestions on this KIP have been addressed > so far, I'd like to start a vote. > The KIP can be found at > https://

Re: [ANNOUNCE] Apache Kafka 1.1.0 Released

Thanks Damian and Rajini for running the release! Congrats and good job everyone! -James Sent from my iPhone > On Mar 29, 2018, at 2:27 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 1.1.0. > > > Kafka 1.1.0 includes a numbe

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Jun, Can you add me as well? wushuja...@gmail.com Thanks, -James > On Apr 5, 2018, at 1:34 PM, Jun Rao wrote: > > Hi, Ted, Vahid, > > Added you to the invite. > > Thanks, > > Jun > > > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian < > vahidhashem...@us.

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Guozhang, in a previous message, you proposed said this: > On Jul 30, 2018, at 3:56 PM, Guozhang Wang wrote: > > 1. We bump up the JoinGroupRequest with additional fields: > > 1.a) a flag indicating "static" or "dynamic" membership protocols. > 1.b) with "static" membership, we also add the p

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong! -James > On Aug 20, 2018, at 3:54 AM, Ismael Juma wrote: > > Hi everyone, > > Dong Lin became a committer in March 2018. Since then, he has remained > active in the community and contributed a number of patches, reviewed > several pull requests and participated in numerous KIP d

Re: [ANNOUNCE] New committer: Colin McCabe

Congrats, Colin! -James > On Sep 25, 2018, at 1:39 AM, Ismael Juma wrote: > > Hi all, > > The PMC for Apache Kafka has invited Colin McCabe as a committer and we are > pleased to announce that he has accepted! > > Colin has contributed 101 commits and 8 KIPs including significant > improvemen

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

dy Chambers, > Apurva Mehta, Armin Braun, Attila Kreiner, Balint Molnar, Bart De Vylder, > Ben Stopford, Bharat Viswanadham, Bill Bejeck, Boyang Chen, Bryan Baugher, > Colin P. Mccabe, Koen De Groote, Dale Peakall, Damian Guy, Dana Powers, > Dejan Stojadinović, Derrick Or, Dong Lin, Zhen

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur! Well deserved! -James > On Nov 6, 2017, at 9:24 AM, Jun Rao wrote: > > Hi, everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur > Karaman. > > Onur's most significant work is the improvement of Kafka controller, which > is the brain of a Kafka

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

How fast does the in-memory cache grow? As a random datapoint... 10 months ago we set our offsets.retention.minutes to 1 year. So, for the past 10 months, we essentially have not expired any offsets. Via JMX, one of our brokers says kafka.coordinator.group:type=GroupMetadataManager,name=NumOff

Re: [DISCUSS] KIP-225 - Use tags for consumer “records.lag” metrics

This KIP will break backwards compatibility for anyone who is using the existing attribute names. Kafka devs, I believe that metrics are a supported interface, and so this would be a breaking change. In order to do this, we would need a deprecation timeframe for the old metric, and a transition

Re: [DISCUSS] KIP-225 - Use tags for consumer “records.lag” metrics

pt my KIP if the community thinks we should duplicate the > metric for a while. > > Thanks! > > On Thu, Nov 16, 2017 at 8:13 AM, James Cheng wrote: > >> This KIP will break backwards compatibility for anyone who is using the >> existing attribute names. >>

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

I think the discussion may have already cover this but just in case... How does the leader decide when a newly written message is "committed" enough to hand out to consumers? When a message is produced and is stored to the disk of the leader, the message is not considered "committed" until it h

Pull Request emails are not being sent to the mailing list

Since Dec 21st, it seems that email notifications of Github pull requests are not being sent to the dev mailing list. Is this a bug, or have PR emails moved to somewhere else? If it’s a bug, can we look into this? The last email PR that was sent was this one: http://mail-archives.apache.org/mod

Re: [DISCUSS] Kafka 2.2.0 in February 2018

Matthias, You said “Friday 2/14”, but 2/14 is this Thursday. -James Sent from my iPhone > On Feb 11, 2019, at 2:31 PM, Matthias J. Sax wrote: > > Hello, > > this is a short reminder, that feature freeze for AK 2.2 release is end > of this week, Friday 2/14. > > Currently, there are two bl

Re: [ANNOUNCE] New Committer: Randall Hauch

Congrats, Randall! Well deserved! -James Sent from my iPhone > On Feb 14, 2019, at 6:16 PM, Guozhang Wang wrote: > > Hello all, > > The PMC of Apache Kafka is happy to announce another new committer joining > the project today: we have invited Randall Hauch as a project committer and > he has

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats!! -James Sent from my iPhone > On Apr 18, 2019, at 2:35 PM, Guozhang Wang wrote: > > Hello Everyone, > > I'm glad to announce that Matthias J. Sax is now a member of Kafka PMC. > > Matthias has been a committer since Jan. 2018, and since then he continued > to be active in the commu

Re: UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

Alexandre, You are right that this is a problem. There is a JIRA on this from a while back. https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4680 I don’t think anyone is currently working on it right now. -James Sent from my iPhone > On Aug 13, 2019, at 1:17 AM, Alexandre

Re: Minimum Replication Factor

The create.topic.policy stuff I think is only used as part of the new CreateTopic broker API that's coming in 0.11. That's one of the administrative APIs which let you create topics by talking directly to the broker, without needing to talk to zookeeper directly. So this means your brokers wil

Re: Mirroring multiple clusters into one

I'm not sure what the "official" recommendation is. At TiVo, we *do* run all our mirrormakers near the target cluster. It works fine for us, but we're still fairly inexperienced, so I'm not sure how strong of a data point we should be. I think the thought process is, if you are mirroring from a

Re: Mirroring multiple clusters into one

art with "mirror." This prevents us from creating mirroring loops. > Thanks. > --Vahid > > > > From: James Cheng > To: us...@kafka.apache.org > Cc: dev > Date: 07/06/2017 12:37 PM > Subject:Re: Mirroring multiple clusters into one > >

Re: Aggressive log compaction ratio appears to have no negative effect on log-compacted topics

Jeff, The discussion thread from a while back on KIP-58 has some discussion around "log.cleaner.min.cleanable.ratio". KIP-58 page: https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable Discussion thread (linked off that page): http://mail-archives.ap

Re: [ANNOUNCE] New Kafka PMC member Jason Gustafson

Congrats Jason! -James > On Jul 11, 2017, at 10:32 PM, Guozhang Wang wrote: > > Hi Everyone, > > Jason Gustafson has been very active in contributing to the Kafka community > since he became a Kafka committer last September and has done lots of > significant work including the most recent exac

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

Congrats Ismael! -James > On Jul 5, 2017, at 1:55 PM, Jun Rao wrote: > > Hi, Everyone, > > Ismael Juma has been active in the Kafka community since he became > a Kafka committer about a year ago. I am glad to announce that Ismael is > now a member of Kafka PMC. > > Congratulations, Ismael! >

Re: Consumer offsets partitions size much bigger than others

It's possible that the log-cleaning thread has crashed. That is the thread that implements log compaction. Look in the log-cleaner.log file in your kafka debuglog directory to see if there is any indication that it has crashed (error messages, stack traces, etc). What version of kafka are you u

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

+1 from me! -James > On Aug 8, 2017, at 5:24 PM, Ewen Cheslack-Postava wrote: > > Hi all, > > I posted a simple new KIP for a problem we see with a lot of users: > KIP-186: Increase offsets retention default to 7 days > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+o

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

Congrats Becket! -James > On Aug 23, 2017, at 10:20 PM, Joel Koshy wrote: > > Hi everyone, > > Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has > contributed significantly to several major patches, reviews and discussions > since. I am glad to announce that Becket is no

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks for the KIP, Randall. The KIP has one MBean per metric name. Can I suggest an alternate grouping? kafka.connect:type=connector-metrics,connector=([-.\w]+) connector-type connector-class connector-version status kafka.connect:type=task-metrics,connector=([-.

Re: 1.0.0 KIPs Update

Hi. I'm a little unclear on what types of things are subject to the "Feature Freeze" today (Sept 20th). For changes such as https://github.com/apache/kafka/pull/3799 and https://issues.apache.org/jira/browse/KAFKA-3480

Re: 1.0.0 KIPs Update

oday. If it's new > functionality, then it depends on size, risk and importance. If you think > it's important and should target 1.0.0, feel free to make your case in the > JIRA or PR. > > Thanks, > Ismael > > On Wed, Sep 20, 2017 at 10:59 PM, James Cheng wrote:

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

We would find this KIP very useful. Our particular use case falls into the "application mistakes" portion of the KIP. We are storing source of truth data in log compacted topics, similar to the Confluent Schema Registry. One situation we had recently was a misbehaving application. It sent data

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

> On May 16, 2016, at 9:21 PM, Eric Wasserman wrote: > > Gwen, > > For simplicity, the example I gave in the gist is for a single table with a > single partition. The salient point is that even for a single topic with one > partition there is no guarantee without the feature that one will be

Re: [DISCUSS] scalability limits in the coordinator

> On May 23, 2016, at 10:59 AM, Jason Gustafson wrote: > > 2. Maybe there's a better way to lay out the assignment without needing to > explicitly repeat the topic? For example, the leader could sort the topics > for each member and just use an integer to represent the index of each > topic with

Re: [VOTE] KIP-58 - Make Log Compaction Point Configurable

I'm not sure what are the rules for who is allowed to vote, but I'm: +1 (non-binding) on the proposal I agree that the "log.cleaner.min.compaction.lag.ms" name is a little confusing. I like Becket's "log.cleaner.compaction.delay.ms", or something similar. The KIP describes it as the portion of

Can I get a code review on my PR for KAFKA-3809? "Auto-generate documentation for topic-level configuration"

Hi Kafka devs, Can I get a code review on my pull request for https://issues.apache.org/jira/browse/KAFKA-3809? The code has been waiting for review for about a week. I think I clicked everything correctly in JIRA/Github to mark it ready for review, but it’s possible I missed something. Than

Re: Can I get a code review on my PR for KAFKA-3809? "Auto-generate documentation for topic-level configuration"

Hi Kafka devs, Checking in again. Can I get a code review on this? Thanks, -James > On Jun 30, 2016, at 1:51 PM, James Cheng wrote: > > Hi Kafka devs, > > Can I get a code review on my pull request for > https://issues.apache.org/jira/browse/KAFKA-3809? > > The c

Re: [DISCUSS] KIP-82 - Add Record Headers

his also customizable makes it a bit >>>>>> harder. >>>>>>>>> Think about making the whole message record custom serializable, >>>> this >>>>>>>> would >>>>>>>>> make it fairly tricky (though it would not b

Re: [DISCUSS] Deprecating the old consumers in trunk

Sorry to self-plug, but I wrote a blog post that talks about this, with respect to mirrormaker. I came to the same 3 solutions that Onur described. https://logallthethings.com/2016/10/07/mirrormaker-gotchas-when-moving-from-the-old-consumer-to-the-new-consumer/

Re: [DISCUSS] Deprecating the old consumers in trunk

follows: >>>> >>>> - Add a “migration mode” to the old consumer that disables its >>> fetchers >>>> and disables offset commits. i.e., the consumers rebalance and own >>>> partitions but do basically nothing. >>&g

Re: [DISCUSS] KIP-82 - Add Record Headers

> On Dec 2, 2016, at 4:57 PM, Michael Pearce wrote: > > Hi Jun. > > RE Mirroring, > > [...] > > Lastly around mirroring we have a partionKey field, as the key used for > portioning logic != compaction key all the time but we want to preserve it > for when we mirror so that if source cluster

Re: Kafka controlled shutdown hangs when there are large number of topics in the cluster

> On Dec 19, 2016, at 10:13 AM, Gwen Shapira wrote: > > Can you try setting auto.leader.rebalance.enable=false in your > configuration (for all brokers) and see if it solves this problem? > We've had some reports regarding this feature interfering with > controlled shutdown. > Gwen, are there

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

> On Feb 23, 2017, at 10:03 PM, Jaikiran Pai wrote: > > (Re)posting this from the user mailing list to dev mailing list, hoping for > some inputs from the Kafka dev team: > > We are on Kafka 0.10.0.1 (server and client) and use Java consumer/producer > APIs. We have an application where we cr

Re: Trying to understand design decision about producer ack and min.insync.replicas

t; > All: > > - *receive.buffer.bytes=-1 (was 102400) > - *send.buffer.bytes=-1 (was 102400) > > > > > On Fri, Feb 3, 2017 at 2:03 AM, Ismael Juma wrote: > >> I'd be in favour too. >> >> Ismael >> >> On 3 Feb 2017 7:33 am, "

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

x27;ll just catch it and repeat the call for the >> fixed number of times. >> >> Thanks again, this was really helpful - I was running out of ideas to come >> up with a clean enough workaround to this problem. Your explanation has >> given me a couple of ideas

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Mayuresh, The Compatibility/Migration section says to upgrade the clients first, before the brokers. Are you talking about implementation or deployment? Do you mean to implement the client changes before the broker changes? That would imply that it would take 2 Kafka releases to implement this

Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled from True to False

+1 I agree with Gwen that this needs to be well communicated to users. Anyone who is using the current default may see their brokers start to go exit when they upgrade to this new version. On a somewhat related subject, occasionally I'll need to look up how my Kafka brokers are configured. Typ

Re: [jira] [Commented] (KAFKA-4497) log cleaner breaks on timeindex

> On Jan 5, 2017, at 3:47 AM, Edoardo Comar wrote: > > "Jiangjie Qin (JIRA)" wrote on 05/01/2017 08:06:58: > >>[ https://issues.apache.org/jira/browse/KAFKA-4497? >> page=com.atlassian.jira.plugin.system.issuetabpanels:comment- >> tabpanel&focusedCommentId=15800696#comment-15800696 ] >>

Re: [ANNOUNCE] New committer: Grant Henke

Congrats, Grant!! -James > On Jan 11, 2017, at 11:51 AM, Gwen Shapira wrote: > > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews, countless great > comments on discu

Re: Is this a bug or just unintuitive behavior?

It's >>> debatable >>>> whether it would even need a KIP. >>>> >>>> We have other settings in MM where we override them if they aren't set >>>> explicitly but we don't want the normal defaults. Most are producer

Re: [DISCUSS] KIP-115: Enforce offsets.topic.replication.factor

> On Jan 25, 2017, at 9:26 PM, Joel Koshy wrote: > > already voted, but one thing worth considering (since this KIP speaks of > *enforcement*) is desired behavior if the topic already exists and the > config != existing RF. > Yeah, I'm curious about this too. -James > On Wed, Jan 25, 2017 at

Re: [VOTE] KIP-115: Enforce offsets.topic.replication.factor

+1 > On Jan 25, 2017, at 9:48 PM, Ewen Cheslack-Postava wrote: > > +1 > > On Wed, Jan 25, 2017 at 9:23 PM, Joel Koshy wrote: > >> +1 >> >> >> On Wed, Jan 25, 2017 at 4:41 PM, Jason Gustafson >> wrote: >> >>> +1 >>> >>> On Wed, Jan 25, 2017 at 4:39 PM, Dong Lin wrote: >>> +1

Re: Trying to understand design decision about producer ack and min.insync.replicas

> On Jan 27, 2017, at 12:18 AM, Ewen Cheslack-Postava wrote: > > On Thu, Jan 26, 2017 at 4:23 PM, Luciano Afranllie > wrote: > >> I was thinking about the situation where you have less brokers in the ISR >> list than the number set in min.insync.replicas. >> >> My idea was that if I, as an adm

Re: Trying to understand design decision about producer ack and min.insync.replicas

t, they may want a stronger guarantee for, e.g., sometime > like chat messages, where they want to have a permanent record of them in > all circumstances. > > -Ewen > > On Fri, Jan 27, 2017 at 12:59 AM, James Cheng wrote: > >> >>> On Jan 27, 2017, at 12:18

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Woohoo! Thanks for running the release, Ewen! -James > On Feb 22, 2017, at 12:33 AM, Ewen Cheslack-Postava wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.2.0. This is a feature release which includes the completion > of 15 KIPs, over 200 bug fix

Re: Kafka Mailing List Filters

Thanks Grant! These are awesome! I can actually see stuff now! :) -James > On Apr 3, 2016, at 9:41 PM, Grant Henke wrote: > > The Kafka development mailing list can be a lot to keep up with. To cope, I > have created a few filters to separate the most important communication > from the noise. I

Re: [ANNOUNCE] New Kafka Committer Ewen Cheslack-Postava

Congrats!! -James > On Dec 8, 2015, at 11:37 AM, Neha Narkhede wrote: > > I am pleased to announce that the Apache Kafka PMC has voted to > invite Ewen Cheslack-Postava as a committer and Ewen has accepted. > > Ewen is an active member of the community and has contributed and reviewed > numerous

Re: [DISCUSS] KIP-42: Add Producer and Consumer Interceptors

Anna, I'm trying to understand a concrete use case. It sounds like producer interceptors could be used to implement part of LinkedIn's Kafak Audit tool? https://engineering.linkedin.com/kafka/running-kafka-scale Part of that is done by a wrapper library around the kafka producer that keeps a c

Does kstreams have a key-value store that is backed by Kafka?

Hi, I think I've heard that the upcoming Kafka Streams has something like Samza's local state key-value database. That is, something that looks like a key-value store that saves all writes to Kafka, and that can then later be reinitialized from Kafka. Does such a thing exist? Can someone point

Please add me to the contributor list

Hi, I'd like to be able to file and assign JIRAs to myself to make some docs. Can you please add me to the contributor list? My Apache JIRA username is wushujames. Thanks, -James This email and any attachments may contain confidential and privileged material

Re: Please add me to the contributor list

, there is both: * http://kafka.apache.org/documentation.html#monitoring * https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics And the two lists are different. -James > Guozhang > > On Fri, Feb 5, 2016 at 9:51 AM, James Cheng wrote: > >> Hi, >> >> I

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

I agree with Sriram and Martin. Kafka is already about providing streams of data, and so Kafka Streams or anything like that is confusing to me. This new library is about making it easier to process the data. -James On Jul 30, 2015, at 9:38 AM, Aditya Auradkar wrote: > Personally, I prefer K

How can I start writing a Copycat connector?

Hi, I'd like to write a Copycat connector. I saw on Github that the Copycat data APIs have been committed to trunk. How can I get started? I've never developed against trunk before. Do I git clone from trunk, or is there a pre-built set of jars from trunk that I can use? I see that there is a

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

This KIP would definitely address a gap in the current functionality, where you currently can't have a tombstone with any associated content. That said, I'd like to talk about use cases, to make sure that this is in fact useful. The KIP should be updated with whatever use cases we come up with.

Re: [DISCUSS] KIP-82 - Add Record Headers

> On Oct 25, 2016, at 10:23 PM, Michael Pearce wrote: > > Hi All, > > In case you hadn't noticed re the compaction issue for non-null values i have > created a separate KIP-87, if you could all contribute to its discussion > would be much appreciated. > > https://cwiki.apache.org/confluence/

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Michael, What information would you want to include on a delete tombstone, and what would you use it for? -James Sent from my iPhone > On Oct 27, 2016, at 9:17 PM, Michael Pearce wrote: > > Hi Jay, > > I think use case that is the issue that Konstantin mentioned in the kip-82 > thread , an

Re: [DISCUSS] KIP-82 - Add Record Headers

You would use avro to deserialize the container, and then potentially use a different deserializer for the payload. Using avro would potentially reduce the overhead of the container format, and let you use complex types in your headers. However, this would mean people would still have to use avro for

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

Congrats, Becket! -James > On Oct 31, 2016, at 10:35 AM, Joel Koshy wrote: > > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > committer and we are pleased to announce that he has accepted! > > Becket has made significant contributions to Kafka over the last two years

Re: [DISCUSS] KIP-82 - Add Record Headers

Let me talk about the container format that we are using here at TiVo to add headers to our Kafka messages. Just some quick terminology, so that I don't confuse everyone. I'm going to use "message body" to refer to the thing returned by ConsumerRecord.value() And I'm going to use "payload" to re

Re: [DISCUSS] KIP-82 - Add Record Headers

ed to ever message but we > don’t want to recode these frameworks. Same answer as above. I think this is just a serialization format. You would use Kafka Streams, but would provide your own serializer/deserializer. Same thing applies to Kafka Connect. -James > > > On 10/29/16, 8:09

How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

I have a JIRA that I am working on, and a pull request available for it. [KAFKA-10473] Website is missing docs on JMX metrics for partition size-on-disk (kafka.log:type=Log,name=*) https://issues.apache.org/jira/browse/KAFKA-10473 https://github.com/apache/kafka/pull/9276 The "Contributing Code

Re: How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

eady for review. Do this by clicking the "Submit Patch" button in JIRA, and then in the resulting dialog, click "Submit Patch". -James > On Sep 9, 2020, at 6:24 PM, John Roesler wrote: > > Hi James, > > I think the button on Jira says “Add Patch” or somethin

Re: How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

n. It’s a > separate account from Jira. > > Thanks, > John > >> On Wed, Sep 9, 2020, at 20:45, James Cheng wrote: >> Thanks John. That worked. >> >> I clicked the button that says "Submit Patch", and a dialog box popped >> up. I didn

Re: How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

til now. I've granted you > access. > > Thanks, > -John > > On Wed, 2020-09-09 at 21:44 -0700, James Cheng wrote: >> Thanks John. My wiki user ID is wushujames >> >> -James >> >> Sent from my iPhone >> >>> On Sep 9, 2020, at

Can I get a review for a documentation update (KAFKA-10473)?

Hi, Can I get a review from one of the commiters for this documentation update? I am adding docs for the following JMX metrics: kafka.log,type=Log,name=Size kafka.log,type=Log,name=NumLogSegments kafka.log,type=Log,name=LogStartOffset kafka.log,type=Log,name=LogEnd

  1   2   3   4   >