Streams Processing meetup on Wednesday, February 5, 2020 at LinkedIn, Sunnyvale

*[bcc: (users,dev)@kafka.apache.org ]* Hello, The Streams Infra team invites you to attend the Streams Processing meetup to be held on Wednesday, February 5, 2020. This meetup will focus on Apache Kafka, Apache Samza and related streaming technologies. *Where*: Unify Con

Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

+1 On Thu, Oct 24, 2019 at 9:33 PM radai wrote: > Hello, > > I'd like to initiate a vote on KIP-514. > > links: > the kip - > https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer > the PR - https://github.com/apache/kafka/pull/7569 > > Than

Streams meetup at LinkedIn Sunnyvale, 6pm, Thursday, October 3, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infra team invites you to attend a Streams Processing meetup on Thursday, October 3, 2019 at LinkedIn's Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related streaming technologies

Streams Meetup at LinkedIn Sunnyvale, 6pm, Wednesday, March 20, 2019

*[bcc: (users,dev)@kafka.apache.org ]* Hi everyone, The Streams Infrastructure team at LinkedIn invites you to attend a Streams Processing meetup on Wednesday, March 20 at LinkedIn’s Sunnyvale campus. (This meetup focuses on Apache Kafka, Apache Samza, and related streami

Re: [VOTE] KIP-354 Time-based log compaction policy

; > > > wrote: > > > > > > > > > > > > > > > Thanks Dong. > > > > > > > > I have updated the KIP. > > > > > > > > > > > > > > > > Xiongqi (Wesley) Wu > > > >

Re: [VOTE] KIP-354 Time-based log compaction policy

+1 with one suggestion on the proposed metric. You should probably include the unit. So for e.g., max-compaction-delay-secs. Joel On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu wrote: > bump > Xiongqi (Wesley) Wu > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu wrote: > > > > > Thanks Eno, Brett, D

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

gt; >> > > >> > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > > > > > >> >> > > >> > > >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

I had some offline discussions with Lucas on this KIP. While it is much more work than the original proposals, separating the control plane entirely removes any interference with the data plane as summarized under the rejected alternatives section. Just a few minor comments: - Can you update t

Re: [VOTE] KIP-340: Allow kafka-reassign-partitions.sh and kafka-log-dirs.sh to take admin client property file

+1 On Tue, Jul 17, 2018 at 1:42 PM, Ted Yu wrote: > +1 > > On Tue, Jul 17, 2018 at 1:40 PM Jason Gustafson > wrote: > > > +1. This is useful (though the naming inconsistencies in the tools are > > vexing, as always). > > > > -Jason > > > > On Tue, Jul 17, 2018 at 12:24 PM, Dong Lin wrote: > >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

est > queue. > > 3. Controller to broker connection failed and the controller reconnected > to > > the broker. > > 4. Controller sends a request R2 to the broker > > 5. Broker receives R2 and add it to the head of the request queue. > > Now on the brok

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

@Mayuresh - I like your idea. It appears to be a simpler less invasive alternative and it should work. Jun/Becket/others, do you see any pitfalls with this approach? On Wed, Jul 18, 2018 at 12:03 PM, Lucas Wang wrote: > @Mayuresh, > That's a very interesting idea that I haven't thought before. >

[jira] [Resolved] (KAFKA-5098) KafkaProducer.send() blocks and generates TimeoutException if topic name has illegal char

[ https://issues.apache.org/jira/browse/KAFKA-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-5098. --- Resolution: Fixed > KafkaProducer.send() blocks and generates TimeoutException if topic name

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

+1 on the KIP. (I'm not sure we actually necessary to introduce the condition variables for the concern that Jun raised, but it's an implementation detail that we can defer to a discussion in the PR.) On Sat, Jul 14, 2018 at 10:45 PM, Lucas Wang wrote: > Hi Jun, > > I agree by using the conditi

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hey Becket - good point. Lucas and I were talking about this offline last week. It is true that there is only one request in flight for processing. However, there may be more during a controller failover but it should not be very high - basically the maximum number of controller failures that can o

[ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

Hi everyone, Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has contributed significantly to several major patches, reviews and discussions since. I am glad to announce that Becket is now a member of the Apache Kafka PMC. Congratulations Becket! Joel

Re: [VOTE] KIP-164 Add unavailablePartitionCount and per-partition Unavailable metrics

+1 On Thu, Jul 20, 2017 at 10:30 AM, Becket Qin wrote: > +1, Thanks for the KIP. > > On Thu, Jul 20, 2017 at 7:08 AM, Ismael Juma wrote: > > > Thanks for the KIP, +1 (binding). > > > > On Thu, Jun 1, 2017 at 9:44 AM, Dong Lin wrote: > > > > > Hi all, > > > > > > Can you please vote for KIP-164

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

+1 On Thu, Jul 13, 2017 at 12:24 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > > > > From: Dong Lin > To: dev@kafka.apache.org > Date: 07/12/2017 10:43 AM > Subject:Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster > > > > +1 (non-bindi

Re: [DISCUSS] KIP-168: Add TotalTopicCount metric per cluster

+1 on the original KIP I actually prefer TotalTopicCount because it makes it clearer that it is a cluster-wide count. OfflinePartitionsCount is global to the cluster (but it is fairly clear that the controller is SoT on that). TopicCount on the other hand could be misread as a local count since Par

Re: [VOTE] KIP-113 - Support replicas movement between log directories

+1 Few additional comments (most of which we discussed offline): - This was summarized in the “discuss” thread, but it is worth recording in the KIP itself that the LEO in DescribeDirsResponse is useful to measure progress of the move. - num.replica.move.threads defaults to #

Re: [VOTE] KIP-144: Exponential backoff for broker reconnect attempts

+1 On Mon, May 8, 2017 at 11:07 AM, Colin McCabe wrote: > +1 (non-binding) > > > > On Sat, May 6, 2017, at 11:13, Dana Powers wrote: > > +1 ! > > > > On May 6, 2017 4:49 AM, "Edoardo Comar" wrote: > > > > > +1 (non binding) > > > thanks > > > -- >

Re: [DISCUSS] KIP-143: Controller Health Metrics

Hi Ismael, > > What about a broker that is not the controller? Would you need a separate > > idle-not-controller state? > > > Do we need a separate state or can users just use the ActiveControllerCount > metric to check if the broker is the controller? > Sure - the ACC metric should be sufficien

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

+1 On Mon, May 8, 2017 at 8:30 AM, Roger Hoover wrote: > +1 > > Sent from my iPhone > > > On May 8, 2017, at 5:00 AM, Edoardo Comar wrote: > > > > +1 > > Many thanks Jun > > -- > > Edoardo Comar > > IBM MessageHub > > eco...@uk.ibm.com > > IBM UK

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

+1 On Thu, May 4, 2017 at 7:00 PM Becket Qin wrote: > Bump. > > On Tue, Apr 25, 2017 at 3:17 PM, Bill Bejeck wrote: > > > +1 > > > > On Tue, Apr 25, 2017 at 4:43 PM, Dong Lin wrote: > > > > > +1 (non-binding) > > > > > > On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin > > wrote: > > > > > > > Hi

Re: [DISCUSS] KIP-143: Controller Health Metrics

would be useful to add a new metrics for the controller queue >> size. >> > kafka.controller:type=ControllerStats,name=QueueSize >> > >> > 11. It would also be useful to know how long an event is waiting in the >> > controller queue before being processing. Perhaps

Re: [DISCUSS] KIP-143: Controller Health Metrics

Thanks for the KIP - couple of comments: - Do you intend to actually use yammer metrics? or use kafka-metrics and split the timer into an explicit rate and time? I think long term we ought to move off yammer and use kafka-metrics only. Actually either is fine, but we should ideally use only one in

Re: [VOTE] KIP-112 - Handle disk failure for JBOD

+1 Discussed a few edits/improvements with Dong. - Rather than a blanket (Error != None) condition for detecting offline replicas you probably want a storage exception-specific error code. - Definitely in favor of improvement #7 and it shouldn’t be too hard to do. When bouncing with a log direct

[jira] [Commented] (KAFKA-5011) Replica fetchers may need to down-convert messages during a selective message format upgrade

[ https://issues.apache.org/jira/browse/KAFKA-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964994#comment-15964994 ] Joel Koshy commented on KAFKA-5011: --- Yes that is correct - it is rare. I think

[jira] [Created] (KAFKA-5011) Replica fetchers may need to down-convert messages during a selective message format upgrade

Joel Koshy created KAFKA-5011: - Summary: Replica fetchers may need to down-convert messages during a selective message format upgrade Key: KAFKA-5011 URL: https://issues.apache.org/jira/browse/KAFKA-5011

Re: [VOTE] KIP-82 Add Record Headers

+1 On Tue, Mar 21, 2017 at 5:01 PM, Jason Gustafson wrote: > Thanks for the KIP! +1 (binding) from me. Just one nit: can we change > `Headers.header(key)` to be `Headers.lastHeader(key)`? It's not a > deal-breaker, but I think it's better to let the name reflect the actual > behavior as clearly

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

If we deprecate KafkaPrincipal, then the Authorizer interface will also need to change - i.e., deprecate the getAcls(KafkaPrincipal) method. On Tue, Feb 28, 2017 at 10:11 AM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Hi Jun/Ismael, > > Thanks for the comments. > > I agree. > What I w

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

> > Lets say we sent the batch over the wire and received a > RecordTooLargeException, how do we split it as once we add the message to > the batch we loose the message level granularity. We will have to > decompress, do deep iteration and split and again compress. right? This > looks like a perfor

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

> > the > > > > > value > > > > > > > length. > > > > > > > However, if we are adding record headers to the > end, > > we > > > > > would > > > > > > need to &g

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Regarding (2) - yes that's a good point. @Onur - I think the KIP should explicitly call this out. It is something that we did consider and decided against optimizing for. i.e., we just wrote that off as a minor caveat of the upgrade path in that there will be a few duplicates, but not too many give

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

I left a couple of comments/questions directly on the google-doc - I found it much more tractable for a proposal of this size to discuss in context within the doc. The permissions on the doc don't let everyone view co

Re: [DISCUSS] KIP-115: Enforce offsets.topic.replication.factor

already voted, but one thing worth considering (since this KIP speaks of *enforcement*) is desired behavior if the topic already exists and the config != existing RF. On Wed, Jan 25, 2017 at 4:30 PM, Dong Lin wrote: > +1 > > On Wed, Jan 25, 2017 at 4:22 PM, Ismael Juma wrote: > > > An important

Re: [VOTE] KIP-115: Enforce offsets.topic.replication.factor

+1 On Wed, Jan 25, 2017 at 4:41 PM, Jason Gustafson wrote: > +1 > > On Wed, Jan 25, 2017 at 4:39 PM, Dong Lin wrote: > > > +1 > > > > On Wed, Jan 25, 2017 at 4:37 PM, Ismael Juma wrote: > > > > > +1 (binding) > > > > > > Ismael > > > > > > On Thu, Jan 26, 2017 at 12:34 AM, Onur Karaman < > >

Stream processing meetup at LinkedIn (Sunnyvale) on Thursday, February 16 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus on Thursday, February 16 at 6pm. Please RSVP here (*only if you intend to attend in person*): https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/237171557/

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

+1 (for the record, I favor the rejected alternative of not awaiting low watermarks to go past the purge offset. I realize it offers a weaker guarantee but it is still very useful, easier to implement, slightly simpler API (no need to return a future) and you can still get access to the current lo

Re: [ANNOUNCE] New committer: Grant Henke

Hey Grant - congrats! On Thu, Jan 12, 2017 at 10:00 AM, Neha Narkhede wrote: > Congratulations, Grant. Well deserved! > > On Thu, Jan 12, 2017 at 7:51 AM Grant Henke wrote: > > > Thanks everyone! > > > > On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy > wrote: > > > > > Congratulations! > > > > >

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

think we have any formal guidance on the scenarios that you highlighted at this point so it may help to have a discussion on a separate thread and codify that in our docs under a new "Kafka message and protocol versioning" section. Thanks, Joel > > > > On Tue, Jan 10, 2017 at

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

t > targeting > > a certain major release in which the old consumer will be removed, I > don't > > see any harm in marking it as deprecated. > > There will be enough time to plan and implement the migration, if the > > community decides that's the way to

Re: [VOTE] KIP-105: Addition of Record Level for Sensors

+1 (although I added a minor comment on the discussion thread) On Mon, Jan 9, 2017 at 3:43 AM, Michael Noll wrote: > +1 (non-binding) > > On Fri, Jan 6, 2017 at 6:12 PM, Matthias J. Sax > wrote: > > > +1 > > > > On 1/6/17 9:09 AM, Neha Narkhede wrote: > > > +1 > > > > > > On Fri, Jan 6, 2017 at

Re: [DISCUSS] KIP-105: Addition of Record Level for Sensors

Didn't get a chance to review this earlier, but this LGTM. Minor comments: - The current name is fine, but I would have preferred calling it RecordingLevel to RecordLevel - first thing that comes to my mind with RecordLevel is Kafka records. - Under future work: it may be useful to allow specifying

Re: [VOTE] KIP-103: Separation of Internal and External traffic

+1 On Fri, Jan 6, 2017 at 4:53 PM, Ismael Juma wrote: > Hi all, > > Since a few people (including myself) felt that listener name was clearer > than protocol label, I updated the KIP to use that (as mentioned in the > discuss thread). Given that this is a minor change, I don't think we need > to

Re: [VOTE] KIP-108: Create Topic Policy

+1 Thanks for the KIP - this is a very useful feature. Thanks, Joel On Mon, Jan 9, 2017 at 2:15 AM, Edoardo Comar wrote: > Ismael, > thanks for the KIP I see it as quite useful in a managed cloud > environment. > > +1 (non-binding) > -- > Edoard

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

While I realize this only marks the old consumer as deprecated and not a complete removal, I agree that it is somewhat premature to do this prior to having a migration process implemented. Onur has described this in detail in the earlier thread: http://markmail.org/message/ekv352zy7xttco5s and I'm

Re: [VOTE] Vote for KIP-101 - Leader Epochs

(adding the dev list back - as it seems to have gotten dropped earlier in this thread) On Thu, Jan 5, 2017 at 6:36 PM, Joel Koshy wrote: > +1 > > This is a very well-written KIP! > Minor: there is still a mix of terms in the doc that references the > earlier LeaderGenerationRe

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

+1 on the KIP. I'll comment more on the formal PR. Also, can you also link a jira for this from the KIP? Thanks, Joel On Tue, Jan 3, 2017 at 11:14 AM, radai wrote: > I've just re-validated the functionality works - broker throttles under > stress instead of OOMs. > > at this point my branch (

Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled from True to False

+1 On Tue, Jan 3, 2017 at 10:54 AM, Ben Stopford wrote: > Hi All > > Please find the below KIP which proposes changing the setting > unclean.leader.election.enabled from true to false. The motivation for > this change is that it catches out new Kafka users who don’t realise the > default favours

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

+1 On Wed, Dec 21, 2016 at 10:26 AM, radai wrote: > +1 > > On Wed, Dec 21, 2016 at 9:51 AM, Dong Lin wrote: > > > +1 (non-binding) > > > > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin > wrote: > > > > > Hi, > > > > > > I want to start a voting thread on KIP-92 which proposes to add per > > > pa

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

> > > @Joel, > > I read over your wiki, and apart from the introduction of the notion of > journal partitions --whose pros and cons are already being discussed-- you > also introduce the notion of a 'producer group' which enables multiple > producers to participate in a single transaction. This is

Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

LGTM. However, can you comment on the effect of releasing ownership of partitions after a rebalance? For e.g., should it reset itself to (say) -1? or removed? This really applies to any per-partition metrics that we intend to maintain in the consumer. On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin w

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Just got some time to go through most of this thread and KIP - great to see this materialize and discussed!! I will add more comments in the coming days on some of the other "tracks" in this thread; but since Radai brought up the double-journaling approach that we had discussed I thought I would mo

[jira] [Resolved] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-4250. --- Resolution: Fixed > make ProducerRecord and ConsumerRecord extensi

[jira] [Reopened] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy reopened KAFKA-4250: --- cc [~becket_qin] [~nsolis] [~radai] > make ProducerRecord and ConsumerRecord extensi

[jira] [Updated] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-4250: -- Fix Version/s: 0.10.2.0 > make ProducerRecord and ConsumerRecord extensi

[jira] [Updated] (KAFKA-1911) Log deletion on stopping replicas should be async

[ https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1911: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Log deletion on stopping repli

Re: [DISCUSS] Deprecating the old consumers in trunk

Not sure it is worth doing, but a simple migration approach that avoids *service* downtime could be as follows: - Add a “migration mode” to the old consumer that disables its fetchers and disables offset commits. i.e., the consumers rebalance and own partitions but do basically nothing.

[jira] [Created] (KAFKA-4409) ZK consumer shutdown/topic event deadlock

Joel Koshy created KAFKA-4409: - Summary: ZK consumer shutdown/topic event deadlock Key: KAFKA-4409 URL: https://issues.apache.org/jira/browse/KAFKA-4409 Project: Kafka Issue Type: Bug

Re: any plans to switch to java 8?

http://markmail.org/message/gnrn5ccql7a2pmc5 We can bump that up to revisit the discussion. That thread didn't have any closure, but has a lot of background information. On Thu, Nov 10, 2016 at 10:37 AM, Sean McCauliff wrote: > Wait for JDK 9 which is supposed to be 4-5 months from now? > > Sean

[jira] [Commented] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645583#comment-15645583 ] Joel Koshy commented on KAFKA-4362: --- In this specific issue, the coordinato

[jira] [Commented] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645427#comment-15645427 ] Joel Koshy commented on KAFKA-4362: --- Sorry - missed this comment. It does not rec

[jira] [Commented] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[ https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645419#comment-15645419 ] Joel Koshy commented on KAFKA-4381: --- This is up to you but I definitely agree

Re: why cant SslTransportLayer be muted before handshake completion?

Sriharsha can validate this, but I think the reason is that if we allow muting/unmuting at will (via those public APIs) that can completely mess up the handshake itself. It should be possible to pause/resume the handshake if that's what you'r elooking for but I'm not sure it is worth it for the pur

[jira] [Updated] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-4362: -- Summary: Consumer can fail after reassignment of the offsets topic partition (was: Offset commits fail

[jira] [Commented] (KAFKA-4362) Offset commits fail after a partition reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15624520#comment-15624520 ] Joel Koshy commented on KAFKA-4362: --- Btw, the summary doesn't make it clear

[jira] [Created] (KAFKA-4362) Offset commits fail after a partition reassignment

Joel Koshy created KAFKA-4362: - Summary: Offset commits fail after a partition reassignment Key: KAFKA-4362 URL: https://issues.apache.org/jira/browse/KAFKA-4362 Project: Kafka Issue Type: Bug

[ANNOUNCE] New committer: Jiangjie (Becket) Qin

The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a committer and we are pleased to announce that he has accepted! Becket has made significant contributions to Kafka over the last two years. He has been deeply involved in a broad range of KIP discussions and has contributed sev

Re: [DISCUSS] KIP-81: Max in-flight fetches

Agreed with this approach. One detail to be wary of is that since we multiplex various other requests (e.g., heartbeats, offset commits, metadata, etc.) over the client that connects to the coordinator this could delay some of these critical requests. Realistically I don't think it will be an issue

[VOTE] KIP-47 - Add timestamp-based log deletion policy

> > - It seems that the consumer will need to write log.retention.min.timestamp > periodically to zookeeper as dynamic configuration of the topic, so that > broker can pick up log.retention.min.timestamp. However, this introduces > dependency of consumer on zookeeper which is undesirable. Note that

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

I'm not sure why it would be useful, but it should be theoretically possible if the attribute bit alone is enough to mark a tombstone. OTOH, we could consider that as invalid if we wish. These are relevant details that I think should be added to the KIP. Also, in the few odd scenarios that I menti

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

A magic byte bump would be required for the addition of new fields; or removal of existing fields. Changing the interpretation of an existing field (e.g., switching from absolute to relative offsets) almost always needs a magic byte bump as well. One concern Nacho alluded to (I think) is if a newe

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

+1 - I was thinking the exact same thing. On Tue, Oct 25, 2016 at 2:52 PM, Jun Rao wrote: > One of the main reasons for retaining messages on the broker after > consumption is to support replay. A common reason for replay is to fix and > application error. So, it seems that it's a bit hard to de

Re: [VOTE] KIP-73 - Replication Quotas

PM, Ben Stopford wrote: > > > > > Thanks everyone. It looks like this KIP has now been accepted. > > > > > > There is a corresponding PR <https://github.com/apache/kafka/pull/1776 > > > > > for the implementation also. > > > > > >

[jira] [Resolved] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-4250. --- Resolution: Fixed Assignee: radai rosenblatt Reviewer: Joel Koshy Fix Version

[jira] [Commented] (KAFKA-1351) String.format is very expensive in Scala

[ https://issues.apache.org/jira/browse/KAFKA-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583851#comment-15583851 ] Joel Koshy commented on KAFKA-1351: --- Yes - this is still an issue. cc [~ns

Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, November 2 at 6pm

Hi everyone, We would like to invite you to a Stream Processing Meetup at LinkedIn’s Sunnyvale campus on Wednesday, November 2 at 6pm. Please RSVP here (if you intend to attend in person): http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/234454163 We have the following three talks

[jira] [Resolved] (KAFKA-4025) build fails on windows due to rat target output encoding

[ https://issues.apache.org/jira/browse/KAFKA-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-4025. --- Resolution: Fixed Assignee: radai rosenblatt Reviewer: Joel Koshy Fix Version

[jira] [Updated] (KAFKA-4293) ByteBufferMessageSet.deepIterator burns CPU catching EOFExceptions

[ https://issues.apache.org/jira/browse/KAFKA-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-4293: -- Assignee: radai rosenblatt It turns out we should be able to handle all of our current codecs by re

[jira] [Updated] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[ https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-3175: -- Resolution: Fixed Reviewer: Joel Koshy Status: Resolved (was: Patch Available) > to

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Jay, Couple of comments inline: One of things that has helped keep Kafka simple is not adding in new > abstractions and concepts except when the proposal is really elegant and > makes things simpler. > I don't quite see how this impacts simplicity because (per your taxonomy) the scope is "com

[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553228#comment-15553228 ] Joel Koshy commented on KAFKA-4254: --- It seems (2) would still help as there are

Re: [DISCUSS] KIP-82 - Add Record Headers

@Nacho > > - Brokers can't see the headers (part of the "V" black box)> > > (Also, it would be nice if we had a way to access the headers from the > > brokers, something that is not trivial at this time with the current > broker > > architecture). > > I think this can be addressed with broker i

Re: Snazzy new look to our website

Looks great! On Tue, Oct 4, 2016 at 4:38 PM, Guozhang Wang wrote: > The new look is great, thanks Derrick and Gwen! > > I'm wondering if we should still consider breaking "document.html" into > multiple pages and indexed as sub-topics on the left bar? > > > Guozhang > > > On Tue, Oct 4, 2016 at

Re: [DISCUSS] Fault injection tests for Kafka

Hi Gwen, I've also seen suggestions of using Jepsen for fault injection, but > I'm not familiar with this framework. > > What do you guys think? Write our own failure injection? or write > Kafka tests in Jepsen? > This would definitely add a lot of value and save a lot on release validation overh

[jira] [Commented] (KAFKA-4207) Partitions stopped after a rapid restart of a broker

[ https://issues.apache.org/jira/browse/KAFKA-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524443#comment-15524443 ] Joel Koshy commented on KAFKA-4207: --- I have a KIP draft that has been sitting ar

[jira] [Commented] (KAFKA-4178) Replication Throttling: Consolidate Rate Classes

[ https://issues.apache.org/jira/browse/KAFKA-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511604#comment-15511604 ] Joel Koshy commented on KAFKA-4178: --- [~junrao] sorry I don't remember, but

[jira] [Resolved] (KAFKA-4158) Reset quota to default value if quota override is deleted for a given clientId

[ https://issues.apache.org/jira/browse/KAFKA-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-4158. --- Resolution: Fixed Pushed to trunk. Not sure if we need this in 0.9 since there is an easy work-around

[jira] [Updated] (KAFKA-4158) Reset quota to default value if quota override is deleted for a given clientId

[ https://issues.apache.org/jira/browse/KAFKA-4158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-4158: -- Fix Version/s: 0.10.1.0 > Reset quota to default value if quota override is deleted for a gi

Re: [VOTE] 0.10.1 Release Plan

+1 On Tue, Sep 13, 2016 at 2:40 PM, Jason Gustafson wrote: > Hi All, > > I'd like to start a vote on the release plan documented here: > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1. I > went ahead and included KIP-55 since Jun said it may have a chance, but > note that

[jira] [Commented] (KAFKA-4074) Deleting a topic can make it unavailable even if delete.topic.enable is false

[ https://issues.apache.org/jira/browse/KAFKA-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15471686#comment-15471686 ] Joel Koshy commented on KAFKA-4074: --- Yes - totally missed KAFKA-3175 > Del

Re: [VOTE] KIP-73 - Replication Quotas

+1 (sent some very minor edits to you off-thread) On Fri, Aug 19, 2016 at 1:21 AM, Ben Stopford wrote: > I’d like to initiate the voting process for KIP-73: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 73+Replication+Quotas confluence/display/KAFKA/KIP-

[jira] [Created] (KAFKA-4074) Deleting a topic can make it unavailable even if delete.topic.enable is false

Joel Koshy created KAFKA-4074: - Summary: Deleting a topic can make it unavailable even if delete.topic.enable is false Key: KAFKA-4074 URL: https://issues.apache.org/jira/browse/KAFKA-4074 Project: Kafka

[jira] [Resolved] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[ https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-4050. --- Resolution: Fixed Fix Version/s: 0.10.0.2 0.10.1.0 Pushed to trunk and

Re: [DISCUSS] KIP-73: Replication Quotas

e configurable, or could be derived, with pros and cons for > > each. > > > > But the more general problem is actually quite hard to reason about, so > > after some discussion we decided to settle on something simple, that we > > felt we could get working, and extend t

Re: [DISCUSS] KIP-73: Replication Quotas

gt; no > > > > extra delay to due throttling, right? For replica fetchers, the > default > > > > min.byte is 1. So, the response is only delayed if there is no byte > in > > > the > > > > response, which is what we want. > > > > > > > >

Re: [ANNOUNCE] New Kafka PMC member Gwen Shapira

Congrats! On Wed, Aug 17, 2016 at 9:28 PM, Sriram Subramanian wrote: > Congratulations Gwen! > > On Wed, Aug 17, 2016 at 9:24 PM, Neha Narkhede wrote: > > > Congratulations and welcome, Gwen! > > On Wed, Aug 17, 2016 at 6:44 PM Jun Rao wrote: > > > > > Hi, Everyone, > > > > > > Gwen Shapira ha

[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[ https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425906#comment-15425906 ] Joel Koshy commented on KAFKA-4050: --- [~toddpalino] I had left a comment about thi

[jira] [Commented] (KAFKA-4050) Allow configuration of the PRNG used for SSL

[ https://issues.apache.org/jira/browse/KAFKA-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423839#comment-15423839 ] Joel Koshy commented on KAFKA-4050: --- A stack trace should help further clarify. (

Re: [VOTE] KIP:71 Enable log compaction and deletion to co-exist

+1 On Mon, Aug 15, 2016 at 4:58 PM, Ewen Cheslack-Postava wrote: > +1 (binding) > > Thanks, > -Ewen > > On Mon, Aug 15, 2016 at 4:26 PM, Jun Rao wrote: > > > Thanks for the proposal. +1 > > > > Jun > > > > On Mon, Aug 15, 2016 at 6:20 AM, Damian Guy > wrote: > > > > > I would like to initiate

  1   2   3   4   5   6   7   8   9   10   >