Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

During the 'consumer-transform-produce' cycle, if the consumer app needs to update an external data store (e.g. update RocksDB in Kafka streams), and that update is not idempotent (e.g. increment a counter in RocksDB), how do you make that update part of Kafka transaction? On Wed, Nov 30, 2016 at

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711144#comment-15711144 ] Abhi commented on KAFKA-1194: - This is my server.log that showed this ? [2016-12-01 11:38:55,4

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711131#comment-15711131 ] Abhi commented on KAFKA-1194: - broker.id=2 port:9095 num.network.threads=3 num.io.threads=8 s

[GitHub] kafka pull request #2164: KAFKA-4438; Cross compile to Scala 2.12.0

Github user leachbj closed the pull request at: https://github.com/apache/kafka/pull/2164 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[jira] [Commented] (KAFKA-4438) BACKPORT - Add scala 2.12 support

[ https://issues.apache.org/jira/browse/KAFKA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711065#comment-15711065 ] ASF GitHub Bot commented on KAFKA-4438: --- Github user leachbj closed the pull request

[jira] [Resolved] (KAFKA-4438) BACKPORT - Add scala 2.12 support

[ https://issues.apache.org/jira/browse/KAFKA-4438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4438. -- Resolution: Fixed Fix Version/s: 0.10.2.0 Issue resolved by pull request 2164 [https://gi

[jira] [Commented] (KAFKA-3637) Add method that checks if streams are initialised

[ https://issues.apache.org/jira/browse/KAFKA-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711053#comment-15711053 ] ASF GitHub Bot commented on KAFKA-3637: --- Github user asfgit closed the pull request

[jira] [Comment Edited] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15702384#comment-15702384 ] Json Tu edited comment on KAFKA-4447 at 12/1/16 6:24 AM: - [~hachik

[GitHub] kafka pull request #2135: KAFKA-3637: Added initial states

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2135 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4447) Controller resigned but it also acts as a controller for a long time

[ https://issues.apache.org/jira/browse/KAFKA-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711052#comment-15711052 ] Json Tu commented on KAFKA-4447: [~hachikuji] could you help review it，thanks. > Controll

[jira] [Resolved] (KAFKA-3637) Add method that checks if streams are initialised

[ https://issues.apache.org/jira/browse/KAFKA-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3637. -- Resolution: Fixed Issue resolved by pull request 2135 [https://github.com/apache/kafka/pull/2135

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Thanks for your comment, I updated the document. Let me know if it is clear now. Apurva On Wed, Nov 30, 2016 at 9:42 PM, Onur Karaman wrote: > @Apurva yep that's what I was trying to say. > > Original message: > If there is already an entry with the AppID in the mapping, increment the > epoch n

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

@Apurva yep that's what I was trying to say. Original message: If there is already an entry with the AppID in the mapping, increment the epoch number and go on to the next step. If there is no entry with the AppID in the mapping, construct a PID with initialized epoch number; append an AppID messa

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

The first line in step 2 of that section is: "If there is already an entry with the AppID in the mapping, increment the epoch number and go on to the next step." Are you suggesting that it be made explicit that 'increment the epoch number' includes persisting the updated value to the log? Thanks,

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Nice google doc! Probably need to go over the google doc a few more times, but a minor comment from the first pass: In Transaction Coordinator Request Handling ( https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#bookmark=id.jro89lml46du), step 2 mentions that if

Jenkins build is back to normal : kafka-trunk-jdk7 #1720

See

When does Produce response get sent back to client if acks is set to -1?

Hello guys, Due to the laziness of updating high watermark and async replicating, please guide me when the Produce response will be sent back to client if I set to acks to -1? Before all the followers have updated their LEOs or after ? Thanks.

Dose kafaka change network configuration?

Hi guys, After deploying kafka cluster on my machines, I found output from java.net.Inetaddress getCanonicalHostName() returns IP address instead of hostname. Before deploying kafka cluster, it really return hostname. As this is system wide change, does kafka change any system co

[jira] [Created] (KAFKA-4472) offsetRetentionMs miscalculated in GroupCoordinator

Jason Aliyetti created KAFKA-4472: - Summary: offsetRetentionMs miscalculated in GroupCoordinator Key: KAFKA-4472 URL: https://issues.apache.org/jira/browse/KAFKA-4472 Project: Kafka Issue Typ

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi, Michael, In order to answer the first two questions, it would be helpful if we could identify 1 or 2 strong use cases for headers in the space for third-party vendors. For use cases within an organization, one could always use other approaches such as company-wise containers to get around w/o

[jira] [Commented] (KAFKA-4465) Create docker image and scripts for running tests locally

[ https://issues.apache.org/jira/browse/KAFKA-4465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710626#comment-15710626 ] ASF GitHub Bot commented on KAFKA-4465: --- GitHub user raghavgautam opened a pull requ

[GitHub] kafka pull request #2197: KAFKA-4465: Create docker image and scripts for ru...

GitHub user raghavgautam opened a pull request: https://github.com/apache/kafka/pull/2197 KAFKA-4465: Create docker image and scripts for running tests locally You can merge this pull request into a Git repository by running: $ git pull https://github.com/raghavgautam/kafka tr

[jira] [Commented] (KAFKA-3910) Cyclic schema support in ConnectSchema and SchemaBuilder

[ https://issues.apache.org/jira/browse/KAFKA-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710509#comment-15710509 ] ASF GitHub Bot commented on KAFKA-3910: --- GitHub user shikhar opened a pull request:

[GitHub] kafka pull request #2196: KAFKA-3910: prototype of another approach to cycli...

GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2196 KAFKA-3910: prototype of another approach to cyclic schemas You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka KAFKA-3910 Alternativ

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Thanks for your comments, let me deal with your second point regarding merging the __consumer-offsets and transactions topic. Needless to say, we considered doing this, but chose to keep them separate for the following reasons: 1. Your assumption that group.id and transaction.app.id can be the

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

Hi all, I updated the KIP to include a command to print out the version information of brokers (KAFKA-4457). This will be a useful command for administrators. best, Colin On Wed, Nov 30, 2016, at 11:17, Colin McCabe wrote: > Thanks, Ashish. I think the idea of having the client make an > ApiV

Build failed in Jenkins: kafka-trunk-jdk7 #1719

See Changes: [wangguoz] KAFKA-4271: Fix the server start script for Windows 32-bit OS -- [...truncated 14305 lines...] org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNega

[jira] [Resolved] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy resolved KAFKA-4250. --- Resolution: Fixed > make ProducerRecord and ConsumerRecord extensible > --

[jira] [Updated] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] radai rosenblatt updated KAFKA-4250: Fix Version/s: (was: 0.10.1.1) > make ProducerRecord and ConsumerRecord extensible > ---

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Thanks for initiating this KIP! I think it is well written and I'm excited to see the first step towards adding an important feature in Kafka. I had a few initial thoughts on the KIP, mostly not as deeply thought through than what you've done - 1. Perhaps you’ve thought about how this would work

[jira] [Reopened] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy reopened KAFKA-4250: --- cc [~becket_qin] [~nsolis] [~radai] > make ProducerRecord and ConsumerRecord extensible > ---

[jira] [Updated] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-4250: -- Fix Version/s: 0.10.2.0 > make ProducerRecord and ConsumerRecord extensible > --

[jira] [Updated] (KAFKA-4250) make ProducerRecord and ConsumerRecord extensible

[ https://issues.apache.org/jira/browse/KAFKA-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] radai rosenblatt updated KAFKA-4250: Fix Version/s: (was: 0.10.2.0) 0.10.1.1 > make ProducerRecord and Con

Re: [DISCUSS] KIP-93: Improve invalid timestamp handling in Kafka Streams

Thanks for this clarification (and your +1) I completely agree and just want to add my thoughts: 1. Yes, it is a backward compatible change but as I discusses with others, we want accept this for now. All previous releases did contain non-compatible changes for Kafka Streams, too. And as Kafka St

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

Right now, there is only one config value and I am open to better suggestions. We did not go for batch.mode=true/false because we might want to have auto stop at specific stop-offsets or stop-timestamp later on. So we can extend the parameter with new values like autostop.at=timestamp in combinati

Re: [DISCUSS] KIP 88: OffsetFetch Protocol Update

Hi Jason, Thanks for the feedback. I believe the proposal now covers what you're suggesting. I also like the AdminClient option better, and that what made me update the KIP to take that approach, and suggest the modification to OffsetFetch protocol as you pointed out. I'll wait until Monday, a

Re: [DISCUSS] KIP-96 - Add per partition metrics for in-sync and assigned replica count

+1 (non-binding) On Wed, Nov 30, 2016 at 1:20 PM, Xavier Léauté wrote: > FYI, Based on internal feedback I renamed AssignedReplicasCount to simply > be called ReplicasCount. > > On Tue, Nov 29, 2016 at 7:56 PM Neha Narkhede wrote: > > > This seems useful, +1 > > > > On Tue, Nov 29, 2016 at 5:39

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

+1 (non-binding) On Wed, Nov 30, 2016 at 2:00 PM, Jason Gustafson wrote: > +1. Thanks for the KIP! > > On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira wrote: > > > +1 (binding) > > > > On Wed, Nov 30, 2016 at 1:34 PM, Xavier Léauté > > wrote: > > > Based on the feedback KIP-96 seems pretty uncon

Re: [DISCUSS] KIP 88: OffsetFetch Protocol Update

Hey Vahid, Sorry to getting to this so late. I don't have a strong preference for keeping it out of the KafkaConsumer API. Having a no-arg committed() method which returns all the committed offsets seems natural and could be useful. Since no one has asked for it, however, I'd lean toward supportin

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

I agree that the metadata topic is required to build a batching semantic that is intuitive. One question on the config - autostop.at I see one value for it - eol. What other values can be used? Instead, why would we not have batch.mode=true/false? On Wed, Nov 30, 2016 at 1:51 PM, Matthias J. Sa

[jira] [Commented] (KAFKA-3834) Consumer should not block in poll on coordinator discovery

[ https://issues.apache.org/jira/browse/KAFKA-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710240#comment-15710240 ] Sönke Liebau commented on KAFKA-3834: - Does anybody know whether this is being discuss

Build failed in Jenkins: kafka-trunk-jdk8 #1069

See Changes: [jason] KAFKA-4469; Fix consumer performance regression from inefficient list -- [...truncated 3910 lines...] kafka.integration.SaslPlaintextTopicMetadataTest > testIsrAfterBroker

[jira] [Commented] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[ https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710135#comment-15710135 ] Jason Gustafson commented on KAFKA-4471: [~salaev] This is actually intentional. W

Build failed in Jenkins: kafka-trunk-jdk8 #1068

See Changes: [wangguoz] KAFKA-4271: Fix the server start script for Windows 32-bit OS -- [...truncated 15992 lines...] org.apache.kafka.connect.runtime.WorkerTest > testCleanupTasksOnStop FAILED

[DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi all, I have just created KIP-98 to enhance Kafka with exactly once delivery semantics: *https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

[jira] [Commented] (KAFKA-3994) Deadlock between consumer heartbeat expiration and offset commit.

[ https://issues.apache.org/jira/browse/KAFKA-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709975#comment-15709975 ] ASF GitHub Bot commented on KAFKA-3994: --- Github user hachikuji closed the pull reque

[GitHub] kafka pull request #1738: KAFKA-3994: Fix deadlock in Watchers by calling tr...

Github user hachikuji closed the pull request at: https://github.com/apache/kafka/pull/1738 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (KAFKA-3994) Deadlock between consumer heartbeat expiration and offset commit.

[ https://issues.apache.org/jira/browse/KAFKA-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709959#comment-15709959 ] ASF GitHub Bot commented on KAFKA-3994: --- GitHub user hachikuji opened a pull request

[GitHub] kafka pull request #2195: KAFKA-3994: Fix deadlock in Watchers by calling tr...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/2195 KAFKA-3994: Fix deadlock in Watchers by calling tryComplete without any locks You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

+1. Thanks for the KIP! On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira wrote: > +1 (binding) > > On Wed, Nov 30, 2016 at 1:34 PM, Xavier Léauté > wrote: > > Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to > > initiate a vote on it. > > > > https://cwiki.apache.org/confl

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

Both types of intermediate topics are handled the exact same way and both types do connect different subtopologies (even if the user might not be aware that there are multiple subtopologies in case of internal data repartitioning). So there is no distinction between user intermediate topics (via th

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

+1 (binding) On Wed, Nov 30, 2016 at 1:34 PM, Xavier Léauté wrote: > Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to > initiate a vote on it. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count >

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

+1 (binding) On Wed, Nov 30, 2016 at 1:43 PM Eno Thereska wrote: > +1 (non binding) > > > On 30 Nov 2016, at 21:34, Xavier Léauté wrote: > > > > Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to > > initiate a vote on it. > > > > > https://cwiki.apache.org/confluence/dis

Re: [VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

+1 (non binding) > On 30 Nov 2016, at 21:34, Xavier Léauté wrote: > > Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to > initiate a vote on it. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+coun

[jira] [Updated] (KAFKA-4451) Recovering empty replica yields negative offsets in index of compact partitions

[ https://issues.apache.org/jira/browse/KAFKA-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-4451: Labels: reliability (was: ) > Recovering empty replica yields negative offsets in index of compact

[VOTE] KIP-96 - Add per partition metrics for in-sync and replica count

Based on the feedback KIP-96 seems pretty uncontroversial, so I'd like to initiate a vote on it. https://cwiki.apache.org/confluence/display/KAFKA/KIP-96+-+Add+per+partition+metrics+for+in-sync+and+assigned+replica+count Xavier

[jira] [Resolved] (KAFKA-4469) Consumer throughput regression caused by inefficient list removal and copy

[ https://issues.apache.org/jira/browse/KAFKA-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-4469. Resolution: Fixed Issue resolved by pull request 2190 [https://github.com/apache/kafka/pull/

[jira] [Commented] (KAFKA-4469) Consumer throughput regression caused by inefficient list removal and copy

[ https://issues.apache.org/jira/browse/KAFKA-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709847#comment-15709847 ] ASF GitHub Bot commented on KAFKA-4469: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #2190: KAFKA-4469: Fix consumer performance regression fr...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2190 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

With the marker we have substituted a failure problem for a liveness problem in that under repeated failure the other instances will not do any useful work. As mentioned in the other email, I don't know if we need to worry about that corner case just yet. Eno > On 30 Nov 2016, at 20:06, Matthi

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

In the KIP, two types of intermediate topics are described, 1) ones that connect two sub-topologies, and 2) others that are internal repartitioning topics (e.g., for joins). I wasn't envisioning stopping the consumption of (2) at the HWM. The HWM can be used for the source topics only (so I agre

Re: [DISCUSS] KIP-96 - Add per partition metrics for in-sync and assigned replica count

FYI, Based on internal feedback I renamed AssignedReplicasCount to simply be called ReplicasCount. On Tue, Nov 29, 2016 at 7:56 PM Neha Narkhede wrote: > This seems useful, +1 > > On Tue, Nov 29, 2016 at 5:39 AM Ismael Juma wrote: > > > Hi Xavier, > > > > Thanks for the KIP. Sounds good to me.

Build failed in Jenkins: kafka-trunk-jdk8 #1067

See Changes: [jjkoshy] KAFKA-1911; Async delete topic - contributed by Mayuresh Gharat -- [...truncated 12186 lines...] org.apache.kafka.common.record.RecordTest > testChecksum[190] PASSED org.

[jira] [Updated] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[ https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-4271: - Resolution: Fixed Fix Version/s: 0.10.1.1 0.10.2.0 Status: Res

[jira] [Commented] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[ https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709663#comment-15709663 ] ASF GitHub Bot commented on KAFKA-4271: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #2189: KAFKA-4271: Fix the server start script for Window...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2189 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

Eno, > So in general, we have the problem of failures during writes to the metadata topic itself. The KIP suggests to use marker messaged for this case. The marker is either written (indicating success) or not. If not, after failure/rebalance the (new) group leader will collect HW again. As long

Re: [DISCUSS] 0.10.1.1 Plan

@Bernard Leach That sounds good, we can consider adding a kafka_2.12-0.10.1.1-beta.tgz into maven for Scala community to test it out. Guozhang On Tue, Nov 29, 2016 at 10:01 PM, Bernard Leach wrote: > Hi Guozhang, > > My suggestion was to not add kafka_2.12-0.10.1.0.tgz to downloads.html but >

[jira] [Updated] (KAFKA-4469) Consumer throughput regression caused by inefficient list removal and copy

[ https://issues.apache.org/jira/browse/KAFKA-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-4469: --- Summary: Consumer throughput regression caused by inefficient list removal and copy (was: Con

[jira] [Commented] (KAFKA-4469) Consumer throughput regression caused by decrease in max.poll.records

[ https://issues.apache.org/jira/browse/KAFKA-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709573#comment-15709573 ] Jason Gustafson commented on KAFKA-4469: [~ijuma] suggests that the inefficiency i

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

Thanks, Ashish. I think the idea of having the client make an ApiVersionRequest call when it starts up is a good one. This idea is described in both KIP-97, and the KAFKA-3600 patches. I also think we ought to maintain per-node version information. It would be good to get that in so that we can

[jira] [Updated] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eno Thereska updated KAFKA-3715: Assignee: aarti gupta (was: Eno Thereska) > Higher granularity streams metrics > -

[jira] [Updated] (KAFKA-1911) Log deletion on stopping replicas should be async

[ https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1911: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Log deletion on stopping replicas s

[jira] [Commented] (KAFKA-1911) Log deletion on stopping replicas should be async

[ https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709379#comment-15709379 ] ASF GitHub Bot commented on KAFKA-1911: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #1664: KAFKA-1911: Async delete topic

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1664 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP-97: Improved Kafka Client RPC Compatibility Policy

Hello Ismael, It is good to know that you are willing to review KAFKA-3600 again. As before, we at Cloudera are highly in support of client compatibility, and KAFKA-3600 has always been a building block for that. Now that client compatibility is at forefront again, thanks to Colin, I will be happy

[jira] [Commented] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[ https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709355#comment-15709355 ] Vahid Hashemian commented on KAFKA-4471: [~salaev] Thanks for reporting the issue

[jira] [Assigned] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[ https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian reassigned KAFKA-4471: -- Assignee: Vahid Hashemian > KafkaConsumer unpauses partitions after subscribe() > -

[jira] [Commented] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[ https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709345#comment-15709345 ] Dong Lin commented on KAFKA-4415: - [~ijuma] As of current Kafka implementation, Kafka does

[jira] [Updated] (KAFKA-4415) Reduce time to create and send MetadataUpdateRequest

[ https://issues.apache.org/jira/browse/KAFKA-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-4415: Description: As of current implementation, when controller receives ControlledShutdownRequest, it will 1) f

[jira] [Commented] (KAFKA-4454) Authorizer should also include the Principal generated by the PrincipalBuilder.

[ https://issues.apache.org/jira/browse/KAFKA-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709335#comment-15709335 ] Mayuresh Gharat commented on KAFKA-4454: [~jjkoshy] [~ashishsinghdev] [~parth.brah

[jira] [Resolved] (KAFKA-3309) Update Protocol Documentation WIP patch

[ https://issues.apache.org/jira/browse/KAFKA-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh resolved KAFKA-3309. --- Resolution: Duplicate > Update Protocol Documentation WIP patch >

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi, Michael, Thanks for the KIP. A few comments below. 1. The message format change contains "HeadersLength Headers". Is that intended? 2. For compressed messageset, is the tombstone bit only set at the shallow level? Do we always leave that bit in the wrapper message unset? An alternative is to

Re: [VOTE] KIP-93: Improve invalid timestamp handling in Kafka Streams

+1 (binding) I second Ewen's point about compatibility. On Wed, Nov 30, 2016 at 2:53 AM, Damian Guy wrote: > +1 > > On Wed, 30 Nov 2016 at 05:58 Ewen Cheslack-Postava > wrote: > > > +1 (binding). > > > > Also, see my notes in discussion thread around future compatibility > > discussions for br

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709138#comment-15709138 ] Soumyajit Sahu commented on KAFKA-1194: --- [~haraldk] could you please provide all of

[jira] [Updated] (KAFKA-4418) Broker Leadership Election Fails If Missing ZK Path Raises Exception

[ https://issues.apache.org/jira/browse/KAFKA-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4418: --- Labels: reliability (was: ) > Broker Leadership Election Fails If Missing ZK Path Raises Exception >

[jira] [Updated] (KAFKA-2758) Improve Offset Commit Behavior

[ https://issues.apache.org/jira/browse/KAFKA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2758: --- Labels: newbiee reliability (was: newbiee) > Improve Offset Commit Behavior > ---

[jira] [Updated] (KAFKA-3795) Transient system test failure upgrade_test.TestUpgrade

[ https://issues.apache.org/jira/browse/KAFKA-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3795: --- Labels: reliability (was: ) > Transient system test failure upgrade_test.TestUpgrade > --

[jira] [Updated] (KAFKA-3039) Temporary loss of leader resulted in log being completely truncated

[ https://issues.apache.org/jira/browse/KAFKA-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3039: --- Labels: reliability (was: ) > Temporary loss of leader resulted in log being completely truncated > -

[jira] [Updated] (KAFKA-1120) Controller could miss a broker state change

[ https://issues.apache.org/jira/browse/KAFKA-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-1120: --- Labels: reliability (was: ) > Controller could miss a broker state change >

[jira] [Updated] (KAFKA-1342) Slow controlled shutdowns can result in stale shutdown requests

[ https://issues.apache.org/jira/browse/KAFKA-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-1342: --- Labels: newbie++ newbiee reliability (was: newbie++ newbiee) > Slow controlled shutdowns can result i

[jira] [Updated] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3919: --- Labels: reliability (was: ) > Broker faills to start after ungraceful shutdown due to non-monotonical

[jira] [Updated] (KAFKA-3693) Lost highwatermark at broker start-up

[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3693: --- Labels: reliability (was: ) > Lost highwatermark at broker start-up > ---

[jira] [Updated] (KAFKA-4150) Facing Message Loss When trying rolling updates

[ https://issues.apache.org/jira/browse/KAFKA-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4150: --- Labels: reliability (was: ) > Facing Message Loss When trying rolling updates > -

[jira] [Updated] (KAFKA-3924) Data loss due to halting when LEO is larger than leader's LEO

[ https://issues.apache.org/jira/browse/KAFKA-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3924: --- Labels: reliability (was: ) > Data loss due to halting when LEO is larger than leader's LEO > ---

[jira] [Updated] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

[ https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-1211: --- Labels: reliability (was: ) > Hold the produce request with ack > 1 in purgatory until replicas' HW h

[jira] [Updated] (KAFKA-4360) Controller may deadLock when autoLeaderRebalance encounter zk expired

[ https://issues.apache.org/jira/browse/KAFKA-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4360: --- Labels: bugfix reliability (was: bugfix) > Controller may deadLock when autoLeaderRebalance encounter

[jira] [Updated] (KAFKA-3083) a soft failure in controller may leave a topic partition in an inconsistent state

[ https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3083: --- Labels: reliability (was: ) > a soft failure in controller may leave a topic partition in an inconsis

[jira] [Updated] (KAFKA-1429) Yet another deadlock in controller shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-1429: --- Labels: reliability (was: ) > Yet another deadlock in controller shutdown > -

[jira] [Updated] (KAFKA-3040) Broker didn't report new data after change in leader

[ https://issues.apache.org/jira/browse/KAFKA-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3040: --- Labels: reliability (was: ) > Broker didn't report new data after change in leader >

  1   2   >