Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

> On May 16, 2016, at 9:21 PM, Eric Wasserman wrote: > > Gwen, > > For simplicity, the example I gave in the gist is for a single table with a > single partition. The salient point is that even for a single topic with one > partition there is no guarantee without the feature that one will be

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

We would find this KIP very useful. Our particular use case falls into the "application mistakes" portion of the KIP. We are storing source of truth data in log compacted topics, similar to the Confluent Schema Registry. One situation we had recently was a misbehaving application. It sent data

[jira] [Commented] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286084#comment-15286084 ] aarti gupta commented on KAFKA-3715: @jklukas, Thanks for the clear description, I wou

[jira] [Assigned] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] aarti gupta reassigned KAFKA-3715: -- Assignee: aarti gupta > Higher granularity streams metrics > -

[jira] [Commented] (KAFKA-2082) Kafka Replication ends up in a bad state

[ https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286045#comment-15286045 ] Raghavendra Nandagopal commented on KAFKA-2082: --- Hi, We are also seeing th

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

I see what you mean, Eric. I was unclear on the specifics of your architecture. It sounds like you have a table somewhere that maps checkpoints to lists of . In that case it is indeed useful to know that if the checkpoint was written N ms ago, you will be able to find the exact offsets by looking

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

Gwen, For simplicity, the example I gave in the gist is for a single table with a single partition. The salient point is that even for a single topic with one partition there is no guarantee without the feature that one will be able to restore some particular checkpoint as the offset indicated

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

Yeah I think I gave a scenario but that is not the same as a concrete use case. I think the question you have is how common is it that people care about this and what concrete things would you build where you had this requirement? I think that would be good to figure out. I think the issue with th

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

I agree that log.cleaner.min.compaction.lag.ms gives slightly more flexibility for potentially-lagging consumers than tuning segment.roll.ms for the exact same scenario. If more people think that the use-case of "consumer which must see every single record, is running on a compacted topic, and is

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

I think it would be good to hammer out some of the practical use cases--I definitely share your disdain for adding more configs. Here is my sort of theoretical understanding of why you might want this. As you say a consumer bootstrapping itself in the compacted part of the log isn't actually trave

[jira] [Updated] (KAFKA-3717) Support building aggregate javadoc for all project modules

[ https://issues.apache.org/jira/browse/KAFKA-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-3717: --- Summary: Support building aggregate javadoc for all project modules (was: On 0.10.0 branch, building

[jira] [Commented] (KAFKA-3717) On 0.10.0 branch, building javadoc results in very small subset of expected javadocs

[ https://issues.apache.org/jira/browse/KAFKA-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285897#comment-15285897 ] Grant Henke commented on KAFKA-3717: Currently the build places a javadoc directory an

Re: [DISCUSS] KIP-58 - Make Log Compaction Point Configurable

Hi Eric, Thank you for submitting this improvement suggestion. Do you mind clarifying the use-case for me? Looking at your gist: https://gist.github.com/ewasserman/f8c892c2e7a9cf26ee46 If my consumer started reading all the CDC topics from the very beginning in which they were created, without

[jira] [Assigned] (KAFKA-3717) On 0.10.0 branch, building javadoc results in very small subset of expected javadocs

[ https://issues.apache.org/jira/browse/KAFKA-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned KAFKA-3717: -- Assignee: Grant Henke > On 0.10.0 branch, building javadoc results in very small subset of expe

Jenkins build is back to normal : kafka-0.10.0-jdk7 #96

See

Help wanted fixing Javadoc build issue

Hi Team Kafka, If anyone feels comfortable with Gradle and has some spare time, maybe help bring back our missing javadocs: https://issues.apache.org/jira/browse/KAFKA-3717 Thanks! Gwen

[jira] [Created] (KAFKA-3717) On 0.10.0 branch, building javadoc results in very small subset of expected javadocs

Gwen Shapira created KAFKA-3717: --- Summary: On 0.10.0 branch, building javadoc results in very small subset of expected javadocs Key: KAFKA-3717 URL: https://issues.apache.org/jira/browse/KAFKA-3717 Proj

[jira] [Commented] (KAFKA-1981) Make log compaction point configurable

[ https://issues.apache.org/jira/browse/KAFKA-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285617#comment-15285617 ] Eric Wasserman commented on KAFKA-1981: --- Thanks, I added your use case. I would like

[DISCUSS] KIP-58 - Make Log Compaction Point Configurable

I would like to begin discussion on KIP-58 The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable Jira: https://issues.apache.org/jira/browse/KAFKA-1981 Pull Request: https://github.com/apache/kafka/pull/1168 Thanks, Eric

Build failed in Jenkins: kafka-trunk-jdk8 #633

See Changes: [wangguoz] MINOR: Add INFO logging if ZK config is not specified -- [...truncated 3934 lines...] kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED kafka.log.Bro

[jira] [Updated] (KAFKA-3714) Allow users greater access to register custom streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3714: - Labels: api (was: ) > Allow users greater access to register custom streams metrics > ---

[jira] [Updated] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3715: - Assignee: (was: Guozhang Wang) > Higher granularity streams metrics > ---

[jira] [Updated] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3715: - Labels: api newbie (was: api) > Higher granularity streams metrics > ---

[jira] [Updated] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3715: - Labels: api (was: ) > Higher granularity streams metrics > --- >

[jira] [Commented] (KAFKA-3716) Check against negative timestamps

[ https://issues.apache.org/jira/browse/KAFKA-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285551#comment-15285551 ] ASF GitHub Bot commented on KAFKA-3716: --- GitHub user guozhangwang opened a pull requ

[VOTE] 0.10.0.0 RC5

Hello Kafka users, developers and client-developers, This is the sixth (!) candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give pe

[GitHub] kafka pull request: KAFKA-3716: Validate all timestamps are not ne...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/1393 KAFKA-3716: Validate all timestamps are not negative You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K3716-check-non-ne

[jira] [Created] (KAFKA-3716) Check against negative timestamps

Guozhang Wang created KAFKA-3716: Summary: Check against negative timestamps Key: KAFKA-3716 URL: https://issues.apache.org/jira/browse/KAFKA-3716 Project: Kafka Issue Type: Bug Com

[GitHub] kafka pull request: MINOR: Add INFO logging if ZK config is not sp...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1392 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Updated] (KAFKA-3444) Figure out when to bump the version on release-candidate artifacts

[ https://issues.apache.org/jira/browse/KAFKA-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-3444: Fix Version/s: (was: 0.10.1.0) > Figure out when to bump the version on release-candidate artifa

[jira] [Commented] (KAFKA-3444) Figure out when to bump the version on release-candidate artifacts

[ https://issues.apache.org/jira/browse/KAFKA-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285297#comment-15285297 ] Gwen Shapira commented on KAFKA-3444: - Resolved. The build process was updated to incl

[jira] [Resolved] (KAFKA-3444) Figure out when to bump the version on release-candidate artifacts

[ https://issues.apache.org/jira/browse/KAFKA-3444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3444. - Resolution: Fixed Assignee: Gwen Shapira > Figure out when to bump the version on release-ca

Jenkins build is back to normal : kafka-trunk-jdk7 #1297

See

[jira] [Commented] (KAFKA-3699) Update protocol page on website to explain how KIP-35 should be used

[ https://issues.apache.org/jira/browse/KAFKA-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285135#comment-15285135 ] Ashish K Singh commented on KAFKA-3699: --- I will have a patch ready hopefully by EOD,

Build failed in Jenkins: kafka-0.10.0-jdk7 #95

See Changes: [cshapi] KAFKA-3704; Revert "Remove hard-coded block size in KafkaProducer" -- [...truncated 39 lines...] :core:clean :examples:clean :log4j-appender:clean :streams:clean :tools:clea

[GitHub] kafka pull request: MINOR: Add INFO logging if ZK config is not sp...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/1392 MINOR: Add INFO logging if ZK config is not specified You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka Kminor-warn-no-zk-

[jira] [Commented] (KAFKA-3699) Update protocol page on website to explain how KIP-35 should be used

[ https://issues.apache.org/jira/browse/KAFKA-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285120#comment-15285120 ] Gwen Shapira commented on KAFKA-3699: - I moved this off the upcoming release. I would

Build failed in Jenkins: kafka-trunk-jdk8 #632

See Changes: [cshapi] KAFKA-3704; Revert "Remove hard-coded block size in KafkaProducer" -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu3

[jira] [Updated] (KAFKA-3699) Update protocol page on website to explain how KIP-35 should be used

[ https://issues.apache.org/jira/browse/KAFKA-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-3699: Fix Version/s: (was: 0.10.0.0) 0.10.0.1 > Update protocol page on website to

[jira] [Resolved] (KAFKA-3633) Kafka Consumer API breaking backward compatibility

[ https://issues.apache.org/jira/browse/KAFKA-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3633. - Resolution: Won't Fix It looks like this won't get fixed (per community discussion here: https://

[jira] [Commented] (KAFKA-3704) Improve mechanism for compression stream block size selection in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285091#comment-15285091 ] ASF GitHub Bot commented on KAFKA-3704: --- Github user asfgit closed the pull request

[GitHub] kafka pull request: KAFKA-3704; Revert "Remove hard-coded block si...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1391 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Resolved] (KAFKA-3704) Improve mechanism for compression stream block size selection in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3704. - Resolution: Fixed Fix Version/s: 0.10.0.0 Issue resolved by pull request 1391 [https://gith

[GitHub] kafka pull request: MINOR: Inline `Record.write` into `Compressor....

Github user ijuma closed the pull request at: https://github.com/apache/kafka/pull/1388 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

Jenkins build is back to normal : kafka-0.10.0-jdk7 #94

See

Build failed in Jenkins: kafka-trunk-jdk7 #1296

See Changes: [ismael] KAFKA-3698; Update the message format section. -- [...truncated 1646 lines...] kafka.log.LogTest > testAsyncDelete PASSED kafka.log.LogTest > testReadOutOfRange PASSED k

Build failed in Jenkins: kafka-trunk-jdk8 #631

See Changes: [ismael] KAFKA-3698; Update the message format section. -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu3 (Ubuntu ubuntu lega

[GitHub] kafka pull request: KAFKA-3701: Expose KafkaStreams Metrics in pub...

Github user jklukas closed the pull request at: https://github.com/apache/kafka/pull/1362 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[jira] [Updated] (KAFKA-3698) Update website documentation when it comes to the message format

[ https://issues.apache.org/jira/browse/KAFKA-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3698: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 1375

[GitHub] kafka pull request: KAFKA-3704; Revert "Remove hard-coded block si...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1391 KAFKA-3704; Revert "Remove hard-coded block size in KafkaProducer" This is not an exact revert as the code changed a bit since the original commit. We also include a note in `upgrade.html`.

[GitHub] kafka pull request: KAFKA-3698: Update the message format section.

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1375 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Jenkins build is back to normal : kafka-trunk-jdk8 #630

See

[jira] [Updated] (KAFKA-3704) Improve mechanism for compression stream block size selection in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3704: --- Summary: Improve mechanism for compression stream block size selection in KafkaProducer (was: Use def

[jira] [Reopened] (KAFKA-3704) Use default block size in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-3704: Assignee: Ismael Juma (was: Guozhang Wang) > Use default block size in KafkaProducer > --

[jira] [Updated] (KAFKA-3704) Use default block size in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3704: --- Fix Version/s: (was: 0.10.0.0) 0.10.1.0 > Use default block size in KafkaProduc

[jira] [Updated] (KAFKA-3704) Use default block size in KafkaProducer

[ https://issues.apache.org/jira/browse/KAFKA-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3704: --- Description: As discovered in https://issues.apache.org/jira/browse/KAFKA-3565, the current default b

Re: reading the consumer offsets topic

Hi Tao, Sorry for the delay. Thanks for pointing out that property. That was the fix. On Mon, May 9, 2016 at 6:00 PM, tao xiao wrote: > You need to enable internal topic in the consumer.properties > > exclude.internal.topics=false > > On Mon, 9 May 2016 at 12:42 Cliff Rhyne wrote: > > > Than

[jira] [Updated] (KAFKA-3714) Allow users greater access to register custom streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Klukas updated KAFKA-3714: --- Issue Type: Improvement (was: Bug) > Allow users greater access to register custom streams metrics >

[jira] [Commented] (KAFKA-3715) Higher granularity streams metrics

[ https://issues.apache.org/jira/browse/KAFKA-3715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284706#comment-15284706 ] Jeff Klukas commented on KAFKA-3715: It would be interesting to work on this, but I wo

[jira] [Created] (KAFKA-3715) Higher granularity streams metrics

Jeff Klukas created KAFKA-3715: -- Summary: Higher granularity streams metrics Key: KAFKA-3715 URL: https://issues.apache.org/jira/browse/KAFKA-3715 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-3714) Allow users greater access to register custom streams metrics

Jeff Klukas created KAFKA-3714: -- Summary: Allow users greater access to register custom streams metrics Key: KAFKA-3714 URL: https://issues.apache.org/jira/browse/KAFKA-3714 Project: Kafka Issu

[jira] [Commented] (KAFKA-3699) Update protocol page on website to explain how KIP-35 should be used

[ https://issues.apache.org/jira/browse/KAFKA-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284508#comment-15284508 ] Ismael Juma commented on KAFKA-3699: Any progress on this [~singhashish]? > Update pr

[jira] [Updated] (KAFKA-3258) BrokerTopicMetrics of deleted topics are never deleted

[ https://issues.apache.org/jira/browse/KAFKA-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3258: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 944

[GitHub] kafka pull request: KAFKA-3258: Delete broker topic metrics of del...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/944 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

Build failed in Jenkins: kafka-0.10.0-jdk7 #93

See Changes: [ismael] MINOR: document increased network bandwidth of 0.10 under replication -- [...truncated 83 lines...] new UpdateMetadataRequest(controllerId, controllerEpoch, liv

Build failed in Jenkins: kafka-trunk-jdk8 #629

See Changes: [ismael] MINOR: document increased network bandwidth of 0.10 under replication -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubun

[GitHub] kafka pull request: MINOR: document increased network bandwidth of...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1389 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: 0.9.0.1 - KafkaProducer keeps requesting for metadata of topic which is no longer present, leads to never ending UNKNOWN_TOPIC_OR_PARTITION logs

On Monday 16 May 2016 04:52 PM, Rajini Sivaram wrote: Hi Jaikiran, 1) If you delete a topic with no outstanding sends, you will see one WARN log entry and the topic is immediately removed from the metadata. If you delete a topic with outstanding sends, you will see a sequence of WARN logs until

Re: 0.9.0.1 - KafkaProducer keeps requesting for metadata of topic which is no longer present, leads to never ending UNKNOWN_TOPIC_OR_PARTITION logs

Hi Jaikiran, 1) If you delete a topic with no outstanding sends, you will see one WARN log entry and the topic is immediately removed from the metadata. If you delete a topic with outstanding sends, you will see a sequence of WARN logs until the outstanding send requests expire ("max.block.ms" pro

Re: 0.9.0.1 - KafkaProducer keeps requesting for metadata of topic which is no longer present, leads to never ending UNKNOWN_TOPIC_OR_PARTITION logs

Thank you for looking into this Rajini. A few questions related to this: 1) Looking at that patch, it looks like that WARN message will still be logged for a "few" times till that topic is removed from the Set maintained in the Metadata. Is my understanding correct? If yes, would it be possib

[jira] [Assigned] (KAFKA-3282) Change tools to use --new-consumer by default and introduce --old-consumer

[ https://issues.apache.org/jira/browse/KAFKA-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Mahadevan reassigned KAFKA-3282: - Assignee: Arun Mahadevan > Change tools to use --new-consumer by default and introduce --

Re: 0.9.0.1 - KafkaProducer keeps requesting for metadata of topic which is no longer present, leads to never ending UNKNOWN_TOPIC_OR_PARTITION logs

Sorry, that was the wrong JIRA. https://issues.apache.org/jira/browse/KAFKA-2948 is the one which addresses this issue. On Mon, May 16, 2016 at 7:52 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > There is an open JIRA for this issue ( > https://issues.apache.org/jira/browse/KAFKA-306