Simple consumer offset help.

Hello All, I am new to Kafka, and confused in knowing things realated to offsets value that we set over partitions in specific topic using SimpleConsumer class. Would be great if anyone can tell me with example how to set an offset for specific partions , for reading from beginig or from last offs

Kangaroo projects with KafkaInputformat - Mapreduce.

Hi Team, Does any one use Kangaroo Project which uses SimpleConsumer of Kafka API, that provides support to integrate with MapReduce Jobs. Facing one issue related to getting an offsets for specific consumer. Also after Re launching, the committed offsets for a consumer is to be read, and start f

[jira] [Created] (KAFKA-3227) Conservative update of Kafka dependencies

Ismael Juma created KAFKA-3227: -- Summary: Conservative update of Kafka dependencies Key: KAFKA-3227 URL: https://issues.apache.org/jira/browse/KAFKA-3227 Project: Kafka Issue Type: Improvement

[GitHub] kafka pull request: KAFKA-3227; Conservative update of Kafka depen...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/903 KAFKA-3227; Conservative update of Kafka dependencies Patch version bumps for bouncy castle, minikdc, snappy, slf4j and powermock: * Snappy: fixes a resource leak * MiniKDC: bumps commons-c

[jira] [Commented] (KAFKA-3227) Conservative update of Kafka dependencies

[ https://issues.apache.org/jira/browse/KAFKA-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142488#comment-15142488 ] ASF GitHub Bot commented on KAFKA-3227: --- GitHub user ijuma opened a pull request:

[jira] [Commented] (KAFKA-3225) Method commit() of class SourceTask never invoked

[ https://issues.apache.org/jira/browse/KAFKA-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142502#comment-15142502 ] Krzysztof Dębski commented on KAFKA-3225: - I am quite new to Kafka (two weeks of e

[jira] [Commented] (KAFKA-3167) Use local to the workspace Gradle cache and recreate it on every build

[ https://issues.apache.org/jira/browse/KAFKA-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142724#comment-15142724 ] Ismael Juma commented on KAFKA-3167: One way to do this would be to set the following

[jira] [Commented] (KAFKA-3167) Use local to the workspace Gradle cache and recreate it on every build

[ https://issues.apache.org/jira/browse/KAFKA-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142761#comment-15142761 ] Flavio Junqueira commented on KAFKA-3167: - we currently have this: {noformat} $G

[jira] [Commented] (KAFKA-3167) Use local to the workspace Gradle cache and recreate it on every build

[ https://issues.apache.org/jira/browse/KAFKA-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142814#comment-15142814 ] Ismael Juma commented on KAFKA-3167: The following is for the gradle binary so it can

[GitHub] kafka pull request: KAFKA-2866: Bump up commons-collections versio...

Github user granthenke closed the pull request at: https://github.com/apache/kafka/pull/564 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (KAFKA-2866) Bump up commons-collections version to 3.2.2 to address a security flaw

[ https://issues.apache.org/jira/browse/KAFKA-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142837#comment-15142837 ] ASF GitHub Bot commented on KAFKA-2866: --- Github user granthenke closed the pull requ

[jira] [Updated] (KAFKA-2866) Bump up commons-collections version to 3.2.2 to address a security flaw

[ https://issues.apache.org/jira/browse/KAFKA-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-2866: --- Status: Open (was: Patch Available) > Bump up commons-collections version to 3.2.2 to address a secur

[jira] [Created] (KAFKA-3228) Partition reassignment failure for brokers freshly added to cluster

Andrew Olson created KAFKA-3228: --- Summary: Partition reassignment failure for brokers freshly added to cluster Key: KAFKA-3228 URL: https://issues.apache.org/jira/browse/KAFKA-3228 Project: Kafka

[jira] [Updated] (KAFKA-3228) Partition reassignment failure for brokers freshly added to cluster

[ https://issues.apache.org/jira/browse/KAFKA-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-3228: Description: After adding about new 20 brokers to double the size of an existing production Kafka d

[jira] [Updated] (KAFKA-3228) Partition reassignment failure for brokers freshly added to cluster

[ https://issues.apache.org/jira/browse/KAFKA-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated KAFKA-3228: Description: After adding about new 20 brokers to double the size of an existing production Kafka d

[jira] [Created] (KAFKA-3229) Root statestore is not registered with ProcessorStateManager, inner state store is registered instead

Tom Dearman created KAFKA-3229: -- Summary: Root statestore is not registered with ProcessorStateManager, inner state store is registered instead Key: KAFKA-3229 URL: https://issues.apache.org/jira/browse/KAFKA-3229

Swift Client

Hi, All, I'm new to this group and have been working an Apache Kafka Client for Swift. See here: https://github.com/kellanburket/franz. I'd like access to get access to the wiki. -Kellan

[jira] [Created] (KAFKA-3230) Unable to commit offsets if partition is paused

Craig W created KAFKA-3230: -- Summary: Unable to commit offsets if partition is paused Key: KAFKA-3230 URL: https://issues.apache.org/jira/browse/KAFKA-3230 Project: Kafka Issue Type: Bug C

[jira] [Commented] (KAFKA-3230) Unable to commit offsets if partition is paused

[ https://issues.apache.org/jira/browse/KAFKA-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142900#comment-15142900 ] Ismael Juma commented on KAFKA-3230: cc [~hachikuji] > Unable to commit offsets if pa

[jira] [Commented] (KAFKA-3229) Root statestore is not registered with ProcessorStateManager, inner state store is registered instead

[ https://issues.apache.org/jira/browse/KAFKA-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142936#comment-15142936 ] ASF GitHub Bot commented on KAFKA-3229: --- GitHub user tomdearman opened a pull reques

[GitHub] kafka pull request: KAFKA-3229 ensure that root statestore is regi...

GitHub user tomdearman opened a pull request: https://github.com/apache/kafka/pull/904 KAFKA-3229 ensure that root statestore is registered with ProcessorStateManager Pass through the root StateStore in the init method so the inner StateStore can register that object. You can merg

[jira] [Updated] (KAFKA-3227) Conservative update of Kafka dependencies

[ https://issues.apache.org/jira/browse/KAFKA-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3227: --- Description: Patch version bumps for bouncy castle, minikdc, snappy, slf4j, scalatest and powermock.

Build failed in Jenkins: kafka_0.9.0_jdk7 #122

See -- [...truncated 3064 lines...] kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup PASSED kafka.coordinator.GroupCoordinatorResponseTest > testListGroupsIncludesRebalancing

[jira] [Created] (KAFKA-3231) describe group IllegalArgumentException for kafka-consumer-groups

Curtis Allen created KAFKA-3231: --- Summary: describe group IllegalArgumentException for kafka-consumer-groups Key: KAFKA-3231 URL: https://issues.apache.org/jira/browse/KAFKA-3231 Project: Kafka

[jira] [Commented] (KAFKA-3231) describe group IllegalArgumentException for kafka-consumer-groups

[ https://issues.apache.org/jira/browse/KAFKA-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142991#comment-15142991 ] Ismael Juma commented on KAFKA-3231: Thanks for the report. Which client did you use t

[jira] [Commented] (KAFKA-3231) describe group IllegalArgumentException for kafka-consumer-groups

[ https://issues.apache.org/jira/browse/KAFKA-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143005#comment-15143005 ] Curtis Allen commented on KAFKA-3231: - Using https://github.com/Shopify/sarama via ht

[jira] [Commented] (KAFKA-3231) describe group IllegalArgumentException for kafka-consumer-groups

[ https://issues.apache.org/jira/browse/KAFKA-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143018#comment-15143018 ] Ismael Juma commented on KAFKA-3231: Thanks. That client is possibly doing something u

[jira] [Commented] (KAFKA-3231) describe group IllegalArgumentException for kafka-consumer-groups

[ https://issues.apache.org/jira/browse/KAFKA-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143116#comment-15143116 ] Curtis Allen commented on KAFKA-3231: - Thanks [~ijuma] ran kafka-consumer-groups from

[jira] [Resolved] (KAFKA-3231) describe group IllegalArgumentException for kafka-consumer-groups

[ https://issues.apache.org/jira/browse/KAFKA-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3231. Resolution: Duplicate Thanks for checking [~curtissallen]. > describe group IllegalArgumentExceptio

[GitHub] kafka pull request: KAFKA-3229 ensure that root statestore is regi...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/904 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Created] (KAFKA-3232) Evaluate if configs with no value exist and if not remove support for such configs in CommanLineUtils

Ashish K Singh created KAFKA-3232: - Summary: Evaluate if configs with no value exist and if not remove support for such configs in CommanLineUtils Key: KAFKA-3232 URL: https://issues.apache.org/jira/browse/KAFKA-3

[jira] [Commented] (KAFKA-3229) Root statestore is not registered with ProcessorStateManager, inner state store is registered instead

[ https://issues.apache.org/jira/browse/KAFKA-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143224#comment-15143224 ] ASF GitHub Bot commented on KAFKA-3229: --- Github user asfgit closed the pull request

[jira] [Commented] (KAFKA-3230) Unable to commit offsets if partition is paused

[ https://issues.apache.org/jira/browse/KAFKA-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143231#comment-15143231 ] Craig W commented on KAFKA-3230: So the problem appears to be on my misunderstanding of co

[jira] [Resolved] (KAFKA-3229) Root statestore is not registered with ProcessorStateManager, inner state store is registered instead

[ https://issues.apache.org/jira/browse/KAFKA-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-3229. - Resolution: Fixed Issue resolved by pull request 904 [https://github.com/apache/kafka/pull/904] >

Jenkins build is back to normal : kafka-trunk-jdk7 #1038

See

Jenkins build is back to normal : kafka-trunk-jdk8 #365

See

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[ https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143527#comment-15143527 ] Vahid Hashemian commented on KAFKA-3129: Some more information on this issue: * I

[jira] [Comment Edited] (KAFKA-3129) Console Producer/Consumer Issue

[ https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143527#comment-15143527 ] Vahid Hashemian edited comment on KAFKA-3129 at 2/11/16 9:24 PM: ---

[jira] [Commented] (KAFKA-3230) Unable to commit offsets if partition is paused

[ https://issues.apache.org/jira/browse/KAFKA-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143649#comment-15143649 ] Jason Gustafson commented on KAFKA-3230: [~codecraig] Yes, you are correct. The co

[jira] [Created] (KAFKA-3233) Per topic consumer metrics missing in new consumer

Yifan Ying created KAFKA-3233: - Summary: Per topic consumer metrics missing in new consumer Key: KAFKA-3233 URL: https://issues.apache.org/jira/browse/KAFKA-3233 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[ https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Ying updated KAFKA-3233: -- Summary: Per topic consumer metrics missing in new consumer's metrics reporter (was: Per topic consumer

[GitHub] kafka pull request: KAFKA-3233: expose per topic consumer metrics ...

GitHub user happymap opened a pull request: https://github.com/apache/kafka/pull/905 KAFKA-3233: expose per topic consumer metrics to metrics reporter In version of 0.8.2.1, the old consumer will provide the metrics reporter per-topic consumer metrics under group 'ConsumerTopicMetri

[jira] [Commented] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[ https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143684#comment-15143684 ] ASF GitHub Bot commented on KAFKA-3233: --- GitHub user happymap opened a pull request:

[jira] [Updated] (KAFKA-3233) Per topic consumer metrics missing in new consumer's metrics reporter

[ https://issues.apache.org/jira/browse/KAFKA-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Ying updated KAFKA-3233: -- Description: In version of 0.8.2.1, the old consumer will provide the metrics reporter per-topic consum

[jira] [Updated] (KAFKA-3221) kafka-acls.sh must verify if a user has sufficient privileges to perform acls CRUD

[ https://issues.apache.org/jira/browse/KAFKA-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3221: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > kafka-acls.sh must verify if a user has suffici

[jira] [Updated] (KAFKA-3187) Make kafka-acls.sh help messages more generic.

[ https://issues.apache.org/jira/browse/KAFKA-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3187: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > Make kafka-acls.sh help messages more generic.

[jira] [Updated] (KAFKA-3190) KafkaProducer should not invoke callback in send()

[ https://issues.apache.org/jira/browse/KAFKA-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3190: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > KafkaProducer should not invoke callback in sen

[jira] [Updated] (KAFKA-3186) Kafka authorizer should be aware of principal types it supports.

[ https://issues.apache.org/jira/browse/KAFKA-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3186: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > Kafka authorizer should be aware of principal t

[jira] [Updated] (KAFKA-3175) topic not accessible after deletion even when delete.topic.enable is disabled

[ https://issues.apache.org/jira/browse/KAFKA-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3175: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > topic not accessible after deletion even when d

[jira] [Updated] (KAFKA-3173) Error while moving some partitions to OnlinePartition state

[ https://issues.apache.org/jira/browse/KAFKA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3173: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > Error while moving some partitions to OnlinePar

[jira] [Updated] (KAFKA-3147) Memory records is not writable in MirrorMaker

[ https://issues.apache.org/jira/browse/KAFKA-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3147: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > Memory records is not writable in MirrorMaker >

[jira] [Updated] (KAFKA-3145) CPU Usage Spike to 100% when network connection is to error port

[ https://issues.apache.org/jira/browse/KAFKA-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3145: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > CPU Usage Spike to 100% when network connection

[jira] [Updated] (KAFKA-3088) 0.9.0.0 broker crash on receipt of produce request with empty client ID

[ https://issues.apache.org/jira/browse/KAFKA-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3088: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > 0.9.0.0 broker crash on receipt of produce requ

[jira] [Updated] (KAFKA-2930) Update references to ZooKeeper in the docs

[ https://issues.apache.org/jira/browse/KAFKA-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2930: --- Fix Version/s: (was: 0.9.0.1) 0.9.1.0 > Update references to ZooKeeper in the docs > --

You're Invited: Apache Kafka Meetup on Tuesday, February 23 at LinkedIn's campus in Mountain View, CA

Hi Apache Kafka Open Source distros, LinkedIn invites you to attend an Apache Kafka meetup on Tuesday, February 23 at 6PM at LinkedIn's campus in Mountain View, CA. Here's are the details: - We have 3 great talks scheduled: - *Loose Coupling with the Jitney Proj

[jira] [Created] (KAFKA-3234) Minor documentation edits: clarify minISR; some topic-level configs are missing

Joel Koshy created KAFKA-3234: - Summary: Minor documentation edits: clarify minISR; some topic-level configs are missing Key: KAFKA-3234 URL: https://issues.apache.org/jira/browse/KAFKA-3234 Project: Kafk

[GitHub] kafka pull request: MINOR: remove streams config params from produ...

GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/906 MINOR: remove streams config params from producer/consumer configs Removing streams' specific config params from producer/consumer configs to reduce warning messages. You can merge this pull reques

Re: KafkaConsumer.poll(long) pauses indefinitely

See https://issues.apache.org/jira/browse/KAFKA-1894. -Jason On Wed, Feb 10, 2016 at 2:08 PM, William Grim wrote: > Hi, > > I'm noticing that when we have a thread that comes up and calls > KafkaConsumer.poll(long), it will pause for a much longer time than our 1s > timeout. It looks like this

Build failed in Jenkins: kafka_0.9.0_jdk7 #123

See Changes: [junrao] change release version to 0.9.0.1 -- [...truncated 3009 lines...] kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED kafka.integration.Primitiv

Re: [DISCUSS] KIP-46: Self Healing

> > 1. Replica Throttling - I agree this is rather important to get done. > However, it may also be argued that this problem is orthogonal. We do not > have these protections currently yet we do run partition reassignment > fairly often. Having said that, I'm perfectly happy to tackle KIP-46 after

[GitHub] kafka pull request: KAFKA-3234; Clarify minISR in documentation an...

GitHub user jjkoshy opened a pull request: https://github.com/apache/kafka/pull/907 KAFKA-3234; Clarify minISR in documentation and auto-generate topic configuration docs Fix design doc comments to clarify minISR and unclean leader election, switch topic configuration doc to be aut

[jira] [Commented] (KAFKA-3234) Minor documentation edits: clarify minISR; some topic-level configs are missing

[ https://issues.apache.org/jira/browse/KAFKA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143925#comment-15143925 ] ASF GitHub Bot commented on KAFKA-3234: --- GitHub user jjkoshy opened a pull request:

0.9.0.1 RC1

This is the first candidate for release of Apache Kafka 0.9.0.1. This a bug fix release that fixes 70 issues. Release Notes for the 0.9.0.1 release https://home.apache.org/~junrao/kafka-0.9.0.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Tuesday, Feb. 16, 7pm PT Kafka's K

[jira] [Updated] (KAFKA-3145) CPU Usage Spike to 100% when network connection is to error port

[ https://issues.apache.org/jira/browse/KAFKA-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-3145: Assignee: Dong Lin > CPU Usage Spike to 100% when network connection is to error port >