Re: How to reduce time for consumer to take over?

Hello Christian, Assuming that you are using new Java kafka-clients with Kafka's consumer group management, you'd have to tune consumers and broker. See - Broker config https://kafka.apache.org/documentation/#brokerconfigs -- group.initial.rebalance.delay.ms -- group.max.session.timeout.ms -- gro

Re: How would Kafka behave in this scenario

ks Stevo for pointing me out to correct link. > > > In this case, how would exactly once feature of streams would behave > > since > > > they configure producers with acks=all. I think they would fail and > would > > > need to be resumed once the broker comes back. >

Re: Streams dependency on RocksDB - licenses

Created https://issues.apache.org/jira/browse/KAFKA-5977 for this issue. On Tue, Sep 26, 2017 at 9:49 AM, Stevo Slavić wrote: > There is legal problem for older rocksdb versions, see > https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16109870&am

Re: Streams dependency on RocksDB - licenses

mit you referenced, adds GPL2 to the list of licenses in > the > > pom, but it's not clear why. > > > > Ismael > > > > On Thu, Sep 21, 2017 at 3:44 PM, Stevo Slavić wrote: > > > >> LevelDB GPL2 notice seems to have been added to rocksdb 5

Re: How would Kafka behave in this scenario

Hello Sameer, Behavior depends on min.insync.replicas configured for the topic. Find more info in the documentation https://kafka.apache.org/documentation/#topicconfigs Kind regards, Stevo Slavic. On Tue, Sep 26, 2017 at 9:01 AM, Sameer Kumar wrote: > In case one of the brokers fail, the brok

Re: Streams dependency on RocksDB - licenses

/6e3ee015fb1ce03e47838e9a3995410ce884c212#diff-1d5a1a614aa158e14292a60d9891f1df Shouldn't similar notice be added to Apache Kafka too? On Thu, Sep 21, 2017 at 3:43 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > Is it on purpose that kafka-streams 0.11.0.1 depends on > org.rocksdb:rocksdbj

Streams dependency on RocksDB - licenses

Hello Apache Kafka community, Is it on purpose that kafka-streams 0.11.0.1 depends on org.rocksdb:rocksdbjni:5.0.1, and not on newer 5.7.3, because 5.0.1 has Apache 2 license while 5.7.3 has also GPL 2.0 parts? Kind regards, Stevo Slavic.

Re: Is CommitFailedException thrown from commitSync with offsets because an offset is out of range?

Reported a bug for this https://issues.apache.org/jira/browse/KAFKA-5835 On Wed, Aug 30, 2017 at 10:30 AM, Stevo Slavić wrote: > Hello Apache Kafka community, > > org.apache.kafka.clients.consumer.CommitFailedException javadoc and > especially exception message suggests that the e

Is CommitFailedException thrown from commitSync with offsets because an offset is out of range?

Hello Apache Kafka community, org.apache.kafka.clients.consumer.CommitFailedException javadoc and especially exception message suggests that the exception can be thrown only due to consumer re-balancing, KafkaConsumer instance not owning TopicPartition for the consumer group of the offset being co

"client" and requests in request.timeout.ms

Hello Apache Kafka community, In Consumer, Producer, AdminClient and Broker configuration documentation there's a common config property, request.timeout.ms, with common description part being: "The configuration controls the maximum amount of time the client will wait for the response of a reques

Re: Message polling behavior when subscribed to a topic pattern

Hello William, You have overriden KafkaConsumer's max.poll.records configuration, to return only up to 10 records in single call to poll, so it does that. Fetcher iterates over completed fetch responses and in first one(s) from single topic finds enough of records to satisfy max limit. See https:/

Re: Kafka consumer offset reset

Hello Jakub, Maybe you're interested in feature that's not yet available but is being proposed and discussed (on dev ML) for future - see https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling Kind regards, Stevo Slavic. On Tue, Mar 21, 2017 at 4:43

Re: Kafka client and ELB (round-robin DNS)

See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html On Fri, Mar 3, 2017, 20:57 BGCH wrote: > Hi all, > > briefly about the problem, if bootstrap.servers is a domain name and > that domain name eventually starts pointing to different IP, kafka > client will never t

Re: Kafka partition no migrating to another broker.

Hello Shri, That behavior is by current Apache Kafka design. At topic creation time, for every topic partition, replication factor is converted to a replica set (set of ids of brokers which should replicate the partition), and those per partition replica sets is the metadata the gets stored in Zoo

Re: zookeeper.set.acl=false not considered by ZookeeperLeaderElector

Thanks Ismael, I've created https://issues.apache.org/jira/browse/KAFKA-4814 Kind regards, Stevo Slavic. On Tue, Feb 28, 2017 at 5:26 PM, Ismael Juma wrote: > Hi Stevo, > > That looks like a bug, can you please file a JIRA? > > Ismael > > On Mon, Feb 27, 2017 at 3:

zookeeper.set.acl=false not considered by ZookeeperLeaderElector

Hello Apache Kafka community, There's nice documentation on enabling ZooKeeper security on an existing Apache Kafka cluster at https://kafka.apache.org/documentation/#zk_authz_migration For your convenience here are the first two steps of migration: 1. Perform a rolling restart setting the J

Re: Is there any command can be used to get broker.id on one of broker with command?

You can just grep it from broker's config/server.properties On Thu, Feb 16, 2017 at 7:42 AM, kay James wrote: > Is there any command can be used to get broker.id on one of broker with > command? > I can check zookeeper or check the broker starting log to find broker.id, > but I wander whether th

Re: Fault tolerance not working in Kafka

Check __consumer-offsets topic, number of partitions and their replica assignment. You may be affected by problem solved by https://cwiki.apache.org/confluence/display/KAFKA/KIP-115%3A+Enforce+offsets.topic.replication.factor+upon+__consumer_offsets+auto+topic+creation On Mon, Feb 6, 2017, 06:32 R

Re: When publishing to non existing topic, TimeoutException is thrown instead of UnknownTopicOrPartitionException

Found somewhat related ticket https://issues.apache.org/ jira/browse/KAFKA-4385 On Fri, Jan 27, 2017 at 1:09 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > When using new async KafkaProducer, one can register callback with send > calls. > > With auto.create.to

When publishing to non existing topic, TimeoutException is thrown instead of UnknownTopicOrPartitionException

Hello Apache Kafka community, When using new async KafkaProducer, one can register callback with send calls. With auto.create.topics.enable set to false, when I try to publish to non existing topic, I expect callback to complete with UnknownTopicOrPartitionException. Instead, I get back "org.apac

Re: Reg: Need info on Kafka Brokers

Hello Sumit, Not yet. AFAIK few topic management requests got introduced in broker side (see https://issues.apache.org/jira/browse/KAFKA-2229 ) but not yet in client APIs. Querying/listing topics request doesn't seem even to be planned yet. AdminUtils which talks with ZooKeeper via ZkClient is Ka

Re: Unable to create new topics, "no brokers available"

Hello Alex, ZooKeeper nodes that Kafka brokers create in /brokers/ids path, upon registering themselves that they are available, are ephemeral, so not persistent. Ephemeral ZooKeeper nodes live as long as ZooKeeper session that created them is active. The /broker/ids child nodes are gone by design

Common client metrics - per request?

Hello Apache Kafka community, Please correct me if wrong, I assume common Kafka client metrics (see https://kafka.apache.org/documentation.html#selector_monitoring ) are aggregated metrics of all different requests particular client instance makes. So e.g. producer common metrics like outgoing-byt

Re: REST API to kafka topic

https://github.com/confluentinc/kafka-rest On Thu, Dec 15, 2016 at 8:12 PM, Gautham Kumaran < gautham.kumara...@gmail.com> wrote: > Hi Everyone, > > I'm trying to learn Kafka. So this might be basic :) > > I've come across the following API call to a kafka topic directly, but I > was under the im

Release Kafka 0.9.0.2?

Hello Apache Kafka community, Would it be possible to release 0.9.0.2? It has few important fixes, like KAFKA-3594 and would be helpful for us that cannot upgrade to 0.10.x yet. Kind regards, Stevo Slavic.

Re: [VOTE] Add REST Server to Apache Kafka

-1 I fully agree with Jay and Jeff. On Wed, Oct 26, 2016 at 12:03 AM, Jeff Widman wrote: > -1 > > As an end-user, while I like the idea in theory, in practice I don't think > it's a good idea just yet. > > Certainly, it'd be useful, enabling things like > https://github.com/Landoop/kafka-topics

Re: Migrating old consumer offsets to new consumer

Hello Nick, There are migration instructions in the docs (see http://kafka.apache.org/documentation.html#offsetmigration ) Kind regards, Stevo Slavic. On Wed, Sep 21, 2016 at 5:32 PM, Nick Kleinschmidt wrote: > I’m running the old 0.8 consumer storing offsets in Zookeeper, want to > migrate to

Re: Balancing based on lag

Hello Jens, By design there can be only one active consumer per consumer group per partion at a time, only one thread after being assigned (acquiring a lock) moves the offset for consumer group for partition, so no concurrency problems. Typically that assignment lasts long until rebalancing gets t

Re: kafka latency

Hello Chao, How did you measure latency? See also https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Kind regards, Stevo Slavic. On Tue, Jul 26, 2016 at 9:52 PM, Luo, Chao wrote: > Dear Kafka guys, > > I measured the latency of kafka

Re: Kafka Replication Factor

Hello Avi, Yes, replication factor of 1 means just one lead replica gets assigned for the topic partitions. To achieve HA of messages there are more things to watch out, not only replication factor - metadata/ZK HA, producer acks, min-in-sync replicas, rack awareness for replica assignment to nam

ZkUtils creating parent path with different ACL

Hello Apache Kafka community, Is there a reason why acls are not passed through in https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L404 call ? If one overrides and uses custom acls for a node, parent if missing might get created with default ACL. Kind reg

Re: PartitionMetadata hasLeader

Correction, leader was an Option, and I guess Optional cannot be used yet because Java language level compatibility is too low. What a mess, Java and Scala in codebase, and even new code being written in old Java. On Mon, Jun 20, 2016 at 12:08 PM, Stevo Slavić wrote: > Hello Apache Ka

PartitionMetadata hasLeader

Hello Apache Kafka community, Scala API had hasLeader method in PartitionMetadata of MetadatResponse. I see new Java API does not have such method and there's no javadoc on the class. Can somebody please explain - will leader reference be null or something when partition has no leader? Why not enc

Re: [DISCUSS] Java 8 as a minimum requirement

+1 on Java 8 already, Scala 2.11 is enough as well On Fri, Jun 17, 2016 at 9:48 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > +1 to Java8 - we have already moved to Java8 last year. > > On Fri, Jun 17, 2016 at 12:48 PM, Ismael Juma wrote: > > > Hi Harsha, > > > > Are you sayin

Re: Three consumers on a single partition

Replication factor should not be a problem, only lead is used for writes and reads. I'd monitor consumer offsets of all consumers of that topic - to make sure groupid is really same. Another thing could be that sarama client is simple, not implementing high level consumer like locking that only o

Kafka controller replica state docs outdated?

Hello Apache Kafka community, Is it intentional that not all states (like ReplicaDeletionIneligible) are documented on https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals or is the page just outdated? Btw I see replica states documented in javadoc https://github.com/apach

UnderReplicatedPartitions 0 while lead is only ISR member

Hello Apache Kafka community, On 0.9.0.1 cluster, with all brokers up, I have a topic with single partition and replication factor of 3, min ISR is 2. When topic was created all 3 assigned replicas are in ISR. Now: 1. All brokers report UnderReplicatedPartitions of 0 while describe topic reports

Re: New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Thanks for sharing Ismael! On Fri, Apr 1, 2016 at 6:24 PM, Ismael Juma wrote: > See: > > https://github.com/apache/kafka/pull/645 > https://github.com/apache/kafka/pull/735 > > Ismael > > On Fri, Apr 1, 2016 at 5:04 PM, Stevo Slavić wrote: > > > Correction, i

Re: New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Apr 1, 2016 at 5:52 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > With kafka-clients 0.9.0.1, async producer doesn't seem to handle deleted > topics well. > > If I create a topic, publish few messages to it, and delete it, > KafkaProducer instance and underlyi

New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Hello Apache Kafka community, With kafka-clients 0.9.0.1, async producer doesn't seem to handle deleted topics well. If I create a topic, publish few messages to it, and delete it, KafkaProducer instance and underlying NetworkClient which was used to publish to that topic, will endlessly every 10

Re: Kafka 0.8.2.2 is using high CPU

Consider upgrading to 0.9.0.1 - our cluster is much more stable since upgrade. Clients can be upgraded later. On Fri, Mar 18, 2016, 23:22 li jinyu wrote: > I want to add more information. > > we've already setup a completely new Kafka cluster with 0.8.2.2, and the > old cluster is still there, b

Re: How to get message count per topic?

See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Using metadata api one can get topic partitions and for each partition which broker is lead. Using offset api one can get partition size. Both apis are low level and to use them directly you would use SimpleConsume

Re: Deletion of topic on 0.9.0.0 spams this exception

I've recently created related ticket https://issues.apache.org/jira/browse/KAFKA-3390 On Mon, Mar 14, 2016, 20:54 Scott Reynolds wrote: > >Conditional update of path > >/brokers/topics/messages.events/partitions/0/state with data > > {"controller_epoch":2,"leader":492687262,"version":1,"leader_e

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

You are affected by this 0.9.0.0 bug https://issues.apache.org/jira/browse/KAFKA-2988 It was fixed for 0.9.0.1. You could just apply same fix to your 0.9.0.0 cluster but I'd recommend upgrading to 0.9.0.1. Kind regards, Stevo Slavic. On Mon, Mar 14, 2016, 07:10 Kunal Gupta wrote: > Hi everyone

Re: Kafka topic deletion still not working well

Created https://issues.apache.org/jira/browse/KAFKA-3389 and https://issues.apache.org/jira/browse/KAFKA-3390 On Sun, Mar 13, 2016 at 10:11 AM, Stevo Slavić wrote: > Is it normal/expected behavior during topic deletion that to-be-deleted > topic partitions ISR will get updated/shr

Re: Kafka topic deletion still not working well

Is it normal/expected behavior during topic deletion that to-be-deleted topic partitions ISR will get updated/shrink as replicas remove fetcher and delete partition log? On Fri, Mar 11, 2016 at 5:32 PM, Stevo Slavić wrote: > Didn't finish sentence, wanted to change font but managed to

Re: Kafka topic deletion still not working well

process deleted the state node. Will check more logs, Kafka and ZK, if I can find what deleted the state node. Any ideas? On Fri, Mar 11, 2016 at 5:22 PM, Stevo Slavić wrote: > Sorry, I do not know if this is related to KAFKA-2937, I let you judge and > decide whether ticket needs to be r

Re: Kafka topic deletion still not working well

; controller and then the controller dies and the new controller gets in to > > the exception and never completes deleting the topic. The topic existed > in > > the cluster and was also marked for delete. > > The case reported here is that the topic does not exist in cluster but i

Re: Kafka topic deletion still not working well

caDeletionSuccessful) (kafka.controller.ReplicaStateMachine) Kind regards, Stevo Slavic. On Sun, Mar 6, 2016 at 12:31 AM, Guozhang Wang wrote: > Thanks Stevo, > > Feel free to paste your findings in KAFKA-2937, we can re-open that ticket > if necessary. > > Guozhang > > On Fri, Mar 4, 2016

test-log4j.properties purpose

Hello Apache Kafka community, Is config/test-log4j.properties used for tests only? If so, would it make sense to remove it from binary distribution archive? It's in kafka_2.11-0.9.0.1.tgz Kind regards, Stevo Slavic.

Kafka topic deletion still not working well

Hell Apache Kafka community, I'm still investigating an incident; from initial findings topic deletion doesn't seem to work well still with Kafka 0.9.0.1, likely some edge case not covered. Before with 0.8.2.x it used to happen that non-lead replica would be stuck in topic deletion process, and w

Re: Apache Kafka Case Studies

Hello Joe, Consider registering for http://www.oreilly.com/online-training/building-distributed-pipelines-for-data-science.html and/or for http://www.kafka-summit.org/ Just as an idea where you could get ideas :) Btw, there's no such thing as best practice - only practice/approach that worked in p

min ISR of 1 for consumer offsets topic and disabling unclean leader election - problem or not?

Hello Apache Kafka community, Consumer offsets topic default configuration is somewhat different from all other topics for a good reason, e.g. replication factor is 3 compared to 1 for other topics, and number of partitions is 50 compared to 1 for other topic. Can somebody please explain why does

Default max retries diff between old and new producer

Hello Apache Kafka community, Old producer had message.send.max.retries defaulting to 3, while new producer retries config property defaults to 0. Can somebody please explain what is the rationale to use 0 retries as default in new producer? Kind regards, Stevo Slavic.

Re: Group coordinator in 0.9

Hello Oleksiy, All offsets for given consumer group are stored in (and fetched from) one and only one partition of all partitions of __consumer_offsets topic. By default __consumer_offsets has 50 partitions and for each of these partitions replication factor is 3, so there is one lead and 2 extra

Re: Migrate a topic which has no brokers

Hello Ben, Yes, you can use apply different replica assignment. See related docs: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Kind regards, Stevo Slavic. On Tue, Jan 5, 2016 at 5:32 PM, Ben Davison wrote: > Hi All, > > On our dev kafka environment our broke

Re: Does MirrorMaker delete topics?

; super-cluster replicate topics onto the secondary site but don't take part > in topic leadership - is this on the cards at all? > > Thanks, > SimonC > > -Original Message- > From: Stevo Slavić [mailto:ssla...@gmail.com] > Sent: 05 January 2016 14:04 > To:

Re: Does MirrorMaker delete topics?

AFAIK "if the right options are set" actually means "if auto.create.topics.enable is left to default, set to true". As on any Kafka cluster with this configuration option, this will allow implicit topic creation, e.g. on first message being published to the topic, if topic does not exist it will ge

Re: How to reset a consumer-group's offset in kafka 0.9?

t be done immediately after the group is > removed, but only after the offsets expiration has elapsed. > > We can add this feature as an admin command in future versions. > > Guozhang > > > On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić wrote: > > > Created related ti

Re: How to reset a consumer-group's offset in kafka 0.9?

; > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. > > And > > > > also > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker > --zookeeper > > > > > localhost:2181 --group group-name` seems broken. >

Re: How to reset a consumer-group's offset in kafka 0.9?

and consumers all the messages. > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić : > > > Have you considered deleting and recreating topic used in test? > > Once topic is clean, read/poll once - any committed offset should be > > outside of the range, and consumer should res

Re: How to reset a consumer-group's offset in kafka 0.9?

Have you considered deleting and recreating topic used in test? Once topic is clean, read/poll once - any committed offset should be outside of the range, and consumer should reset offset. On Tue, Dec 29, 2015 at 4:11 PM, Han JU wrote: > Hello, > > For local test purpose I need to frequently res

Re: Unable to set log.cleanup.policy on topic

Hello Rakesh, log.cleanup.policy is broker configuration property, while cleanup.policy is topic configuration property (see http://kafka.apache.org/documentation.html#topic-config ). Since you are configuring particular topic, you need to use second one. Kind regards, Stevo Slavic. On Mon, Dec

Re: Failed attempt to delete topic

Delete was actually considered to be working since Kafka 0.8.2 (although there are still not easily reproducible edge cases when it doesn't work well even in in 0.8.2 or newer). In 0.8.1 one could request topic to be deleted (request gets stored as entry in ZooKeeper), because of presence of the re

Re: Problems getting offsets

ystem? Will it affect how logs are managed, > and when “older” messages are purged? Or are they two independent systems? > > On 11/2/15, 03:51, "Stevo Slavić" wrote: > > >Here is a bit longer and more detailed, not necessarily better > >understandable explanatio

Re: Blocked consumer - next offset same as current offset

It's > the responsibility of the client to skip over those messages. Note that the > high level consumer handles that logic already. > > Thanks, > > Jun > > On Wed, Nov 11, 2015 at 12:40 AM, Stevo Slavić wrote: > > > Hello Apache Kafka community, > > &g

Re: Blocked consumer - next offset same as current offset

would get triggered only when consumer lag is of appropriate size - I guess async producer made this issue more likely, because of batch sending multiple messages to same partition. On Wed, Nov 11, 2015 at 9:40 AM, Stevo Slavić wrote: > Hello Apache Kafka community, > > > I'm usin

Blocked consumer - next offset same as current offset

Hello Apache Kafka community, I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some conditions fetch response message set for a partition can contain at least one (if not all) MessageAndOffset with nextOffset being equal to current (committed) offset, offset used in fetch reque

Scheduled job in Kafka, every 2h?

Hello Apache Kafka community, I'm seeing some cyclic load on cluster, trying to determine cause. Couldn't determine from docs - is there some internal scheduled job in Kafka (0.8.2.1) executing on every broker every two hours? Kind regards, Stevo Slavic.

Re: Problems getting offsets

m is not with your topic, but with internal consumer offsets topic. On Mon, Nov 2, 2015 at 1:56 AM, Stevo Slavić wrote: > Hello David, > > In short, problem is not with your topic, it is with consumer offsets > topic initialization. > > You could modify your code to just retry fetchin

Re: Problems getting offsets

Hello David, In short, problem is not with your topic, it is with consumer offsets topic initialization. You could modify your code to just retry fetching offsets (until successful where successful is also return of -1 offset, or timeout), or alternatively you could trigger consumer offsets topic

New producer and storing offsets in Kafka - previously committed offsets fetched as uncommitted

Hello Apache Kafka community, I'm trying to use new producer, from kafka-clients 0.8.2.2, together with simple consumer to fetch and commit offsets stored in Kafka, and I'm seeing strange behavior - a committed offset/message gets read multiple times, offset fetch requests do not always see commit

Re: ("offsets.storage","kafka") not working with Consumer API

Hello Kiran, Check how many brokers you have in the cluster. Consumer offsets topic requires by default at least 3. In dev environment you could lower replication factor for that topic (see broker config options). Kind regards, Stevo Slavic. On Fri, Oct 16, 2015, 07:31 Kiran Singh wrote: > Hel

Re: How to verify offsets topic exists?

32 PM, Stevo Slavić wrote: > There's another related bug - triggering offsets topic creation through > requesting metadata about that topic does not work in case of single broker > clean (no topics created yet) Kafka cluster running. In that case sequence > returned by KafkaApis.ge

Re: How to verify offsets topic exists?

ation factor for offsets topic. Does this make sense? As workaround, I guess I will have to resort to explicitly creating offsets topic if it doesn't exist already. Kind regards, Stevo Slavic. On Tue, Oct 6, 2015 at 11:34 AM, Stevo Slavić wrote: > Debugged, and found in KafkaApis.handl

Re: How to verify offsets topic exists?

o Slavic. On Tue, Oct 6, 2015 at 10:02 AM, Stevo Slavić wrote: > Thanks Grant for quick reply! > > I've used AdminUtils.topicExists("__consumer_offsets") check and even > 10sec after Kafka broker startup, the check fails. > > When, on which event, does this inte

Re: How to verify offsets topic exists?

ers/topics/__consumer_offsets > > get /brokers/topics/__consumer_offsets > > > Thanks, > Grant > > On Mon, Oct 5, 2015 at 10:44 AM, Stevo Slavić wrote: > > > Hello Apache Kafka community, > > > > In my integration tests, with single 0.8.2.2 broker, for newly

How to verify offsets topic exists?

Hello Apache Kafka community, In my integration tests, with single 0.8.2.2 broker, for newly created topic with single partition, after determining through topic metadata request that partition has lead broker assigned, when I try to reset offset for given consumer group, I first try to discover o

Re: New Consumer & committed offsets

Hello Damian, Yes, there's a +1 difference. See related discussion http://mail-archives.apache.org/mod_mbox/kafka-users/201507.mbox/%3CCAOeJiJh2SMzVn23JsoWiNk3sfsw82Jr_-kRLcNRd-oZ7pR1yWg%40mail.gmail.com%3E Kind regards, Stevo Slavic. On Tue, Sep 15, 2015 at 3:56 PM, Damian Guy wrote: > I turn

Re: Unclean leader election docs outdated

Created https://issues.apache.org/jira/browse/KAFKA-2551 On Mon, Sep 14, 2015 at 7:22 PM, Guozhang Wang wrote: > Yes you are right. Could you file a JIRA to edit the documents? > > Guozhang > > On Fri, Sep 11, 2015 at 4:41 PM, Stevo Slavić wrote: > > > That sente

Re: 0.9.0.0 remaining jiras

has "Affects Version/s" set to 0.9, maybe "Fix Version/s" should be set to that value instead. Kind regards, Stevo Slavic. On Mon, Sep 14, 2015 at 9:43 AM, Stevo Slavić wrote: > Hello Jason, > > Maybe this answers your question: > http://mail-archives.apache.or

Re: 0.9.0.0 remaining jiras

Hello Jason, Maybe this answers your question: http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAFc58G-UScVKrSF1kdsowQ8Y96OAaZEdiZsk40G8fwf7iToFaw%40mail.gmail.com%3E Kind regards, Stevo Slavic. On Mon, Sep 14, 2015 at 8:56 AM, Jason Rosenberg wrote: > Hi Jun, > > Can you cl

Re: open source projects based on kafka

Have a look at https://github.com/allegro/hermes On Mon, Sep 14, 2015, 01:28 David Luu wrote: > The toy project idea is good. Another option I think could be to look at > the various Kafka client langague bindings and/or utilities (like > kafkacat). And from there, another option is to build a c

Re: Unclean leader election docs outdated

mentions ability to disable unclean leader election, so likely just this one reference needs to be updated. On Sat, Sep 12, 2015 at 1:05 AM, Guozhang Wang wrote: > Hi Stevo, > > Could you point me to the link of the docs? > > Guozhang > > On Fri, Sep 11, 2015 at 5:47 AM, Stev

Unclean leader election docs outdated

Hello Apache Kafka community, Current unclean leader election docs state: "In the future, we would like to make this configurable to better support use cases where downtime is preferable to inconsistency. " If I'm not mistaken, since 0.8.2, unclean leader election strategy (whether to allow it or

Re: [DISCUSSION] Kafka 0.8.2.2 release?

> committed yet. > > > > Hope that helps the effort. > > > > Thanks, > > Grant > > > > On Mon, Aug 17, 2015 at 12:09 AM, Grant Henke > wrote: > > > >> +1 to that suggestion. Though I suspect that requires a committer to do. > &g

Re: [DISCUSSION] Kafka 0.8.2.2 release?

+1 (non-binding) for 0.8.2.2 release Would be nice to include in that release new producer resiliency bug fixes https://issues.apache.org/jira/browse/KAFKA-1788 and https://issues.apache.org/jira/browse/KAFKA-2120 On Fri, Aug 14, 2015 at 4:03 PM, Gwen Shapira wrote: > Will be nice to include Ka

Re: 0.8.3 ETA?

updated > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan). > > As we are getting closer, we will clean up the 0.8.3 jiras and push > non-critical ones to future releases. > > Thanks, > > Jun > > On Mon, Aug 3, 2015 at 5:52 AM, Stevo Slavić wrote: > >

0.8.3 ETA?

Hello Apache Kafka community, If I recall well, two weeks ago it was mentioned in a discussion that Kafka 0.8.3 might be released in a month time. Is this still Kafka dev team goal, in few weeks time to have Kafka 0.8.3 released? Or is more (re)work (e.g. more new consumer API changes) planned fo

Re: 0.8.2 Producer sends blocking when brokers are unavailable

Hello David, It's a known issue, see https://issues.apache.org/jira/browse/KAFKA-1788 and https://issues.apache.org/jira/browse/KAFKA-2120 Kind regards, Stevo Slavic. On Fri, Jul 31, 2015 at 10:15 AM, David KOCH wrote: > Hello, > > The new producer org.apache.kafka.clients.producer.KafkaProduc

Re: New consumer - offset one gets in poll is not offset one is supposed to commit

set(partition))? > > Thanks, > Jason > > On Fri, Jul 24, 2015 at 10:11 AM, Stevo Slavić wrote: > > > Hello Apache Kafka community, > > > > Say there is only one topic with single partition and a single message on > > it. > > Result of calling a poll with

New consumer - offset one gets in poll is not offset one is supposed to commit

Hello Apache Kafka community, Say there is only one topic with single partition and a single message on it. Result of calling a poll with new consumer will return ConsumerRecord for that message and it will have offset of 0. After processing message, current KafkaConsumer implementation expects o

Re: New consumer - partitions auto assigned only on poll

ubscriptions() other than to keep it > non-blocking. Perhaps you can open a ticket and we can get feedback from > some other devs? > > Thanks, > Jason > > On Wed, Jul 22, 2015 at 2:09 AM, Stevo Slavić wrote: > > > Hello Apache Kafka community, > > >

Re: New consumer - poll/seek javadoc confusing, need clarification

Strange, if after seek I make several poll requests, eventually it will read/return messages from offset that seek set. On Thu, Jul 23, 2015 at 11:03 AM, Stevo Slavić wrote: > Thanks Ewen for heads up. > > It's great that seek is not needed in between poll when business goes as &

Re: New consumer - poll/seek javadoc confusing, need clarification

unclean leader election is not > enabled). > > On Tue, Jul 21, 2015 at 9:11 PM, James Cheng wrote: > > > > > > On Jul 21, 2015, at 9:15 AM, Ewen Cheslack-Postava > > wrote: > > > > > > On Tue, Jul 21, 2015 at 2:38 AM, Stevo Slavić > wrote: &g

Re: High Level Consumer use case

Hello Sreenivasulu, It's automatic, just start them, and as each HLC starts it registers in ZK, rebalancing of the HLC to partition assignments happens. Be gentle when starting consumers, there is a bug reported that if multiple HLCs are started in short time, some of them may end up without any

Re: New consumer - ConsumerRecords partitions

ions which had messages. > Would you mind creating a jira for the feature request? You're welcome to > submit a patch as well. > > -Jason > > On Tue, Jul 21, 2015 at 2:27 AM, Stevo Slavić wrote: > > > Hello Apache Kafka community, > > > > New HLC poll r

New consumer - partitions auto assigned only on poll

Hello Apache Kafka community, In the new consumer I encountered unexpected behavior. After constructing KafakConsumer instance with configured consumer rebalance callback handler, and subscribing to a topic with "consumer.subscribe(topic)", retrieving subscriptions would return empty set and callb

New consumer - offset storage options

Hello Apache Kafka community, It seems new high level consumer coming in 0.8.3 will support only offset storage in Kafka topic. Can somebody please confirm/comment? Kind regards, Stevo Slavic.

New producer hangs inifitely when it looses connection to Kafka cluster

Hello Apache Kafka community, Just noticed that : - message is successfully published using new 0.8.2.1 producer - and then Kafka is stopped next attempt to publish message using same instance of new producer hangs forever, and following stacktrace gets logged repeatedly: [WARN ] [o.a.kafka.comm

  1   2   >