Re: How to reset a consumer-group's offset in kafka 0.9?

; > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. > > And > > > > also > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker > --zookeeper > > > > > localhost:2181 --group group-name` seems broken. >

Re: How to reset a consumer-group's offset in kafka 0.9?

t be done immediately after the group is > removed, but only after the offsets expiration has elapsed. > > We can add this feature as an admin command in future versions. > > Guozhang > > > On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić wrote: > > > Created related ti

Re: Does MirrorMaker delete topics?

AFAIK "if the right options are set" actually means "if auto.create.topics.enable is left to default, set to true". As on any Kafka cluster with this configuration option, this will allow implicit topic creation, e.g. on first message being published to the topic, if topic does not exist it will ge

Re: Does MirrorMaker delete topics?

; super-cluster replicate topics onto the secondary site but don't take part > in topic leadership - is this on the cards at all? > > Thanks, > SimonC > > -Original Message- > From: Stevo Slavić [mailto:ssla...@gmail.com] > Sent: 05 January 2016 14:04 > To:

Re: Migrate a topic which has no brokers

Hello Ben, Yes, you can use apply different replica assignment. See related docs: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Kind regards, Stevo Slavic. On Tue, Jan 5, 2016 at 5:32 PM, Ben Davison wrote: > Hi All, > > On our dev kafka environment our broke

Re: Group coordinator in 0.9

Hello Oleksiy, All offsets for given consumer group are stored in (and fetched from) one and only one partition of all partitions of __consumer_offsets topic. By default __consumer_offsets has 50 partitions and for each of these partitions replication factor is 3, so there is one lead and 2 extra

Default max retries diff between old and new producer

Hello Apache Kafka community, Old producer had message.send.max.retries defaulting to 3, while new producer retries config property defaults to 0. Can somebody please explain what is the rationale to use 0 retries as default in new producer? Kind regards, Stevo Slavic.

min ISR of 1 for consumer offsets topic and disabling unclean leader election - problem or not?

Hello Apache Kafka community, Consumer offsets topic default configuration is somewhat different from all other topics for a good reason, e.g. replication factor is 3 compared to 1 for other topics, and number of partitions is 50 compared to 1 for other topic. Can somebody please explain why does

Re: Apache Kafka Case Studies

Hello Joe, Consider registering for http://www.oreilly.com/online-training/building-distributed-pipelines-for-data-science.html and/or for http://www.kafka-summit.org/ Just as an idea where you could get ideas :) Btw, there's no such thing as best practice - only practice/approach that worked in p

Kafka topic deletion still not working well

Hell Apache Kafka community, I'm still investigating an incident; from initial findings topic deletion doesn't seem to work well still with Kafka 0.9.0.1, likely some edge case not covered. Before with 0.8.2.x it used to happen that non-lead replica would be stuck in topic deletion process, and w

test-log4j.properties purpose

Hello Apache Kafka community, Is config/test-log4j.properties used for tests only? If so, would it make sense to remove it from binary distribution archive? It's in kafka_2.11-0.9.0.1.tgz Kind regards, Stevo Slavic.

Re: Kafka topic deletion still not working well

caDeletionSuccessful) (kafka.controller.ReplicaStateMachine) Kind regards, Stevo Slavic. On Sun, Mar 6, 2016 at 12:31 AM, Guozhang Wang wrote: > Thanks Stevo, > > Feel free to paste your findings in KAFKA-2937, we can re-open that ticket > if necessary. > > Guozhang > > On Fri, Mar 4, 2016

Re: Kafka topic deletion still not working well

; controller and then the controller dies and the new controller gets in to > > the exception and never completes deleting the topic. The topic existed > in > > the cluster and was also marked for delete. > > The case reported here is that the topic does not exist in cluster but i

Re: Kafka topic deletion still not working well

process deleted the state node. Will check more logs, Kafka and ZK, if I can find what deleted the state node. Any ideas? On Fri, Mar 11, 2016 at 5:22 PM, Stevo Slavić wrote: > Sorry, I do not know if this is related to KAFKA-2937, I let you judge and > decide whether ticket needs to be r

Re: Kafka topic deletion still not working well

Is it normal/expected behavior during topic deletion that to-be-deleted topic partitions ISR will get updated/shrink as replicas remove fetcher and delete partition log? On Fri, Mar 11, 2016 at 5:32 PM, Stevo Slavić wrote: > Didn't finish sentence, wanted to change font but managed to

Re: Kafka topic deletion still not working well

Created https://issues.apache.org/jira/browse/KAFKA-3389 and https://issues.apache.org/jira/browse/KAFKA-3390 On Sun, Mar 13, 2016 at 10:11 AM, Stevo Slavić wrote: > Is it normal/expected behavior during topic deletion that to-be-deleted > topic partitions ISR will get updated/shr

Re: Need a help in understanding __consumer_offsets topic creation in Kafka Cluster

You are affected by this 0.9.0.0 bug https://issues.apache.org/jira/browse/KAFKA-2988 It was fixed for 0.9.0.1. You could just apply same fix to your 0.9.0.0 cluster but I'd recommend upgrading to 0.9.0.1. Kind regards, Stevo Slavic. On Mon, Mar 14, 2016, 07:10 Kunal Gupta wrote: > Hi everyone

Re: Deletion of topic on 0.9.0.0 spams this exception

I've recently created related ticket https://issues.apache.org/jira/browse/KAFKA-3390 On Mon, Mar 14, 2016, 20:54 Scott Reynolds wrote: > >Conditional update of path > >/brokers/topics/messages.events/partitions/0/state with data > > {"controller_epoch":2,"leader":492687262,"version":1,"leader_e

Re: How to get message count per topic?

See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Using metadata api one can get topic partitions and for each partition which broker is lead. Using offset api one can get partition size. Both apis are low level and to use them directly you would use SimpleConsume

Re: Kafka 0.8.2.2 is using high CPU

Consider upgrading to 0.9.0.1 - our cluster is much more stable since upgrade. Clients can be upgraded later. On Fri, Mar 18, 2016, 23:22 li jinyu wrote: > I want to add more information. > > we've already setup a completely new Kafka cluster with 0.8.2.2, and the > old cluster is still there, b

"client" and requests in request.timeout.ms

Hello Apache Kafka community, In Consumer, Producer, AdminClient and Broker configuration documentation there's a common config property, request.timeout.ms, with common description part being: "The configuration controls the maximum amount of time the client will wait for the response of a reques

Is CommitFailedException thrown from commitSync with offsets because an offset is out of range?

Hello Apache Kafka community, org.apache.kafka.clients.consumer.CommitFailedException javadoc and especially exception message suggests that the exception can be thrown only due to consumer re-balancing, KafkaConsumer instance not owning TopicPartition for the consumer group of the offset being co

Re: Is CommitFailedException thrown from commitSync with offsets because an offset is out of range?

Reported a bug for this https://issues.apache.org/jira/browse/KAFKA-5835 On Wed, Aug 30, 2017 at 10:30 AM, Stevo Slavić wrote: > Hello Apache Kafka community, > > org.apache.kafka.clients.consumer.CommitFailedException javadoc and > especially exception message suggests that the e

Streams dependency on RocksDB - licenses

Hello Apache Kafka community, Is it on purpose that kafka-streams 0.11.0.1 depends on org.rocksdb:rocksdbjni:5.0.1, and not on newer 5.7.3, because 5.0.1 has Apache 2 license while 5.7.3 has also GPL 2.0 parts? Kind regards, Stevo Slavic.

Re: Streams dependency on RocksDB - licenses

/6e3ee015fb1ce03e47838e9a3995410ce884c212#diff-1d5a1a614aa158e14292a60d9891f1df Shouldn't similar notice be added to Apache Kafka too? On Thu, Sep 21, 2017 at 3:43 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > Is it on purpose that kafka-streams 0.11.0.1 depends on > org.rocksdb:rocksdbj

Re: How would Kafka behave in this scenario

Hello Sameer, Behavior depends on min.insync.replicas configured for the topic. Find more info in the documentation https://kafka.apache.org/documentation/#topicconfigs Kind regards, Stevo Slavic. On Tue, Sep 26, 2017 at 9:01 AM, Sameer Kumar wrote: > In case one of the brokers fail, the brok

Re: Streams dependency on RocksDB - licenses

mit you referenced, adds GPL2 to the list of licenses in > the > > pom, but it's not clear why. > > > > Ismael > > > > On Thu, Sep 21, 2017 at 3:44 PM, Stevo Slavić wrote: > > > >> LevelDB GPL2 notice seems to have been added to rocksdb 5

Re: Streams dependency on RocksDB - licenses

Created https://issues.apache.org/jira/browse/KAFKA-5977 for this issue. On Tue, Sep 26, 2017 at 9:49 AM, Stevo Slavić wrote: > There is legal problem for older rocksdb versions, see > https://issues.apache.org/jira/browse/LEGAL-303?focusedCommentId=16109870&am

Re: How would Kafka behave in this scenario

ks Stevo for pointing me out to correct link. > > > In this case, how would exactly once feature of streams would behave > > since > > > they configure producers with acks=all. I think they would fail and > would > > > need to be resumed once the broker comes back. >

Re: How to reduce time for consumer to take over?

Hello Christian, Assuming that you are using new Java kafka-clients with Kafka's consumer group management, you'd have to tune consumers and broker. See - Broker config https://kafka.apache.org/documentation/#brokerconfigs -- group.initial.rebalance.delay.ms -- group.max.session.timeout.ms -- gro

New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Hello Apache Kafka community, With kafka-clients 0.9.0.1, async producer doesn't seem to handle deleted topics well. If I create a topic, publish few messages to it, and delete it, KafkaProducer instance and underlying NetworkClient which was used to publish to that topic, will endlessly every 10

Re: New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Apr 1, 2016 at 5:52 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > With kafka-clients 0.9.0.1, async producer doesn't seem to handle deleted > topics well. > > If I create a topic, publish few messages to it, and delete it, > KafkaProducer instance and underlyi

Re: New async producer NetworkClient endlessly trying to fetch metadata for deleted topics

Thanks for sharing Ismael! On Fri, Apr 1, 2016 at 6:24 PM, Ismael Juma wrote: > See: > > https://github.com/apache/kafka/pull/645 > https://github.com/apache/kafka/pull/735 > > Ismael > > On Fri, Apr 1, 2016 at 5:04 PM, Stevo Slavić wrote: > > > Correction, i

UnderReplicatedPartitions 0 while lead is only ISR member

Hello Apache Kafka community, On 0.9.0.1 cluster, with all brokers up, I have a topic with single partition and replication factor of 3, min ISR is 2. When topic was created all 3 assigned replicas are in ISR. Now: 1. All brokers report UnderReplicatedPartitions of 0 while describe topic reports

Kafka controller replica state docs outdated?

Hello Apache Kafka community, Is it intentional that not all states (like ReplicaDeletionIneligible) are documented on https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals or is the page just outdated? Btw I see replica states documented in javadoc https://github.com/apach

Re: Three consumers on a single partition

Replication factor should not be a problem, only lead is used for writes and reads. I'd monitor consumer offsets of all consumers of that topic - to make sure groupid is really same. Another thing could be that sarama client is simple, not implementing high level consumer like locking that only o

Re: [DISCUSS] Java 8 as a minimum requirement

+1 on Java 8 already, Scala 2.11 is enough as well On Fri, Jun 17, 2016 at 9:48 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > +1 to Java8 - we have already moved to Java8 last year. > > On Fri, Jun 17, 2016 at 12:48 PM, Ismael Juma wrote: > > > Hi Harsha, > > > > Are you sayin

PartitionMetadata hasLeader

Hello Apache Kafka community, Scala API had hasLeader method in PartitionMetadata of MetadatResponse. I see new Java API does not have such method and there's no javadoc on the class. Can somebody please explain - will leader reference be null or something when partition has no leader? Why not enc

Re: PartitionMetadata hasLeader

Correction, leader was an Option, and I guess Optional cannot be used yet because Java language level compatibility is too low. What a mess, Java and Scala in codebase, and even new code being written in old Java. On Mon, Jun 20, 2016 at 12:08 PM, Stevo Slavić wrote: > Hello Apache Ka

ZkUtils creating parent path with different ACL

Hello Apache Kafka community, Is there a reason why acls are not passed through in https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/ZkUtils.scala#L404 call ? If one overrides and uses custom acls for a node, parent if missing might get created with default ACL. Kind reg

Re: Kafka Replication Factor

Hello Avi, Yes, replication factor of 1 means just one lead replica gets assigned for the topic partitions. To achieve HA of messages there are more things to watch out, not only replication factor - metadata/ZK HA, producer acks, min-in-sync replicas, rack awareness for replica assignment to nam

Re: kafka latency

Hello Chao, How did you measure latency? See also https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Kind regards, Stevo Slavic. On Tue, Jul 26, 2016 at 9:52 PM, Luo, Chao wrote: > Dear Kafka guys, > > I measured the latency of kafka

Re: Balancing based on lag

Hello Jens, By design there can be only one active consumer per consumer group per partion at a time, only one thread after being assigned (acquiring a lock) moves the offset for consumer group for partition, so no concurrency problems. Typically that assignment lasts long until rebalancing gets t

Re: Migrating old consumer offsets to new consumer

Hello Nick, There are migration instructions in the docs (see http://kafka.apache.org/documentation.html#offsetmigration ) Kind regards, Stevo Slavic. On Wed, Sep 21, 2016 at 5:32 PM, Nick Kleinschmidt wrote: > I’m running the old 0.8 consumer storing offsets in Zookeeper, want to > migrate to

Re: [VOTE] Add REST Server to Apache Kafka

-1 I fully agree with Jay and Jeff. On Wed, Oct 26, 2016 at 12:03 AM, Jeff Widman wrote: > -1 > > As an end-user, while I like the idea in theory, in practice I don't think > it's a good idea just yet. > > Certainly, it'd be useful, enabling things like > https://github.com/Landoop/kafka-topics

Release Kafka 0.9.0.2?

Hello Apache Kafka community, Would it be possible to release 0.9.0.2? It has few important fixes, like KAFKA-3594 and would be helpful for us that cannot upgrade to 0.10.x yet. Kind regards, Stevo Slavic.

Re: REST API to kafka topic

https://github.com/confluentinc/kafka-rest On Thu, Dec 15, 2016 at 8:12 PM, Gautham Kumaran < gautham.kumara...@gmail.com> wrote: > Hi Everyone, > > I'm trying to learn Kafka. So this might be basic :) > > I've come across the following API call to a kafka topic directly, but I > was under the im

Common client metrics - per request?

Hello Apache Kafka community, Please correct me if wrong, I assume common Kafka client metrics (see https://kafka.apache.org/documentation.html#selector_monitoring ) are aggregated metrics of all different requests particular client instance makes. So e.g. producer common metrics like outgoing-byt

Re: Unable to create new topics, "no brokers available"

Hello Alex, ZooKeeper nodes that Kafka brokers create in /brokers/ids path, upon registering themselves that they are available, are ephemeral, so not persistent. Ephemeral ZooKeeper nodes live as long as ZooKeeper session that created them is active. The /broker/ids child nodes are gone by design

Re: Reg: Need info on Kafka Brokers

Hello Sumit, Not yet. AFAIK few topic management requests got introduced in broker side (see https://issues.apache.org/jira/browse/KAFKA-2229 ) but not yet in client APIs. Querying/listing topics request doesn't seem even to be planned yet. AdminUtils which talks with ZooKeeper via ZkClient is Ka

When publishing to non existing topic, TimeoutException is thrown instead of UnknownTopicOrPartitionException

Hello Apache Kafka community, When using new async KafkaProducer, one can register callback with send calls. With auto.create.topics.enable set to false, when I try to publish to non existing topic, I expect callback to complete with UnknownTopicOrPartitionException. Instead, I get back "org.apac

Re: When publishing to non existing topic, TimeoutException is thrown instead of UnknownTopicOrPartitionException

Found somewhat related ticket https://issues.apache.org/ jira/browse/KAFKA-4385 On Fri, Jan 27, 2017 at 1:09 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > When using new async KafkaProducer, one can register callback with send > calls. > > With auto.create.to

Re: Fault tolerance not working in Kafka

Check __consumer-offsets topic, number of partitions and their replica assignment. You may be affected by problem solved by https://cwiki.apache.org/confluence/display/KAFKA/KIP-115%3A+Enforce+offsets.topic.replication.factor+upon+__consumer_offsets+auto+topic+creation On Mon, Feb 6, 2017, 06:32 R

Re: Is there any command can be used to get broker.id on one of broker with command?

You can just grep it from broker's config/server.properties On Thu, Feb 16, 2017 at 7:42 AM, kay James wrote: > Is there any command can be used to get broker.id on one of broker with > command? > I can check zookeeper or check the broker starting log to find broker.id, > but I wander whether th

zookeeper.set.acl=false not considered by ZookeeperLeaderElector

Hello Apache Kafka community, There's nice documentation on enabling ZooKeeper security on an existing Apache Kafka cluster at https://kafka.apache.org/documentation/#zk_authz_migration For your convenience here are the first two steps of migration: 1. Perform a rolling restart setting the J

Re: zookeeper.set.acl=false not considered by ZookeeperLeaderElector

Thanks Ismael, I've created https://issues.apache.org/jira/browse/KAFKA-4814 Kind regards, Stevo Slavic. On Tue, Feb 28, 2017 at 5:26 PM, Ismael Juma wrote: > Hi Stevo, > > That looks like a bug, can you please file a JIRA? > > Ismael > > On Mon, Feb 27, 2017 at 3:

Re: Kafka partition no migrating to another broker.

Hello Shri, That behavior is by current Apache Kafka design. At topic creation time, for every topic partition, replication factor is converted to a replica set (set of ids of brokers which should replicate the partition), and those per partition replica sets is the metadata the gets stored in Zoo

Re: Kafka client and ELB (round-robin DNS)

See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html On Fri, Mar 3, 2017, 20:57 BGCH wrote: > Hi all, > > briefly about the problem, if bootstrap.servers is a domain name and > that domain name eventually starts pointing to different IP, kafka > client will never t

Re: Kafka consumer offset reset

Hello Jakub, Maybe you're interested in feature that's not yet available but is being proposed and discussed (on dev ML) for future - see https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling Kind regards, Stevo Slavic. On Tue, Mar 21, 2017 at 4:43

Re: Message polling behavior when subscribed to a topic pattern

Hello William, You have overriden KafkaConsumer's max.poll.records configuration, to return only up to 10 records in single call to poll, so it does that. Fetcher iterates over completed fetch responses and in first one(s) from single topic finds enough of records to satisfy max limit. See https:/

Building Kafka 0.8.1 branch with specific Scala version

Hello Kafka community, I'm trying to import Kafka, 0.8.1 branch, in eclipse IDE. Gradle eclipse plugin is already applied in Kafka build script. If I just run "gradle eclipse", default scala 2.8.0 will be used to generate eclipse project files, so classpath will exclude kafka/utils/Annotations_2.

Re: Building Kafka 0.8.1 branch with specific Scala version

Found that the posh-git shell which comes with GitHub for Windows is causing this odd CLI behavior (see here for related discussion on Gradle forum: http://forums.gradle.org/gradle/topics/which_characters_are_allowed_in_value_of_gradle_project_properties) On Wed, Apr 9, 2014 at 3:13 PM, Stevo

<    1   2