does kafka support "COMMIT" of a batch ?

nting. does kafka support a "BATCH mode" with commit ? so basically at the start of my session, I declare "start transaction", and after all records have been pulled and sent to kafka, I declare "commit" . Thanks Yang

Re: does kafka support "COMMIT" of a batch ?

well I guess this is what I need https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka anyone knows how far this effort has gone? thanks Yang On Mon, Jan 26, 2015 at 12:06 PM, Yang wrote: > we are pulling from a DB, maybe 100k rows at a time. for each row we s

Re: does kafka support "COMMIT" of a batch ?

thanks for the info yang On Mon, Jan 26, 2015 at 2:09 PM, Jay Kreps wrote: > We did a relatively complete prototype but it isn't integrated into the > mainline code yet and there isn't a target release date. There is rather a > lot of testing and compatability work that wo

generics type for Producer and Consumer do not need to match?

words, how do we determine the K,V types on consumer side? thanks Yang

Re: generics type for Producer and Consumer do not need to match?

byte[] it is effectively a no-op > conversion. > > On Thu, Feb 05, 2015 at 12:11:03PM -0800, Yang wrote: > > in the example > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example > > > > we use a String,String for > > > > in the same

java API equivalent of --from-beginning?

we are trying to achive the equivalent of bin/kafka-console-consumer.sh we tried " props.put("auto.offset.reset", "smallest"); but it still doesn't work. how can I achieve --from-beginning in java code? thanks (we are using 8.1.1)

Re: java API equivalent of --from-beginning?

@allthingshadoop <http://www.twitter.com/allthingshadoop> > ****/ > > On Fri, Feb 6, 2015 at 5:48 AM, Yang wrote: > > > we are trying to achive the equivalent of > > bin/kafka-console-consumer.sh > > > > > >

one message consumed by both consumers in the same group?

--. all the parameters are the same then I use console producer to dump some messages into the queue. the message does show up on both ends. where am I doing wrong? thanks Yang

Camus add third party decoder jar, doesn't work

Yang

what groupID does camus use?

I looked at the example camus.properties, it does not specify this. how could we set the groupId ? thanks Yang

latest messages do not show up on consumers

;t I expect the consumer on the 3rd line to read out the messages? in fact if I give it --from-beginning, it works fine this is kafka_2.9.2-0.8.1.1 Thanks Yang

Re: latest messages do not show up on consumers

ing either. looks like a bug? On Tue, Feb 24, 2015 at 1:51 PM, Yang wrote: > in the following code, I just create a new topic, send some messages onto > the new topic with console producer, then read it with console consumer. > but after the 3rd line, nothing is printed out. > &

Re: what groupID does camus use?

d coded and is "hadoop-etl" in some places (when it > initializes the offsets) and "camus" in other places. > The name "camus" is defined in camus.properties in "kafka.client.name", so > you can override it. > > Gwen > > On Tue, Feb 24,

Re: latest messages do not show up on consumers

JiangJie: thanks for the info. it looks I can change the default behavior and let a new group read from earliest offset by setting auto.offset.reset=smallest Yang On Tue, Feb 24, 2015 at 3:06 PM, Jiangjie Qin wrote: > If a consumer comes from a new consumer group, it will by default cons

cross-colo writing/reading?

are any techniques/properties to set , or maybe enhancements to be made to the kafka code, that could cater to such situations ? Thanks Yang

Camus reads from multiple offsets in parallel?

group, which a previous thead has clarified to be not the case, i.e. camus does not use group api), it would be very slow. Thanks Yang

kafka producer does not distribute messages to partitions evenly?

? thanks Yang

Re: kafka producer does not distribute messages to partitions evenly?

nd try > to do the same for your keys before you send them and check which > partitions are those going to. > > The default hashpartitioner does something like this : > > hash(key) % numPartitions. > > Thanks, > > Mayuresh > > On Mon, Mar 2, 2015 at 3:52 PM,

Re: kafka producer does not distribute messages to partitions evenly?

nhjc340y4qbqoqcismm43u...@mail.gmail.com%3E > has details. So with the default 10 minute refresh if your test is only an > hour or two with a single producer you would not expect to see all > partitions be hit. > > Christian > > On Mon, Mar 2, 2015 at 4:23 PM, Yang wrote: > > >

Re: cross-colo writing/reading?

together into one place. > This way clients can pick a local or a global view of the data, and we can > have everyone still stay in their local Kafka clusters. > > -Todd > > > On Fri, Feb 27, 2015 at 4:21 PM, Yang wrote: > > > we tested our new application that reads and

Re: Camus reads from multiple offsets in parallel?

Thanks for that info Jun. On Tue, Mar 3, 2015 at 3:56 PM, Jun Rao wrote: > Camus only fetches from different partitions in parallel. > > Thanks, > > Jun > > On Fri, Feb 27, 2015 at 4:24 PM, Yang wrote: > > > we have a single partition, and the topic contains 300k

integrate Camus and Hive?

s kind of not clean.... thanks yang

at-least-once guarantee?

able to assert between 2 cases "nothing" vs ">=1 delivered", which can be seen as 2 different answers 0 and 1. isn't this as hard as the common Byzantine general problem? Thanks Yang

Re: Message Serialization

I did a comparison between Thrift vs PB vs Avro about 3 years ago. at the time, Avro was faster than PB than Thrift. Avro has schema evolution (mentioned in the kafka paper). On Thu, Aug 8, 2013 at 10:08 AM, Mark wrote: > I've read that LinkedIn uses Avro for their message serialization. Was >

default producer to retro-fit existing log files collection process?

; producer included from kafka to just read the existing rotated logs and send through kafka API? this would avoid having to touch the existing java code thanks Yang

how can I let camus etl point to hdfs instead of local?

Job(getConf()); get a conf that points to a hdfs setup? I already set my env HADOOP_CONF_DIR to my running hadoop thanks Yang

Re: how can I let camus etl point to hdfs instead of local?

...@googlegroups.com seems less active :) thanks Yang On Tue, Sep 3, 2013 at 6:31 PM, Neha Narkhede wrote: > I'm sorry I'm not too familiar with the camus code base. Have you tried > pinging camus_...@googlegroups.com ? > > Thanks, > Neha > > > On Tue, Sep 3, 2013 at 1

Re: at-least-once guarantee?

her perspective, the consumer has to take on the responsibility to make sure of "at least once consumption" (if this is required by application), by means such as atomic commit of the offset and data, as you mentioned. Regards Yang On Wed, Aug 7, 2013 at 4:26 PM, Jay Kreps wrote: &g

ConsumerOffsetChecker returns negative value for log lag

Hi, I have a Kafka 0.8.1 cluster. I used the ConsumerOffsetChecker tool to check the lag of consumer groups. I found that for some partition, the tool returns negative value for the "lag " column. Is this a known issue that has been seen before? I find that the negative value prevents the consume

how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Hi all, We have a kafka 0.8.1 cluster. We implemented a consumers for the topics on the Kafka 0.8 cluster using high-level consumer api. We observed that if the Kafka cluster was down and got rebooted and the consumer was running, the consumer will fail to read a few topic partitions due to negati

Re: ConsumerOffsetChecker returns negative value for log lag

, 2014 at 8:06 PM, Jun Rao wrote: > Haven't seen this one before. Are you manually committing offsets yourself? > > Thanks, > > Jun > > On Fri, Oct 17, 2014 at 11:10 PM, Yu Yang wrote: > > > Hi, > > > > I have a Kafka 0.8.1 cluster. I used the Con

Re: how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Thanks, Jun! Yes, I set the topic replication factor to 3. On Sun, Oct 19, 2014 at 8:09 PM, Jun Rao wrote: > Did you set the replication factor to be more than 1? > > Thanks, > > Jun > > On Sat, Oct 18, 2014 at 2:32 AM, Yu Yang wrote: > > > Hi all, > > &

Re: kafka web console running error

do you see error msg "Too many open files"? it tips you should modify nofile On Tue, Nov 25, 2014 at 1:26 PM, Jun Rao wrote: > Which web console are you using? > > Thanks, > > Jun > > On Fri, Nov 21, 2014 at 8:34 AM, Sa Li wrote: > > > Hi, all > > > > I am trying to get kafka web console work,

Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

Hi, We have a kafka cluster that runs Kafka 0.8.1 that we are considering upgrade to 0.8.1.1. The Kafka documentation mentions upgrading from 0.8 to 0.8.1, but not from 0.8.1 to 0.8.1.1. Will doing one broker at a time by brining the broker dow

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

Narkhede wrote: > Will doing one broker at > a time by brining the broker down, updating the code, and restarting it be > sufficient? > > Yes this should work for the upgrade. > > On Mon, Dec 1, 2014 at 10:23 PM, Yu Yang wrote: > > > Hi, > > > > We have a k

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

; > Are you enabling message compression in 0.8.1 now? If you have already then > upgrading to 0.8.2 will not change its behavior. > > Guozhang > > On Tue, Dec 2, 2014 at 4:21 PM, Yu Yang wrote: > > > Hi Neha, > > > > Thanks for the reply! We know that Ka

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

ed on the > message type when consuming the data. > > Guozhang > > On Wed, Dec 3, 2014 at 11:33 AM, Yu Yang wrote: > > > Guozhang, > > > > We haven't enable message compression yet. In this case, what shall we do > > when we upgrade to 0.8.2? Must we l

Re: How to handle broker disk failure

I think the best way is raid not jbod. If one disk of jbod goes wrong , broker shutdown, then it takes long time to recovery . Brokes which run for long time will be more and more leaders of partitions. I/O pressure will be unbalanced. btw, I use kafka 0.8.0-beta1

Re: delete log file

YuanJia, I want to know why your broker's disk is almost full . Maybe your issue is the same with mine. Is it one broker out of service for a long time, then it fails back, at first it looks good, but after some hours, its disk is almost full. Other brokers are all fine, avg remaining capacity is

Immediate Kafka security solution before 0.9 release

Hi All, Before Kafka 0.9 release is available, is there an immediate security solution that we can leverage? I've come across https://github.com/relango/kafka/tree/kafka_security and the IP address filter patch from Kafka 0.8.3, which has not have a set release date. Thanks, Connie

Re: [VOTE] 0.8.2.1 Candidate 1

+1 On Feb 18, 2015 7:23 PM, "Matt Narrell" wrote: > +1 > > > On Feb 18, 2015, at 7:56 PM, Jun Rao wrote: > > > > This is the first candidate for release of Apache Kafka 0.8.2.1. This > > only fixes one critical issue (KAFKA-1952) in 0.8.2.0. > > > > Release Notes for the 0.8.2.1 release > > > ht

Re: Got java.util.IllegalFormatConversionException when running MirrorMaker off trunk code

When will KAFKA-1997 be available? Thanks Connie On Sat, Mar 7, 2015 at 12:48 AM, Jiangjie Qin wrote: > Hi Tao, > > Thanks a lot for finding the bug. We are actually rewriting the mirror > maker in KAFKA-1997 with a much simplified solution using the newly added > flush() call in new java produ

Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

Hi, Kafka 0.8.1.1 allows us to send a list of messages in sync mode: public void send(List messages); I did not find a counter-part of this api in the new producer that is introduced in kafka 0.8.2. It seems that we can use the following method to do sync send in kafka 0.8.2: producer.

Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

ived msg3 before msg2? On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang wrote: > Hi, > > Kafka 0.8.1.1 allows us to send a list of messages in sync mode: > > public void send(List messages); > > I did not find a counter-part of this api in the new producer that is > introduc

Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

Manikumar Reddy > wrote: > > > 1 . > > > > > > > > On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang wrote: > > > >> The confluent blog > >> < > http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/> > >>

Re: Kafka throw InvalidMessageException and lost data

; > at > > > > > > > > > > > > > > > > > > > > > > > > > > > kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaRequestHandlers.scala:59) > > > > > > > > > > > > Or this: > > > > > > > > > > > > 1830146 [kafka-processor-0] ERROR kafka.network.Processor - > > Closing > > > > > socket > > > > > > for /10.0.2.140 because of error > > > > > > java.lang.IllegalArgumentException > > > > > > at java.nio.Buffer.limit(Buffer.java:266) > > > > > > at > > kafka.api.ProducerRequest$.readFrom(ProducerRequest.scala:33) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > kafka.api.MultiProducerRequest$$anonfun$readFrom$1.apply$mcVI$sp(MultiProducerRequest.scala:28) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > scala.collection.immutable.Range$ByOne$class.foreach$mVc$sp(Range.scala:282) > > > > > > at > > > > > > > > > > > > scala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265) > > > > > > at > > > > > > > > > kafka.api.MultiProducerRequest$.readFrom(MultiProducerRequest.scala:27) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > kafka.server.KafkaRequestHandlers.handleMultiProducerRequest(KafkaRequestHandlers.scala:59) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaRequestHandlers.scala:41) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$4.apply(KafkaRequestHandlers.scala:41) > > > > > > at kafka.network.Processor.handle(SocketServer.scala:296) > > > > > > at kafka.network.Processor.read(SocketServer.scala:319) > > > > > > at kafka.network.Processor.run(SocketServer.scala:214) > > > > > > at java.lang.Thread.run(Thread.java:636) > > > > > > > > > > > > It bothers us for a few days, and at first we thought it might be > > > some > > > > > > wrong configuration settings, and we changed to the wiki's > > > recommended > > > > > > configuration, but unfortunately the exceptions still came out. > > > > > > > > > > > > In what situation can these exceptions be thrown out ? What can > we > > > do > > > > to > > > > > > avoid these exceptions ? > > > > > > > > > > > > THANKS > > > > > > > > > > > > -- > > > > > > *Best Regards > > > > > > > > > > > > Xiang Helin* > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Best Regards > > > > > > > > 向河林* > > > > > > > > > > > > > > > -- > > Guodong Wang > > 王国栋 > > > -- Yang Zhou（周阳） Department of Computer Science and Engineering Shanghai Jiao Tong University

Re: Kafka throw InvalidMessageException and lost data

ala.collection.immutable.Range$$anon$2.foreach$mVc$sp(Range.scala:265) > > > > > > at > > > > > > > > > kafka.api.MultiProducerRequest$.readFrom(MultiProducerRequest.scala:27) > > > > > > at > > > > > > > > > >

cannot make another partition reassignment due to the previous partition reassignment failure

HI, We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment for some topic partitions in the cluster. Due to broker failure, the partition reassignment failed. I cannot do another partition assignment now, and always get errors as follows. How can we work around this? I have tried g

Re: cannot make another partition reassignment due to the previous partition reassignment failure

nikumar > > > On Tue, Jun 16, 2015 at 8:15 AM, Yu Yang wrote: > > > HI, > > > > We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment > for > > some topic partitions in the cluster. Due to broker failure, the > partition > > reassignme

Re: Support customized security protocol

@Ismael, what's the status of the SASL/PLAIN PR, https://github.com/apache/kafka/pull/341? On Tue, Jan 19, 2016 at 6:25 PM, tao xiao wrote: > The PR provides a new SASL mech but it doesn't provide a pluggable way to > implement user's own logic to do authentication. So I don't think the PR > w

Re: Not able to read committed offset on nodes failures with replication setup.

Hi Here is the doc in the ReplicaFetcherThread.handleOffsetOutOfRange which maybe the answer: /** * Handle a partition whose offset is out of range and return a new fetch offset. */ def handleOffsetOutOfRange(topicAndPartition: TopicAndPartition): Long = { val replica = replicaMgr.g

Re: Topic Creation fails - Need help

your broker is not running 发自我的 iPhone > 在 2017年8月18日，下午3:14，Raghav 写道： > > Hi > > I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. > I am trying to create a topic using below command: > > "bin/kafka-topics.sh --create --zookeeper localhost:2181 > --replication-facto

Re: Topic Creation fails - Need help

please use zk client to check the path：/brokers/ids in ZK 发自我的 iPhone > 在 2017年8月18日，下午3:14，Raghav 写道： > > Hi > > I have a 1 broker and 1 zookeeper on the same VM. I am using Kafka 10.2.1. > I am trying to create a topic using below command: > > "bin/kafka-topics.sh --create --zookeeper local

Why doesn't MirrorMaker apply the Exactly once and transactional semantics at 0.11.x?

semantics on production work. Thanks & Have a good day! -- Yang Cui FreeWheel | Beijing +86 1381-1441-685

kafka controller setting for detecting broker failure and re-electing a new leader for partitions?

Hi everyone, Recently we had a cluster in which the controller failed to connect to a broker A for an extended period of time. I had expected that the controller would identify the broker as a failed broker, and re-elect another broker as the leader for partitions that were hosted on broker A. Ho

Re: 答复: kafka controller setting for detecting broker failure and re-electing a new leader for partitions?

ka/controller/ControllerChannelManager.scala#L222 <https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/ControllerChannelManager.scala#L245> On Thu, Jan 25, 2018 at 12:02 AM, Hu Xi wrote: > Yu Yang, > > > There does exist a broker-side config named 'contr

slow log segment loading during kafka 0.10.2 -> 1.0.1 upgrade

Hi, We are experimenting upgrading our kafka service from 0.10.2 to 1.0.1, and noticed that it was slow in restarting the broker after updating the kafka binary. On a test cluster that have ~120G data on each broker, it took about 20 minutes to load log segments. I am wondering if there is any wo

Kafka 0.8.2.1 fatal error under high cpu load

Earlier this year, our cluster crashed multiple times due to the same Fatal error regarding replica offset. In our production cluster, we set replication factor to 3 and min isr to 2. On the day this error occurred we temporarily doubled the number of producers and all kafka brokers were under hig

Kafka running on Ceph

Hi All, Does anyone have any performance metrics running Kafka on Ceph? I briefly gathered at the 2016 Kafka Summit that there's an ongoing work between the Kafka community and RedHat in getting Kafka running successfully on Ceph. Is this correct? If so, what's timeline for that? Thanks Connie

consumer with version 0.10.0

Hi all, I am using kafka 0.10.0.1, and I set up my listeners like: listeners=PLAINTEXT://myhostName:9092 then I have one consumer going using the new api. However, I did not see anything return for the api. The log from kafka is: [2016-08-26 14:39:28,548] INFO [GroupCoordinator 0]: Preparing to

RE: consumer with version 0.10.0

with version 0.10.0 Is anyone producing any (new) messages to the topics you are subscribing to in that consumer? -Jaikiran On Friday 26 August 2016 10:14 AM, Jack Yang wrote: > Hi all, > I am using kafka 0.10.0.1, and I set up my listeners like: > > listeners=PLAINTEXT://myh

RE: consumer with version 0.10.0

No worries. I figure that out already. Thanks all. Best regards, Jack -Original Message- From: Jack Yang [mailto:j...@uow.edu.au] Sent: Monday, 29 August 2016 10:13 AM To: users@kafka.apache.org Subject: RE: consumer with version 0.10.0 Hi there, My fault. When I produce messages

metrics for checking whether a broker throttles requests based on its quota limits?

Hi, Recently we enabled Kafka quota management for our Kafka clusters. We are looking for Kafka metrics that can be used for alerting on whether a Kafka broker throttles requests based on quota. There are a few throttle related metrics on Kafka. But none of them can tell accurately whether the br

graceful shutdown may be cause loss message

hello, currently CONTROLLED_SHUTDOWN API does not ensure the current leader sends all data to followers which in isr. it may cause data loss if the client does not use ack=all. I think semantic is a little weak. I can accept data loss if the broker crash, but we can try best to avoid data loss when

RE: [VOTE] 2.7.0 RC4

unsubsribe

Enable TLS Authentication + Enable External Access using LoadBalancer

lancer IPs + port, or any 1 LoadBalancer IP + port? * On external client side, does it need all 3 broker’s certificates? * How does the client know using which certificate while creating request to Kafka cluster? Thanks and regards, Wei Yang Cloud Infrastructure Engineer [/var/

Re: Enable TLS Authentication + Enable External Access using LoadBalancer

: Tuesday, December 21, 2021 at 8:55 AM To: Kafka Users Subject: Re: Enable TLS Authentication + Enable External Access using LoadBalancer Hi Wei Yang, > On Kafka cluster side, how to configure *advertised.listeners* for external access? All 3 LoadBalancer IPs + port, or any 1 LoadBalancer IP + p

Request to be added to kafka contributors list

Dear Apache Kafka Team, I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I kindly request to be added to the contributors list for Apache Kafka. Being part of this list would allow me to be assigned to JIRA tickets and work on them. Thank you for considering my request

回复: Request to be added to kafka contributors list

ka contributors list Thanks for reaching out Yang. You should be all set. -Matthias On 5/16/24 7:40 AM, Yang Fan wrote: > Dear Apache Kafka Team, > > I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I > kindly request to be added to the contributors list for

Re: Request to be added to kafka contributors list

: Request to be added to kafka contributors list Did you sign out and sign in again? On 5/17/24 9:49 AM, Yang Fan wrote: > Thanks Matthias, > > I still can't find "Assign to me" button beside Assignee and Reporter. Could > you help me set it again? > > Best regards

Re: Request to be added to kafka contributors list

5/21/24 1:14 AM, Fan Yang wrote: > Hi Matthia, > > I tried sign out and sign in, still can't find the "Assign" button, my JIRA > ID is fanyan, could you help me set it again? > > Best, > Fan > > > From: Matthias

how to ingest a database with a Kafka Connect cluster in parallel?

ill have no way to figure out if the ingestion is parallelized. I called the status API and received the following result: { "name":"test", "connector":{ "state":"RUNNING", "worker_id":"connect2:8083" }, "tasks":[ { "state":"RUNNING", "id":0, "worker_id":"connect3:8083" } ] } This result is the same for all instances. Does it mean the ingestion tasks are not parallelized? Is there anything important I am missing or this type of clustering is simply not supported? Any comments and suggestions are highly appreciated. Have a nice day! Best regards, Yang

Re: how to ingest a database with a Kafka Connect cluster in parallel?

Hi Ewen, Thanks a lot for your reply. So it means we cannot parallelize ingestion of one table with multiple processes. Is it because of Kafka Connect or the JDBC connector? Have a nice day. Best regards, Yang 2017-01-03 20:55 GMT+01:00 Ewen Cheslack-Postava : > The unit of parallelism

Re: how to ingest a database with a Kafka Connect cluster in parallel?

Hi Ewen, OK. Thanks a lot for your feedback! Best regards, Yang 2017-01-03 22:42 GMT+01:00 Ewen Cheslack-Postava : > It's an implementation detail of the JDBC connector. You could potentially > write a connector that parallelizes at that level, but you lose other > pote

Kafka mirror maker issue

Hi I am doing some experiments on kafka mirror maker. I have two Kafka cluster, one is kafka_2.11-0.9.0.1 with 3 nodes as source cluster, another is kafka_2.10-0.9.0.1 with one node as target cluster. First I mirror the message for topic TEST_TOPIC from source cluster to target cluster with co

How many levels can Message Set nest recursively?

I am thinking about that: 1 If a Producer compresses a Message set which is nested more than 2 levels records recursively and sends it to broker, how does broker know which offset should be allocated to this message set without uncompressed all levels and getting the all records? 2 Assumi

recommendation on kafka go client : sarama or confluent-kafka-go

Hi all, We need to write a service in Go to consume from kafka. There are two popular kafka clients: sarama and confluent-kafka-go . Sarama client is written purely in Go. This allow it to show the full Go st

About "org.apache.kafka.common.protocol.types.SchemaException" Problem

Dear All, I am using Kafka cluster 2.11_0.9.0.1, and the new consumer of 2.11_0.9.0.1. When I set the quota configuration is: quota.producer.default=100 quota.consumer.default=100 And I used the new consumer to consume data, then the error happened sometimes: org

Re: About "org.apache.kafka.common.protocol.types.SchemaException" Problem

Hi All, Have anyone can help answer this question? Thanks a lot! On 26/04/2017, 8:00 PM, "Yang Cui" wrote: Dear All, I am using Kafka cluster 2.11_0.9.0.1, and the new consumer of 2.11_0.9.0.1. When I set the quota configuration is: quota.produc

How to rolling restart Kafka cluster?

Hi guys, I have a confusion about whether it will lost data when I rolling restart Kafka cluster. Suppose that Kafka cluster has 4 brokers: A, B, C, D, and 4 partitions of a topic which name is “test-topic” are allocated on these brokers with balance: p1 for A, p2 for B, p3 for C, p4 for

Brokers is down by “java.io.IOException: Too many open files”

Our Kafka cluster is broken down by the problem “java.io.IOException: Too many open files” three times in 3 weeks. We encounter these problem on both 0.9.0.1 and 0.10.2.1 version. The error is like: java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0

Re: Brokers is down by “java.io.IOException: Too many open files”

# /etc/security/limits.conf * - nofile 65536 On Fri, May 12, 2017 at 6:34 PM, Yang Cui wrote: > Our Kafka cluster is broken down by the problem “java.io.IOException: Too > many open files” three times in 3 weeks. > > We encounter

Should set type of log.segment.bytes from INT to LONG

We wish to enlarge the segment file size from 1GB to 2GB, but we found that it will throw the exception as: “Invalid value 2147483648 for configuration log.segment.bytes: Not a number of type INT”. It is because of the overflow of INT, if we set l”og.segment.bytes” to 2147483648. Now we are s

What happens to Kafka when ZK lost its quorum?

Hi all, Can Kafka producers, brokers and consumers still be processing messages and functioning in their normal states if Zookeeper lost its quorum? Thanks, Connie

What happens to Kafka when ZK lost its quorum or becomes unstable?

Hi, Can the producers, brokers and consumers still be processing messages when their ZK cluster lost its quorum or becomes unstable? I know this is rather general question as it may depends on what configuration these use. So, please enumerate all of those combinations. Thanks, Connie

producer & consumer fail if the leader failed-over

Hi List, I'm new to kafka, feel sorry if this has been asked, I didn't figure out my answer by googling, so asked here. Thanks in advance! I'm following the kafka quick start http://kafka.apache.org/documentation.html#quickstart and managed setup a kafka cluster with two brokers, which connected

Re: producer & consumer fail if the leader failed-over

the consumer failed? -- Thanks, Chengwei > > Thanks, > > Jun > > > On Thu, May 29, 2014 at 2:38 AM, Chengwei Yang > wrote: > > > Hi List, > > > > I'm new to kafka, feel sorry if this has been asked, I didn't figure out > > my answe

Message lost during leader fail-over

gwei On Thu, May 29, 2014 at 07:52:11AM -0700, Jun Rao wrote: > You need to pass in multiple brokers in --broker-list in the producer. To > obtain the new leaders, the producer needs to talk to a live broker to get > the new metadata. > > Thanks, > > Jun > > > On Th

Re: Message lost during leader fail-over

On Thu, May 29, 2014 at 08:39:47PM -0700, Jun Rao wrote: > Have you looked at request.required.acks in > http://kafka.apache.org/documentation.html#producerconfigs ? Thanks Jun, that's make sense to me. -- Thanks, Chengwei > > Thanks, > > Jun

How to recover from a disk full situation in Kafka cluster?

Hi All, Our Kafka cluster ran out of disk space yesterday. After we freed up the cluster disk space and adjusted the broker data retention policy, we noticed that the cluster partition was not balanced based on topic describe script came from Kafka 0.8.1.1 distribution. So, we tried to rebalance

Re: How to recover from a disk full situation in Kafka cluster?

of 1 inadvertently? > > Thanks, > Neha > > > On Thu, Jul 17, 2014 at 11:57 AM, Connie Yang > wrote: > > > Hi All, > > > > Our Kafka cluster ran out of disk space yesterday. After we freed up the > > cluster disk space and adjusted the broker d

Re: How to recover from a disk full situation in Kafka cluster?

oducer > sticks to certain partitions, is 10 mins. So it effectively does not > distribute data evenly across all partitions. > > If you see the same behavior next time, try to take a snapshot of data > distribution across all partitions to verify this theory. > > Thanks, > N

Re: How to recover from a disk full situation in Kafka cluster?

s gone through some level of testing now > and we will appreciate any feedback/bugs that you can report back. > > Thanks, > Neha > > > On Fri, Jul 18, 2014 at 4:23 PM, Connie Yang > wrote: > > > Sure, I will try to take a snapshot of the data dis

How to upgrade Kafka broker to a newer version without impacting live traffic?

Hi All, If I want to upgrade from Kafka 0.8.1.1 to a newer version, what are the steps that I need to do without impacting production traffic? More specifically, how do I have the new brokers to continue writing to the existing logs assuming the file format has not changed? Is there a migration

Re: [ANNOUNCE] New committer: TengYao Chi

Congratulation! Well-deserved! PoAn On 2025/04/10 15:54:58 "Matthias J. Sax" wrote: > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce TengYao Chi as a new Kafka committer. > > TengYao has been active in the project since April of 2024 and has > contributed a gr

Re: [VOTE] 3.9.1 RC1

Hi TengYao, I have done the following tests: - Built from 3.9.1-rc1 with OpenJDK-17 and OpenJDK-21. - Ran all unit and integration tests. - Verified basic produce/consume. - Validated the checksums and keys. - Ran KRaft with kafka_2.13-3.9.1.tgz Thanks, PoAn > On Apr 23, 2025, at 5:00 PM, TengY