Palur,
Just adding to what Guozhang said, the answer to your question might
depend on which producer you are using.
Assuming you are producing messages without keys to the same topic, in new
producer(KafkaProducer), the messages will go to brokers in a round robin
way, so the messages will end up
r[pos=0 lim=50 cap=50])*
>
>Thanks
>Sandeep
>
>On Tue, Nov 25, 2014 at 7:07 PM, Jiangjie Qin
>wrote:
>
>> Palur,
>>
>> Just adding to what Guozhang said, the answer to your question might
>> depend on which producer you are using.
>> Assuming you
sages evenly distributed among partitions?
>do you mean that it is not possible in 0.8.1.1 version?
>
>On Wed, Nov 26, 2014 at 12:03 PM, Jiangjie Qin
>wrote:
>
>> Hi Sandeep,
>>
>> If you are sending messages to different topics, each topic will stick
>>to
>&
nt
>MessageandMetadata
>
>*Thread 0: Message(magic = 0, attributes = 0, crc = 127991357, key =
>java.nio.HeapByteBuffer[pos=0 lim=1 cap=55], payload =
>java.nio.HeapByteBuffer[pos=0 lim=50 cap=50])*
>
>Can you please tell me where can I find partition number in this?
>
>
>
&g
Has the message successfully produced to broker? You might need to change
producer settings as well. Otherwise the message could have been dropped.
‹Jiangjie (Becket) Qin
On 12/1/14, 8:09 PM, "Palur Sandeep" wrote:
>Yeah I did. I made the following changes to server.config:
>
>message.max.bytes
I'm just thinking instead of binding serialization with producer, another
option is to bind serializer/deserializer with
ProducerRecord/ConsumerRecord (please see the detail proposal below.)
The arguments for this option is:
A. A single producer could send different message type
One thing might worth checking is the offset.storage value in your
consumer config. Is it set to ³kafka² or ³zookeeper² for you consumer? If
the offset is set to kafka, then the offset will be stored in the offset
topic in Kafka instead of in zookeeper. So deleting zookeeper path won¹t
work for tho
ValueSerializer, Properties properties) can be useful.
>
>Thanks,
>
>Jun
>
>On Thu, Dec 4, 2014 at 10:33 AM, Jiangjie Qin
>wrote:
>
>>
>> I'm just thinking instead of binding serialization with producer,
>>another
>> option is to bind serializer/de
KAFKA-1476 is addressing this need, but it¹s not checked in yet. Currently
maybe you can use zookeeper client to check the zookeeper path.
‹Jiangjie (Becket) Qin
On 12/12/14, 2:45 AM, "svante karlsson" wrote:
>Disregard the creation question - we must have done something wrong
>because
>now our
Hi Rajiv,
You can send messages without keys. Just provide null for key.
Jiangjie (Becket) Qin
On 12/19/14, 10:14 AM, "Rajiv Kurian" wrote:
>Hi all,
>
>I was wondering what why every ProducerRecord sent requires a serialized
>key. I am using kafka, to send opaque bytes and I am ending up crea
In meetup we said that KAFKA-1650 and follow up patches is included in
0.8.2, but it seems not on the list.
On 1/29/15, 1:01 AM, "Magnus Edenhill" wrote:
>+1 on librdkafka interop
>
>Minor nitpick:
> KAFKA-1781 (state required gradle version in README) is included in the
>Release notes but is
I think this is the offset checker bug.
The offset checker will
1. first check if the offset exists in offset topic on broker or not.
2. If it is on broker then it will just return that offset.
3. Otherwise it goes to zookeeper.
So the problem you saw was actually following this logic.
After dual
t there a way that I can wipe out the offset
>stored in kafka so that the checker can continue to work again?
>
>On Fri, Feb 13, 2015 at 1:31 PM, Jiangjie Qin
>wrote:
>
>> I think this is the offset checker bug.
>> The offset checker will
>> 1. first check if the
I think your log did show that your are connecting to localhost:9092:
[2015-02-17 20:43:32,622] WARN Fetching topic metadata with correlation id
0 for topics [Set(test)] from broker [id:0,host:localhost,port:9092]
failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
Can yo
If you are using old producer for mirror maker, you can specify a custom
partitioner for the mirror maker producer which has exact same logic to
partition message as your custom producer does. If you are using new java
producer, currently there is no way to do it. We are working on adding a
message
+1, I¹m very interested.
On 2/23/15, 3:05 PM, "Jay Kreps" wrote:
>+1
>
>I think something like "Kafka on AWS at Netflix" would be hugely
>interesting to a lot of people.
>
>-Jay
>
>On Mon, Feb 23, 2015 at 3:02 PM, Allen Wang
>wrote:
>
>> We (Steven Wu and Allen Wang) can talk about Kafka use ca
the
committed offset and consume from 0. In this case you will see the
produced messages. If you use a different consumer group in your second
run of consumer, you will not be able to see the messages because the
offset will be reset to latest like in the first scenario.
-Jiangjie Qin
On 2/24/15
There is this ProducerPerformance class coming with new java producer. You
can go to KAFKA_HOME/bin and use the following command:
./kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
USAGE: java org.apache.kafka.clients.tools.ProducerPerformance topic_name
num_records record_si
These messages are usually caused by leader migration. I think as long as
you don¹t see this lasting for ever and got a bunch of under replicated
partitions, it should be fine.
Jiangjie (Becket) Qin
On 2/25/15, 4:07 PM, "Zakee" wrote:
>Need to know if I should I be worried about this or ignore
y to reduce it?
>
>How does "num.replica.fetchers" affect the replica sync? Currently have
>configured 7 each of 5 brokers.
>
>-Zakee
>
>On Wed, Feb 25, 2015 at 4:17 PM, Jiangjie Qin
>wrote:
>
>> These messages are usually caused by leader migration. I th
Some times an ephemeral ZK path does not go away after a consumer is
closed. You can check the log for each rebalance to see if it complains
some conflict data of ZK Path. If all the complaints are pointing to the
same consumer, bounce that consumer. Otherwise you can try to remove the
ZK path manu
Can you paste the error log for each rebalance try?
You may search for keyword ³exception during rebalance².
On 2/26/15, 7:41 PM, "Ashwin Jayaprakash"
wrote:
>Just give you some more debugging context, we noticed that the "consumers"
>path becomes empty after all the JVMs have exited because of
Do you mean you were not able to connect to zookeeper after retry?
We see this error in the log from time to time, but the zkClient will
retry and usually it will succeed. Can you verify if you were finally be
able to connect or not?
Jiangjie (Becket) Qin
On 2/27/15, 12:53 AM, "Victor L" wrote:
Can you check if you replica fetcher thread is still running on broker 1?
Also, you may check the public access log on broker 5 to see if there are
fetch requests from broker 1.
On 2/28/15, 12:39 AM, "tao xiao" wrote:
>Thanks Harsha. In my case the replica doesn't catch up at all. the last
>log
Are you using Kafka based offset commit or ZK based offset commit?
On 2/28/15, 6:16 AM, "Gene Robichaux" wrote:
>What is the best way to detect consumer lag?
>
>We are running each consumer as a separate group and I am running the
>ConsumerOffsetChecker to assess the partitions and the lag for e
Hi Puneet,
One of the conditions for K3 back to ISR is K3¹s log end offset to be
higher than the K1(leaderReplica)¹s high watermark.
If batch 2 is committed, then the leader high watermark will be above the
offsets of messages in batch 2.
In order to be added into ISR again, K3 has to at least hav
gt;>
>>> Gene Robichaux
>>> Manager, Database Operations
>>> Match.com
>>> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
>>>
>>> -Original Message-
>>> From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID]
>>> S
Which server version are you running?
On 3/2/15, 2:05 AM, "Jianshi Huang" wrote:
>Hi,
>
>I'd like to use Scala 2.11 with Kafka, which is only supported from 0.8.2.
>
>Can I use 0.8.2 jars for my consumer with producer of older version (mine
>is 0.8.1.1), which I have no control over it.
>
>Thank
There are two algorithms: range and round robin.
Range algorithm does balance for each topic independently.
Round robin balance across all the topics the consumer is consuming from.
Jiangjie (Becket) Qin
On 3/2/15, 2:05 AM, "sunil kalva" wrote:
>Is kafka load balancing based on number of partit
In this case you have data loss. In step 6, when broker 1 comes up, it
becomes the leader and has log end offset 1000. When broker 0 comes up, it
becomes follower and will truncate its log to 1000, i.e. 1000 messages
were lost. Next time when the consumer starts, its offset will be reset to
either
for a partition. Is
>there any mechanism built around Kafka to ensure that only the in-sync
>broker can be chosen to be a leader? If no, what is the best practice to
>restart brokers if some of the replicas are out of sync?
>
>On Tue, Mar 3, 2015 at 2:35 AM, Jiangjie Qin
>wrote:
>
3, 2015 at 2:07 AM, Jiangjie Qin
>wrote:
>
>> Which server version are you running?
>>
>> On 3/2/15, 2:05 AM, "Jianshi Huang" wrote:
>>
>> >Hi,
>> >
>> >I'd like to use Scala 2.11 with Kafka, which is only supported from
&g
What do you mean by Kafka embedded broker?
Anyway, this could happen. For example, producer sends message to broker.
After that some network issue occurs and the producer did not got
confirmation from broker, so the producer thought the send failed. But the
broker actually got the message. The prod
Hi Tao,
Yes, your understanding is correct. We probably should update the document
to make it more clear. Could you open a ticket for it?
Jiangjie (Becket) Qin
On 3/6/15, 1:23 AM, "tao xiao" wrote:
>Hi team,
>
>After reading the source code of AbstractFetcherManager I found out that
>the usage
labalance to false. I am still
>noticing the rebalance happening. My understanding was the rebalance will
>not happen when this is set to false.
>
>Thanks
>Zakee
>
>
>
>> On Feb 25, 2015, at 5:17 PM, Jiangjie Qin
>>wrote:
>>
>> I don’t think num.replica
Yet another option is camus, that we are using in LinkedIn:
https://github.com/linkedin/camus
Jiangjie (Becket) Qin
On 3/6/15, 10:01 PM, "max square" wrote:
>This presentation from a recent Kafka meetup in NYC describes different
>approaches.
>http://www.slideshare.net/gwenshap/kafka-hadoop-fo
Hi Tao,
Thanks a lot for finding the bug. We are actually rewriting the mirror
maker in KAFKA-1997 with a much simplified solution using the newly added
flush() call in new java producer.
Mirror maker in current trunk is also missing one necessary
synchronization - the UncheckedOffsets.removeOffse
I don¹t think we can specify partition to disk mapping now. All the
partition will resides in the same directory.
Here is a wild idea but I haven¹t tried this.
1. Create the topic and make sure all the log files are created.
2. Move each partition log directory to the disk that you want them to
res
For data not showing up, you need to make sure mirror maker consumer
auto.offset.reset is set to smallest, otherwise when you run mirror maker
for the first time, all the pre-existing messages won¹t be consumed.
For partition sticking, can you verify if your messages are keyed messages
or not? If t
>Zakee
>
>
>
>> On Mar 6, 2015, at 3:51 PM, Jiangjie Qin
>>wrote:
>>
>> Yes, the rebalance should not happen in that case. That is a little bit
>> strange. Could you try to launch a clean Kafka cluster with
>> auto.leader.election disabled and try pu
insync with source cluster.
>
>SunilKalva
>
>On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin
>wrote:
>
>> For data not showing up, you need to make sure mirror maker consumer
>> auto.offset.reset is set to smallest, otherwise when you run mirror
>>maker
>> fo
;s fix it now or
>revert
>that change.
>
>-Jay
>
>On Sat, Mar 7, 2015 at 12:48 AM, Jiangjie Qin
>wrote:
>
>> Hi Tao,
>>
>> Thanks a lot for finding the bug. We are actually rewriting the mirror
>> maker in KAFKA-1997 with a much simplified solution using
HI Phill,
Do you mean you are using 6 consumers with the same group id? Or you have
3 consumers using one group id, and another 3 using another different
group id?
For the example you mentioned, what you can do is to run several consumers
on different physical machine with the same group id, they
Hi Tao,
That is expected behavior. You can use set
partition.assignment.strategy=roundrobin in consumer config. It will take
all the partitions from all topics and do a round robin assignment,
whereas range only take partitions for each individual topic for
assignment.
Jiangjie (Becket) Qin
On 3
;
>> > On Mon, Mar 9, 2015 at 6:41 AM, tao xiao wrote:
>> >
>> > > Ctrl+c is clean shutdown. kill -9 is not
>> > >
>> > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville
>> > wrote:
>> > >
>> &g
Hi Yang,
In the code suggested by Manikumar, yes, it is possible message 3 still
got sent even message 2 failed.
There is no single line code for send a batch of message synchronously
now, but after KAFKA-1660 is checked in, you may be able to achieve this
by doing the following:
Set a callback fo
The stickiness of partition only applies to old producer. In new producer
we have the round robin for each message. The batching in new producer is
per topic partition, the batch size it is controlled by both max batch
size and linger time config.
Jiangjie (Becket) Qin
On 3/9/15, 10:10 AM, "Corey
>>> 2. In zookeeper path, can you verify /admin/preferred_replica_election
>>> does not exist?
>> ls /admin
>> [delete_topics]
>> ls /admin/preferred_replica_election
>> Node does not exist: /admin/preferred_replica_election
>>
>>
>> Tha
Yes, Kevin is right. It does not matter whether you run the consumer from
the same JVM or not, as long as the consumers has same group id, they are
in the same group. So in your case, you have 6 consumers in the same
consumer group. Since you have 6 partitions in the topic, assuming you
have only o
Hi Kevin,
You can use partition.assignment.strategy=roundrobin.
This will balance all the partition of all the topics across consumer
thread.
I think the rationale behind using default consumer id is that you will
have better information to identify a consumer. But if you want to have
some specif
Hi Mark,
You’ve already asked a question in the right place – sending email to
users@kafka.apache.org is the right way.
If it is a development question, you can send to d...@kakfa.apache.org.
Jiangjie (Becket) Qin
From: Mark Flores
mailto:mark.flo...@expeditors.com>>
Reply-To: "users@kafka.apa
Hi Mark,
For global Pub/Sub between clusters, I think you might need another layer of
service to direct users to the right Kafka cluster.
Yes, mirror maker could be used for cross colo replication. Currently mirror
maker cannot be used for bi-direction mirror yet, after KAFKA-1997 get checked
i
, 8:31 PM, "sy.pan" wrote:
>Hi, tao xiao and Jiangjie Qin
>
>I encounter with the same issue, my node had recovered from high load
>problem (caused by other application)
>
>this is the kafka-topic show:
>
>Topic:ad_click_sts PartitionCount:6ReplicationF
I would suggest to use the new java producer if possible. It is more
efficient and does the round robin by default.
Jiangjie (Becket) Qin
On 3/10/15, 3:28 AM, "tao xiao" wrote:
>The default partitioner of old producer API is a sticky partitioner that
>keeps sending messages to the same partitio
Hi Xiao,
For z/OS, do you mean z/VM or native z/OS? For z/VM, it probably will work
fine, but for z/OS, I would be surprised if Kafka can run directly on it.
I think Guozhang¹s approach for cross colo replication is worth trying.
One thing might need to be aware of for deploying Kafka cluster cro
>(kafka.utils.ZkUtils$)
>
>[2015-03-11 11:00:40,086] INFO Partition [ad_click_sts,4] on broker 1:
>Cached zkVersion [564] not equal to that in zookeeper, skip updating ISR
>(kafka.cluster.Partition)
>>>>>>>>>>>>>>>>>>>
>
>fina
No sure if it is an option. But does filtering out topics with message
handler works for you? Are you going to resume consuming from a topic
after you stop consuming from it?
Jiangjie (Becket) Qin
On 3/12/15, 8:05 AM, "tao xiao" wrote:
>Yes, you are right. a dynamic topicfilter is more appropri
>> topics, you will lose the data that gets consumed during the period of
>>the
>> blacklist.
>>
>> Guozhang
>>
>> On Thu, Mar 12, 2015 at 10:01 PM, tao xiao wrote:
>>
>> > Yes, that will work. message handle can filter out message sent
ctrl+c should work. Did you see any issue for that?
On 3/12/15, 11:49 PM, "tao xiao" wrote:
>Hi,
>
>I wanted to know that how I can shutdown mirror maker safely (ctrl+c) when
>there is no message coming to consume. I am using mirror maker off trunk
>code.
>
>--
>Regards,
>Tao
Can you reproduce this problem? Although the the fix is strait forward we
would like to understand why this happened.
On 3/13/15, 3:56 PM, "Zakee" wrote:
>Just found there is a known issue to be resolved in future kafka version:
> https://issues.apache.org/jira/browse/KAFKA-1554
>
>The workaroun
We are planning to develop a Chaos Monkey test in LinkedIn and will open
source it. You can check out KAFKA-2014.
Jiangjie (Becket) Qin
On 3/16/15, 1:24 PM, "John Lonergan" wrote:
>Re kafka-1539
>
>Is the community executing random failure testing for Kafka?
>
>It would seem that sick testing w
Yeah, using different partition assignment algorithms in the same consumer
group won¹t work. Is there a particular reason you want to do this?
On 3/17/15, 8:32 AM, "tao xiao" wrote:
>This is the corrected zk result
>
>Here is the result from zk
>[zk: localhost:2181(CONNECTED) 0] get
>/consumers/
ot; wrote:
>Probably we should return an error response if you already have a
>partition
>assignment strategy inplace for a group and you try to use other strategy.
>
>Thanks,
>
>Mayuresh
>
>On Tue, Mar 17, 2015 at 2:10 PM, Jiangjie Qin
>wrote:
>
>> Yeah, using diff
ol to
>> verify if all configuration settings are correct, even if the broker can
>> return an error message to the consumers.
>>
>> Thanks,
>>
>> Xiao Li
>>
>> On Mar 17, 2015, at 5:18 PM, Jiangjie Qin
>> wrote:
>>
>> > The p
Hi Jason,
The round-robin strategy first takes the partitions of all the topics a
consumer is consuming from, then distributed them across all the consumers.
If different consumers are consuming from different topics, the assigning
algorithm will generate different answers on different consumers.
Hi Vamsi,
The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh still
works. You can use them to check the offsets.
If you need to check the offsets programmatically, you can send
OffsetsFetcheRequest to broker using simple consumer. You may refer the
ConsumerOffsetChecker.scala to s
This is correct when you send to a topic for the first time. After that
the metadata will be cached, the metadata cache has an age and after it
expires, metadata will be refreshed.
So the time a producer found a broker is not reachable is the minimum
value of the following times:
1. Linger.ms + ret
an easy to understand algorithm would work fine
>with
>a heterogeneous set of selected topics between consumers.
>
>Jason
>
>On Thu, Mar 19, 2015 at 8:07 PM, Mayuresh Gharat
>> wrote:
>
>> Hi Becket,
>>
>> Can you list down an example for this. It would be
ill have a temporary outage consuming
>data until all nodes have been updated. So, it makes it difficult to do
>rolling restarts, or canary updates on a subset of nodes, etc.
>
>Jason
>
>Jason
>
>On Fri, Mar 20, 2015 at 10:15 PM, Jiangjie Qin
>wrote:
>
>> Hi Jaso
Yes, Kafka use replica to tolerate node failures. Depending on which level
of durability and availability guarantee you need, you might need
different settings on the broker and producer. Kafka cluster will
automatically take care of node failure in a cluster for you.
Jiangjie (Becket) Qin
On 3/2
Hi Xiao,
I think the fix for IllegalStateExcepetion is correct.
Can you also create a ticket and submit a patch?
Thanks.
Jiangjie (Becket) Qin
On 3/24/15, 4:31 PM, "tao xiao" wrote:
>Hi community,
>
>I wanted to know if the solution I supplied can fix the
>IllegalMonitorStateException
>issue.
m up while fetcher thread is down.
>
>On Wed, Mar 25, 2015 at 8:00 AM, tao xiao wrote:
>
>> Thanks JIanjie. Can I reuse KAFKA-1997 or should I create a new ticket?
>>
>> On Wed, Mar 25, 2015 at 7:58 AM, Jiangjie Qin
>>
>> wrote:
>>
>>> Hi Xiao,
If you want to do a seamless migration. I think a better way is to build a
cross datacenter Kafka cluster temporarily. So the process is:
1. Add several new Kafka brokers in your new datacenter and add them to
the old cluster.
2. Use replica assignment tool to reassign all the partitions to brokers
Hi,
KafkaConsumer is still under development and not ready for wide use case.
Currently, it can be used to replace SimpleConsumer (low level consumer),
but can not replace ZookeeperConsumerConnector(high level consumer). So if
you need to use simple consumer, I would suggest using KafkaConsumer
in
The async send() put the message into a message queue then returns. When
the messages are pulled out of the queue by the sender thread, it still
uses SyncProducer to send ProducerRequests to brokers.
Jiangjie (Becket) Qin
On 3/30/15, 10:44 PM, "Madhukar Bharti" wrote:
>Hi All,
>
>I am using *as
The Mirror maker in trunk now supports mirroring with preserved partition.
You can wire in a message handler to assign partitions for each producer
record before handing them to producer.
Jiangjie (Becket) Qin
On 3/31/15, 3:41 AM, "Ivan Balashov" wrote:
>Alex,
>
>Just wondering, did you have an
Yes, KafkaConsumer in 0.8.2 is still in development. You probably still
want to use ZookeeperConsumerConnector for now.
On 4/1/15, 9:28 AM, "Mark Zang" wrote:
>I found the 0.8.2.0 and 0.8.2.1 has a KafkaConsumer. But this class seems
>not completed and not functional. Lots of method returns null
Another thing is that the active log segment would not be deleted, so if
there is only one log segment, it will not honor the retention policy. You
may config log.roll.ms to make sure you have a rolled over log segment.
On 4/3/15, 5:20 AM, "Madhukar Bharti" wrote:
>Hi Gaurav,
>
>What is your "lo
This sounds a very serious issueŠ Could you provide the controller log and
the log for the first broker on which you tried controlled shutdown and
upgrade?
On 4/3/15, 8:57 AM, "Jason Rosenberg" wrote:
>I'm preparing a longer post here, but we recently ran into a similar
>scenario. Not sure yet
Hey Liang,
Have you looked at the quick start here:
https://kafka.apache.org/documentation.html#quickstart
In Kafka, on the producer side, there is no concept of ³commit². If you
are producing using KafkaProducer, you can do a send.get(), this is a
synchronized send so if no exception was thrown,
Also if you are using Kafka from the latest trunk, KafkaProducer has a
flush() interface that you may call. This will ensure all the message
previously sent from send() methods are sent to Kafka server.
On 4/3/15, 3:38 PM, "Sun, Joey" wrote:
>Hello, group
>
>I am a newbie to Kafka. I am research
og to Kafka's producer?
>
>Thanks
>Liang
>
>-----Original Message-
>From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID]
>Sent: Monday, April 06, 2015 11:46 AM
>To: users@kafka.apache.org
>Subject: Re: question about Kafka
>
>Hey Liang,
>
>
A quick reference.
http://www.slideshare.net/junrao/kafka-replication-apachecon2013
On 4/12/15, 11:36 PM, "Bill Hastings" wrote:
>Hi Guys
>
>How do topics get assigned to brokers? I mean if I were to create a topic
>X
>and publish to it how does Kafka assign the topic and the message to a
>part
cting to 192.168.1.201:2181
>Welcome to ZooKeeper!
>JLine support is disabled
>
>WATCHER::
>
>WatchedEvent state:SyncConnected type:None path:null
>ls /config/topics
>[my-control]
>ls /consumers
>[]
>quit
>
>
>Thanks.
>Mayank.
>
>On Fri, Mar 20, 2015 at
Hey Evan,
Is this issue only observed when mirror maker is consuming? It looks that
for Cluster A you have some other consumers.
Do you mean if you stop mirror maker the problem goes away?
Jiangjie (Becket) Qin
On 4/14/15, 6:55 AM, "Evan Huus" wrote:
>Any ideas on this? It's still occurring...
It seems there are many different symptoms you see...
Maybe we can start from leader flapping issue. Any findings in controller
log?
Jiangjie (Becket) Qin
On 4/16/15, 12:09 PM, "Kyle Banker" wrote:
>Hi,
>
>I've run into a pretty serious production issue with Kafka 0.8.2, and I'm
>wondering w
Producers usually do not query zookeeper at all.
Consumers usually query zookeeper at beginning or rebalance. It is
supposed to be in frequent if you don¹t have consumers come and go all the
time. One exception is that if you are using zookeeper based consumer
offset commit, it will commit offset t
2 partitions should be OK.
On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash"
wrote:
>We are planning to have ~2 partitions. Will it be a bottleneck?
>
>On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin
>wrote:
>
>> Producers usually do not query zookeeper at
Those .index files are for different partitions and
they should be generated if new replicas is assigned to the broker.
We might want to know what caused the UnknownException. Did you see any
error log on broker 28?
Jiangjie (Becket) Qin
On 4/21/15, 9:16 AM, "Wes Chow" wrote
Apis.handleFetchRequest(KafkaApis.scala:424)
at kafka.server.KafkaApis.handle(KafkaApis.scala:186)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:745)
What does this mean?
Thanks!
Wes
[cid:part1.08040305.06010608@chartbeat.com]
Jiangjie Qin&l
your consumer offsets in kafka itself instead of Zk which
>>further
>> brings down write load on ZKs.
>>
>> Regards,
>> Pushkar
>>
>> On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin
>>
>> wrote:
>>
>> > 2 partitions should be OK.
n 116's replica set is: [4, 7,
25]. /brokers/topics/click_engage/partitions/116/state says the leader is 28
and the ISR is [28, 15]. Does this need to be resolved, and if so how?
Thanks,
Wes
[cid:part1.03010908.07060808@chartbeat.com]
Jiangjie Qin<mailto:j...@linkedin.com.INVALID>
lier this morning. Do the other
>brokers have to be bounced after the controller situation is fixed? I did
>not do that for all brokers.
>
>Wes
> On Apr 21, 2015 8:25 PM, "Jiangjie Qin"
>wrote:
>
>> Yes, should be broker 25 thread 0 from the log.
>> This ne
Hi Tao,
KAFKA-2150 has been filed.
Jiangjie
On 4/24/15, 12:38 PM, "tao xiao" wrote:
>Hi team,
>
>I observed java.lang.IllegalMonitorStateException thrown
>from AbstractFetcherThread in mirror maker when it is trying to build the
>fetchrequst. Below is the error
>
>[2015-04-23 16:16:02,049] ERR
Hi Jay,
Does o.a.k.clients.tools.ProducerPerformance provide multi-thread test? I
did not find it.
I tweaked the test a little bit to make it multi-threaded and what I found
is that in a single thread case, with each message of 10 bytes, single
caller thread has ~2M messages/second throughput. 2
Yes, if you set the offset storage to Kafka, high level consumer will be
using Kafka for all offset related operations.
Jiangjie (Becket) Qin
On 4/27/15, 7:03 PM, "Gomathivinayagam Muthuvinayagam"
wrote:
>I am trying to commit offset request in a background thread. I am able to
>commit it so f
OK, so you turned off auto.offset.commit, and set the auto.offset.reset to
largest.
That means when you consume,
1. If you did not commit offsets manually, no offsets will be committed to
Kafka.
2. If you do not have an offset stored in Kafka, you will start from the
log end and ignore the existin
Roshan,
If I understand correctly, you just want to make sure a number of messages
has been sent successfully. Using callback might be easier to do so.
Public class MyCallback implements Callback {
public Set failedSend;
@Override
Public void onCompletion(RecordMetadata me
Which mirror maker version did you look at? The MirrorMaker in trunk
should not have data loss if you just use the default setting.
On 4/30/15, 7:53 PM, "Joong Lee" wrote:
>Hi,
>We are exploring Kafka to keep two data centers (primary and DR) running
>hosts of elastic search nodes in sync. One k
You can use ELB address as metadata broker list. But you still need to
allow direct traffic from clients to a brokers because clients need to
talk to a specific broker who has the interested partition.
Jiangjie (Becket) Qin
On 5/2/15, 11:10 AM, "Chandrashekhar Kotekar"
wrote:
>Hi,
>
>I am new t
1 - 100 of 172 matches
Mail list logo
