Jun,
I hear you say "partitions are evenly distributed among all consumers in
the same group", yet I did bump into a case where launching a process with
X high level consumer API threads took over all partitions, sending
existing consumers to be unemployed.
According to the claim above, and if I
| | |
|
| |
hello，
I have two kafka_server，IP address is 192.168.100.91 192.168.100.92
broker 0：192.168.100.91broker 1 ：192.168.100.92
At broker 0：
bin/kafka-topics.sh --create --zookeeper 192.168.100.91:2181 --topic qaz
--partition 3 --replication-factor 2
bin/kafk
alipayhu...@gmail.com
Hi Guozhang,
I was wondering if there is anything wrong in the above --describe data.
thanks
On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami wrote:
> This is what I get -
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe
> Topic:HeartbeatPartitionCount:2ReplicationFactor:1
note that --zookeeper is the location of the zookeeper server, not Kafka broker.
Are you running zookeeper on both 192.168.100.91 and 192.168.100.92?
Zookeeper is based on simple majority, therefore you can't run it with
2 nodes (well you can, but it will freeze if you lose one node), you
need ei
Is Compression happening on Producer Side (on application thread meaning
thread that call send method or background Kafka thread ) and where does
decompression Consumer side ?
Application thread in both cases.
Is there any Compression/Decompression happening on Brokers Side when
receiving messag
queued.max.message.chunks controls the consumer's fetcher queue.
On Mon, Oct 27, 2014 at 9:32 PM, Bhavesh Mistry
wrote:
> HI Neha,
>
> If I solved the problem number 1 think and number 2 will be solved (prob
> 1 is causing problem number 2(blocked)). Can you please let me know what
> controls
Sameer,
The topic name is not consistent from your describe outputs. Normally they
should be either
Topic:*Test*PartitionCount:2ReplicationFactor:1Configs:
Topic: *Test*Partition: 0Leader: 0Replicas: 0Isr: 0
Topic: *Test*Partition: 1Leader: 0
There was a typo earlier.
This is the output -
Topic:Test PartitionCount:2ReplicationFactor:1Configs:
Topic: TestPartition: 0Leader: 0Replicas: 0Isr: 0
Topic: TestPartition: 1Leader: 0Replicas: 0Isr: 0
thanks
Sameer
On Tue, Oct 28, 2014
Just a guess but could it be a firewall issue? Did you enable connections
to port 9092 from outside EC2 in a security group? Can you telnet to each
broker IP and port?
On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami wrote:
> There was a typo earlier.
>
> This is the output -
>
> Topic:Test Parti
Hi,
I have the following Kafka Setup
Number of producer : 1
Number of topics : 1
Number of partitions : 2
Number of consumers : 3 (with same group id)
Number of Kafka cluster : none(single Kafka server)
Zookeeper.session.timeout : 1000
Producer produces messages without any specific partitioning
Yes. I can telnet from outside using the public ip of the instance and port
of 9092
On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover
wrote:
> Just a guess but could it be a firewall issue? Did you enable connections
> to port 9092 from outside EC2 in a security group? Can you telnet to each
> br
Hi Neha,
Thanks for your answer. Can you please let me know how I can resolve the
Iterator IllegalStateException ? I would appreciate your is this is bug I
can file one or let me know if this is use case specific ?
Thanks,
Bhavesh
On Tue, Oct 28, 2014 at 9:30 AM, Neha Narkhede
wrote:
> queu
Hi Kafka Team,
What is expected behavior when you close *ConsumerConnector* and auto
commit is on ? Basically, when auto commit interval is set to 5 seconds
and shutdown is called (before 5 seconds elapses) does ConsumerConnector
commit the offset of message consumed by (next()) method or consume
High level consumer commits before shutting down.
If you'll look at ZookeeperConsumerConnector.scala (currently the only
implementation of ConsumerConnector) you'll see shutdown() includes
the following:
if (config.autoCommitEnable)
commitOffsets()
Gwen
On Tue, Oct 28, 201
HI Gwen,
Thanks for info appreciate. Sometime, I have observed duplicated events...
so that is why I ask.
Thanks,
Bhavesh
On Tue, Oct 28, 2014 at 11:51 AM, Gwen Shapira
wrote:
> High level consumer commits before shutting down.
>
> If you'll look at ZookeeperConsumerConnector.scala (currently
Sameer, could you check broker logs and see if it has similar
LeaderNotAvailable exceptions?
Guozhang
On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami wrote:
> Yes. I can telnet from outside using the public ip of the instance and port
> of 9092
>
> On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover
Is this what you are looking for -
2014-10-28 21:46:24,146 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092 for producing
2014-10-28 21:46:24,149 pool-13-thread-1 INFO
kafka.producer.SyncProducer-68: Disconnecting from 172.31.16.31:9092
2014-10-28 21:46:24,1
Sameer, the one you sent me is from the producer logs, not the broker logs.
And /tmp/kafka-logs actually store the message log (i.e. the "data")
instead of the server logs. Their locations should be specified in your
log4j.properties file.
On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami wrote:
> Is
This is the one from the file specified in log4j.properties -
[2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on Controller 0]:
Topic change listener fired for path /brokers/topics with children Test
(kafka.controller.PartitionStateMachine$TopicChangeListener)
[2014-10-28 22:38:36,094] INFO [
20 matches
Mail list logo
