I usually get this exception when I define > 2 partitions ..
Current configuration :
Single Topic - 4 partitions
1 Consumers Group - 10 Threads
On Wed, Sep 11, 2013 at 10:24 PM, prashant amar wrote:
> From the broker log:
>
>
> INFO Reconnect due to socket error: (kafka.consumer.SimpleCons
>From the broker log:
INFO Reconnect due to socket error: (kafka.consumer.SimpleConsumer)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:72)
at
kafka.consumer.SimpleConsu
1. You can put multiple directories, each on a separate volume, in log.dirs.
2. Yes, our replica assignment logic will try to spread the partitions and
the leaders evenly among the brokers.
Thanks,
Jun
On Wed, Sep 11, 2013 at 9:59 PM, Lu Xuechao wrote:
> Hi Team,
>
> I have some questions re
This means the broker somehow closed the socket connection. Anything in the
broker log around the same time?
Thanks,
Jun
On Wed, Sep 11, 2013 at 6:07 PM, prashant amar wrote:
> Also noticed another issue
>
> Specified below is the current configuration
>
> Topic1 -> n Partitions -> 2 Consumer
Hi Team,
I have some questions regarding Kafka partitions:
1. Based on my understanding, the partitions of the same broker have
contention on disk IO. Say If I have 10 hard drives, can I specify all the
partitions spread evenly on those drives?
2. If I configure default.replication.factor=2, the
Also noticed another issue
Specified below is the current configuration
Topic1 -> n Partitions -> 2 Consumer Groups (gr1 and gr2)
Topic2 -> n Partitions -> 2 Consumer Groups (gr1 and gr2)
Notice that I have used the same naming convention on the consumer group
set i.e. 'gr1' and 'gr2' are consum
I think you are hitting this -
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whysomeoftheconsumersinaconsumergroupneverreceiveanymessage%3F
Let us know if we can improve the documentation to make it clearer.
Thanks,
Neha
On Wed, Sep 11, 2013 at 5:28 PM, prashant amar wrote:
> Also
Take a look at the "Producer Load balancing" section of the documentation -
http://kafka.apache.org/08/documentation.html#theproducer
Let us know how we can improve it to answer your question.
Thanks,
Neha
On Wed, Sep 11, 2013 at 4:45 PM, Xuyen On wrote:
> Hi Neha,
>
> How do you get metadata
Also attempted another pattern where
1. Created a topic T with 'n' partitions.
2. Created a consumer group process with 'n + 1' threads subscribing from
topic 'T' with a groupID 'y'
3. Added another consumer group process with 'n + 1' threads subscribing
from same topic 'T' with same groupID 'z'
(
A Design Question that needs verification:
1. Created a topic T with 'n' partitions.
2. Created a consumer group process with 'n + 1' threads subscribing from
topic 'T' with a groupID 'y'
3. Added another consumer group process with 'n + 1' threads subscribing
from same topic 'T' with same groupID
Hi Neha,
How do you get metadata about the cluster from the hardware load balancer? A
load balancer should direct a message to brokers that are up but I don't know
how you would detect for things like adding/deleting topics/partitions from the
cluster through the hardware load balancer. Can you
Actually an intermediate consumer would be useful for new topics, in which case
it would probably need to watch the relevant zookeeper nodes. I could see a
number of use cases where consumption strategies across a consumer group would
deviate from the resident partition balancing algorithm. Perh
Thanks Guozhang, Yes we had set to 'largest' and changing it to 'smallest'
resolved the issue. So it was due to the jira
https://issues.apache.org/jira/browse/KAFKA-1006
Thanks,
Raja.
On Tue, Sep 10, 2013 at 1:18 PM, Guozhang Wang wrote:
> Oh got it. Did you set auto.offset.reset = smallest or
Kafka 08 doesn't use zookeeper based load balancing on the producer side.
We rely on a hardware load balancer to be able to talk to any broker and
get metadata about the cluster. This happens on startup, for new
topics/partitions and if the producer runs into an error while sending data.
Thanks,
N
Hi Joel,
Thanks for helping.
I think we found the reason here:
We check out and build kafka before jmx metrics added to
AbstractFetcherManager.
Thanks,
Siyuan
On Wednesday, September 11, 2013, Joel Koshy wrote:
> That's weird - it should just work - can you try this:
>
> you would have to
Correct - but since you wanted sticky allocation rebalancing wouldn't
really be necessary.
Thanks,
Joel
On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi wrote:
> Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as
> suggested below) does not join in partition rebalancing
Can anyone from LinkedIn let me know if they are using Hardware Loadbalancer to
distribute the producer messages or if they're using zookeeper?
I'm trying to find out what the pros and cons are of using a hardware
loadbalancer vs. using zookeeper to distribute the load of producers sending
messa
That's weird - it should just work - can you try this:
you would have to start your Kafka broker with a JMX port - should be
by default, otherwise use JMX_PORT=
./bin/kafka-run-class.sh kafka.tools.JmxTool --object-name
'"kafka.server":type="ReplicaFetcherManager",name="Replica-MaxLag"'
--jm
Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as
suggested below) does not join in partition rebalancing when another consumer
in it's group joins or a new broker joins or a new topic is created?
From: Joel Koshy
To: "users@kafka.ap
Hi Joel,
I cannot find such a value. My teammate also cannot find it.
There is no ReplicaFetcherManager underl kafka.server.
There is only a ReplicaManager but there is no MaxLag
under it.
Regards,
Libo
-Original Message-
From: Joel Koshy [mailto:jjkosh...@gmail.com]
Sent: Tuesday, S
We're writing an F# client for 0.8. Cooperate?
On 11 sep 2013, at 17:57, Richard Park wrote:
> At Ancestry we're starting a project using Kafka where a large majority of
> out producer processes are going to be in C#.
>
> I've done a bit of research and have found a few repos that have c# cli
At Ancestry we're starting a project using Kafka where a large majority of out
producer processes are going to be in C#.
I've done a bit of research and have found a few repos that have c# clients
which all seem to be at different points of development.
So I have two questions:
1. Is there a
22 matches
Mail list logo
