Is there a way to reduce the replication count? I'm trying to spread
existing partitions across more brokers, but it's hard to decomm a broker.
Reducing repl count would suffice for now.
Any tips?
I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
Thanks. It works.
- Original Message -
From: Jun Rao
At: Tuesday, May 20, 2014 23:33
You can issue a TopicMetadataRequest. See
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
Thanks,
Jun
On Tue, May 20, 2014 at 3:41 PM, Saurabh Agarwal (BLOOMBERG/ 7
Thanks, you are right, my consumer config was pointing to the 0.8! Cheers!
On 20 May 2014 05:36, Jun Rao wrote:
> It seems that you may have set the zk connection string to one used by 0.8
> Kafka brokers.
>
> Thanks,
>
> Jun
>
>
> On Mon, May 19, 2014 at 8:34 AM, Mo Firouz
> wrote:
>
> > Hi a
Hi all,
We are using kafka - 0.8 in our production setup. We have 2 kafka servers
and 2 zookeepers. There seem to be some sort of network problem in our DC. As a
result of this, connection refused exception in replica which was thrown , is
shown here. Our producer didnt throw any exceptio
Restarting the partition leaders cleared things up. We were hesitant to do
that at first because it was unclear what would happen with the
availability of the partitions in question since there were no other
replicas in the ISR (at least according to ZK). From what we observed, the
partitions did r
We don't have such throttling right now. Could you file a jira?
Thanks,
Jun
On Tue, May 20, 2014 at 10:29 PM, Marcos Juarez Lopez wrote:
> Hi,
>
> We have several Kafka clusters in production, and we've had to reassign
> replication a few times now in production. Some of our topic/partitions
We don't have an exact tool for doing this. You may be able to do that
through
http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
specifying fewer replicas.
Thanks,
Jun
On Wed, May 21, 2014 at 1:23 AM, David Birdsong wrote:
> Is there a way to reduce the replica
Hi Hangjun
I've explored deploying kafka on yarn and current YARN does not support long
running services with locality constraints. Deploying kafka producers /
consumers (not brokers) is supported in the apache incubator samza project.
Background on YARN limitations can be found here: YARN-371,
I did that and so now the topic has 4 replicas for a repl count of 3, but
only the 'new' replicas exist in the ISR.
The old broker that I want to clear disk space and generally free up
resources has fully synced a topic that I want to disassociate from it.
Is there a way to do this?
On Wed, May
It is possible to do this using the kafka-reassign-partitions admin
command. You can explicitly set the replica list for a partition to be
less than the current replication factor, and that will effectively reduce
it. However, I will say that you should be really sure that you want to do
this. If,
Do you see any exceptions from the consumer logs?
Guozhang
On Wed, May 21, 2014 at 4:15 AM, Ranjith Venkatesan
wrote:
> Hi all,
>
> We are using kafka - 0.8 in our production setup. We have 2 kafka
> servers and 2 zookeepers. There seem to be some sort of network problem in
> our DC. As a
During the re-assignment, did you move the replica off the old broker?
Thanks,
Jun
On Wed, May 21, 2014 at 8:21 AM, David Birdsong wrote:
> I did that and so now the topic has 4 replicas for a repl count of 3, but
> only the 'new' replicas exist in the ISR.
>
> The old broker that I want to cl
Here's the reassignment json and current "describe" output:
https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
In my re-assignment json, I tried to re-assign to 2 when the repl is set to
3. Once I noticed the the completely new node "133" had appeared in the
ISR, I tried stopping 224, wip
On Wed, May 21, 2014 at 9:11 AM, David Birdsong wrote:
> Here's the reassignment json and current "describe" output:
> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
>
>
> In my re-assignment json, I tried to re-assign to 2 when the repl is set
> to 3. Once I noticed the the completely
Thanks for your response Jun.
JIRA has been filed (see link below). Please let me know if I should add
more details/context:
https://issues.apache.org/jira/browse/KAFKA-1464
Thanks,
Marcos Juarez
On Wed, May 21, 2014 at 8:40 AM, Jun Rao wrote:
> We don't have such throttling right now. Co
Hi All,
What is the good way to measure the producer throughput?
We are currently accumulating time take by each individual send in the producer
and then dividing it with the total volume to calculate the producer
throughput. We seems to be getting 20% more throughput than our network can
su
>
> What is the good way to measure the producer throughput?
The producer exposes a number of mbeans that you can poke to monitor
throughput (e.g., bytes-per-sec, messages-per-sec, etc.) and it does
so on an aggregate as well as per-topic basis.
> We also find compression (such as GZip) does no
Any suggestions? I'm kind of in a bind in that I don't understand how to
grow the cluster when more capacity is needed--which happens to be right
now for me.
The only thing I can think that might work is to create new brokers,
cherry-pick topic/partition pairs and move them, then turn off the old
Expansion can be done by following
http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
If you just want to free up a server, you can stop the broker there and
start a broker using the same broker id on a new server. Data should be
automatically replicated to the new server.
Tha
On Wed, May 21, 2014 at 9:06 PM, Jun Rao wrote:
> Expansion can be done by following
> http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
>
> If you just want to free up a server, you can stop the broker there and
> start a broker using the same broker id on a new server. Data
20 matches
Mail list logo
