In 0.7, one way to do this is to use a vip. All producers send data to the
vip. To decommission a broker, you first take the broker out of vip so no
new data will be produced to it. Then you let the consumer drain the data
(you can use ConsumerOffsetChecker to check if all data has been consumed).
Please can you try following the updated wiki -
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)?
Thanks,
Neha
On Sat, Jan 5, 2013 at 3:32 AM, CuiLiang wrote:
> Hi! I've been tasked with setting up a kafka cluster and mirroring. My
> Kafka version is 0.7.2. But
In 0.8, we will provide a way for your to shutdown the broker in a
controlled fashion. What that would include is moving all the leaders away
from the broker so that it does not take any more produce requests. Once
that is done, you can shutdown the broker normally. You don't have to wait
until the
Storm has support for Kafka, if that's the sort of thing you're looking
for. Maybe you could describe your use case a bit more?
On Sunday, January 6, 2013, Guy Doulberg wrote:
> Hi
>
> I am looking for an ETL tool that can connect to kafka, as a consumer and
> as a producer,
>
> Have you heard of
You can use Kafka to store data on Hadoop via the Hadoop consumer in
contrib, and then use Talend or Pig to ETL it, before finally emitting the
ETL's records via the Hadoop producer in contrib.
https://github.com/kafka-dev/kafka/tree/master/contrib
http://docs.hortonworks.com/CURRENT/index.htm#Dat
Hi,
Thanks David,
I am looking for a product (open source or not), something like Talend
or Pentaho that in which I can design the ETL (from and to kafka), and
run the the ETL in Storm/ IronCount or even maybe I can run it in Hadoop
Map/Reduce.
The product should be complete and supports man
Hi Jun,
thanks for your answers, they helped a lot.
Regards,
Benjamin
On 01/03/2013 05:16 PM, Jun Rao wrote:
Ben,
For 1), yes.
For 2) and 3), you can shutdown your consumer cleanly, do the code upgrade
and start the consumer again. On restart, Kafka will make sure that the
consumer picks up
