I am trying to mirror a three node kafka cluster to a single node cluster.
When I starts the mirror maker I got the exception LeaderNotAvailable, but
at the same time topics are created and messages are commited.
Please help me out.
Best Regards
Anshul Bhatnagar
need to make sure mirror maker
>> > consumer
>> > > > > >> auto.offset.reset is set to smallest, otherwise when you run
>> > mirror
>> > > > > >>maker
>> > > > > >> for the first time, all the pr
smallest, otherwise when you run
> > mirror
> > > > > >>maker
> > > > > >> for the first time, all the pre-existing messages won¹t be
> > consumed.
> > > > > >> For partition sticking, can you verify if your messages are
>
sages
> > > > >> or not? If they are not keyed messages, can you check if you are
> > using
> > > > >>old
> > > > >> producer or new producer? For old producer, the default behavior
> is
> > > > >> sticking to one partition for
u check if you are
> using
> > > >>old
> > > >> producer or new producer? For old producer, the default behavior is
> > > >> sticking to one partition for 10 min and then move to the next
> > > >>partition.
> > > >
gt;> So if you wait for more than 10 min, you should see messages in two
> > >> different partitions.
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On 3/7/15, 8:28 AM, "sunil kalva" wrote:
> > >>
> > >>
t;> different partitions.
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On 3/7/15, 8:28 AM, "sunil kalva" wrote:
> >>
> >> >And i also observed ,all the data is moving to one partition in
> >> >destination
>
28 AM, "sunil kalva" wrote:
>>
>> >And i also observed ,all the data is moving to one partition in
>> >destination
>> >cluster though i have multiple partitions for that topic in source and
>> >destination clusters.
>> >
>> >
gt; >And i also observed ,all the data is moving to one partition in
> >destination
> >cluster though i have multiple partitions for that topic in source and
> >destination clusters.
> >
> >SunilKalva
> >
> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva
ar 7, 2015 at 9:54 PM, sunil kalva wrote:
>
>> I ran kafka mirroring tool after producing data in source cluster, and
>> this is not copied to destination cluster. If i produce data after
>>running
>> tool those data are copied to destination cluster. Am i missing
>>something ?
>>
>> --
>> SunilKalva
>>
>
>
>
>--
>SunilKalva
And i also observed ,all the data is moving to one partition in destination
cluster though i have multiple partitions for that topic in source and
destination clusters.
SunilKalva
On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva wrote:
> I ran kafka mirroring tool after producing data in sou
I ran kafka mirroring tool after producing data in source cluster, and this
is not copied to destination cluster. If i produce data after running tool
those data are copied to destination cluster. Am i missing something ?
--
SunilKalva
apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
>>> > > ?
>>> > >
>>> > > Guozhang
>>> > >
>>> > >
>>> > > On Thu, Aug 21, 2014 at 12:09 AM,
t;> > > On Thu, Aug 21, 2014 at 12:09 AM, Ashutosh Prasad Panigrahi <
>> > > ashutosh1...@gmail.com> wrote:
>> > >
>> > > > Hello All,
>> > > >
>> > > > When i start the mirroring process, events are written only to
>&
ote:
> > >
> > > > Hello All,
> > > >
> > > > When i start the mirroring process, events are written only to
> > partition
> > > > "1" on the target cluster and no data is written to partition 0.
> > > >
> >
gt; >
> > > My production setup looks something like this
> > > 1. Datacenter1 :3 kafka nodes/3 zookeeper nodes
> > > 2. Datacenter 2 : 3 kafka nodes/3 zookeeper nodes
> > >
> > > All the kafka nodes are configured with auto.create
eper nodes
> >
> > All the kafka nodes are configured with auto.create.topics.enable=true
> and
> > the default partition is 2 .
> >
> > I start the kafka mirroring tool with the following command
> >
> > bin/kafka-run-class.sh kafka.tools.MirrorMaker
gt; 1. Datacenter1 :3 kafka nodes/3 zookeeper nodes
> 2. Datacenter 2 : 3 kafka nodes/3 zookeeper nodes
>
> All the kafka nodes are configured with auto.create.topics.enable=true and
> the default partition is 2 .
>
> I start the kafka mirroring tool with the followin
ka nodes/3 zookeeper nodes
2. Datacenter 2 : 3 kafka nodes/3 zookeeper nodes
All the kafka nodes are configured with auto.create.topics.enable=true and
the default partition is 2 .
I start the kafka mirroring tool with the following command
bin/kafka-run-class.sh kafka.tools.M
Great, thanks for the answers all!
What about just for redundancy/maintenance purposes? I know that with
replication, individual brokers are redundant anyway, but might it ever be nice
to be able to take the analytics kafka cluster offline completely without
worrying about losing data from the
We sort of have the same situation where our analytics DC is one of the
main producer DC too. If you use Kafka only for analytics it is fine to
produce directly to the analytics cluster from that DC and mirror the rest.
However we also want to be able to run things locally that will consume
local
In our case, our aggregator/analytics cluster is in our main datacenter, so
there's no risk of the main producers becoming disconnected from it. It seems
nicer to have a dedicated aggregator cluster, that only only gets its data via
MirrorMaker (Option A), but in our case this isn't necessary.
We do something like A (though I'm not sure I understand B):
http://kafka.apache.org/documentation.html#datacenters
Essentially what we wanted was that each datacenter stood alone so that we
would not lose data if the datacenters became disconnected. Network
partitions within our data centers are
Hi all!
Wikimedia is investigating how best to set up Broker clusters in multiple data
centers. Our main analytics Broker cluster is currently in our main
datacenter. It is possible for all of the main DC's frontend producers to
produce directly to our analytics cluster, but we're not sure if
Thanks Jun.
We have a cluster as well. However, in our setup each Kafka node has both
the mirror maker and broker process running, with the mirror maker writing
only to the local broker. The main rationale was to limit further use of
network. But, I guess that wasn't a good idea given the failure s
Yes, this is true if you have only 1 broker in the target cluster. If you
set up multiple brokers in the target cluster, mirror maker will send
messages to available brokers.
Thanks,
Jun
On Fri, Mar 22, 2013 at 12:02 PM, Riju Kallivalappil <
riju.kallivalap...@corp.247customer.com> wrote:
> Hi,
Hi,
I've a question about fault tolerance of Kafka mirror maker (0.7.1)
Let's say that I've a mirroring setup with topics in broker B1 mirrored to
broker B2.
On B2, I've Kafka mirror maker and Kafka broker process running. Now,
following is what I noticed when the Kafka broker process on B2 is
r
Please can you try following the updated wiki -
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)?
Thanks,
Neha
On Sat, Jan 5, 2013 at 3:32 AM, CuiLiang wrote:
> Hi! I've been tasked with setting up a kafka cluster and mirroring. My
> Kafka version is
28 matches
Mail list logo
