Hello,
I am in the process of setting up a kafka cluster which is configured to
use KRaft. There is a set of three controller nodes and a set of six
brokers. Both the controllers and the brokers are configured to use mTLS
(Mutual TLS). So the part of the controller config looks like:
listeners
that could be(come)
duplicated or out-of-order in cases of outages. At least that will likely
happen at the boundaries where you switch to / back from a secondary backup
storage, and in the process of replaying data from that backup storage back
to Kafka.
For certain kinds of network
t; impact our services. To mitigate the impact of such outages and ensure
> continuity, we are exploring the possibility of leveraging Cassandra as a
> backup solution.
>
> Our proposed approach involves storing messages in Cassandra during Kafka
> outages. Subsequently, we
is susceptible to
occasional outages which can disrupt our operations and impact our
services. To mitigate the impact of such outages and ensure continuity, we
are exploring the possibility of leveraging Cassandra as a backup solution.
Our proposed approach involves storing messages in Cassandra
t; > It'll require having a HA cluster running in another region, of course.
> > One other caveat is that it doesn't preserve the offsets of the records
> >
> > -> I believe I can't afford to keep another cluster running due to cost
> > reasons.Can you elaborat
27;t preserve the offsets of the records
>
> -> I believe I can't afford to keep another cluster running due to cost
> reasons.Can you elaborate on the offset part, if offset is not preserved
> then how the backup cluster know where to start processing for each topic?
>
&
u elaborate on the offset part, if offset is not preserved
then how the backup cluster know where to start processing for each topic?
For example, you could use a Kafka Connect s3 sink. You'd have to write
some disaster-recovery code to restore lost data from s3 into Kafka.
-> again here the s
probably okay for your use-case, but you should be aware of it.
>
> Since what you want is a backup, there are many ways to do that which might
> be cheaper than another Kafka cluster.
>
> For example, you could use a Kafka Connect s3 sink. You'd have to write
> some dis
t preserve the offsets of the records. That's
probably okay for your use-case, but you should be aware of it.
Since what you want is a backup, there are many ways to do that which might
be cheaper than another Kafka cluster.
For example, you could use a Kafka Connect s3 sink. You'd have to
s://kafka.apache.org/documentation/#georeplication-mirrormaker
>>>
>>> Thanks.
>>> Luke
>>>
>>> On Fri, Mar 5, 2021 at 1:50 PM Pushkar Deole
>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I was looking for so
e Chen wrote:
> >
> > > Hi Pushkar,
> > > MirrorMaker is what you're looking for.
> > > ref:
> https://kafka.apache.org/documentation/#georeplication-mirrormaker
> > >
> > > Thanks.
> > > Luke
> > >
> > &g
te:
>
> > Hi Pushkar,
> > MirrorMaker is what you're looking for.
> > ref: https://kafka.apache.org/documentation/#georeplication-mirrormaker
> >
> > Thanks.
> > Luke
> >
> > On Fri, Mar 5, 2021 at 1:50 PM Pushkar Deole
> wrote:
> >
>
for.
> ref: https://kafka.apache.org/documentation/#georeplication-mirrormaker
>
> Thanks.
> Luke
>
> On Fri, Mar 5, 2021 at 1:50 PM Pushkar Deole wrote:
>
> > Hi All,
> >
> > I was looking for some options to backup a running kafka cluster, for
> >
Hi Pushkar,
MirrorMaker is what you're looking for.
ref: https://kafka.apache.org/documentation/#georeplication-mirrormaker
Thanks.
Luke
On Fri, Mar 5, 2021 at 1:50 PM Pushkar Deole wrote:
> Hi All,
>
> I was looking for some options to backup a running kafka cluster, for
>
Hi All,
I was looking for some options to backup a running kafka cluster, for
disaster recovery requirements. Can someone provide what are the available
options to backup and restore a running cluster in case the entire cluster
goes down?
Thanks..
sumer code
can help.
However, if you want to keep a backup of all messages to sort through later
outside of Kafka's interface, you can wire up a Kafka Connect sink to send
the data to an external source. Given that Kafka isn't designed for
indefinite data storage, this can also be
rnally. Thus, doing backups seems not to be necessary.
>
> Backups are usually used for single server system that may loose data if for
> example a disk goes bad, because those systems don't replicate data
> themselves, and the backup is a form of "manual data replication".
Thank you for your message ,Matthias J.
I understand the concept of server and cluster, but we need to backup the
entire kafka service (topic, configurations in cloudera)
I have read 3 options
1.Kafka Backup
2.Snapshots of the file system
3.Mirror Maker2
but there is not much information
, and the backup is a form of "manual data replication".
-Matthias
On 11/22/20 11:01 AM, WILSON BARBOSA RIAÑO wrote:
> Good Morning
>
>
>
>
>
> I need to subscribe to the kafka community to make a query how to backup
> topics and in general the kafka servic
Good Morning
I need to subscribe to the kafka community to make a query how to backup topics
and in general the kafka service thanks
Name: Wilson Barbosa
Correo warbar...@hotmail.com<mailto:warbar...@hotmail.com>
Country : Colombia
Enviado desde Correo<https://go.micr
Good Morning
I need to subscribe to the kafka community to make a query how to backup topics
and in general the kafka service thanks
Name: Wilson Barbosa
Correo warbar...@hotmail.com<mailto:warbar...@hotmail.com>
Country : Colombia
C -> SO instead for
obvious reasons. All that said, I've never had multiple Kafka nodes fail
to the extent that I need my S3 backup. But it's good to have it, for
peace of mind. Thanks, Liam Clarke-Hutchinson On Mon, 10 Aug. 2020, 2:27
pm kumar, <oracl...@gmail.co
Hi Kumar,
You can restore but you'll lose data since your last snapshot. I use KC ->
SO instead for obvious reasons.
All that said, I've never had multiple Kafka nodes fail to the extent that
I need my S3 backup. But it's good to have it, for peace of mind.
Thanks,
Liam Cla
backup+archive logs) of their database. is it possible to recover kafka
like that?
We had storage failure of the entire site. We were not confident on the
data recovered on kafka compared to oracle database. We had 3 kafka nodes
with no mirror maker.
My understanding is replication and mirror
might be some overhead around getting your Kafka Connect
workers deployed and configured correctly. But if you're looking to run a
hot/warm (main/backup) cluster scenario, MM 2 is ideal.
Kind regards,
Liam Clarke-Hutchinson
On Sun, Aug 9, 2020 at 11:56 PM Dor Ben Dov
some overhead around getting your Kafka Connect
workers deployed and configured correctly. But if you're looking to run a
hot/warm (main/backup) cluster scenario, MM 2 is ideal.
Kind regards,
Liam Clarke-Hutchinson
On Sun, Aug 9, 2020 at 11:56 PM Dor Ben Dov wrote:
> Hi Liam,
>
-Hutchinson
Sent: Sunday, August 9, 2020 2:52 PM
To: users@kafka.apache.org
Subject: Re: backup
Hi Dor,
There are multiple approaches.
1) Clone your Kafka broker volumes
2) Use Kafka Connect to stream all data to a different storage system such as
Hadoop, S3, etc.
3) Use Mirrormaker to replicate all
Hi Dor,
There are multiple approaches.
1) Clone your Kafka broker volumes
2) Use Kafka Connect to stream all data to a different storage system such
as Hadoop, S3, etc.
3) Use Mirrormaker to replicate all data to a backup cluster.
Which approach is right for you really depends on your needs
nt: Sunday, August 9, 2020 2:48 PM
To: users@kafka.apache.org
Subject: Re: backup
Hi Don,
Kafka is not meant to be a general purpose database. Its a streaming platform.
So think about retention of kafka messages rather than taking backup.
Kafka itself has retention capability. So you can tune it as per
Hi Don,
Kafka is not meant to be a general purpose database. Its a streaming
platform. So think about retention of kafka messages rather than taking
backup.
Kafka itself has retention capability. So you can tune it as per your need.
Regards,
Sunil.
On Sun, 9 Aug 2020 at 5:13 PM, Dor Ben Dov
Hi All,
What is the best recommended way, and tool to backup kafka in production?
Regards,
Dor
This email and the information contained herein is proprietary and confidential
and subject to the Amdocs Email Terms of Service, which you may review at
https://www.amdocs.com/about/email-terms-of
Hello,
I've got an topic with the default retention rate of 7 days.
Now on production we need to retrieve the events because of an incident.
Can i just spin up a new consumer (no group id), offset beginning and
consume the data?
It should not have any effect on other running consumers right?
--
Hi All,
Is there any community preferred tooling for doing point in time backups of
kafka? (ideally without downtime)
We've looked at https://github.com/pinterest/secor but refeeding ~500gb+ of
data doesn't seem too neat.
Thanks,
Ian.
kind of disaster).
- Mirror Maker is great, but does it keep the topic configuration,
partitioning and offsets? We basically want our consumers to keep on
working like they used to in case we lose a whole cluster.
- Secor as you said, will allow me to backup the data, but not to reload it
to a
the data and you would have to write
your own.
Let me know if that helps!
Thanks much,
Andrew Clarkson
On Wed, Dec 21, 2016 at 9:32 PM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:
> Hi,
>
> I have Kafka running on EC2 in AWS.
> I would like to backup my data volum
Hi,
I have Kafka running on EC2 in AWS.
I would like to backup my data volumes daily in order to recover to a point
in time in case of a disaster.
One thing I’m worried about is that if I do an EBS snapshot while Kafka is
running, it seems a Kafka that recovers on it will have to deal with
Hi,
I have 2 kafka nodes(node0--xx.host.com:9092,node1--localhost:9093) and 1
zookeeper node,and the producer configuration is below:
bootstrap.servers=xx.host.com:9092,localhost:9093
it's works fine when the 2 kafka nodes are running. But when I shutdown the
node0,and I got the error logs belo
Thanks for the quick anwser.
The only think I'm not happy about is _no_ env separation.
As far as I do understand, this is "online" operation, am I right?
So the best would be to create mirror, and then cut it off form Production.
greetings
Jakub
2015-06-09 11:17 GMT+02:00 Stevo Slavić :
> Hel
Hello Jakub,
Maybe it will work for you to combine MirrorMaker
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330
and Burrow: https://github.com/linkedin/Burrow
See recent announcement for Burrow
http://mail-archives.apache.org/mod_mbox/kafka-users/201506.mbox/%3CCACrdVJpS3k
Hi
I'm looking for the best way to "dump" current system state, and recreate
it on the new, autonomic environment.
Lets say I'd like to create a copy of Production, and based on that, create
new, separate environment for testing.
Can You suggest some solutions?
greetings
Jakub
There is a patch for KAFKA-1013 that has a tool for importing and exporting
offsets to and from kafka. Using that tool you can write
topic/partion/offset to a file and import it back to kafka.
Thanks,
Mayuresh
On Thu, Jan 8, 2015 at 1:21 PM, Kyle Banker wrote:
> We currently checkpoint partiti
We currently checkpoint partition offsets from ZK into an external
database, and we have a process for restoring those offsets to ZK in case
we ever need to replay data from Kafka.
Since offsets are now stored on the broker, the ConsumerOffsetChecker no
longer works. Is there a new way (tool) to d
will be preserved.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Sun, Jun 16, 2013 at 3:02 PM, Ran RanUser wrote:
>>
>> > I've been researching Kafka for our requirements and am trying to figure
>> > out the best way to implement multi-regi
:
>
> > I've been researching Kafka for our requirements and am trying to figure
> > out the best way to implement multi-region failover (lowest complexity).
> >
> > One requirement we have is that the offsets of the backup must match the
> > primary. As I understand
3:02 PM, Ran RanUser wrote:
> I've been researching Kafka for our requirements and am trying to figure
> out the best way to implement multi-region failover (lowest complexity).
>
> One requirement we have is that the offsets of the backup must match the
> primary. As I unders
I've been researching Kafka for our requirements and am trying to figure
out the best way to implement multi-region failover (lowest complexity).
One requirement we have is that the offsets of the backup must match the
primary. As I understand it, MirrorMaker does not (currently) guarantee
One possibility is to copy the data to another cluster using MirrorMaker.
Thanks,
Jun
On Fri, Jun 14, 2013 at 10:47 AM, Scott Clasen wrote:
> So despite 0.8 being a release that will give much higher availability do
> people do anything at all to back up the data?
>
> For instance if any of
So despite 0.8 being a release that will give much higher availability do
people do anything at all to back up the data?
For instance if any of you are running on EC2 and using ephemeral disk for
perf reasons, what do you do about messages that you absolutely cant afford
to lose.
Basically look
48 matches
Mail list logo
