Hey Ben, I think the distinction between your and Matthias' points is analogous to, borrowing from the OSI model, transport- vs presentation/application-level concerns. When configured appropriately, Kafka durably provides transport for messages from point A to point B, with such optional goodies as exactly-once delivery if desired. However, if the application -- especially ie a routing application that forwards messages from topic to topic -- has a bug, then that can cause problems not related directly to Kafka's functionality. This is where design elements like idempotent consumer code can help.
However, if you want to keep a backup of all messages to sort through later outside of Kafka's interface, you can wire up a Kafka Connect sink to send the data to an external source. Given that Kafka isn't designed for indefinite data storage, this can also be useful if you want a persistent record of activity in your system. Cheers, Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any unauthorized or improper disclosure, copying, distribution, or use of the contents of this message is prohibited. The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If you have received this message in error, please notify the sender immediately and delete the original message. On Sun, Nov 22, 2020 at 11:57 PM Young, Ben <ben.yo...@fisglobal.com.invalid> wrote: > Hi, > > We love Kafka but I would pretty strongly disagree with that statement. > Backups are not really about recovering from local failures and lots of > systems that are resilient and redundant have backups as well (e.g. > CockroachDB). Uses of backups in a system like Kafka would include: > > 1) Recovering from application errors that have damaged the record and > history (e.g. an accidental day of streaming incorrect data into Kafka) > 2) Recovering from accidentally applied admin errors (e.g. deleting a > topic) > 3) Recovering from DC level failures (yes MirrorMaker works, but is not > always appropriate) > 4) Transferring point in time images to other systems for analysis (e.g. > production data to development for analysis of issues) > > So for us backups would be a fantastic addition to Kafka > > Thanks, > Ben > > > Kafka is a highly available and fault-tolerant system, that replicates > data internally. Thus, doing backups seems not to be necessary. > > > > Backups are usually used for single server system that may loose data if > for example a disk goes bad, because those systems don't replicate data > themselves, and the backup is a form of "manual data replication". > > > > > > -Matthias > > On 11/22/20 11:01 AM, WILSON BARBOSA RIAÑO wrote: > > Good Morning > > > > > > > > > > > > I need to subscribe to the kafka community to make a query how to > > backup topics and in general the kafka service thanks > > > > > > > > > > > > Name: Wilson Barbosa > > > > Correo warbar...@hotmail.com<mailto:warbar...@hotmail.com> > > > > Country : Colombia > > > > > > Enviado desde > > Correo<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F > > %2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986&data=04%7C01%7CB > > en.Young%40fisglobal.com%7C9960343890c5473a004608d88f332daf%7Ce3ff91d8 > > 34c84b15a0b418910a6ac575%7C0%7C0%7C637416797268360993%7CUnknown%7CTWFp > > bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn > > 0%3D%7C1000&sdata=nM2ad5iG6DIgAVVNTRS1EEA9WO3FKmBCzX%2BGnOmfYMo%3D > > &reserved=0> para Windows 10 > > > > > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete the > message and all copies; (ii) do not disclose, distribute or use the message > in any manner; and (iii) notify the sender immediately. In addition, please > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. FIS is a trading > name of the following companies: Advanced Portfolio Technologies Ltd (No: > 6312142) | Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: > 2567370) | FIS Apex (International) Limited (No: 2999960) | FIS Apex (UK) > Limited (No. 3573008) | FIS Consulting Services (UK) Limited (No: 2486794) > | FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS Energy > Solutions Limited (No: 1889028) | FIS Global Execution Services Limited > (No. 3127109) | FIS Global Trading (UK) Limited (No: 2523114) | FIS > Investment Systems (UK) Limited (No: 1366010) | FIS Sherwood Systems Group > Limited (No: 982833) | FIS Systems Limited (No: 1937159) | FIS Treasury > Systems (Europe) Limited (No: 2624209) | FIS Treasury Systems (UK) Limited > (No: 2893376) | GL Settle Limited (No: 2396127) | Integrity Treasury > Solutions Europe Limited (No: 3289271) | Monis Software Limited (No: > 2333925) | Reech Capital Limited (No: 3649490) | Solutions Plus Consulting > Services Limited (No: 3839487) | Valuelink Information Services Limited > (No: 3827424) all registered in England & Wales with their registered > office at 25 Canada Square, London E14 5LQ | FIS Global Execution Services > Limited is authorised and regulated by the Financial Conduct Authority | > Certegy Card Services Limited (No: 3517639) | Certegy France Limited (No: > 2557650) | eFunds International Limited (No: 1930117) | Fidelity > Information Services Limited (No: 2225203) | FIS Payments (UK) Limited (No: > 4215488) | Metavante Technologies Limited (No: 2659326) all registered in > England & Wales with their registered office at 1st Floor Tricorn House, > 51-53 Hagley Road, Edgbaston, Birmingham, West Midlands, B16 8TU, United > Kingdom | FIS Payments (UK) Limited is authorised and regulated by the > Financial Conduct Authority; some services are covered by the Financial > Ombudsman Service (in the UK) | Worldpay (UK) Limited (No: 07316500 / FCA > No. 530923) | Worldpay Limited (No. 03424752 / FCA No. 504504) | Worldpay > AP Limited (No: 05593466 / FCA No: 502597) all registered in England & > Wales with their registered office at The Walbrook Building, 25 Walbrook, > London EC4N 8AF and authorised by the Financial Conduct Authority under the > Payment Service Regulations 2017 for the provision of payment services. > Worldpay (UK) Limited is authorised and regulated by the Financial Conduct > Authority for consumer credit activities | Worldpay B.V. has its registered > office in Amsterdam, the Netherlands (Handelsregister KvK no. 60494344). > WPBV holds a licence from and is included in the register kept by De > Nederlandsche Bank, which registration can be consulted through www.dnb.nl. > Clear2Pay Limited, Registered in Scotland (No SC157659), Registered Office: > Clear2Pay House, Pitreavie Court, Pitreavie Business Park Queensferry Rd, > Dunfermline, Fife, SS, KY11 8UU, Scotland | FIS eProcess Intelligence LLC > (UK Branch), UK Establishment Registered in England & Wales (No: > FC16527/Branch No. BR000355), Registered Branch Office: 25 Canada Square, > London, E14 5LQ; FIS eProcess Intelligence LLC is a limited liability > company formed in the USA registered on file with the Office of the > Delaware Secretary of State, Division of Corporations (File No. 2032143), > Head Office: 601 Riverside Avenue, Jacksonville Florida, FL32204, USA | FIS > Investment Systems LLC, UK Establishment Registered in England & Wales (No: > FC033836/Branch No. BR018923), Registered Branch Office: 25 Canada Square, > London, E14 5LQ; FIS Investment Systems LLC is a limited liability company > formed in the USA registered on file with the Office of the Delaware > Secretary of State, Division of Corporations (File No. 0881255), Head > Office: 377 E. Butterfield Road, Suite 800, Lombard, IL 60148, USA | Calls > to and from the companies may be recorded for quality purposes. | All of > the named companies are part of FIS (Fidelity National Information > Services, Inc.). >
