Re: Kleppmann's idea of data loss - is it valid?

regards, Sebastian > Am 29.10.2024 um 16:04 schrieb Mike James : > > https://martin.kleppmann.com/2017/01/26/data-loss-in-large-clusters.html > > Is this article based on any experimental data? What are the real-world stats > on probability of data loss in large clusters. A dis

Kleppmann's idea of data loss - is it valid?

https://martin.kleppmann.com/2017/01/26/data-loss-in-large-clusters.html Is this article based on any experimental data? What are the real-world stats on probability of data loss in large clusters. A discussion of this is taking place within the company but I wanted to get real-world experiences

data loss

I am inserting to Cassandra by a simple insert query and an update counter query for every input record. input rate is so high. I've configured the update query with idempotent = true (no config for insert query, default is false IMHO) I've seen multiple records having rows in counter table (ide

?????? data loss in different DC

Thanks All -- -- ??: "Jeff Jirsa";; : 2017??9??28??(??) 9:16 ??: "user"; : Re: data loss in different DC Your quorum writers are only guaranteed to be on half+1 nodes - there??s no guarantee which node

Re: 回复： data loss in different DC

user@cassandra.apache.org>; *主题:* Re: data loss in different DC If you're writing into DC1 with CL = LOCAL_xxx, there is no guarantee to be sure to read the same data in DC2. Only repair will help you On Thu, Sep 28, 2017 at 11:41 AM, Peng Xiao <2535...@qq.com

Re: data loss in different DC

Your quorum writers are only guaranteed to be on half+1 nodes - there’s no guarantee which nodes those will be. For strong consistency with multiple DCs, You can either: - write at quorum and read at quorum from any dc, or - write each_quorum and read local_quorum from any dc, or - write at loc

Re: 回复： data loss in different DC

Reynald > > > On 28/09/2017 13:46, Peng Xiao wrote: > > even with CL=QUORUM,there is no guarantee to be sure to read the same data > in DC2,right? > then multi DCs looks make no sense? > > > -- 原始邮件 -- > *发件人:* "DuyHai Doan";

Re: 回复： data loss in different DC

? --?0?2?0?2-- *??:*?0?2"DuyHai Doan";; *:*?0?22017??9??28??(??) 5:45 *??:*?0?2"user"; *????:*?0?2Re: data loss in different DC If you're writing into DC1 with CL = LOCAL_xxx, there is no guarantee to be sure to read t

?????? data loss in different DC

even with CL=QUORUM,there is no guarantee to be sure to read the same data in DC2,right? then multi DCs looks make no sense? -- -- ??: "DuyHai Doan";; : 2017??9??28??(??) 5:45 ??: "user"; : Re: data

Re: data loss in different DC

very sorry for the duplicate mail. -- Original -- From: "";<2535...@qq.com>; Date: Thu, Sep 28, 2017 07:41 PM To: "user"; Subject: data loss in different DC Dear All, We have a cluster with one DC1:RF=3,another DC

data loss in different DC

Dear All, We have a cluster with one DC1:RF=3,another DC DC2:RF=1,DC2 only for ETL,but we found that sometimes we can query records in DC1,while not able not find the same record in DC2 with local_quorum.How it happens?looks data loss in DC2. Could anyone please advise? looks we can only run

Re: data loss in different DC

If you're writing into DC1 with CL = LOCAL_xxx, there is no guarantee to be sure to read the same data in DC2. Only repair will help you On Thu, Sep 28, 2017 at 11:41 AM, Peng Xiao <2535...@qq.com> wrote: > Dear All, > > We have a cluster with one DC1:RF=3,another DC DC2:RF=1 only for ETL,but > w

data loss in different DC

Dear All, We have a cluster with one DC1:RF=3,another DC DC2:RF=1 only for ETL,but we found that sometimes we can query records in DC1,while not able not find the same record in DC2 with local_quorum.How it happens? Could anyone please advise? looks we can only run repair to fix it. Thanks, P

回复： Data Loss irreparabley so

Due to the tombstone,we have set GC_GRACE_SECONDS to 6 hours.And for a huge table with 4T size,repair is a hard thing for us. -- 原始邮件 -- 发件人: "kurt";; 发送时间: 2017年8月3日(星期四) 中午12:08 收件人: "User"; 主题: Re: Data Loss irreparabley so You shou

Re: Data Loss irreparabley so

You should run repairs every GC_GRACE_SECONDS. If a node is overloaded/goes down, you should run repairs. LOCAL_QUORUM will somewhat maintain consistency within a DC, but certainly doesn't mean you can get away without running repairs. You need to run repairs even if you are using QUORUM or ONE.​

回复： Data Loss irreparabley so

-- 发件人: "Jon Haddad";; 发送时间: 2017年7月28日(星期五) 凌晨1:37 收件人: "user"; 主题: Re: Data Loss irreparabley so We (The Last Pickle) maintain an open source tool to help manage repairs across your clusters called Reaper. It’s a lot easier to set up and manage than trying to ma

Re: Cassandra data loss in come DC

> anything exceptional. > we only upgrade this DC from 2.1.13 to 2.1.18,but this seems won't cause data > loss. > > Could anyone please advise? > > Thanks, > Peng - To unsubscribe, e-mail: user-

Cassandra data loss in come DC

upgrade this DC from 2.1.13 to 2.1.18,but this seems won't cause data loss. Could anyone please advise? Thanks, Peng

Re: Data Loss irreparabley so

We (The Last Pickle) maintain an open source tool to help manage repairs across your clusters called Reaper. It’s a lot easier to set up and manage than trying to manage it through cron. http://thelastpickle.com/reaper.html > On Jul 27, 2017, at 12:38 AM,

Re: Data Loss irreparabley so

In that vein, Cassandra support Auto compaction and incremental repair. Does this mean I have to set up cron jobs on each node to do a nodetool repair or is this taken care of by Cassandra anyways? How often should I run nodetool repair Greetings Daniel Jeff Jirsa schrieb am Do. 27. Juli 2017 um

Re: Data Loss irreparabley so

On 2017-07-25 15:49 (-0700), Roger Warner wrote: > This is a quick informational question. I know that Cassandra can detect > failures of nodes and repair them given replication and multiple DC. > > My question is can Cassandra tell if data was lost after a failure and > node(s) “fixed

Re: Data Loss irreparabley so

Cassandra doesn't do any automatic repairing. It can tell if your data is inconsistent, however it's really up to you to manage consistency through repairs and choice of consistency level for queries. If you lose a node, you have to manually repair the cluster after replacing the node, but really y

Data Loss irreparabley so

This is a quick informational question. I know that Cassandra can detect failures of nodes and repair them given replication and multiple DC. My question is can Cassandra tell if data was lost after a failure and node(s) “fixed” and resumed operation? If so where would it log or flag it? O

Data loss after system_keyspace-table removal?

Hi experts! Seems, that we have a data loss after a rolling update of our Cassandra 2.2.7 nodes. The nodes failed to start with an exception: IllegalStateException: empty rows returned when reading system.schema_keyspaces à ( https://issues.apache.org/jira/browse/CASSANDRA-12351

Data loss bug in upgrade to 3.0

We have discovered a critical bug for upgrade in the 3.0.0, 3.0.1 and 3.1 release. This bug, https://issues.apache.org/jira/browse/CASSANDRA-10822, only affects upgrades from a 2.x version (to any of the currently released 3.x version) and might cause data loss. As such, we strongly advice waiting

RE: Insert (and delete) data loss?

2015 15.25 To: user@cassandra.apache.org Subject: Re: Insert (and delete) data loss? Can you try two more tests: 1) Write the way you are, perform a repair on all nodes, then read the way you are. 2) Write with CL quorum, read with CL quorum. On Tue, Jun 30, 2015 at 8:34 AM, Mauri Moreno

RE: Insert (and delete) data loss?

2015 15.25 To: user@cassandra.apache.org Subject: Re: Insert (and delete) data loss? Can you try two more tests: 1) Write the way you are, perform a repair on all nodes, then read the way you are. 2) Write with CL quorum, read with CL quorum. On Tue, Jun 30, 2015 at 8:34 AM, Mauri Moreno

RE: Insert (and delete) data loss?

Hi Carlos, first I used Consistency ONE, then ALL (I retry with ALL in order to be sure that problem doesn’t disappear). Thanks Moreno From: Carlos Alonso [mailto:i...@mrcalonso.com] Sent: martedì 30 giugno 2015 15.24 To: user@cassandra.apache.org Subject: Re: Insert (and delete) data loss

Re: Insert (and delete) data loss?

Can you try two more tests: 1) Write the way you are, perform a repair on all nodes, then read the way you are. 2) Write with CL quorum, read with CL quorum. On Tue, Jun 30, 2015 at 8:34 AM, Mauri Moreno Cesare < morenocesare.ma...@italtel.com> wrote: > Hi all, > > I configured a Cassandra Cl

Re: Insert (and delete) data loss?

Hi Moreno, Which consistency level are you using? If you're using ONE, that may make sense, as, depending on the partitioning and the cluster coordinating the query, different values may be received. Hope it helps. Regards Carlos Alonso | Software Engineer | @calonso

Insert (and delete) data loss?

Hi all, I configured a Cassandra Cluster (3 nodes). Then I created a KEYSPACE: cql> CREATE KEYSPACE test WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; and a table: cql> CREATE TABLE chiamate_stabili (chiave TEXT PRIMARY KEY, valore BLOB); I inserted

Re: Cassandra version 1.0.10 Data Loss upon restart

it is > being replicated with another node in another data center. We are using the > thrift API in java application to retrieve and modify the data from the > primary node only. We are not using the other node in the other data center > for any operations. The data loss well exceeds the commit

Re: Cassandra version 1.0.10 Data Loss upon restart

using the thrift API in java application to retrieve and modify the data from the primary node only. We are not using the other node in the other data center for any operations. The data loss well exceeds the commit log sync period. Thanks, Ankit On Mon, Nov 24, 2014 at 8:53 PM, Robert Coli

Re: Cassandra version 1.0.10 Data Loss upon restart

On Mon, Nov 24, 2014 at 5:51 PM, Robert Coli wrote: > What is your replication factor? What CL are you using to read? > Ah, I see from OP that RF is 1. As a general statement, RF=1 is an edge case which very, very few people have ever operated in production. It is relatively likely that there a

Re: Cassandra version 1.0.10 Data Loss upon restart

On Mon, Nov 24, 2014 at 3:19 PM, Ankit Patel wrote: > We are experiencing data loss with Cassandra 1.0.10 when we had restarted > the without flushing. We see in the cassandra logs that the commitlogs were > read back without any problems. Until the restart the data was correct. > Ho

Cassandra version 1.0.10 Data Loss upon restart

We are experiencing data loss with Cassandra 1.0.10 when we had restarted the without flushing. We see in the cassandra logs that the commitlogs were read back without any problems. Until the restart the data was correct. However, after the node restarted we retrieved older version of the data

RE: NTS, vnodes and 0% chance of data loss

014 11:11 PM To: user@cassandra.apache.org Subject: NTS, vnodes and 0% chance of data loss I found this: http://mail-archives.apache.org/mod_mbox/cassandra-user/201404.mbox/%3ccaeduwd1erq-1m-kfj6ubzsbeser8dwh+g-kgdpstnbgqsqc...@mail.gmail.com%3E I read the three referenced cases. In addit

NTS, vnodes and 0% chance of data loss

one replica. At this point, my "0% chance of data loss" scenarios are: 1.) Failure of two nodes at random 2.) Failure of 2 racks (6 nodes!) Vnodes: my "0% chance of data loss" scenarios are: 1.) Failure of two nodes at random Which means a rack failure (3 nodes) has a non-zero ch

Re: NTS, vnodes and 0% chance of data loss

: I could do token selection using ideas from case 3810 such that > each rack has one replica. At this point, my "0% chance of data loss" > scenarios are: > 1.) Failure of two nodes at random > 2.) Failure of 2 racks (6 nodes!) > > Vnodes: my "0% chance of data loss

Re: Data loss when swapping out cluster

On Fri, Nov 29, 2013 at 6:36 PM, Anthony Grasso wrote: > In this case would it be possible to do the following to replace a seed > node? > With the quoted procedure, you are essentially just "changing the ip address of a node", which will work as long as you set auto_bootstrap:false in cassandra.

Re: Data loss when swapping out cluster

Hi Robert, In this case would it be possible to do the following to replace a seed node? nodetool disablethrift nodetool disablegossip nodetool drain stop Cassandra deep copy /var/lib/cassandra/* on old seed node to new seed node start Cassandra on new seed node Regards, Anthony On Wed, Nov

Re: Data loss when swapping out cluster

7;s no warning, no logging, no documentation; just something that you might accidentally do and which will manifest itself as random data loss. /Janne On 26 Nov 2013, at 21:20, Robert Coli wrote: > On Tue, Nov 26, 2013 at 9:48 AM, Christopher J. Bottaro > wrote: > One thing that I didn&#x

Re: Data loss when swapping out cluster

On Tue, Nov 26, 2013 at 9:48 AM, Christopher J. Bottaro < cjbott...@academicworks.com> wrote: > One thing that I didn't mention, and I think may be the culprit after > doing a lot or mailing list reading, is that when we brought the 4 new > nodes into the cluster, they had themselves listed in the

Re: Data loss when swapping out cluster

We ran repair -pr on each node after we realized there was data loss and we added the 4 original nodes back in the cluster. I.e. we ran repair on the 8 node cluster that consisted of the 4 old and 4 new nodes, once we realized there was a problem. We are using quorum reads and writes. One thing

Re: Data loss when swapping out cluster

> We recently experienced (pretty severe) data loss after moving our 4 node > Cassandra cluster from one EC2 availability zone to another. Our strategy > for doing so was as follows: > One at a time, bring up new nodes in the new availability zone and have them > join the clu

Re: Data loss when swapping out cluster

That sounds bad! Did you run repair at any stage? Which CL are you reading with? /Janne On 25 Nov 2013, at 19:00, Christopher J. Bottaro wrote: > Hello, > > We recently experienced (pretty severe) data loss after moving our 4 node > Cassandra cluster from one EC2 availabi

Data loss when swapping out cluster

Hello, We recently experienced (pretty severe) data loss after moving our 4 node Cassandra cluster from one EC2 availability zone to another. Our strategy for doing so was as follows: - One at a time, bring up new nodes in the new availability zone and have them join the cluster. - One

Re: Simple test of adding a node causes data loss

bootstrap > operation. > > > > Julien Campan > > > > > 2013/11/21 Tamar Rosen > >> Hi, >> >> We are testing the process of adding a node to a cluster using a simple >> procedure, and seeing data loss. >> >> System: Ubuntu 12.04 on AWS &

Re: Simple test of adding a node causes data loss

a simple > procedure, and seeing data loss. > > System: Ubuntu 12.04 on AWS > Version: Cassandra + dsc 1.2.10 > > Here is what we did: > Created 2 new m1.large instances > Installed Java > Installed Cassandra 1.2.10 (the version we are using in our production > system) &g

Simple test of adding a node causes data loss

Hi, We are testing the process of adding a node to a cluster using a simple procedure, and seeing data loss. System: Ubuntu 12.04 on AWS Version: Cassandra + dsc 1.2.10 Here is what we did: Created 2 new m1.large instances Installed Java Installed Cassandra 1.2.10 (the version we are using in

Re: Data Loss/Missing With Cassandra

On Fri, Jun 14, 2013 at 2:05 PM, Tyler Hobbs wrote: > Your data is not marked as deleted. sstableloader will stream all of the > data on that node to all other nodes that need it. http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra Has some details on the difference between

Re: Data Loss/Missing With Cassandra

On Mon, Jun 10, 2013 at 1:11 PM, Nimi Wariboko Jr wrote: > > 1.) Seeing as how I only had a single node cluster previously, if I just > `nodetool move 0` the original node that should be an easy fix then? > > You don't need to move the node (and it won't fix everything), just run sstableloader.

Re: Data Loss/Missing With Cassandra

r Hobbs wrote: > > > > > > > > On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr > > (mailto:nimiwaribo...@gmail.com)> wrote: > > > > If I had to do a repair after upping the RF, than that is probably what > > > > caused the data loss. Wish

Re: Data Loss/Missing With Cassandra

This is true unless you have ran cleanup. If you have run 'nodetool cleanup' then there is data loss. On Mon, Jun 10, 2013 at 1:40 PM, Tyler Hobbs wrote: > > On Mon, Jun 10, 2013 at 11:39 AM, Nimi Wariboko Jr < > nimiwaribo...@gmail.com> wrote: > >> How can

Re: Data Loss/Missing With Cassandra

On Mon, Jun 10, 2013 at 11:39 AM, Nimi Wariboko Jr wrote: > How can I recover that data? Can I assume they are still in the sstables? Yes > Would doing a sstable2json then reading and reinserting be an optimal > solution? You can just use the sstableloader utility to load them directly. It

Re: Data Loss/Missing With Cassandra

AM, Tyler Hobbs wrote: > > > On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr > wrote: > > If I had to do a repair after upping the RF, than that is probably what > caused the data loss. Wish I had been more careful. > > I'm guessing the data is irrevocably lost, I

Re: Data Loss/Missing With Cassandra

waribo...@gmail.com)> wrote: > > If I had to do a repair after upping the RF, than that is probably what > > caused the data loss. Wish I had been more careful. > > > > I'm guessing the data is irrevocably lost, I didn't make any any snapshots. > > > > Would

Re: Data Loss/Missing With Cassandra

e RF, than that is probably what >> caused the data loss. Wish I had been more careful. >> >> I'm guessing the data is irrevocably lost, I didn't make any any >> snapshots. >> >> Would it be possible to figure out if only a certain part of the ring was

Re: Data Loss/Missing With Cassandra

On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr wrote: > If I had to do a repair after upping the RF, than that is probably what > caused the data loss. Wish I had been more careful. > > I'm guessing the data is irrevocably lost, I didn't make any any snapshots. >

Re: Data Loss/Missing With Cassandra

If I had to do a repair after upping the RF, than that is probably what caused the data loss. Wish I had been more careful. I'm guessing the data is irrevocably lost, I didn't make any any snapshots. Would it be possible to figure out if only a certain part of the ring was effected?

Re: Data Loss/Missing With Cassandra

ikely have a data loss. If you did repair all nodes before the shuffle, I do not know then the shuffle must have went wrong. If your reading at CL.ALL and still seeing inconsistencies that is bad. Possible try raising the read repair chance to 100% and continue reading and see if the data becomes

Data Loss/Missing With Cassandra

Hi, We are seeing an issue where data that was written to the cluster is no longer accessible after trying to expand the size of the cluster. I will try and provide as much information as possible, I am just starting at with Cassandra and I'm not entirely sure what data is relevant. All Cassa

Re: potential data loss in Cassandra 1.1.0 .. 1.1.4

On Thu, Oct 18, 2012 at 7:30 AM, Alain RODRIGUEZ wrote: > Hi Jonathan. > > We are currently running the datastax AMI on amazon. Cassandra is in version > 1.1.2. > > I guess that the datastax repo (deb http://debian.datastax.com/community > stable main) will be updated directly in 1.1.6 ? Yes. >

Re: potential data loss in Cassandra 1.1.0 .. 1.1.4

ier 1.1 release, you should upgrade > to 1.1.6. > > Explanation: > > A rewrite of the commitlog code for 1.1.0 used Java's nanotime api to > generate commitlog segment IDs. This could cause data loss in the > event of a power failure, since we assume commitlog IDs are strict

potential data loss in Cassandra 1.1.0 .. 1.1.4

re okay as far as this is concerned. But if you used an earlier 1.1 release, you should upgrade to 1.1.6. Explanation: A rewrite of the commitlog code for 1.1.0 used Java's nanotime api to generate commitlog segment IDs. This could cause data loss in the event of a power failure, since we a

Re: would it possible for this kind of data loss?

Ok, will look into it, thx for the heads up. Sent from a mobile device, please forgive typos. On Aug 21, 2011 6:45 PM, "Yan Chunlu" wrote: > I was aware of the deleted items might be come back alive without proper > node repair. > > how about modified items, for example 'A'=>{1,2,3}. then 'A'=>{4

would it possible for this kind of data loss?

I was aware of the deleted items might be come back alive without proper node repair. how about modified items, for example 'A'=>{1,2,3}. then 'A'=>{4,5}. if that possible 'A' change back to {1,2,3}? I have encountered this mystery problem after go through a mess procedure with cassandra nodes

Re: Data loss and corruption

Evan was just doing a ritual CYA and saying "this is new technology" [about a year ago]. On Tue, Jun 8, 2010 at 10:45 PM, Hector Urroz wrote: > Hi all, > We're starting to prototype Cassandra for use in a production system and > became concerned about data corruption after reading the excellent a

Re: Data loss and corruption

In my opinion the #1 risk for corruption is user/client error with the timestamps. Over time, Cassandra flushes data from memory to disks. After it flushes to disk, Cassandra doesn't go back to delete or modify that data. Because of this, deletes are performed by writing a "tombstone" to disk.

Data loss and corruption

Hi all, We're starting to prototype Cassandra for use in a production system and became concerned about data corruption after reading the excellent article: http://blog.evanweaver.com/articles/2009/07/06/up-and-running-with-cassandra/ where Evan Weaver writes: "Cassandra is an alpha product and

Re: Cassandra data loss

You also need to set CommitLogSync to batch instead of periodic if you Absolutely Cannot Lose Data. On Mon, May 24, 2010 at 10:51 AM, Joe Stump wrote: > This is largely FUD. Cassandra let's you choose how consistent you want > writes to be. The more consistency you choose, the slower the writes,

Re: Cassandra data loss

m in Cassandra (which will likely cause data loss)? Or human > intervention is needed? > > Steve > > > On Mon, May 24, 2010 at 11:51 AM, Joe Stump wrote: > >> This is largely FUD. Cassandra let's you choose how consistent you want >> writes to be. The more cons

Re: Cassandra data loss

en if I set it up as eventual consistent? > Will the data become inconsistent after a crash/reboot, similar to the case > of asynchronous replication? Is there an automated conflict resolution > algorithm in Cassandra (which will likely cause data loss)? Or human > intervention

Re: Cassandra data loss

replication? Is there an automated conflict resolution algorithm in Cassandra (which will likely cause data loss)? Or human intervention is needed? Steve On Mon, May 24, 2010 at 11:51 AM, Joe Stump wrote: > This is largely FUD. Cassandra let's you choose how consistent you want > writes to b

Re: Cassandra data loss

This is largely FUD. Cassandra let's you choose how consistent you want writes to be. The more consistency you choose, the slower the writes, but it's very unlikely with high consistency that you'll lose data. That being said, if you write with a consistency level of 0 then, yes, you could lose

Cassandra data loss

I am evaluating Cassandra as a candidate for our next-gen database. One of my colleagues told me that "it's not recommended to use it as your system of Record because it CAN lose data". Can someone with architecture understanding shed some light on under what circumstance Cassandra cluster can eith