Re: Lots of write timeouts and missing data during decomission/bootstrap

WOW.. nice. you rock!! On Wed, Jul 1, 2015 at 3:18 PM, Robert Coli wrote: > On Wed, Jul 1, 2015 at 2:58 PM, Kevin Burton wrote: > >> Looks like all of this is happening because we’re using CAS operations >> and the driver is going to SERIAL consistency level. >> ... >> This is Cassandra 2.0.9 b

Re: Lots of write timeouts and missing data during decomission/bootstrap

On Wed, Jul 1, 2015 at 2:58 PM, Kevin Burton wrote: > Looks like all of this is happening because we’re using CAS operations and > the driver is going to SERIAL consistency level. > ... > This is Cassandra 2.0.9 btw. > > https://issues.apache.org/jira/browse/CASSANDRA-8640 =Rob (credit to iamal

Re: Lots of write timeouts and missing data during decomission/bootstrap

Looks like all of this is happening because we’re using CAS operations and the driver is going to SERIAL consistency level. SERIAL and LOCAL_SERIAL write failure scenarios¶ > >

Lots of write timeouts and missing data during decomission/bootstrap

We get lots of write timeouts when we decommission a node. About 80% of them are write timeout and just about 20% of them are read timeout. We’ve tried to adjust streamthroughput (and compaction throughput) for that matter and that doesn’t resolve the issue. We’ve increased write_request_timeout

Re: Missing data

Jun 15, 2015, at 10:50 AM, Jean Tremblay mailto:jean.tremb...@zen-innovations.com>> wrote: Dear all, I identified a bit more closely the root cause of my missing data. The problem is occurring when I use com.datastax.cassandra cassandra-driver-core 2.1.6 on my client against Cassandra 2

Re: Missing data

trating. If you are doing a simple loop on a "select * " query, then the DataStax java driver will only process 2^31 rows (e.g. the Java Integer Max (2,147,483,647)) before it stops w/o any error or output in the logs. The fact that you said you only had about 2 billion rows but you ar

Re: Missing data

cause of my missing data. The problem is occurring when I use com.datastax.cassandra cassandra-driver-core 2.1.6 on my client against Cassandra 2.1.6. I did not have the problem when I was using the driver 2.1.4 with C* 2.1.4. Interestingly enough I don’t have the problem with the driver 2.1.

Re: Missing data

(2,147,483,647)) before it stops w/o any error or output in the logs. The fact that you said you only had about 2 billion rows but you are seeing missing data is a red flag. I found the only way around this is to do your "select *" in chunks based on the token range (see this g

Re: Missing data

Dear all, I identified a bit more closely the root cause of my missing data. The problem is occurring when I use com.datastax.cassandra cassandra-driver-core 2.1.6 on my client against Cassandra 2.1.6. I did not have the problem when I was using the driver 2.1.4 with C* 2.1.4. Interestingly

Re: Missing data

ts to see how many tombstones per read slice you have. This is, probably, also the cause of your missing data. Data was tombstoned, so it is not available. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.c

Missing data

Hi, I have reloaded the data in my cluster of 3 nodes RF: 2. I have loaded about 2 billion rows in one table. I use LeveledCompactionStrategy on my table. I use version 2.1.6. I use the default cassandra.yaml, only the ip address for seeds and throughput has been change. I loaded my data with si

Re: MIssing data in range query

Nope. No secondary index. Just a slice query on the PK. On Tuesday, October 7, 2014, Robert Coli wrote: > On Tue, Oct 7, 2014 at 3:11 PM, Owen Kim > wrote: > >> Sigh, it is a bit grating. I (genuinely) appreciate your acknowledgement >> of that. Though, I didn't intend for the question to be

Re: MIssing data in range query

On Tue, Oct 7, 2014 at 3:11 PM, Owen Kim wrote: > Sigh, it is a bit grating. I (genuinely) appreciate your acknowledgement > of that. Though, I didn't intend for the question to be "about" > supercolumns. > (Yep, understand tho that if you hadn't been told that advice before, it would grate a lo

Re: MIssing data in range query

Sigh, it is a bit grating. I (genuinely) appreciate your acknowledgement of that. Though, I didn't intend for the question to be "about" supercolumns. It is possible I'm hitting an odd edge case though I'm having trouble reproducing the issue in a controlled environment since there seems to be a t

Re: MIssing data in range query

On Tue, Oct 7, 2014 at 2:03 PM, Owen Kim wrote: > I'm aware. I've had the system up since pre-composite columns and haven't > had the cycles to do a major data and schema migration. > > And that's not "slightly" non-responsive. > "There may be unknown bugs in the code you're using, especially be

Re: MIssing data in range query

I'm aware. I've had the system up since pre-composite columns and haven't had the cycles to do a major data and schema migration. And that's not "slightly" non-responsive. On Tue, Oct 7, 2014 at 1:49 PM, Robert Coli wrote: > On Tue, Oct 7, 2014 at 1:38 PM, Owen Kim wrote: > >> I'm running Cass

Re: MIssing data in range query

On Tue, Oct 7, 2014 at 1:38 PM, Owen Kim wrote: > I'm running Cassandra 1.2.16 with supercolumns and Hector. > Slightly non-responsive response : In general supercolumn use is not recommended. It makes it more difficult to get support when one uses a feature no one else uses. =Rob

MIssing data in range query

Hello, I'm running Cassandra 1.2.16 with supercolumns and Hector. create column family CFName with column_type = 'Super' and comparator = 'UTF8Type' and subcomparator = 'UTF8Type' and default_validation_class = 'UTF8Type' and key_validation_class = 'UTF8Type' and read_repair_cha

Re: restoring from snapshot - missing data

Thanks. After creating the data model and matching the correct snapshot with the correct new node (same token) all worked fine! *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956

Re: restoring from snapshot - missing data

On Mon, May 21, 2012 at 12:01 AM, Tamar Fraenkel wrote: > If I am putting the snapshots on a clean ring, I need to first create the > data model? Yes. -- Tyler Hobbs DataStax

Re: restoring from snapshot - missing data

Thanks. Just figured out yesterday that I switched the snapshots mixing the tokens. Will try again today. And another question. If I am putting the snapshots on a clean ring, I need to first create the data model? Thanks *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1]

Re: restoring from snapshot - missing data

Did you use the same tokens for the nodes in both clusters? On Sun, May 20, 2012 at 1:25 PM, Tamar Fraenkel wrote: > Hi! > I am testing backup and restore. > I created the restore using parallel ssh on all 3 nodes. > I created a new 3 ring setup and used the snapshot to test recover. > Snapshot

restoring from snapshot - missing data

Hi! I am testing backup and restore. I created the restore using parallel ssh on all 3 nodes. I created a new 3 ring setup and used the snapshot to test recover. Snapshot from every original node went to one of the new nodes. When I compare the content of the data dir it seems that all files from t

Node OOM, Slice query - missing data?

Hi there, We run a 3 node cluster with 0.7.8 with replication factor 3 for all key spaces. We store external->internal key mappings in a column family with one row for each customer. The largest row contains abount 200k columns. If we import external data we load the whole row and map externa

Re: commitlog replay missing data

> # wait for a bit until no one is sending it writes anymore More accurately, until all other nodes have realized it's down (nodetool ring on each respective host). -- / Peter Schuller (@scode on twitter)

Re: commitlog replay missing data

> What are the other ways to stop Cassandra? nodetool disablegossip nodetool disablethrift # wait for a bit until no one is sending it writes anymore nodetool flush # only relevant if in periodic mode # then kill it > What's the difference between batch vs periodic? Search for "batch" on http://

Re: commitlog replay missing data

Peter Schuller wrote: > >> Recently upgraded to 0.8.1 and noticed what seems to be missing data >> after a >> commitlog replay on a single-node cluster. I start the node, insert a >> bunch >> of stuff (~600MB), stop it, and restart it. There are log messages >

Re: commitlog replay missing data

> Recently upgraded to 0.8.1 and noticed what seems to be missing data after a > commitlog replay on a single-node cluster. I start the node, insert a bunch > of stuff (~600MB), stop it, and restart it. There are log messages If you stop by a kill, make sure you use batched commitlog s

Re: commitlog replay missing data

orton http://www.thelastpickle.com On 12/07/2011, at 3:28 PM, Jeffrey Wang wrote: > Hey all, > > > > Recently upgraded to 0.8.1 and noticed what seems to be missing data after a > commitlog replay on a single-node cluster. I start the node, insert a bunch > of stuff (~

commitlog replay missing data

Hey all, Recently upgraded to 0.8.1 and noticed what seems to be missing data after a commitlog replay on a single-node cluster. I start the node, insert a bunch of stuff (~600MB), stop it, and restart it. There are log messages pertaining to the commitlog replay and no errors, but some of the