Debugging high tail read latencies (internal timeout)

2016-07-06 Thread Nimi Wariboko Jr
Hi, I've begun experiencing very high tail latencies across my clusters. While Cassandra's internal metrics report <1ms read latencies, measuring responses from within the driver in my applications (roundtrips of query/execute frames), have 90% round trip times of up to a second for very basic que

Re: Cassandra 3.6: Compaction/Write Requests fail with org.xerial.snappy.SnappyError: [NOT_A_DIRECT_BUFFER] destination is not a direct buffer

2016-06-07 Thread Nimi Wariboko Jr
I also found CASSANDRA-9548, "Doing a bit more digging into this, the SnappyCompressor does only support OFF_HEAP buffers.", however I already am using offheap_buffers. I also got this error with offheap_objects. On Tue, Jun 7, 2016 at 4:58 PM, Nimi Wariboko Jr wrote: > Has any

Cassandra 3.6: Compaction/Write Requests fail with org.xerial.snappy.SnappyError: [NOT_A_DIRECT_BUFFER] destination is not a direct buffer

2016-06-07 Thread Nimi Wariboko Jr
Has anyone seen this issue since upgrading to 3.6? I don't have a good idea what is going on yet, as my logs have just been spewing the following error. WARN [SharedPool-Worker-174] 2016-06-07 16:54:07,651 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPoo

IF NOT EXISTS with multiple static columns confusion

2016-02-23 Thread Nimi Wariboko Jr
I have a table with 2 static columns, and I write to either one of them, if I then write to the other one using IF NOT EXISTS, it fails even though it has never been written too before. Is it the case that all static columns share the same "written too" marker? Given a table like so: CREATE TABLE

NullPointerException when trying to compact under 3.2

2016-01-25 Thread Nimi Wariboko Jr
Hi, I recently upgraded from 2.1.12 to 3.2, and one issue I'm having is I can no longer read certain rows from a table. A simple SELECT * FROM `table` times out, only when the bad partition keys are reached. Trying to query the affected partition keys directly also causes a timeout. I think the S

Performance tuning for read throughput w/ token scans?

2015-06-19 Thread Nimi Wariboko Jr
[Cassandra 2.1.5] I'm trying to explore my options for increasing read throughput with token scans (SELECT * FROM x WHERE token(y) > L AND token(y) < L). So far I've started by reading an entire virtual token range from a single node. Currently on a single query I can read about 57,286.03 rows/s

Re: Rebalancing vnodes cluster

2013-09-19 Thread Nimi Wariboko Jr
of this thread) is growing faster than the others (which is leading to hot spotting as well). I guess this has to do with the unfinished shuffle? Are there any remedies for this? On Thursday, September 19, 2013 at 9:50 AM, Robert Coli wrote: > On Wed, Sep 18, 2013 at 4:26 PM, Nimi Waribo

Re: Rebalancing vnodes cluster

2013-09-18 Thread Nimi Wariboko Jr
> > > On Wed, Sep 18, 2013 at 6:26 PM, Nimi Wariboko Jr (mailto:nimiwaribo...@gmail.com)> wrote: > > Hi, > > > > When I started with cassandra I had originally set it up to use tokens. I > > then migrated to vnodes (using shuffle), but my cluster isn't balanc

Rebalancing vnodes cluster

2013-09-18 Thread Nimi Wariboko Jr
Hi, When I started with cassandra I had originally set it up to use tokens. I then migrated to vnodes (using shuffle), but my cluster isn't balanced (http://imgur.com/73eNhJ3). What steps can I take to balance my cluster? Thanks, Nimi

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Nimi Wariboko Jr
whatever tokens they were > before the switches, since the old data is still there. > > Either way both recovery options are long,painful, and a good amount of > manual steps. I would not want to do either one. > > > On Mon, Jun 10, 2013 at 12:39 PM, Nimi Wariboko Jr (mailto:n

Re: Data Loss/Missing With Cassandra

2013-06-10 Thread Nimi Wariboko Jr
How can I recover that data? Can I assume they are still in the sstables? Would doing a sstable2json then reading and reinserting be an optimal solution? On Monday, June 10, 2013 at 9:18 AM, Tyler Hobbs wrote: > > On Sun, Jun 9, 2013 at 3:19 PM, Nimi Wariboko Jr (mailto:nimi

Re: Data Loss/Missing With Cassandra

2013-06-09 Thread Nimi Wariboko Jr
consistent (though I do > not know why repair would not do it). > > > > > > On Sat, Jun 8, 2013 at 8:56 PM, Nimi Wariboko Jr (mailto:nimiwaribo...@gmail.com)> wrote: > > Hi, > > > > We are seeing an issue where data that was written to the cluster

Data Loss/Missing With Cassandra

2013-06-08 Thread Nimi Wariboko Jr
Hi, We are seeing an issue where data that was written to the cluster is no longer accessible after trying to expand the size of the cluster. I will try and provide as much information as possible, I am just starting at with Cassandra and I'm not entirely sure what data is relevant. All Cassa