1. I wrote the list a while back about less-than-great performance when
reading thousands of columns even on cache hits. Last night, I decided to
try to get to the bottom of why.
I tested this by setting the row cache capacity on a TimeUUIDType-sorted CF
to 10, filling up a single row wi
Not sure why the first paragraph turned in to a numbered bullet...
On Sun, May 2, 2010 at 11:00 AM, James Golick wrote:
>
>1. I wrote the list a while back about less-than-great performance when
>reading thousands of columns even on cache hits. Last night, I decided to
>
close to zero.
But, the performance doesn't seem to have noticeably improved. So, I'm not
sure what I'm missing here. Either my test methodology is broken or I
completely misread the profile.
On Sun, May 2, 2010 at 11:01 AM, James Golick wrote:
> Not sure why the first paragr
Got a ~50% improvement by making UUID comparison less heavy-weight.
https://issues.apache.org/jira/browse/CASSANDRA-1043
On Sun, May 2, 2010 at 7:49 PM, James Golick wrote:
> Just an update on this. I wrote a patch which attempts to solve this
> problem by keeping an index of columns th
When I first brought this cluster online, the storage-conf.xml file had a
few cache capacities set. Since then, we've completely changed how we use
cassandra's caching, and no longer use any of the caches I setup in the
original configuration.
I'm finding that cassandra doesn't want to keep my new
-ryan
>
> On Wed, May 12, 2010 at 8:16 AM, James Golick
> wrote:
> > When I first brought this cluster online, the storage-conf.xml file had a
> > few cache capacities set. Since then, we've completely changed how we use
> > cassandra's caching, and no longer
Picked up out of config, I mean.
On Wed, May 12, 2010 at 11:10 AM, James Golick wrote:
> Hmmm that's definitely what we're seeing. Although, we aren't seeing
> cache settings being picked up properly on a restart either.
>
>
> On Wed, May 12, 2010 at 8:42 AM,
We're seeing RAM usage continually climb until eventually, cassandra becomes
unresponsive.
The JVM isn't OOM'ing. It has only committed 14/24GB of memory. So, I am
assuming that the memory usage is related to mmap'd IO. Fair assumption?
I tried setting the IO mode to standard, but it seemed to be
JDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)
Here is the tpstats from a node that was restarted last night:
http://gist.github.com/415987
On Thu, May 27, 2010 at 8:51 AM, Robert Coli wrote:
> On 5/26/10 11:32 PM, James Golick wrote:
>
>> We're seeing RAM usage continu
d Cassandra 0.6.1 fixed a lot of memory problems I ran into.
>
> *From:* James Golick [mailto:jamesgol...@gmail.com]
> *Sent:* Thursday, May 27, 2010 9:05 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Continuously increasing RAM usage
>
> When I say unresponsive, I mean
I just experienced a compaction that brought a node to 100% of its IO
capacity and made its responses incredibly slow.
It wasn't enough to make the node actually appear as down, though, so it
slowed down the operation of the cluster considerably.
The CF being compacted contains a lot of relativel
Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks
to be about the same:
http://skitch.com/jamesgolick/df46f/munin-fetlife.com-cassandra0.fetlife.com-cassandra-memory
There's only one cache turned on, and it's a row cache, but the sizes of the
rows are identical and it's
I was getting a ton of these in my logs:
INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line
499) DigestMismatchException: Mismatch for key 142667/everything
(5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17)
The nodes were all getting really overloaded, p
repair` call is not blocking before 0.6.3 (unreleased): see
> https://issues.apache.org/jira/browse/CASSANDRA-1090
>
> -Original Message-
> From: "James Golick"
> Sent: Sunday, May 30, 2010 3:43am
> To: cassandra-u...@incubator.apache.org
> Subject: Inconsistency
Our cassandra client fails over if a node times out. Aside from actual
failure, repair and major compactions can make a node so slow that it
affects application performance.
One problem we've run in to is that a node in the midst of repair will still
have requests routed to it internally, even if
e-
> > From: "Benjamin Black"
> > Sent: Friday, June 18, 2010 12:32pm
> > To: user@cassandra.apache.org
> > Subject: Re: Failover and slow nodes
> >
> > Would be interesting to have a snitch that manipulated responses for
> > read nodes based on histo
I just increased my cluster from 2 to 4 nodes, and RF=2 to RF=3, using RP.
The tokens seem pretty even on the ring, but two of the nodes are far more
heavily loaded than the others. I understand that there are a variety of
possible reasons for this, but I'm wondering whether anybody has suggestion
On Sun, Jun 20, 2010 at 4:16 PM, James Golick wrote:
>
>> I just increased my cluster from 2 to 4 nodes, and RF=2 to RF=3, using RP.
>>
>> The tokens seem pretty even on the ring, but two of the nodes are far more
>> heavily loaded than the others. I understand that ther
ding to the formula i *
> (2**127 / 4) for i=1..4
>
>
> On Sun, Jun 20, 2010 at 4:31 PM, James Golick wrote:
>
>> I ran cleanup on all of them and the distribution looked roughly even
>> after that, but a couple of days later, it's looking pretty uneven.
>&g
As I alluded to in another post, we just moved from 2-4 nodes. Since then,
the cluster has been incredibly
The memory problems I've posted about before have gotten much worse and our
nodes are becoming incredibly slow/unusable every 24 hours or so. Basically,
the JVM reports that only 14GB is comm
Thanks for your thoughts. Answers below:
On Sun, Jun 20, 2010 at 2:21 PM, Peter Schuller wrote:
> > The memory problems I've posted about before have gotten much worse and
> our
> > nodes are becoming incredibly slow/unusable every 24 hours or so.
> Basically,
> > the JVM reports that only 14GB
n out of memory, and there's no way at all to
control it.
Can we consider changing the default?
On Sun, Jun 20, 2010 at 3:37 PM, James Golick wrote:
> Thanks for your thoughts. Answers below:
>
> On Sun, Jun 20, 2010 at 2:21 PM, Peter Schuller <
> peter.schul...@infidyne.com
I opened #1214 about this. I hope people will take a look and provide their
feedback.
https://issues.apache.org/jira/browse/CASSANDRA-1214
Thanks.
On Sun, Jun 20, 2010 at 3:58 PM, James Golick wrote:
> uh. wow. I just read up on all this again, and read the code, and I'm a
> littl
Just an update here. We're now entirely on standard IO mode, and everything
is stable and happy. There hasn't been much of a performance hit, if at all.
- James
On Mon, Jun 21, 2010 at 3:30 AM, Peter Schuller wrote:
> > How much of your physical RAM is dedicatd to the JVM?
> >
> > I forgot to s
On Mon, Jun 21, 2010 at 12:24 PM, Peter Schuller <
peter.schul...@infidyne.com> wrote:
> > Just an update here. We're now entirely on standard IO mode, and
> everything
> > is stable and happy. There hasn't been much of a performance hit, if at
> all.
>
> Cool. Just be aware that if my speculation
are much bigger? The keys will be distributed without
> considering the size of the data.
>
> The RP is random though, i do not think it tries to evenly distribute the
> keys. So some variance with a small number of nodes should be expected IMHO.
>
> Aaron
>
> On 21 Jun 2010, at 02:
RackUnaware, currently
On Tue, Jun 22, 2010 at 1:26 PM, Robert Coli wrote:
> On 6/22/10 10:07 AM, James Golick wrote:
>
>> This node's load is now growing at a ridiculous rate. It is at 105GB, with
>> the next most loaded node at 70.63GB.
>>
>> Given that RF=
d is growing
as quickly as that one.
- James
On Tue, Jun 22, 2010 at 1:31 PM, James Golick wrote:
> RackUnaware, currently
>
>
> On Tue, Jun 22, 2010 at 1:26 PM, Robert Coli wrote:
>
>> On 6/22/10 10:07 AM, James Golick wrote:
>>
>>> This node's load is no
Jun 22, 2010 at 5:27 PM, Jeremy Dunck wrote:
> On Tue, Jun 22, 2010 at 4:08 PM, James Golick
> wrote:
> > Turns out that this is due to a larger proportion of the wide rows in the
> > system being located on that node. I moved its token over a little to
> > compensate for
We had to take a node down for an upgrade last night. When we brought it
back online in the morning, it got slammed by HH data all day so badly that
it was compacting near constantly, and the pending compactions pool was
piling up. I shut most of the writes down to let things catch up, which they
m
4 nodes, RF=2, 1 node down.
How can I get an UnavailableException in that scenario?
- J.
Oops. I meant to say that I'm reading with CL.ONE.
J.
Sent from my iPhone.
On 2010-07-01, at 1:39 AM, Benjamin Black wrote:
> .QUORUM or .ALL (they are the same with RF=2).
>
> On Wed, Jun 30, 2010 at 10:22 PM, James Golick wrote:
>> 4 nodes, RF=2, 1 node down.
It's happening consistently when I take any node out of rotation.
On Thu, Jul 1, 2010 at 2:24 AM, Jonathan Ellis wrote:
> Presumably the failure detector generated a false positive for a
> second node temporarily
>
> On Wed, Jun 30, 2010 at 10:55 PM, James Golick
> wrote:
Hey,
I put our first cluster in to production (writing but not reading) a couple
of days ago. Right now, it's got two pretty sizeable nodes taking about 200
writes per second each and virtually no reads.
Eventually, though, (and this has happened twice), both nodes seem to start
timing out. If I
, Mar 27, 2010 at 12:29 PM, James Golick
> wrote:
> > Hey,
> > I put our first cluster in to production (writing but not reading) a
> couple
> > of days ago. Right now, it's got two pretty sizeable nodes taking about
> 200
> > writes per second each and vi
unlimited
On Sat, Mar 27, 2010 at 12:09 PM, Chris Goffinet wrote:
> what's the ulimit set to?
>
> -Chris
>
> On Mar 27, 2010, at 10:29 AM, James Golick wrote:
>
> Hey,
>
> I put our first cluster in to production (writing but not reading) a couple
> of days
Oops, I was doing ulimit. ulimit -n returns 1024.
On Sun, Mar 28, 2010 at 3:25 AM, Benoit Perroud wrote:
> ulimit -n returns you unlimited ?
>
>
> 2010/3/28 James Golick :
> > unlimited
> >
> > On Sat, Mar 27, 2010 at 12:09 PM, Chris Goffinet
> wrote:
&
We are starting to use cassandra to power our activity feed. The way we
organize our data is simple. "Event"s live in a CF called Events and are
keyed by a UUID. The timelines themselves live in a CF called Timelines,
which is keyed by user id (i.e. "1229") and contains a event uuids as column
name
"no change observed" or "faster but not
> fast enough?"
>
> On Tue, Mar 30, 2010 at 10:47 PM, James Golick
> wrote:
> > We are starting to use cassandra to power our activity feed. The way we
> > organize our data is simple. "Event"s live in a
Okay, so now my row cache hit rate jumps between 1.0, 99.5, 95.6, and NaN.
Seems like that stat is a little broken.
Still seeing around 35ms to multiget 20 rows.
- James
On Tue, Mar 30, 2010 at 9:22 PM, Ryan King wrote:
> On Tue, Mar 30, 2010 at 9:11 PM, James Golick
> wrote:
> &g
I'm testing on the live cluster, but most of the production reads are being
served by the cache. It's definitely the right CF.
On Wed, Mar 31, 2010 at 8:30 AM, Jonathan Ellis wrote:
> On Wed, Mar 31, 2010 at 12:01 AM, James Golick
> wrote:
> > Okay, so now my row cache
f the reads to replay asynchronously against Cassandra.
> >
> > On Wed, 2010-03-31 at 11:04 -0500, Jonathan Ellis wrote:
> >> Can you redirect some of the reads from memcache to cassandra? Sounds
> >> like the cache isn't getting warmed up.
> >>
>
Standard CF. 10 columns per row. Between about 800 bytes and 2k total per
row.
On Wed, Mar 31, 2010 at 3:06 PM, Chris Goffinet wrote:
> How many columns in each row?
>
> -Chris
>
> On Mar 31, 2010, at 2:54 PM, James Golick wrote:
>
> I just tried running the same multi_
Row cache hit rate: 0.007716049382716049
On Wed, Mar 31, 2010 at 4:15 PM, Jonathan Ellis wrote:
> What does the CFS mbean think read latencies are? Possibly something
> else is introducing latency after the read.
>
> On Wed, Mar 31, 2010 at 5:37 PM, James Golick
> wrote:
>
> times (read/write in ms) in your previous mails. Can you explain a little?
> Iike to work on it also.
>
> CD
>
>
> On Thu, Apr 1, 2010 at 4:15 AM, Jonathan Ellis wrote:
>
>> On Wed, Mar 31, 2010 at 6:21 PM, James Golick
>> wrote:
>> > Keyspa
Damnit!
On Thu, Apr 1, 2010 at 2:05 PM, Jeremy Dunck wrote:
> Or rackspace. ;)
>
> On Thu, Apr 1, 2010 at 2:49 PM, Joseph Stump wrote:
> > Taking our flamewar offline. :-D
> >
> > On Thu, Apr 1, 2010 at 1:36 PM, James Golick
> wrote:
> >> I don&#
ot of time was being spent in thrift. Turns
out, that's where pretty much all the time was going.
I just ran the same test using java (scala) and the load is taking around
2-4ms.
On Thu, Apr 1, 2010 at 4:37 PM, Peter Chang wrote:
> pwned.
>
>
> On Thu, Apr 1, 2010 at 2:09 PM, Jame
Yes.
J.
Sent from my iPhone.
On 2010-04-01, at 9:21 PM, Brandon Williams wrote:
On Thu, Apr 1, 2010 at 9:37 PM, James Golick
wrote:
Well, folks, I'm feeling a little stupid right now (adding to the
injury inflicted by one Mr. Stump :-P).
So, here's the story. The cache hit
Yes.
On Fri, Apr 2, 2010 at 10:35 AM, Ryan King wrote:
> On Thu, Apr 1, 2010 at 8:37 PM, James Golick
> wrote:
> > Well, folks, I'm feeling a little stupid right now (adding to the injury
> > inflicted by one Mr. Stump :-P).
> > So, here's the story. The c
Hi All,
I'm seeing about 35-50ms to read 1000 columns from a CF using
get_range_slices. The columns are TimeUUIDType with empty values.
The row cache is enabled and I'm running the query 500 times in a row, so I
can only assume the row is cached.
Is that about what's expected or am I doing somet
eason that
> oracle doesn't cache tuples from sequential scans -- blowing away
> 1000s of rows worth of recently used rows queried by key, for a swath
> of rows from the scan, is the wrong call more often than it is the
> right one.
>
> On Tue, Apr 13, 2010 at 1:00 PM, James G
That helped a little. But, it's still quite slow. Now, it's around 20-35ms
on average, sometimes as high as 70ms.
On Wed, Apr 14, 2010 at 8:50 AM, James Golick wrote:
> Right - that make sense. I'm only fetching one row. I'll give it a try with
> get_slice().
>
>
Just for the record, I am able to repeat this locally.
I'm seeing around 150ms to read 1000 columns from a row that has 3000 in it.
If I enable the rowcache, that goes down to about 90ms. According to my
profile, 90% of the time is being spent waiting for cassandra to respond, so
it's not thrift.
The values are empty. It's 3000 UUIDs.
On Wed, Apr 14, 2010 at 12:40 PM, Avinash Lakshman <
avinash.laksh...@gmail.com> wrote:
> How large are the values? How much data on disk?
>
> On Wednesday, April 14, 2010, James Golick wrote:
> > Just for the record, I am a
54 matches
Mail list logo
