Yes, I'm reading from a single partition.
-Original Message-
From: Hiller, Dean [mailto:dean.hil...@nrel.gov]
Sent: 21 March 2013 01:38
To: user@cassandra.apache.org
Subject: Re: Unable to fetch large amount of rows
Is your use case reading from a single partition? If so, you may want t
It's clear you are out of memory. How big is your data size?
heap of 1867M is kind of small. According to the discussion on this list, it's
advisable to have m1.xlarge.
Attached please find the related thread.
-Wei
- Original Message -
From: "Dane Miller"
To: user@cassandra.apache.o
Thanks Aaron, appreciate the advice.
On Tue, Mar 19, 2013 at 3:14 AM, aaron morton wrote:
> I think I understand what it means for
> application-level data, but the part I'm not entirely sure about is
> what it could mean for Cassandra internals.
>
> Internally it means the write will not be ret
After having just solved one repair problem, I immediately hit
another. Again, much appreciation for suggestions...
I'm having problems repairing a CF, and the failure consistenly brings
down 2 of the 6 nodes in the cluster. I'm running "repair -pr" on a
single CF on node2, the repair starts str
Hello,
Also have a look at
http://www.datastax.com/docs/1.2/install/recommended_settings
On 21 Mar 2013 00:06, "S C" wrote:
> Apparently max user process was set very low on the machine.
>
> How to check?
> ulimit -u
>
> Set it to unlimited /etc/security/limits.conf
>
> * soft nprocs unlimited
Apparently max user process was set very low on the machine.
How to check?ulimit -u
Set it to unlimited /etc/security/limits.conf
* soft nprocs unlimited* hard nprocs unlimited
From: as...@outlook.com
To: user@cassandra.apache.org
Subject: RE: java.lang.OutOfMemoryError: unable to create new n
On Wed, Mar 20, 2013 at 3:18 PM, Tycen Stafford wrote:
> I don’t think that’s correct for a mult-dc ring, but you’ll want to hear a
> final answer from someone more authoritative. I could easily be wrong.
> Try using the built in token generating tool (token-generator) – I don’t
> seem to have i
I don't think that's correct for a mult-dc ring, but you'll want to hear a
final answer from someone more authoritative. I could easily be wrong. Try
using the built in token generating tool (token-generator) - I don't seem to
have it on my hosts (1.1.6 also) so I can't confirm. I used the to
Is your use case reading from a single partition? If so, you may want to
switch to something like playorm which does virtual partitions so you still get
the performance of multiple disks when reading from a single partition. My
understanding is a single cassandra partition exists on a single n
On Tue, Mar 19, 2013 at 1:01 AM, aaron morton wrote:
> It's easier to understand what's happening if you provide the full error
> message.
>
> It looks like out of order data in the files, nodetool scrub can fix that
> error.
>
> Try repairing a single CF at a time so you can work out which one is
[ Jeffrey Fass ]
> Unsubscribe
Unlikely.
--
Eric Evans
eev...@sym-link.com
Yes – just when the node goes down. I’ll check 4705.
Thanks
From: Derek Williams [mailto:de...@fyrie.net]
Sent: Wednesday, March 20, 2013 12:14 PM
To: user@cassandra.apache.org
Subject: Re: Question regarding multi datacenter and LOCAL_QUORUM
Are those timeouts happening right when the node goe
Are those timeouts happening right when the node goes down? If so it might
be https://issues.apache.org/jira/browse/CASSANDRA-4705
I don't think that issue applies if the node has been down long enough to
be marked as down though.
On Wed, Mar 20, 2013 at 12:53 PM, Dwight Smith
wrote:
> Further
Further information, in AZ1, when 143, 145, and 146 are up, all goes well. But
when, say 143, fails, the client receives a TIMEOUT failure – even though 145
and 146 are up.
From: Derek Williams [mailto:de...@fyrie.net]
Sent: Wednesday, March 20, 2013 11:50 AM
To: user@cassandra.apache.org
Subjec
Yes that is correct – the log is from 146. The client connects to nodes in AZ1.
From: Derek Williams [mailto:de...@fyrie.net]
Sent: Wednesday, March 20, 2013 11:50 AM
To: user@cassandra.apache.org
Subject: Re: Question regarding multi datacenter and LOCAL_QUORUM
I'm think I need help with pointi
I'm think I need help with pointing out what the problem is. The log you
posted only contains references to 143, 145, and 146, which all appear to
be in the same datacenter as 146?
On Wed, Mar 20, 2013 at 11:29 AM, Dwight Smith
wrote:
> Hi
>
> ** **
>
> I have 2 data centers – with 3 nodes
Hmm - the ring output follows, the tokens in AZ2 are offset by 100:
Address DC RackStatus State Load
Effective-Ownership Token
113427455640312821154458202477256070585
Actually the tokens in AZ2 are not correct.
I'll get those corrected - thanks for the pointer.
From: Tycen Stafford [mailto:tstaff...@medio.com]
Sent: Wednesday, March 20, 2013 11:25 AM
To: user@cassandra.apache.org
Subject: RE: Question regarding multi datacenter and LOCAL_QUORUM
Okay - that lo
Okay - that looks alternated to me. I'm assuming that 147, 148 and 149 are
this then:
28356863910078205288614550619314017621
85070591730234615865843651857942052864
141784319550391026443072753096570088106
I'm out of ideas - sorry I couldn't help more.
-Tycen
From: Dwight Smith [mailto:dwight.s
>From the yamls
.143
initial_token: 0
.145
initial_token: 56713727820156410577229101238628035242
.146
initial_token: 113427455640312821154458202477256070485
From: Tycen Stafford [mailto:tstaff...@medio.com]
Sent: Wednesday, March 20, 2013 10:43 AM
To: user@cassandra.apache.org
Subject: RE:
Slight correction - replication factor is 3.
Will obtain nodetool ring info to verify tokens.
From: Dwight Smith [mailto:dwight.sm...@genesyslab.com]
Sent: Wednesday, March 20, 2013 10:30 AM
To: user@cassandra.apache.org
Subject: Question regarding multi datacenter and LOCAL_QUORUM
Hi
I have 2
Did you alternate your tokens? I may be off base - but if not then that's why
you might be seeing cross-dc request.
-Tycen
From: Dwight Smith [mailto:dwight.sm...@genesyslab.com]
Sent: Wednesday, March 20, 2013 10:30 AM
To: user@cassandra.apache.org
Subject: Question regarding multi datacenter
Hi aaron.
I added pagination, and things seem to have started performing much better.
With 1000 page size, now able to fetch 500K records in 25-30 seconds.
However, I'd like to point you to some interesting observations:
+ Did run cfhistograms, the results are interesting (Note: row cache i
Hi
I have 2 data centers - with 3 nodes in each DC - version 1.1.6 - replication
factor 2 - topology properties:
# Cassandra Node IP=Data Center:Rack
xx.yy.zz.143=AZ1:RAC1
xx.yy.zz.145=AZ1:RAC1
xx.yy.zz.146=AZ1:RAC1
xx.yy.zz.147=AZ2:RAC2
xx.yy.zz.148=AZ2:RAC2
xx.yy.zz.149=AZ2:RAC2
Using LOCAL
Hi Aaron,
I did mean 1000 columns. But I see your point.
The current CF schema has user_id as the row key and unnamed column
order_num = order info as the col-val pair. The plan is to add named
columns order_attr1, order_attr2... order_attr18. When I query for user_id
= "user1" and order_attr1 = 1
I am using LCS so bloom filter fp default for 1.2.2 is 0.1 so my
bloomfilter size is 1.27G RAM(nodetool cfstats)1.7 billion rows each
node.
My cfstats for this CF is attached(Since cut and paste screwed up the
formatting). During testing in QA, we were not sure if index_interval
change was wo
I am curious, thanks. ( I am in the same situation, big nodes choking
under 300-400G data load, 500mil keys )
How does your "cfhistograms Keyspace CF" output look like? How many
sstable reads ?
What is your bloom filter fp chance ?
Regards,
Andras
On 20/03/13 13:54, "Hiller, Dean" wrote:
>Oh,
Oh, and to give you an idea of memory savings, we had a node at 10G RAM
usage...we had upped a few nodes to 16G from 8G as we don't have our new
nodes ready yet(we know we should be at 8G but we would have a dead
cluster if we did that).
On startup, the initial RAM is around 6-8G. Startup with
in
Yup, we are rolling it slowly. In production, we have 2 nodes out of 6
switched already and so far have no website degredation at all. We have
narrow rows as well and as the comment says in the props file, "if you
have narrow rows going to 512 sometimes has no impact on performance" and
in our ca
2. Upping index_interval from 128 to 512 (this seemed to reduce our memory
usage significantly!!!)
I'd be very careful with that as a one-stop improvement solution for two
reasons AFAIK
1) you have to rebuild stables ( not an issue if you are evaluating, doing
test writes.. Etc, not so much in pr
Also, look at the cassandra logs. I bet you see the typicalŠblah blah is
at 0.85, doing memory cleanup which is not exactly GC but cassandra memory
managementŠ..and of course, you have GC on top of that.
If you need to get your memory down, there are multiple ways
1. Switching size tiered compact
I've added the node with a different IP address and after disabling the
firewall data is being streamed from the existing nodes to the wiped node.
I'll do a cleanup, followed by remove node once it's done.
I've also added the new node to the existing nodes'
cassandra-topology.properties file and r
What is in your Cassandra log right before and after that freeze?
-Tupshin
On Mar 20, 2013 8:06 AM, "Joel Samuelsson"
wrote:
> Hello,
>
> I've been trying to load test a one node cassandra cluster. When I add
> lots of data, the Cassandra node freezes for 4-5 minutes during which
> neither reads
I'd say GC. Please fill in form CASS-FREEZE-001 below and get back to us
:-) ( sorry )
How big is your JVM heap ? How many CPUs ?
Garbage collection taking long ? ( look for log lines from GCInspector)
Running out of heap ? ( "heap is .. full" log lines )
Any tasks backing up / being dropped ? (
Makes senseŠ.thanks!!! I will note that for our future replacements(we
still have to test a full replacement out).
Dean
On 3/19/13 11:06 AM, "Wei Zhu" wrote:
>Hi Dean,
>If you are not using VNode and try to replace the node, use the new token
>as old token -1, not +1. The reason is that, the as
Hey,
try this blog post by Datastax, it provides a good explanation of CQL3
abstractions.
http://www.datastax.com/dev/blog/cql3-for-cassandra-experts
regards,
ondrej cernos
On Wed, Mar 20, 2013 at 8:50 AM, Thierry Templier wrote:
> Hello,
>
> I have a question regarding composite columns. Wha
Hi, morton:
Thank you for your reply.
I have been using OPP for a long time. In my column family, I use
the fingerprint string as the key. So OPP should be OK.
Now the cassandra can be restarted, but the cpu usage continue to be
very high.
On 03/20/2013 06:05 PM, aaron morton wrote:
Thanks, Aaron.
I re-enabled hinted handoff and noted the following
* no host is marked down in nodetool ring
* No host is logged as down or dead in logs
* No "started hinted handoff for.." is logged
* The hinted handoff manager Mbean lists pending hints to .. (drumroll) 3
non-exi
> ERROR 16:32:16,265 Exception in thread Thread[GossipStage:1,5,main]
> java.lang.RuntimeException: The provided key was not UTF8 encoded.
> at
> org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:170)
> at
> org.apache.cassandra.dht.OrderPreservi
On 03/20/2013 11:34 AM, 杨辉强 wrote:
Hi,
Every time I restart the cassandra server, the cpu usage continue
to be very high(98%) for days. But I have no reading or writing to
this server.
I have tried the follow cmd:
date; date `date +"%m%d%H%M%C%y.%S"`; date;
It doesn't work.
The tail of s
> (probably
> worth mentioning that we have a lot more cf's than the 4 in that log
> excerpt):
Yeah, that and running on multiple SSD's per node :)
With SSD's I would look at increasing the memtable_flush_writers and leaving
the memtable_flush_queue_size in place. When you see IO util too high th
> The query returns fine if I request for lesser number of entries (takes 15
> seconds for returning 20K records).
That feels a little slow, but it depends on the data model, the query type and
the server and a bunch of other things.
> However, as I increase the limit on
> number of entries, th
> On average, this involves abandoning 20k mutations, for a total of 14Mb of
> data.
That's too many mutations to be practical. Each row mutation becomes a single
task in the mutation thread pool. When you send so many risk flooding the
mutation thread pool and starving other requests. Each node
On Mar 20, 2013, at 5:21 AM, aaron morton wrote:
> By design. There may be a plan to change in the future, I'm not aware of one
> though.
bad news. If someone else has more information about that, don't hesitate !
Do you know how hard it would be to change this behaviour ? to not skip tables
w
Yes.
The block size is specified as part of the compression options for the CF /
Table.
Cheers
-
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 20/03/2013, at 5:31 AM, Drew Kutcharian wrote:
> Thanks Sylvain. So C* compre
Hello,
I have a question regarding composite columns. What is the way to create
and use them basing on CQL3? Are there some documentations regarding
this feature? Is it supported with both versions 1.1 and 1.2 of Cassandra?
Thanks very much for your help!
Thierry
46 matches
Mail list logo
