Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread myreasoner
Thanks. I will upgrade to 0.8.1 then. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Read-latency-is-over-1-minute-on-a-column-family-with-400-000-rows-tp6639649p6642582.html Sent from the cassandra-u...@incubator.apache.org mailing list archive

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread Jonathan Ellis
You really need to upgrade from 0.8.0 to fix that. Restarting won't help, much (you'll get exactly one compaction against a given sstable, before it stops working again). On Mon, Aug 1, 2011 at 2:34 PM, myreasoner wrote: > All compaction related settings in the yaml were untouched.  The fingerpr

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread myreasoner
All compaction related settings in the yaml were untouched. The fingerprint column family has been populated three days ago and the cpu/disk usage were pretty low. I'd think Cassandra will silently start the compaction thread on my behalf and try to preserve the min/max thresholds, rather than wa

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread Jonathan Ellis
Why do you think it's ignoring it? In the output you pasted it said "I'm currently busy with a compaction and I have a backlog of 66 more to get to after that." On Mon, Aug 1, 2011 at 1:51 PM, myreasoner wrote: > It was set the min/4 max/32 > > Current compaction thresholds for MyKeyspace/Finger

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread myreasoner
It was set the min/4 max/32 Current compaction thresholds for MyKeyspace/Fingerprint: min = 4, max = 32 What could possibly cause cassandra to ignore these settings? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Read-latency-is-over-1-minute

Odp: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread Meler Wojciech
Upgrade at least to 0.8.1 as your version has broken compaction... Reagards Wojtek Meler - Reply message - Od: "myreasoner" Data: pon., sie 1, 2011 04:23 Temat: Read latency is over 1 minute on a column family with 400,000 rows Do: "cassandra-u...@incubator.apache.org

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-08-01 Thread aaron morton
Having 2056 live SSTables is very odd. Minor compaction should automatically reduce that number. What settings for min_compaction_threshold and max_compaction_threshold did you use when creating the CF ? You can check them with node tool getcompactionthreshold . The default is min 4 and max 32

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread Teijo Holzer
Looks like a broken node, just restart Cassandra on that node. Might want to wait for the compaction to finish on the other nodes. Also, don't forget to JMX gc() manually after the compaction has finished to delete the files on each node. On 01/08/11 16:29, myreasoner wrote: On the node that

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread myreasoner
On the node that the compaction returned almost immediately: *woot@n50:~$ /opt/cassandra/bin/nodetool -h localhost compactionstats pending tasks: 66* However, messages shown on other nodes are: compaction type: Major keyspace: MyKeyspace column family: Fingerprint bytes compacted: 25505066421 byt

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread Mina Naguib
Did you run that verbatim ? Or you appropriately substituted "keyspace" and "columnfamily1" ? Also, anything in cassandra's log file (system.log) ? Compacting 150Gb over 2057 SSTables should take a reasonable bit of time... On 2011-07-31, at 11:47 PM, myreasoner wrote: > Thanks. > > I did

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread Teijo Holzer
Hi, try nodetool -h localhost compact check progress with nodetool -h localhost compactionstats and check system.log Cheers, T. On 01/08/11 15:47, myreasoner wrote: Thanks. I did *./nodetool -h localhost compact keyspace columnfamily1 *. But it came back really quick and the cfs

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread myreasoner
Thanks. I did *./nodetool -h localhost compact keyspace columnfamily1 *. But it came back really quick and the cfstats doesn't seem change much. After compaction: Column Family: Fingerprint SSTable count: 2057 Space used (live): 164351343468

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread Teijo Holzer
Compaction is machine-local, you need to run it on every node. Do it as a rolling compaction (or in parallel if you can take the performance hit). Cheers, T. On 01/08/11 15:31, myreasoner wrote: If I do ./nodetool -h localhost compact keyspace columnfamily1 it will go out and compac

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread myreasoner
If I do ./nodetool -h localhost compact keyspace columnfamily1 it will go out and compact coumnfamily1 on all the nodes not just the localhost, correct? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Read-latency-is-over-1-minute-on-a-column-fa

Re: Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread Teijo Holzer
Hi, try running a major compaction via nodetool on this Column family. The number of SSTables seems quite large. Considering the space used, this might take a few hours and might also impact performance. Cheers, T. On 01/08/11 14:23, myreasoner wrote: Hi, my read latency is reall

Read latency is over 1 minute on a column family with 400,000 rows

2011-07-31 Thread myreasoner
Hi, my read latency is really horrible and I can't figure out what went wrong. I'm running cassandra 0.8.0 on a 5 machine cluster. The Fingerprint ColumnFamily has 400,000 rows, each row has about 4,000 Super columns, and each super column has 1 to 4 columns. One row looks like: RowKey: 00c26f