Re: Read Latency Degradation

2010-12-17 Thread Daniel Doubleday
On Dec 16, 2010, at 11:35 PM, Wayne wrote: > I have read that read latency goes up with the total data size, but to what > degree should we expect a degradation in performance? What is the "normal" > read latency range if there is such a thing for a small slice of scol/cols? > Can we really pu

Cassandra Monitoring

2010-12-17 Thread Daniel Doubleday
Hi all just wanted to share a simple way we use to monitor cassandra internals with zabbix. We use a minimal http server which reads jmx and shows returns them in a property form. Thats read by zabbix every 30secs. That's started together with cassandra: https://gist.github.com/744761 Output

Re: Read Latency Degradation

2010-12-17 Thread Wayne
We have been testing Cassandra for 6+ months and now have 10TB in 10 nodes with rf=3. It is 100% real data generated by real code in an almost production level mode. We have gotten past all our stability issues, java/cmf issues, etc. etc. now to find the one thing we "assumed" may not be true. Our

Re: Read Latency Degradation

2010-12-17 Thread Edward Capriolo
On Fri, Dec 17, 2010 at 8:21 AM, Wayne wrote: > We have been testing Cassandra for 6+ months and now have 10TB in 10 nodes > with rf=3. It is 100% real data generated by real code in an almost > production level mode. We have gotten past all our stability issues, > java/cmf issues, etc. etc. now t

Re: Cassandra Monitoring

2010-12-17 Thread Edward Capriolo
On Fri, Dec 17, 2010 at 5:48 AM, Daniel Doubleday wrote: > Hi all > just wanted to share a simple way we use to monitor cassandra internals with > zabbix. > We use a minimal http server which reads jmx and shows returns them in a > property form. Thats read by zabbix every 30secs. > That's started

Re: Read Latency Degradation

2010-12-17 Thread Wayne
Below are some answers to your questions. We have wide rows (what we like about Cassandra) and I wonder if that plays into this? We have been loading 1 keyspace in our cluster heavily in the last week so it is behind in compaction for that keyspace. I am not even looking at those read latency times

Single Column vs Multiple Columns

2010-12-17 Thread Ericson, Doug
I have a data model where each read and each write will read and write all data for a given key. Thus, does it make more sense to separate the data into multiple columns, or does it make more sense to store all data in a single column? My understanding is that breaking data into multiple colum

Re: Dynamic Snitch / Read Path Questions

2010-12-17 Thread Daniel Doubleday
> the purpose of your thread is: How far are you away from being I/O > bound (say in terms of % utilization - last column of iostat -x 1 - > assuming you don't have a massive RAID underneath the block device) No my cheap boss didn't want to by me a stack of these http://www.ocztechnology.com/prod

Re: Single Column vs Multiple Columns

2010-12-17 Thread Jonathan Ellis
The performance difference is negligible and the other drawbacks are significant, e.g., losing the ability to create indexes on a column. On Fri, Dec 17, 2010 at 11:49 AM, Ericson, Doug wrote: > I have a data model where each read and each write will read and write > all data for a given key. T

Re: Read Latency Degradation

2010-12-17 Thread Daniel Doubleday
> How much ram is dedicated to cassandra? 12gb heap (probably too high?) > What is the hit rate of caches? high, 90%+ If your heap allows it I would definitely try to give more ram for fs cache. Your not using row cache so I don't see what cassandra would gain from so much memory. A question ab

Re: Read Latency Degradation

2010-12-17 Thread Edward Capriolo
On Fri, Dec 17, 2010 at 12:26 PM, Daniel Doubleday wrote: > How much ram is dedicated to cassandra? 12gb heap (probably too high?) > What is the hit rate of caches? high, 90%+ > > If your heap allows it I would definitely try to give more ram for fs cache. > Your not using row cache so I don't see

Re: Cassandra Monitoring

2010-12-17 Thread Dan Kuebrich
Is anyone using cassandra with monit? All I have is this embarrassing bit of monit config: check process cassandra with pidfile /var/run/cassandra.pid start program = "/etc/init.d/cassandra start" with timeout 60 seconds stop program = "/etc/init.d/cassandra stop" if failed port 9160 type

Re: [SOLVED] Very high memory utilization (not caused by mmap on sstables)

2010-12-17 Thread Zhu Han
Seems like the problem there after I upgrade to "OpenJDK Runtime Environment (IcedTea6 1.9.2)". So it is not related to the bug I reported two days ago. Can somebody else share some info with us? What's the java environment you used? Is it stable for long-lived cassandra instances? best regards,