Fwd: Loading SSTables failing via Cassandra SSTableLoader on mulit-node cluster.

2012-12-04 Thread Pradeep Kumar Mantha
Hi! I am trying to load sstables generated onto a running multi-node Cassandra cluster. But I see problems only with multi-cluster and single node works fine. Cassandra version used is 1.1.2 . The cassandra cluster seems to be active. -bash-3.2$ nodetool -host 129.56.57.45 -p 7199 ring Address

Re: strange row cache behavior

2012-12-04 Thread Yiming Sun
Got it. Thanks again, Aaron. -- Y. On Tue, Dec 4, 2012 at 3:07 PM, aaron morton wrote: > Does this mean we should not enable row caches until we are absolutely > sure about what's hot (I think there is a reason why row caches are > disabled by default) ? > > Yes and Yes. > Row cache takes mem

Re: Diagnosing memory issues

2012-12-04 Thread aaron morton
For background, a discussion on estimating working set http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html . You can also just look at the size of tenured heap after a CMS. Are you seeing lots of ParNew or CMS ? GC activity is a result of configuration *and* workload. Look in y

Re: Data backup and restore

2012-12-04 Thread aaron morton
I wrote a script to sym link the snapshots together the other day https://github.com/amorton/cass_snapshot_link I've not really used it in anger yet. That is to say I wrote it for fun and it worked on my mac book. If you use it let me know if it works. Cheers A - Aaron Morton

Re: strange row cache behavior

2012-12-04 Thread aaron morton
> Does this mean we should not enable row caches until we are absolutely sure > about what's hot (I think there is a reason why row caches are disabled by > default) ? Yes and Yes. Row cache takes memory and CPU, unless you know you are getting a benefit from it leave it off. The key cache and

Re: Nodes not synced

2012-12-04 Thread aaron morton
> Node B stil not get any update/data from Node A. Do we need to execute any > command to sync both nodes? Are you seeing the MutationStage completed tasks count change in nodetool tpstats ? > # /opt/apache-cassandra-1.1.4/bin/nodetool -h localhost ring > Address DC Rack

Re: Help on MMap of SSTables

2012-12-04 Thread aaron morton
> Will MMapping data files be detrimental for reads, in this case? No. > In general, when should we opt for MMap data files and what are the factors > that need special attention when enabling the same? mmapping is the default, so I would say use it until you have a reason not to. mmapping wil

Re: Having issues with node token collisions when starting up Cassandra nodes in a cluster on VMWare.

2012-12-04 Thread aaron morton
> We do not want to manually set the value for initial_token for each node > (kind of defeats the goal of being dynamic..) You *really* do want to do this. Adding without setting a token will result in an unbalanced cluster. The 1.1 distro includes a token generator in tools/bin/token-generator

[BETA RELEASE] Apache Cassandra 1.2.0-beta3 released

2012-12-04 Thread Sylvain Lebresne
The Cassandra team is pleased to announce the release of the third beta for the future Apache Cassandra 1.2.0. Let me first stress that this is beta software and as such is *not* ready for production use. This release is still beta and as such may contain bugs. Any help testing this beta would be

Diagnosing memory issues

2012-12-04 Thread Mike
Hello, Our Cassandra cluster has, relatively recently, started experiencing memory pressure that I am in the midsts of diagnosing. Our system has uneven levels of traffic, relatively light during the day, but extremely heavy during some overnight processing. We have started getting a message

Re: Row caching + Wide row column family == almost crashed?

2012-12-04 Thread Mike
Thanks for all the responses! On 12/3/2012 6:55 PM, Bill de hÓra wrote: A Cassandra JVM will generally not function well with with caches and wide rows. Probably the most important thing to understand is Ed's point, that the row cache caches the entire row, not just the slice that was read out

Re: Data backup and restore

2012-12-04 Thread Tomas Nunez
Hi I think he was talking about the "fragmentation" of the snapshot. In cassandra 1.0.X all ColumnFamilies are in the same directory, but in cassandra 1.1.X each ColumnFamily is in its own directory, and snapshots of each ColumnFamily are inside this directory. 1.0.X Snapshot directory: /cassandr

Re: Row caching + Wide row column family == almost crashed?

2012-12-04 Thread Yiming Sun
Yup, got it. Thanks Aaron. On Tue, Dec 4, 2012 at 4:47 AM, aaron morton wrote: > I responded on your other thread. > > Cheers > >- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 4/12/2012, at 5:31 PM, Yimin

Re: strange row cache behavior

2012-12-04 Thread Yiming Sun
Hi Aaron, Thank you,and your explanation makes sense. At the time, I thought having 1GB of row cache on each node was plenty enough, because there was an aggregated 6GB cache, but you are right, with each row in 10's of MBs, some of the nodes can go into a constant load and evict cycle and would

Nodes not synced

2012-12-04 Thread Adeel Akbar
Hi, I have setup 2 nodes cluster with Replica factor 2. I have restored snapshot of another cluster on Node A and restarted cassandra process. Node B stil not get any update/data from Node A. Do we need to execute any command to sync both nodes? # /opt/apache-cassandra-1.1.4/bin/nodetool -h

Re: Row caching + Wide row column family == almost crashed?

2012-12-04 Thread aaron morton
I responded on your other thread. Cheers - Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 4/12/2012, at 5:31 PM, Yiming Sun wrote: > I ran into a different problem with Row cache recently, sent a message to the > list, but

Re: strange row cache behavior

2012-12-04 Thread aaron morton
> Row Cache: size 1072651974 (bytes), capacity 1073741824 (bytes), 0 > hits, 2576 requests, NaN recent hit rate, 0 save period in seconds So the cache is pretty much full, there is only 1 MB free. There were 2,576 read requests that tried to get a row from the cache. Zero of those had

Re: Data not replicating to all datacenters

2012-12-04 Thread Owen Davies
In our main application we are using local quorum to read. I realise the default for cli is one, the point is that we want all our data on all the servers, hence specifying [dc1:3, dc2:3] as the replication strategy. After a couple of days we would expect it to have replicated. As I said, we will

Re: Data not replicating to all datacenters

2012-12-04 Thread aaron morton
> A few days after writing the data, we tried this on cassandra-cli The default consistency level in the CLI is ONE, did you change it to LOCAL QUOURM ? (I'm assuming your example is for two reads from the same CF) It looks like the first read was done at a lower CL, and the value returned is