> During tests I've done mass mutations using an import of data. Using > CL.QUORUM the import takes around 3 times longer that using CL.ONE on a > cluster with 3 nodes.
Is the test sequential or multi-threaded? A factor 3 performance difference seems like a lot in terms of total throughput; but it's easier to buy if it is due to higher latency resulting from use of QUORUM. In other words: What I'm saying is that assuming bottlenecks are are being saturated (sufficient concurrency, etc) I would expect that CL.ONE and CL.QUORUM be roughly similar in their *throughput* (writes per second). However for a single sequential client, or fewer concurrent clients than necessary, it makes sense to see a significantly worse throughput as a result of higher latency, as low concurrency means you're not saturating the system. If you're really seeing a 3x difference with concurrency cranked up that might be worth investigating IMO. > I think this is an expected behavior though, in the CL.ONE case more or less > all reads / writes will be done locally whereas using QUORUM all operations > are done locally and remotely. It's expected that latency on an individual request will be higher because (1) you are doing another level of network round-trips, and (2) there is always variation in latency and having to wait for more than one means a higher probability of having to wait a bit longer just for that reason. If you use case is that you're e.g. doing a few writes, a few reads, and returning a page - higher latencies on these reads and writes are expected to translate to slower page (or whatever) load times. But - I still think that the overall throughput, as long as concurrency is high enough, should be roughly similar for ONE and QUORUOM *for writes*. For reads, CL.ONE is definitely expected to be faster even in terms of throughput, since you are in fact doing 1/RF times the amount of reads. > Giving the locally consistent ONE approach we avoid consistency problems > when the local node is slow, and the import time is more or less the same as > with CL.ONE. We do however introduce potentially slower reads / writes when > the response time of local node it slower that any of the remote nodes, but > because cassandra is embedded in SiteVision the entire jvm will probably > react slowly at this stage - a fast read from a remote node will not help > much. Makes sense. > An other problem with using QUORUM is that it does not scale well in the > case of big production environments. There are cases when a customer > temporarily uses up to 10 nodes where QUORUM would mean reading / writing to > 6 nodes. So writes would still be expensive, but CL.ONE should get you 1/6 the cost (CPU/disk) related to QUORUM in this case. I do want to point out though that it is somewhat of a fundamental limitation as long as you tie RF to cluster size, and will become a scalability problem even if you use CL.ONE due to reads (and for that matter due to repair becoming more expensive). If I may ask, what are you primarly targeting here - is it about shaving a single millisecond or two off of the average request, or is it the number of requests per second in total for a single "task" (page load, etc), or is it overall throughput of the system (total amount of h/w required)? I'm mostly trying to figure out whether you are having some problem with latency or throughput that is not just a result of an extra few milli seconds of an additional round-trip time. If there is some other solution to the problem, the patching you describe, and limiting yourself in scaling by fixing RF=number of nodes, seems like kind of a high price to pay. Obviously I don't know fully your situation so take that with a grain of salt. Just my knee-jerk reaction from what I've read so far. -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
