Re: High disk I/O during reads

2013-03-23 Thread igor
You can try to disable readahead on cassandra data disk. Jon Scarborough написал(а): >Checked tpstats, there are very few dropped messages. > >Checked histograms. Mostly nothing surprising. The vast majority of >rows >are small, and most reads only access one or two SSTables. > >What I did disco

Many to one type of replication.

2013-03-23 Thread Francois Richard
Hi, We currently run our Cassandra deployment with multiple independent clusters. The clusters are totally self contain in terms of redundancy and independent from each others. We have a "sharding "layer higher in our stack to dispatch the requests to the right application stack and this stack c

Cassandra - conflict resolution for column updates with identical timestamp

2013-03-23 Thread dong.yajun
Hello, I would like to know which write wins in case of two updates with the same client timestamp in Cassandra. Initial data: KeyA: { col1:"val AA", col2:"val BB", col3:"val CC"} Client 1 sends update: KeyA: { col1:"val C1", col2:"val B1"} on Sx Client 2 sends update: KeyA: { col1:"val C2", co

Re: Lots of Deleted Rows Came back after upgrade 1.1.6 to 1.1.10

2013-03-23 Thread aaron morton
> Beside the joke, would hinted handoff really have any role in this issue? I could imagine a scenario where a hint was replayed to a replica after all replicas had purged their tombstones. That seems like a long shot, it would need one node to be down for the write and all up for the delete and

Re: Question regarding multi datacenter and LOCAL_QUORUM

2013-03-23 Thread aaron morton
> and read_repair_chance = 0.1 > and dclocal_read_repair_chance = 0.0 So now I am a little confused. With LOCAL_QUOURM I expect the block for to be 2. And with global read repair active (read_reapir_chance) I expect all up nodes in all DC's to be active. With DC_LOCAL read repair (dclocal_rea

Re: Incompatible Gossip 1.1.6 to 1.2.1 Upgrade?

2013-03-23 Thread aaron morton
So all nodes are 1.2 and some are still being marked as down ? I would try a rolling restart with -Dcassandra.load_ring_state=false added as a JVM _OPT in cassandra-env.sh. There is no guarantee it will fix it, but it's a simple thing to try. Cheers - Aaron Morton Freelance C

Re: Recovering from a faulty cassandra node

2013-03-23 Thread aaron morton
With thins like cleanup (and upgradesstables) I then to run them on every RF'th node. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 22/03/2013, at 10:59 AM, Jabbar Azam wrote: > nodetool cleanup command removes

Re: Unable to fetch large amount of rows

2013-03-23 Thread aaron morton
> [pp] no, I didn’t look at proxyhistogram, in fact I don’t know how to run it. > Can you give me insights of how to run it? It's available on nodetool but I cannot remember the version it was added. If it's not there the information has always been available on the StorageProxyMBean. Cheers

Re: create secondary index on column family

2013-03-23 Thread aaron morton
> But a error is thrown saying "can not parse name as hex bytes". If the comparator is Bytes then the column names need to be a hex string. The easiest thing to do is create a CF where the comparator is UTF8Type so you can use string column names. > just that the UTF8Type needs to be validated

Re: Thrift key

2013-03-23 Thread aaron morton
> When I used Thrift, the key is changed by Cassandra. Cassandra Thrift add a > prefix control bytes to Partitioner key and send the object always to the > same machine because the key prefix is the same. Where do you think this is happening in the code? Remember the key received by thrift is tr

Re: Cassandra - conflict resolution for column updates with identical timestamp

2013-03-23 Thread Edward Capriolo
The value that sorts higher, this way it is deterministic. On Sat, Mar 23, 2013 at 12:12 PM, dong.yajun wrote: > Hello, > > I would like to know which write wins in case of two updates with the > same client timestamp in Cassandra. > > Initial data: KeyA: { col1:"val AA", col2:"val BB", col3:"va

Re: Stream fails during repair, two nodes out-of-memory

2013-03-23 Thread Dane Miller
On Fri, Mar 22, 2013 at 5:58 PM, Wei Zhu wrote: > compaction needs some disk I/O. Slowing down our compaction will improve > overall > system performance. Of course, you don't want to go too slow and fall behind > too much. Hmm. Even after making the suggested configuration changes, repair sti

Observation on shuffling vs adding/removing nodes

2013-03-23 Thread Andrew Bialecki
Just curious if anyone has any thoughts on something we've observed in a small test cluster. We had around 100 GB of data on a 3 node cluster (RF=2) and wanted to start using vnodes. We upgraded the cluster to 1.2.2 and then followed the instructions for using vnodes. We initially tried to run a s

Re: High disk I/O during reads

2013-03-23 Thread Matt Kap
Having battled similar issues with read latency recently, here's some general things to look out for. - At 118ms, something is definitely broken. You should be looking at under 10ms or lower, depending on hardware. - Do "nodetool info" on all 5 nodes. Is the load distributed evenly? Is it reasonab

Re: Cassandra - conflict resolution for column updates with identical timestamp

2013-03-23 Thread dong.yajun
Thanks Capriolo, Umm.. so is there any background or history abort this issue? On Sun, Mar 24, 2013 at 2:32 AM, Edward Capriolo wrote: > The value that sorts higher, this way it is deterministic. > > > On Sat, Mar 23, 2013 at 12:12 PM, dong.yajun wrote: > >> Hello, >> >> I would like to know wh