Works.. But during the night my setup encountered a problem. I have two VMs on my cluster (running on VmWare ESXi). Each VM has1GB memory, and two Virtual Disks of 16 GB They are running on a small server with 4CPUs (2.66 GHz), and 4 GB memory (together with two other VMs) I put cassandra data on the second disk of each machine. VMs are running Ubuntu 11.10 and cassandra 1.0.7.
I left them running overnight and this morning when I came: In one node cassandra was down, and the last thing in the system.log is: INFO [CompactionExecutor:150] 2012-03-06 00:55:04,821 CompactionTask.java (line 113) Compacting [SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1243-Data.db'), SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1245-Data.db'), SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1242-Data.db'), SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db')] INFO [CompactionExecutor:150] 2012-03-06 00:55:07,919 CompactionTask.java (line 221) Compacted to [/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1246-Data.db,]. 32,424,771 to 26,447,685 (~81% of original) bytes for 58,938 keys at 8.144165MB/s. Time: 3,097ms. The other node was using all it's CPU and I had to restart it. After that, I can see that the last lines in it's system.log are that the other node is down... INFO [FlushWriter:142] 2012-03-06 00:55:02,418 Memtable.java (line 246) Writing Memtable-tk_vertical_tag_story_indx@1365852701(1122169/25154556 serialized/live bytes, 21173 ops) INFO [FlushWriter:142] 2012-03-06 00:55:02,742 Memtable.java (line 283) Completed flushing /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db (2075930 bytes) INFO [GossipTasks:1] 2012-03-06 08:02:18,584 Gossiper.java (line 818) InetAddress /10.0.0.31 is now dead. How can I trace why that happened? Also, I brought cassandra up in both nodes. They both spend long time reading commit logs, but now they seem to run. Any idea how to debug or improve my setup? Thanks, Tamar *Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Mon, Mar 5, 2012 at 7:30 PM, aaron morton <aa...@thelastpickle.com>wrote: > Create nodes that do not share seeds, and give the clusters different > names as a safety measure. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 6/03/2012, at 12:04 AM, Tamar Fraenkel wrote: > > I want tow separate clusters. > *Tamar Fraenkel * > Senior Software Engineer, TOK Media > > <tokLogo.png> > > > ta...@tok-media.com > Tel: +972 2 6409736 > Mob: +972 54 8356490 > Fax: +972 2 5612956 > > > > > > On Mon, Mar 5, 2012 at 12:48 PM, aaron morton <aa...@thelastpickle.com>wrote: > >> Do you want to create two separate clusters or a single cluster with two >> data centres ? >> >> If it's the later, token selection is discussed here >> http://www.datastax.com/docs/1.0/install/cluster_init#token-gen-cassandra >> >> >> Moreover all tokens must be unique (even across datacenters), although - >> from pure curiosity - I wonder what is the rationale behind this. >> >> Otherwise data is not evenly distributed. >> >> By the way, can someone enlighten me about the first line in the output >> of the nodetool. Obviously it contains a token, but nothing else. It seems >> like a formatting glitch, but maybe it has a role. >> >> It's the exclusive lower bound token for the first node in the ring. This >> also happens to be the token for the last node in the ring. >> >> In your setup >> 10.0.0.19 "owns" (85070591730234615865843651857942052864+1) to 0 >> 10.0.0.28 "owns" (0 + 1) to 85070591730234615865843651857942052864 >> >> (does not imply primary replica, just used to map keys to nodes.) >> >> >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 5/03/2012, at 11:38 PM, Hontvári József Levente wrote: >> >> You have to use PropertyFileSnitch and NetworkTopologyStrategy to create >> a multi-datacenter setup with two circles. You can start reading from this >> page: >> >> http://www.datastax.com/docs/1.0/cluster_architecture/replication#about-replica-placement-strategy >> >> Moreover all tokens must be unique (even across datacenters), although - >> from pure curiosity - I wonder what is the rationale behind this. >> >> By the way, can someone enlighten me about the first line in the output >> of the nodetool. Obviously it contains a token, but nothing else. It seems >> like a formatting glitch, but maybe it has a role. >> >> On 2012.03.05. 11:06, Tamar Fraenkel wrote: >> >> Hi! >> I have a Cassandra cluster with two nodes >> >> nodetool ring -h localhost >> Address DC Rack Status State Load >> Owns Token >> >> 85070591730234615865843651857942052864 >> 10.0.0.19 datacenter1 rack1 Up Normal 488.74 KB >> 50.00% 0 >> 10.0.0.28 datacenter1 rack1 Up Normal 504.63 KB >> 50.00% 85070591730234615865843651857942052864 >> >> I want to create a second ring with the same name but two different >> nodes. >> using tokengentool I get the same tokens as they are affected from the >> number of nodes in a ring. >> >> My question is like this: >> Lets say I create two new VMs, with IPs: 10.0.0.31 and 10.0.0.11 >> *In 10.0.0.31 cassandra.yaml I will set* >> initial_token: 0 >> seeds: "10.0.0.31" >> listen_address: 10.0.0.31 >> rpc_address: 0.0.0.0 >> >> *In 10.0.0.11 cassandra.yaml I will set* >> initial_token: 85070591730234615865843651857942052864 >> seeds: "10.0.0.31" >> listen_address: 10.0.0.11 >> rpc_address: 0.0.0.0 >> >> *Would the rings be separate?* >> >> Thanks, >> >> *Tamar Fraenkel * >> Senior Software Engineer, TOK Media >> >> <Mail Attachment.png> >> >> >> ta...@tok-media.com >> Tel: +972 2 6409736 >> Mob: +972 54 8356490 >> Fax: +972 2 5612956 >> >> >> >> >> >> > >
<<tokLogo.png>>
