Corollary: what is getting shipped over the wire? The ganglia screenshot shows the network traffic on all the three hosts on which I ran the nodetool repair.
[image: Inline image 1] remember UN 10.1.2.11 107.47 KB 256 32.9% 1f800723-10e4-4dcd-841f-73709a81d432 rack1 UN 10.1.2.10 127.67 KB 256 32.4% bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639 rack1 UN 10.1.2.12 107.62 KB 256 34.7% 5258f178-b20e-408f-a7bf-b6da2903e026 rack1 Much appreciated. Sven On Wed, Dec 11, 2013 at 3:56 PM, Sven Stark <sven.st...@m-square.com.au>wrote: > Howdy! > > Not a matter of life or death, just curious. > > I've just stood up a three node cluster (v1.2.8) on three c3.2xlarge boxes > in AWS. Silly me forgot the correct replication factor for one of the > needed keyspaces. So I changed it via cli and ran a nodetool repair. > Well .. there is no data at all in the keyspace yet, only the definition > and nodetool repair ran about 20minutes using 2 of the 8 CPU fully. > > Any hints what nodetool repair is doing on an empty cluster that makes the > host spin so hard? > > Cheers, > Sven > > ========================== > > Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie > Cpu(s): 22.7%us, 1.0%sy, 2.9%ni, 73.0%id, 0.0%wa, 0.0%hi, 0.4%si, > 0.0%st > Mem: 15339196k total, 7474360k used, 7864836k free, 251904k buffers > Swap: 0k total, 0k used, 0k free, 798324k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 10840 cassandr 20 0 8354m 4.1g 19m S 218 28.0 35:25.73 jsvc > 16675 kafka 20 0 3987m 192m 12m S 2 1.3 0:47.89 java > 20328 root 20 0 5613m 569m 16m S 2 3.8 1:35.13 jsvc > 5969 exhibito 20 0 6423m 116m 12m S 1 0.8 0:25.87 java > 14436 tomcat7 20 0 3701m 167m 11m S 1 1.1 0:25.80 java > 6278 exhibito 20 0 6487m 119m 9984 S 0 0.8 0:22.63 java > 17713 storm 20 0 6033m 159m 11m S 0 1.1 0:10.99 java > 18769 storm 20 0 5773m 156m 11m S 0 1.0 0:10.71 java > > root@xxx-01:~# nodetool -h `hostname` status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID > Rack > UN 10.1.2.11 107.47 KB 256 32.9% > 1f800723-10e4-4dcd-841f-73709a81d432 rack1 > UN 10.1.2.10 127.67 KB 256 32.4% > bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639 rack1 > UN 10.1.2.12 107.62 KB 256 34.7% > 5258f178-b20e-408f-a7bf-b6da2903e026 rack1 > > root@xxx-01:~# nodetool -h `hostname` compactionstats > pending tasks: 1 > compaction type keyspace column family completed > total unit progress > Active compaction remaining time : n/a > > root@xxx-01:~# nodetool -h `hostname` netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed > Commands n/a 0 57155 > Responses n/a 0 14573 >
<<image.png>>
