Hi -
Operations like repair and bootstrap on nodes in our cluster (average
load 150GB each) take a very long time.
By long I mean 1-2 days. With nodetool "netstats" I can see the
progress % very slowly progressing.
I guess there are some throttling mechanisms built into cassandra.
And yes there is also production load on these nodes so it is somewhat
understandable. Also some of out compacted data files are as 50-60 GB
each.
I was just wondering if these times are similar to what other people
are experiencing or if there is a serious configuration problem with
our setup.
So what have you guys seen with operations like loadbalance,repair,
cleanup, bootstrap on nodes with large amounts of data??
I'm not seeing too many full garbage collections. Other minor GCs are
well under a second.
Setup info:
0.7.4
5 GB heap
8 GB ram
64 bit linux os
AMD quad core HP blades
CMS Garbage collector with default cassandra settings
1 TB raid 0 sata disks
across 2 datacenters, but operations within the same dc take very long too.
This is a netstat output of a bootstrap that has been going on for 3+ hours:
Mode: Normal
Streaming to: /10.47.108.103
/var/lib/cassandra/data/DFS/main-f-1541-Data.db/(0,32842490722),(32842490722,139556639427),(139556639427,161075890783)
progress=94624588642/161075890783 - 58%
/var/lib/cassandra/data/DFS/main-f-1455-Data.db/(0,660743002)
progress=0/660743002 - 0%
/var/lib/cassandra/data/DFS/main-f-1444-Data.db/(0,32816130132),(32816130132,71465138397),(71465138397,90968640033)
progress=0/90968640033 - 0%
/var/lib/cassandra/data/DFS/main-f-1540-Data.db/(0,931632934),(931632934,2621052149),(2621052149,3236107041)
progress=0/3236107041 - 0%
/var/lib/cassandra/data/DFS/main-f-1488-Data.db/(0,33428780851),(33428780851,110546591227),(110546591227,110851587206)
progress=0/110851587206 - 0%
/var/lib/cassandra/data/DFS/main-f-1542-Data.db/(0,24091168),(24091168,97485080),(97485080,108233211)
progress=0/108233211 - 0%
/var/lib/cassandra/data/DFS/main-f-1544-Data.db/(0,3646406),(3646406,18065308),(18065308,25776551)
progress=0/25776551 - 0%
/var/lib/cassandra/data/DFS/main-f-1452-Data.db/(0,676616940)
progress=0/676616940 - 0%
/var/lib/cassandra/data/DFS/main-f-1548-Data.db/(0,6957269),(6957269,48966550),(48966550,51499779)
progress=0/51499779 - 0%
/var/lib/cassandra/data/DFS/main-f-1552-Data.db/(0,237153399),(237153399,750466875),(750466875,898056853)
progress=0/898056853 - 0%
/var/lib/cassandra/data/DFS/main-f-1554-Data.db/(0,45155582),(45155582,195640768),(195640768,247592141)
progress=0/247592141 - 0%
/var/lib/cassandra/data/DFS/main-f-1449-Data.db/(0,2812483216)
progress=0/2812483216 - 0%
/var/lib/cassandra/data/DFS/main-f-1545-Data.db/(0,107648943),(107648943,434575065),(434575065,436667186)
progress=0/436667186 - 0%
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 134283
Responses n/a 0 192438
