Problem:
I am attempting to compare a data model of SuperColumn family with a normal
Column Family with Secondary Indexes. I did not have insert issues with the
SuperColumn family. The problem I am having seems to be inserting into the
Column Family with indexes. Seems to be very slow and getting slower. Also,
seems like from some previous test, I did not have issue with the normal column
family without indexes. About 24hrs after I started the inserts it is taking 7x
longer to do the same size insert. Progressively getting slower and slower.
Cluster config:
I am using cassandra 0.7.6 for a test on a 4 node cluster with replication set
at 2. The nodes are 32-bit, quad-core, Linux, 4GB ram, single hard drive.
Some settings:
MAX_HEAP_SIZE="2000m"
HEAP_NEWSIZE="400m"
memtable_flush_queue_size: 10 (was 4)
Everything else is pretty much default - Random partitioner, etc.
What I am seeing:
On one machine in particular, it seems to have a bit of IO contention and
waits. The other machines don't exhibit this problem.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
..
1 32 79384 110272 23900 1477272 0 0 3972 4 2927 1659 47 2 0 51
3 31 79384 110520 23892 1476788 0 0 4420 0 1723 622 52 2 0 46
4 29 79384 111512 23892 1475788 0 0 3876 0 1579 576 53 2 0 44
the other machines look like
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 96120 1032600 13100 581160 0 0 72 8 1598 1325 30 4 65 0
1 0 96120 1029252 13108 584224 0 0 0 144 609 155 23 2 75 0
1 0 96120 1027012 13108 587308 0 0 68 0 3437 6890 37 6 57 0
doing an iostat -x on the machine that is bogged down from an io standpoint
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 364.00 0.00 8264.00 0.00 22.70
109.95 149.60 2.75 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
36.86 0.00 1.23 61.92 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 326.00 0.00 7832.00 0.00 24.02
118.45 180.10 3.07 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
40.29 0.00 1.47 58.23 0.00 0.00
****** Additionally, and very strange to me, I see on one machine about 60000+
files representing the test column family(and growing with the test). This does
not seem like it would be normal? I've shown a few with typical sizes(very
small).
mycolumnfam.646f6d61696e4964-f-1943-Data.db 11M
mycolumnfam.646f6d61696e4964-f-1943-Filter.db 40K
mycolumnfam.646f6d61696e4964-f-1943-Index.db 1.3K
mycolumnfam.646f6d61696e4964-f-1943-Statistics.db 4.2K
mycolumnfam-f-993-Index.db
mycolumnfam-f-993-Statistics.db
mycolumnfam-f-994-Data.db
mycolumnfam-f-994-Filter.db
etc, etc. repeating
The test:
I have a process that has only 2 threads that is attempting to load about
300million rows(22GB of data). This is using the Hector java client. I am
doing batch inserts of 1000 rows at a time. I am inserting the column values as
bytes. The column names are strings. The column family has a total of 15
columns(each row). 9 of those columns have indexes.
The column family stats while under test look like the following. I note that
the key cache hit rate is very large. I haven't done any reads yet. None of my
other families have this.
Column Family: mycolumnfam
SSTable count: 11
Space used (live): 13297907918
Space used (total): 13385402196
Memtable Columns Count: 287238
Memtable Data Size: 8778990
Memtable Switch Count: 1036
Read Count: 22211625
Read Latency: 0.347 ms.
Write Count: 22211625
Write Latency: 0.026 ms.
Pending Tasks: 0
Key cache capacity: 200000
Key cache size: 6086
Key cache hit rate: 3.6248105493445186E-5
Row cache: disabled
Compacted row minimum size: 447
Compacted row maximum size: 642
Compacted row mean size: 634
I'm trying to understand why doing the inserts into a column family with
indexes seems to jam things up and am wondering if there are any settings that
I could tweak to help. It seems that the 4 node cluster should be able to
handle 2 threads of data coming at it. Has anyone had any experience with this
number of indexes per column family? Any insight or suggestions would be
appreciated.
Thanks in advance--
