We occasionally see fairly poor compaction performance on random nodes in our
7-node cluster, and I have no idea why. This is one example from the log:
[CompactionExecutor:45] 2012-06-26 13:40:18,721 CompactionTask.java
(line 221) Compacted to
[/raid00/cassandra_data/main/basic/main-basic.basic_id_index-hd-160-Data.db,].
26,632,210 to 26,679,667 (~100% of original) bytes for 2 keys at 0.006250MB/s.
Time: 4,071,163ms.
That particular event took over an hour to compact only 25 megabytes. During
that time, there was very little disk IO, and the java process (OpenJDK 7) was
pegged at 200% CPU. The node was also completely unresponsive to network
requests until the compaction was finished. Most compactions run just over
7MB/s. This is an extreme outlier, but users definitely notice the hit when it
occurs.
I grabbed a sample of the process using jstack, and this was the only thread in
CompactionExecutor:
"CompactionExecutor:54" daemon prio=1 tid=41247522816 nid=0x99a5ff740
runnable [140737253617664]
java.lang.Thread.State: RUNNABLE
at org.xerial.snappy.SnappyNative.rawCompress(Native Method)
at org.xerial.snappy.Snappy.rawCompress(Snappy.java:358)
at
org.apache.cassandra.io.compress.SnappyCompressor.compress(SnappyCompressor.java:80)
at
org.apache.cassandra.io.compress.CompressedSequentialWriter.flushData(CompressedSequentialWriter.java:89)
at
org.apache.cassandra.io.util.SequentialWriter.flushInternal(SequentialWriter.java:196)
at
org.apache.cassandra.io.util.SequentialWriter.reBuffer(SequentialWriter.java:260)
at
org.apache.cassandra.io.util.SequentialWriter.writeAtMost(SequentialWriter.java:128)
at
org.apache.cassandra.io.util.SequentialWriter.write(SequentialWriter.java:112)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
- locked <36527862064> (a java.io.DataOutputStream)
at
org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:142)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:156)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:159)
at
org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:150)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Is it possible that there is an issue with snappy compression? Based on the
lousy compression ratio, I think we could get by without it just fine. Can
compression be changed or disabled on-the-fly with cassandra?
- .Dustin
