MAX_HEAP_SIZE="9G" HEAP_NEWSIZE="800M" 2 socket quad core 44GB RAM Cassandra 1.0.7
[edward@cdbla120 cassandra]$ free -g total used free shared buffers cached Mem: 43 34 8 0 0 25 -/+ buffers/cache: 8 34 Swap: 0 0 0 # GC logging options -- uncomment to enable JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps" JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC" JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime" JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure" JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1" JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log" JVM_OPTS="$JVM_OPTS -verbose:gc" JVM_OPTS="$JVM_OPTS -XX:+PrintSafepointStatistics" Three major column families using compression on two out of three. I never really watched the gc logs too much before since i was disk bound, but in this case I have all the data in main memory so limiting pauses would be big for me. I see a good number of these: Total time for which application threads were stopped: 0.0301370 seconds Total time for which application threads were stopped: 0.0046390 seconds [edward@cdbla120 cassandra]$ tail -400 gc-1329578417.log Total Free Space: 637346254 Max Chunk Size: 138752484 Number of Blocks: 482066 Av. Block Size: 1322 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10516.947: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 48752 bytes, 48752 total : 7689K->5164K(737280K), 0.0173210 secs] 3594487K->3597704K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 636661736 Max Chunk Size: 138752484 Number of Blocks: 481773 Av. Block Size: 1321 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0204840 secs] [Times: user=0.19 sys=0.00, real=0.02 secs] Heap after GC invocations=3856 (full 4): par new generation total 737280K, used 5164K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 6% used [0x00000005e2e00000, 0x00000005e330b030, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3592540K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0251700 seconds Total time for which application threads were stopped: 0.0046210 seconds {Heap before GC invocations=3856 (full 4): par new generation total 737280K, used 660839K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 100% used [0x00000005bae00000, 0x00000005e2e00000, 0x00000005e2e00000) from space 81920K, 6% used [0x00000005e2e00000, 0x00000005e3359df0, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3592540K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:35.980-0500: 10518.844: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 636661736 Max Chunk Size: 138752484 Number of Blocks: 481773 Av. Block Size: 1321 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10518.846: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 7128016 bytes, 7128016 total : 660839K->7853K(737280K), 0.0176930 secs] 4253380K->3600423K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 636661235 Max Chunk Size: 138752484 Number of Blocks: 481772 Av. Block Size: 1321 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0210680 secs] [Times: user=0.18 sys=0.00, real=0.02 secs] Heap after GC invocations=3857 (full 4): par new generation total 737280K, used 7853K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 9% used [0x00000005e7e00000, 0x00000005e85ab660, 0x00000005ece00000) to space 81920K, 0% used [0x00000005e2e00000, 0x00000005e2e00000, 0x00000005e7e00000) concurrent mark-sweep generation total 8617984K, used 3592570K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0256030 seconds {Heap before GC invocations=3857 (full 4): par new generation total 737280K, used 7876K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae05ac0, 0x00000005e2e00000) from space 81920K, 9% used [0x00000005e7e00000, 0x00000005e85ab660, 0x00000005ece00000) to space 81920K, 0% used [0x00000005e2e00000, 0x00000005e2e00000, 0x00000005e7e00000) concurrent mark-sweep generation total 8617984K, used 3592570K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:36.006-0500: 10518.870: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 636661235 Max Chunk Size: 138752484 Number of Blocks: 481772 Av. Block Size: 1321 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10518.871: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 4912 bytes, 4912 total : 7876K->2197K(737280K), 0.0168170 secs] 3600446K->3601858K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 635807044 Max Chunk Size: 138752484 Number of Blocks: 481304 Av. Block Size: 1321 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0199700 secs] [Times: user=0.19 sys=0.00, real=0.02 secs] Heap after GC invocations=3858 (full 4): par new generation total 737280K, used 2197K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 2% used [0x00000005e2e00000, 0x00000005e3025410, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3599661K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0261540 seconds {Heap before GC invocations=3858 (full 4): par new generation total 737280K, used 657515K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 99% used [0x00000005bae00000, 0x00000005e2df5b20, 0x00000005e2e00000) from space 81920K, 2% used [0x00000005e2e00000, 0x00000005e3025410, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3599661K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:39.918-0500: 10522.782: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 635807044 Max Chunk Size: 138752484 Number of Blocks: 481304 Av. Block Size: 1321 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10522.784: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 18138696 bytes, 18138696 total : 657515K->18303K(737280K), 0.0225670 secs] 4257177K->3617966K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 635807044 Max Chunk Size: 138752484 Number of Blocks: 481304 Av. Block Size: 1321 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0259020 secs] [Times: user=0.25 sys=0.00, real=0.03 secs] Heap after GC invocations=3859 (full 4): par new generation total 737280K, used 18303K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 22% used [0x00000005e7e00000, 0x00000005e8fdffa8, 0x00000005ece00000) to space 81920K, 0% used [0x00000005e2e00000, 0x00000005e2e00000, 0x00000005e7e00000) concurrent mark-sweep generation total 8617984K, used 3599662K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0310430 seconds Total time for which application threads were stopped: 0.0044970 seconds {Heap before GC invocations=3859 (full 4): par new generation total 737280K, used 673663K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 100% used [0x00000005bae00000, 0x00000005e2e00000, 0x00000005e2e00000) from space 81920K, 22% used [0x00000005e7e00000, 0x00000005e8fdffa8, 0x00000005ece00000) to space 81920K, 0% used [0x00000005e2e00000, 0x00000005e2e00000, 0x00000005e7e00000) concurrent mark-sweep generation total 8617984K, used 3599662K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:43.662-0500: 10526.526: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 635807044 Max Chunk Size: 138752484 Number of Blocks: 481304 Av. Block Size: 1321 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10526.528: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 6231016 bytes, 6231016 total : 673663K->9643K(737280K), 0.0313290 secs] 4273326K->3621439K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 634281586 Max Chunk Size: 138752484 Number of Blocks: 480350 Av. Block Size: 1320 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0347120 secs] [Times: user=0.31 sys=0.00, real=0.03 secs] Heap after GC invocations=3860 (full 4): par new generation total 737280K, used 9643K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 11% used [0x00000005e2e00000, 0x00000005e376ade8, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3611796K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0398640 seconds {Heap before GC invocations=3860 (full 4): par new generation total 737280K, used 664966K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 99% used [0x00000005bae00000, 0x00000005e2df6be8, 0x00000005e2e00000) from space 81920K, 11% used [0x00000005e2e00000, 0x00000005e376ade8, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3611796K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:46.806-0500: 10529.671: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 634281586 Max Chunk Size: 138752484 Number of Blocks: 480350 Av. Block Size: 1320 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10529.672: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 5578880 bytes, 5578880 total : 664966K->8084K(737280K), 0.0217310 secs] 4276762K->3624063K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 633726004 Max Chunk Size: 138752484 Number of Blocks: 480114 Av. Block Size: 1319 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0251170 secs] [Times: user=0.22 sys=0.00, real=0.03 secs] Heap after GC invocations=3861 (full 4): par new generation total 737280K, used 8084K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 9% used [0x00000005e7e00000, 0x00000005e85e52f8, 0x00000005ece00000) to space 81920K, 0% used [0x00000005e2e00000, 0x00000005e2e00000, 0x00000005e7e00000) concurrent mark-sweep generation total 8617984K, used 3615978K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0301370 seconds Total time for which application threads were stopped: 0.0046390 seconds {Heap before GC invocations=3861 (full 4): par new generation total 737280K, used 663583K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 100% used [0x00000005bae00000, 0x00000005e2e00000, 0x00000005e2e00000) from space 81920K, 10% used [0x00000005e7e00000, 0x00000005e8607de8, 0x00000005ece00000) to space 81920K, 0% used [0x00000005e2e00000, 0x00000005e2e00000, 0x00000005e7e00000) concurrent mark-sweep generation total 8617984K, used 3615978K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:49.225-0500: 10532.089: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 633726004 Max Chunk Size: 138752484 Number of Blocks: 480114 Av. Block Size: 1319 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10532.099: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 4626680 bytes, 4626680 total : 663583K->7027K(737280K), 0.0214590 secs] 4279561K->3626859K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 633246817 Max Chunk Size: 138752484 Number of Blocks: 479921 Av. Block Size: 1319 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0248210 secs] [Times: user=0.21 sys=0.00, real=0.03 secs] Heap after GC invocations=3862 (full 4): par new generation total 737280K, used 7027K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 0% used [0x00000005bae00000, 0x00000005bae00000, 0x00000005e2e00000) from space 81920K, 8% used [0x00000005e2e00000, 0x00000005e34dcd88, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3619831K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) } Total time for which application threads were stopped: 0.0374180 seconds {Heap before GC invocations=3862 (full 4): par new generation total 737280K, used 662376K [0x00000005bae00000, 0x00000005ece00000, 0x00000005ece00000) eden space 655360K, 99% used [0x00000005bae00000, 0x00000005e2dfd2b8, 0x00000005e2e00000) from space 81920K, 8% used [0x00000005e2e00000, 0x00000005e34dcd88, 0x00000005e7e00000) to space 81920K, 0% used [0x00000005e7e00000, 0x00000005e7e00000, 0x00000005ece00000) concurrent mark-sweep generation total 8617984K, used 3619831K [0x00000005ece00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 35796K, used 21382K [0x00000007fae00000, 0x00000007fd0f5000, 0x0000000800000000) 2012-02-18T13:15:52.121-0500: 10534.985: [GC Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 633246817 Max Chunk Size: 138752484 Number of Blocks: 479921 Av. Block Size: 1319 Tree Height: 34 Before GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 10534.987: [ParNew Desired survivor size 41943040 bytes, new threshold 1 (max 1) - age 1: 8678968 bytes, 8678968 total : 662376K->9883K(737280K), 0.0205490 secs] 4282207K->3632214K(9355264K)After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 632933226 Max Chunk Size: 138752484 Number of Blocks: 479784 Av. Block Size: 1319 Tree Height: 34 After GC: Statistics for BinaryTreeDictionary: ------------------------------------ Total Free Space: 1829376 Max Chunk Size: 1829376 Number of Blocks: 1 Av. Block Size: 1829376 Tree Height: 1 , 0.0239560 secs] On Sat, Feb 4, 2012 at 6:25 AM, Jonathan Ellis <jbel...@gmail.com> wrote: > We put a bunch of effort into reducing GC pauses in 1.0, which worked > great on our synthetic benchmarks but I don't have much data on how > well that's working in practice, on real-world workloads. > > If you're running a Cassandra app in production or testing, I'd > appreciate it if you could send me [either personally or via the list] > the results of enabling GC logging on one of your nodes by > uncommenting these lines in cassandra-env.sh: > > # JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails" > # JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps" > # JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC" > # JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution" > # JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime" > # JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure" > # JVM_OPTS="$JVM_OPTS -XX:PrintFLSStatistics=1" > # JVM_OPTS="$JVM_OPTS -Xloggc:/var/log/cassandra/gc-`date +%s`.log" > > (If you're running on Windows, you'll need to add these options > manually to cassandra.bat, and hardcode the log filename.) > > Besides the log file itself it would be useful to know JVM version, > how many cores your machine has, any GC options you have changed from > the default, and a high-level description of your workload (write > heavy? read heavy? slice reads, or query columns by name? row cache > enabled, and if so, serializingcache with jna, or on-heap cache?) > > Thanks! > > P.S. It would be simplest if you can provide results for 1.0.7, which > IMO you would be well-served by upgrading to anyway, but all results > for 1.0.2+ are useful if you specify the version you are on. > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com