Hi guys, Has anybody got fix for this issue? Recently we upgraded our c* cluster from 2.0.17 to 2.1.17. And we saw increase in read latency on few tables, read latency increased almost 2 to 3 times more than what it was in 2.0.17.
Kindly let me know the fix for it if anybody knows it. Thanks TechPyaasa On Wed, Nov 9, 2016 at 1:28 AM, Dikang Gu <dikan...@gmail.com> wrote: > Michael, thanks for the info. It sounds to me a very serious performance > regression. :( > > On Tue, Nov 8, 2016 at 11:39 AM, Michael Kjellman < > mkjell...@internalcircle.com> wrote: > >> Yes, We hit this as well. We have a internal patch that I wrote to mostly >> revert the behavior back to ByteBuffers with as small amount of code change >> as possible. Performance of our build is now even with 2.0.x and we've also >> forward ported it to 3.x (although the 3.x patch was even more complicated >> due to Bounds, RangeTombstoneBound, ClusteringPrefix which actually >> increases the number of allocations to somewhere between 11 and 13 >> depending on how I count it per indexed block -- making it even worse than >> what you're observing in 2.1. >> >> We haven't upstreamed it as 2.1 is obviously not taking any changes at >> this point and the longer term solution is https://issues.apache.org/jira >> /browse/CASSANDRA-9754 (which also includes the changes to go back to >> ByteBuffers and remove as much of the Composites from the storage engine as >> possible.) Also, the solution is a bit of a hack -- although it was a >> blocker from us deploying 2.1 -- so i'm not sure how "hacky" it is if it >> works.. >> >> best, >> kjellman >> >> >> On Nov 8, 2016, at 11:31 AM, Dikang Gu <dikan...@gmail.com<mailto:dik >> an...@gmail.com>> wrote: >> >> This is very expensive: >> >> "MessagingService-Incoming-/2401:db00:21:1029:face:0:9:0" prio=10 >> tid=0x00007f2fd57e1800 nid=0x1cc510 runnable [0x00007f2b971b0000] >> java.lang.Thread.State: RUNNABLE >> at org.apache.cassandra.db.marshal.IntegerType.compare(IntegerT >> ype.java:29) >> at org.apache.cassandra.db.composites.AbstractSimpleCellNameTyp >> e.compare(AbstractSimpleCellNameType.java:98) >> at org.apache.cassandra.db.composites.AbstractSimpleCellNameTyp >> e.compare(AbstractSimpleCellNameType.java:31) >> at java.util.TreeMap.put(TreeMap.java:545) >> at java.util.TreeSet.add(TreeSet.java:255) >> at org.apache.cassandra.db.filter.NamesQueryFilter$Serializer. >> deserialize(NamesQueryFilter.java:254) >> at org.apache.cassandra.db.filter.NamesQueryFilter$Serializer. >> deserialize(NamesQueryFilter.java:228) >> at org.apache.cassandra.db.SliceByNamesReadCommandSerializer. >> deserialize(SliceByNamesReadCommand.java:104) >> at org.apache.cassandra.db.ReadCommandSerializer.deserialize( >> ReadCommand.java:156) >> at org.apache.cassandra.db.ReadCommandSerializer.deserialize( >> ReadCommand.java:132) >> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) >> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessag >> e(IncomingTcpConnection.java:195) >> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessag >> es(IncomingTcpConnection.java:172) >> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingT >> cpConnection.java:88) >> >> >> Checked the git history, it comes from this jira: >> https://issues.apache.org/jira/browse/CASSANDRA-5417 >> >> Any thoughts? >> >> >> On Fri, Oct 28, 2016 at 10:32 AM, Paulo Motta <pauloricard...@gmail.com >> <mailto:pauloricard...@gmail.com>> wrote: >> Haven't seen this before, but perhaps it's related to CASSANDRA-10433? >> This is just a wild guess as it's in a related codepath, but maybe worth >> trying out the patch available to see if it helps anything... >> >> 2016-10-28 15:03 GMT-02:00 Dikang Gu <dikan...@gmail.com<mailto:dik >> an...@gmail.com>>: >> We are seeing huge cpu regression when upgrading one of our 2.0.16 >> cluster to 2.1.14 as well. The 2.1.14 node is not able to handle the same >> amount of read traffic as the 2.0.16 node, actually, it's less than 50%. >> >> And in the perf results, the first line could go as high as 50%, as we >> turn up the read traffic, which never appeared in 2.0.16. >> >> Any thoughts? >> Thanks >> >> >> Samples: 952K of event 'cycles', Event count (approx.): 229681774560 >> Overhead Shared Object Symbol >> 6.52% perf-196410.map [.] >> Lorg/apache/cassandra/db/marshal/IntegerType;.compare in >> Lorg/apache/cassandra/db/composites/AbstractSimpleCellNameType;.compare >> 4.84% libzip.so [.] adler32 >> 2.88% perf-196410.map [.] >> Ljava/nio/HeapByteBuffer;.get in Lorg/apache/cassandra/db/marsh >> al/IntegerType;.compare >> 2.39% perf-196410.map [.] >> Ljava/nio/Buffer;.checkIndex in Lorg/apache/cassandra/db/marsh >> al/IntegerType;.findMostSignificantByte >> 2.03% perf-196410.map [.] >> Ljava/math/BigInteger;.compareTo in Lorg/apache/cassandra/db/Decor >> atedKey;.compareTo >> 1.65% perf-196410.map [.] vtable chunks >> 1.44% perf-196410.map [.] >> Lorg/apache/cassandra/db/DecoratedKey;.compareTo in >> Ljava/util/concurrent/ConcurrentSkipListMap;.findNode >> 1.02% perf-196410.map [.] >> Lorg/apache/cassandra/db/composites/AbstractSimpleCellNameType;.compare >> 1.00% snappy-1.0.5.2-libsnappyjava.so<http://snappy-1.0.5.2-libsna >> ppyjava.so/> [.] 0x0000000000003804 >> 0.87% perf-196410.map [.] >> Ljava/io/DataInputStream;.readFully in Lorg/apache/cassandra/db/Abstr >> actCell$1;.computeNext >> 0.82% snappy-1.0.5.2-libsnappyjava.so<http://snappy-1.0.5.2-libsna >> ppyjava.so/> [.] 0x00000000000036dc >> 0.79% [kernel] [k] >> copy_user_generic_string >> 0.73% perf-196410.map [.] vtable chunks >> 0.71% perf-196410.map [.] >> Lorg/apache/cassandra/db/OnDiskAtom$Serializer;.deserializeFromSSTable >> in Lorg/apache/cassandra/db/AbstractCell$1;.computeNext >> 0.70% [kernel] [k] find_busiest_group >> 0.69% perf-196410.map [.] <80>H<F4>3<AE>^? >> 0.68% perf-196410.map [.] >> Lorg/apache/cassandra/db/DecoratedKey;.compareTo >> 0.65% perf-196410.map [.] >> jbyte_disjoint_arraycopy >> 0.64% [kernel] [k] _raw_spin_lock >> 0.63% [kernel] [k] __schedule >> 0.45% snappy-1.0.5.2-libsnappyjava.so<http://snappy-1.0.5.2-libsna >> ppyjava.so/> [.] 0x00000000000036df >> >> On Fri, Jan 29, 2016 at 2:11 PM, Corry Opdenakker <co...@bestdata.be >> <mailto:co...@bestdata.be>> wrote: >> @JC, Get the pid of your target java process (something like "ps -ef | >> grep -i cassandra") . >> Then do a kill -3 <pid> (at unix/linux) >> Check the stdout logfile of the process. >> it should contain the threaddump. >> If you found it, then great! >> Let that kill -3 loop for about 2 or 3 minutes. >> Herafter copy paste and load the stdout file into one if the mentioned >> tools. >> If you are not familiar with the java internals, then those threaddumps >> will learn you a lot:) >> >> >> >> >> Op vrijdag 29 januari 2016 heeft Jean Carlo <jean.jeancar...@gmail.com >> <mailto:jean.jeancar...@gmail.com>> het volgende geschreven: >> I am having the same issue after upgrade cassandra 2.1.12 from 2.0.10. I >> am not good on jvm so I would like to know how to do what @CorryOpdenakker >> propose with cassandra. >> >> :) >> >> I check concurrent_compactors >> >> >> Saludos >> >> Jean Carlo >> >> "The best way to predict the future is to invent it" Alan Kay >> >> On Fri, Jan 29, 2016 at 9:24 PM, Corry Opdenakker <co...@bestdata.be> >> wrote: >> Hi guys, >> Cassandra is still new for me, but I have a lot of java tuning experience. >> >> For root cause detection of performance degradations its always good to >> start with collecting a series of java thread dumps. Take at problem >> occurrence using a loopscript for example 60 thread dumps with an interval >> of 1 or 2 seconds. >> Then load those dumps into IBM thread dump analyzer or in "eclipse mat" >> or any similar tool and see which methods appear to be most active or >> blocking others. >> >> Its really very useful >> >> Same can be be done in a normal situation to compare the difference. >> >> That should give more insights. >> >> Cheers, Corry >> >> >> Op vrijdag 29 januari 2016 heeft Peddi, Praveen <pe...@amazon.com> het >> volgende geschreven: >> Hello, >> We have another update on performance on 2.1.11. compression_chunk_size >> didn’t really help much but We changed concurrent_compactors from default >> to 64 in 2.1.11 and read latencies improved significantly. However, 2.1.11 >> read latencies are still 1.5 slower than 2.0.9. One thing we noticed in JMX >> metric that could affect read latencies is that 2.1.11 is running >> ReadRepairedBackground and ReadRepairedBlocking too frequently compared to >> 2.0.9 even though our read_repair_chance is same on both. Could anyone shed >> some light on why 2.1.11 could be running read repair 10 to 50 times more >> in spite of same configuration on both clusters? >> >> dclocal_read_repair_chance=0.100000 AND >> read_repair_chance=0.000000 AND >> >> Here is the table for read repair metrics for both clusters. >> 2.0.9 2.1.11 >> ReadRepairedBackground 5MinAvg 0.006 0.1 >> 15MinAvg 0.009 0.153 >> ReadRepairedBlocking 5MinAvg 0.002 0.55 >> 15MinAvg 0.007 0.91 >> >> Thanks >> Praveen >> >> From: Jeff Jirsa <jeff.ji...@crowdstrike.com> >> Reply-To: <user@cassandra.apache.org> >> Date: Thursday, January 14, 2016 at 2:58 PM >> To: "user@cassandra.apache.org" <user@cassandra.apache.org> >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Sorry I wasn’t as explicit as I should have been >> >> The same buffer size is used by compressed reads as well, but tuned with >> compression_chunk_size table property. It’s likely true that if you lower >> compression_chunk_size, you’ll see improved read performance. >> >> This was covered in the AWS re:Invent youtube link I sent in my original >> reply. >> >> >> >> From: "Peddi, Praveen" >> Reply-To: "user@cassandra.apache.org" >> Date: Thursday, January 14, 2016 at 11:36 AM >> To: "user@cassandra.apache.org", Zhiyan Shao >> Cc: "Agrawal, Pratik" >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Hi, >> We will try with reduced “rar_buffer_size” to 4KB. However >> CASSANDRA-10249<https://issues.apache.org/jira/browse/CASSANDRA-10249> >> says "this only affects users who have 1. disabled compression, 2. switched >> to buffered i/o from mmap’d”. None of this is true for us I believe. We use >> default disk_access_mode which should be mmap. We also used LZ4Compressor >> when created table. >> >> >> We will let you know if this property had any effect. We were testing >> with 2.1.11 and this was only fixed in 2.1.12 so we need to play with >> latest version. >> >> Praveen >> >> >> >> >> >> From: Jeff Jirsa <jeff.ji...@crowdstrike.com> >> Reply-To: <user@cassandra.apache.org> >> Date: Thursday, January 14, 2016 at 1:29 PM >> To: Zhiyan Shao <zhiyan.s...@gmail.com>, "user@cassandra.apache.org" < >> user@cassandra.apache.org> >> Cc: "Agrawal, Pratik" <paagr...@amazon.com> >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> This may be due to https://issues.apache.org/jira/browse/CASSANDRA-10249 >> / https://issues.apache.org/jira/browse/CASSANDRA-8894 - whether or not >> this is really the case depends on how much of your data is in page cache, >> and whether or not you’re using mmap. Since the original question was asked >> by someone using small RAM instances, it’s possible. >> >> We mitigate this by dropping compression_chunk_size in order to force a >> smaller buffer on reads, so we don’t over read very small blocks. This has >> other side effects (lower compression ratio, more garbage during >> streaming), but significantly speeds up read workloads for us. >> >> >> From: Zhiyan Shao >> Date: Thursday, January 14, 2016 at 9:49 AM >> To: "user@cassandra.apache.org" >> Cc: Jeff Jirsa, "Agrawal, Pratik" >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Praveen, if you search "Read is slower in 2.1.6 than 2.0.14" in this >> forum, you can find another thread I sent a while ago. The perf test I did >> indicated that read is slower for 2.1.6 than 2.0.14 so we stayed with >> 2.0.14. >> >> On Tue, Jan 12, 2016 at 9:35 AM, Peddi, Praveen <pe...@amazon.com> wrote: >> Thanks Jeff for your reply. Sorry for delayed response. We were running >> some more tests and wanted to wait for the results. >> >> So basically we saw higher CPU with 2.1.11 was higher compared to 2.0.9 >> (see below) for the same exact load test. Memory spikes were also >> aggressive on 2.1.11. >> >> So we wanted to rule out any of our custom setting so we ended up doing >> some testing with Cassandra stress test and default Cassandra installation. >> Here are the results we saw between 2.0.9 and 2.1.11. Both are default >> installations and both use Cassandra stress test with same params. This is >> the closest apple-apple comparison we can get. As you can see both read and >> write latencies are 30 to 50% worse in 2.1.11 than 2.0.9. Since we are >> using default installation. >> >> Highlights of the test: >> Load: 2x reads and 1x writes >> CPU: 2.0.9 (goes upto 25%) compared to 2.1.11 (goes upto 60%) >> Local read latency: 0.039 ms for 2.0.9 and 0.066 ms for 2.1.11 >> Local write Latency: 0.033 ms for 2.0.9 Vs 0.030 ms for 2.1.11 >> One observation is, As the number of threads are increased, 2.1.11 read >> latencies are getting worse compared to 2.0.9 (see below table for 24 >> threads vs 54 threads) >> Not sure if anyone has done this kind of comparison before and what their >> thoughts are. I am thinking for this same reason >> >> 2.0.9 Plain type total ops op/s pk/s >> row/s mean med 0.95 0.99 0.999 >> max time >> 16 threadCount READ 66854 7205 7205 7205 1.6 1.3 >> 2.8 3.5 9.6 85.3 9.3 >> 16 threadCount WRITE 33146 3572 3572 3572 1.3 1 >> 2.6 3.3 7 206.5 9.3 >> 16 threadCount total 100000 10777 10777 10777 1.5 1.3 >> 2.7 3.4 7.9 206.5 9.3 >> 2.1.11 Plain >> 16 threadCount READ 67096 6818 6818 6818 1.6 1.5 >> 2.6 3.5 7.9 61.7 9.8 >> 16 threadCount WRITE 32904 3344 3344 3344 1.4 1.3 >> 2.3 3 6.5 56.7 9.8 >> 16 threadCount total 100000 10162 10162 10162 1.6 1.4 >> 2.5 3.2 6 61.7 9.8 >> 2.0.9 Plain >> 24 threadCount READ 66414 8167 8167 8167 2 1.6 >> 3.7 7.5 16.7 208 8.1 >> 24 threadCount WRITE 33586 4130 4130 4130 1.7 1.3 >> 3.4 5.4 25.6 45.4 8.1 >> 24 threadCount total 100000 12297 12297 12297 1.9 1.5 >> 3.5 6.2 15.2 208 8.1 >> 2.1.11 Plain >> 24 threadCount READ 66628 7433 7433 7433 2.2 2.1 >> 3.4 4.3 8.4 38.3 9 >> 24 threadCount WRITE 33372 3723 3723 3723 2 1.9 >> 3.1 3.8 21.9 37.2 9 >> 24 threadCount total 100000 11155 11155 11155 2.1 2 >> 3.3 4.1 8.8 38.3 9 >> 2.0.9 Plain >> 54 threadCount READ 67115 13419 13419 13419 2.8 2.6 >> 4.2 6.4 36.9 82.4 5 >> 54 threadCount WRITE 32885 6575 6575 6575 2.5 2.3 >> 3.9 5.6 15.9 81.5 5 >> 54 threadCount total 100000 19993 19993 19993 2.7 2.5 >> 4.1 5.7 13.9 82.4 5 >> 2.1.11 Plain >> 54 threadCount READ 66780 8951 8951 8951 4.3 3.9 >> 6.8 9.7 49.4 69.9 7.5 >> 54 threadCount WRITE 33220 4453 4453 4453 3.5 3.2 >> 5.7 8.2 36.8 68 7.5 >> 54 threadCount total 100000 13404 13404 13404 4 3.7 >> 6.6 9.2 48 69.9 7.5 >> >> >> From: Jeff Jirsa <jeff.ji...@crowdstrike.com> >> Date: Thursday, January 7, 2016 at 1:01 AM >> To: "user@cassandra.apache.org" <user@cassandra.apache.org>, Peddi >> Praveen <pe...@amazon.com> >> Subject: Re: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Anecdotal evidence typically agrees that 2.1 is faster than 2.0 (our >> experience was anywhere from 20-60%, depending on workload). >> >> However, it’s not necessarily true that everything behaves exactly the >> same – in particular, memtables are different, commitlog segment handling >> is different, and GC params may need to be tuned differently for 2.1 than >> 2.0. >> >> When the system is busy, what’s it actually DOING? Cassandra exposes a >> TON of metrics – have you plugged any into a reporting system to see what’s >> going on? Is your latency due to pegged cpu, iowait/disk queues or gc >> pauses? >> >> My colleagues spent a lot of time validating different AWS EBS configs >> (video from reinvent at https://www.youtube.com/watch?v=1R-mgOcOSd4), >> 2.1 was faster in almost every case, but you’re using an instance size I >> don’t believe we tried (too little RAM to be viable in production). c3.2xl >> only gives you 15G of ram – most “performance” based systems want 2-4x that >> (people running G1 heaps usually start at 16G heaps and leave another >> 16-30G for page cache), you’re running fairly small hardware – it’s >> possible that 2.1 isn’t “as good” on smaller hardware. >> >> (I do see your domain, presumably you know all of this, but just to be >> sure): >> >> You’re using c3, so presumably you’re using EBS – are you using GP2? >> Which volume sizes? Are they the same between versions? Are you hitting >> your iops limits? Running out of burst tokens? Do you have enhanced >> networking enabled? At load, what part of your system is stressed? Are you >> cpu bound? Are you seeing GC pauses hurt latency? Have you tried changing >> memtable_allocation_type -> offheap objects (available in 2.1, not in 2.0)? >> >> Tuning gc_grace is weird – do you understand what it does? Are you >> overwriting or deleting a lot of data in your test (that’d be unusual)? Are >> you doing a lot of compaction? >> >> >> From: "Peddi, Praveen" >> Reply-To: "user@cassandra.apache.org" >> Date: Wednesday, January 6, 2016 at 11:41 AM >> To: "user@cassandra.apache.org" >> Subject: Slow performance after upgrading from 2.0.9 to 2.1.11 >> >> Hi, >> We have upgraded Cassandra from 2.0.9 to 2.1.11 in our loadtest >> environment with pretty much same yaml settings in both (removed unused >> yaml settings and renamed few others) and we have noticed performance on >> 2.1.11 is worse compared to 2.0.9. After more investigation we found that >> the performance gets worse as we increase replication factor on 2.1.11 >> where as on 2.0.9 performance is more or less same. Has anything >> architecturally changed as far as replication is concerned in 2.1.11? >> >> All googling only suggested 2.1.11 should be FASTER than 2.0.9 so we are >> obviously doing something different. However the client code, load test is >> all identical in both cases. >> >> Details: >> Nodes: 3 ec2 c3.2x large >> R/W Consistency: QUORUM >> Renamed memtable_total_space_in_mb to memtable_heap_space_in_mb and >> removed unused properties from yaml file. >> We run compaction aggressive compaction with low gc_grace (15 mins) but >> this is true for both 2.0.9 and 2.1.11. >> >> As you can see, all p50, p90 and p99 latencies stayed with in 10% >> difference on 2.0.9 when we increased RF from 1 to 3, where as on 2.1.11 >> latencies almost doubled (especially reads are much slower than writes). >> >> # Nodes RF # of rows 2.0.9 2.1.11 >> READ >> P50 P90 P99 P50 P90 P99 >> 3 1 450 306 594 747 425 849 1085 >> 3 3 450 358 634 877 708 1274 2642 >> >> WRITE >> 3 1 10 26 80 179 37 131 196 >> 3 3 10 31 96 184 46 166 468 >> >> Any pointers on how to debug performance issues will be appreciated. >> >> Praveen >> >> >> >> -- >> ---------------------------------- >> Bestdata.be<http://Bestdata.be> >> Optimised ict >> Tel:+32(0)496609576<tel:%2B32%280%29496609576> >> co...@bestdata.be<mailto:co...@bestdata.be> >> ---------------------------------- >> >> >> >> >> -- >> ---------------------------------- >> Bestdata.be<http://Bestdata.be> >> Optimised ict >> Tel:+32(0)496609576<tel:%2B32%280%29496609576> >> co...@bestdata.be<mailto:co...@bestdata.be> >> ---------------------------------- >> >> >> >> >> -- >> Dikang >> >> >> >> >> >> -- >> Dikang >> >> >> > > > -- > Dikang > >
