Concurrent write performance
Folks, I would like to write(insert or update) to a single row in a column family. I have concurrent requests which will write to a single row. Do we see any performance implications because of concurrent writes to a single row where comparator has to sort the columns at the same time? Please share your thoughts. Thanks, Jay
High Read and write through put
Folks, For given situation I am expecting multiple read and write request to a same cluster. What are primary design or configuration consideration we should make? Any thoughts or links to such documentation is appreciated. Thanks, Jay
conditional update or insert
Hi All, On each row I have a column which maintains the timestamp like "lastUpdated" etc. While inserting such row I want to make sure that the row should be only updated if the lastUpdated is older than the new one I am inserting. One way to do this is - Read the record first check the timestamp if newer is latest then update. Since I have higher volume of read and writes load. This additional read will add to it. Any alternative to achieve this? Thanks, Jay
Insert v/s Update performance
Hi Team, I have this 3 node cluster. I am writing data to these node at the rate of 2,000 records/second. What I observed that if I do inserts. (Means records for those keys does not exist, my column family has 0 records to start with) then I have better write performacne, low SSTable count, low pending compaction and write latency is acceptable and CPU utilization on each node between 35% to 85%. When I ran same test but for update this time (means records already exists in Column family with same key), I observed that my SSTable count gone high 3 times. Pending compactions gone high more than 2 times and write latency has gone high too and CPU utilization was almost 92% to 100%. What is a reason of deteriorating Update performance v/s Insert performance. Since this is critical you help is highly appriciated. P.S. I also observed that high number of pending Mutation Stage on my nodetool tpstats. Thanks, Jay
Re: Insert v/s Update performance
Thanks Dean, I have used LCS with sstable_size_in_mb of 15. I have also tried bigger sstable_size_in_mb and observed simillar behavior. Does compaction works differently for update v/s Insert? I belive all keys goes to single SST. What other options I should look into? Thanks, Jay On Tue, Mar 26, 2013 at 6:18 PM, Hiller, Dean wrote: > Most likely compaction kicks in as updates cause duplicated rows in STCS > and compaction causes load that may not have been there before(check your > logs). Also, you can increase the number of nodes in your cluster as well > to better handle the load. > > Later, > Dean > > From: Jay Svc mailto:jaytechg...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Tuesday, March 26, 2013 5:05 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: Insert v/s Update performance > > Hi Team, > > I have this 3 node cluster. I am writing data to these node at the rate of > 2,000 records/second. What I observed that if I do inserts. (Means records > for those keys does not exist, my column family has 0 records to start > with) then I have better write performacne, low SSTable count, low pending > compaction and write latency is acceptable and CPU utilization on each node > between 35% to 85%. > > When I ran same test but for update this time (means records already > exists in Column family with same key), I observed that my SSTable count > gone high 3 times. Pending compactions gone high more than 2 times and > write latency has gone high too and CPU utilization was almost 92% to 100%. > > What is a reason of deteriorating Update performance v/s Insert > performance. Since this is critical you help is highly appriciated. > > P.S. I also observed that high number of pending Mutation Stage on my > nodetool tpstats. > > Thanks, > Jay >
Re: Insert v/s Update performance
Thanks Dean again! My use case is high number of reads and writes out of that I am just focusing on write now. I thought LCS is a suitable for my situation. I tried simillar on STCS and results are same. I ran nodetool for tpstats and MutationStage pending are very high. At the same time the SSTable count and Pending Compaction are high too during my updates. Please find the snapshot of my syslog. INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line 116) OpsCenter.rollups864000,0 INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264) Writing Memtable-InventoryPrice@1051586614(11438914/129587272 serialized/live bytes, 404320 ops) INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java (line 658) 2701 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line 57) Pool NameActive Pending Blocked INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line 72) ReadStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line 72) RequestResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) ReadRepairStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line 72) MutationStage 32 19967 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) ReplicateOnWriteStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) GossipStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line 72) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MigrationStage0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line 72) MemtablePostFlusher 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) FlushWriter 1 1 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line 72) commitlog_archiver0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 72) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 72) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line 77) CompactionManager 127 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,675 StatusLogger.java (line 89) MessagingServicen/a 0,22 INFO [ScheduledTasks:1] 2013-03-26 15:05:53,724 StatusLogger.java (line 99) Cache Type Size Capacity KeysToSave Provider INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 100) KeyCache 142315 2118997 all INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 106) RowCache 0 0 all org.apache.cassandra.cache.SerializingCacheProvider INFO [ScheduledTasks:1] 2013-03-26 15:05:53,725 StatusLogger.java (line 113) ColumnFamilyMemtable ops,data INFO [ScheduledTasks:1] 2013-03-26 15:0 Thanks, Jay On Tue, Mar 26, 2013 at 7:15 PM, Hiller, Dean wrote: > LCS is generally used for high read vs. write ratio though it sounds like > you may be doing a heavy write load instead. LCS will involve more > compactions as you write to the system compared to STCS because LCS is > always trying to keep a 1 to 10 ratio between levels. While LCS will > involve more compaction in general(more I/o, more cpu), I am not sure on > update vs. insert though From what I understand STCS will happily duplicate > rows across SS tables while LCS does not like to do this so as you update > you will constantly compact….well, that is my understanding. Have you > tried STCS out at all? (ps. This is just from what I understand so take > with a grain of salt). > > Also, there are some great tools in the nodetool tool as well so you can > get nodetool compactionstats, etc. etc. and see how backlogged you are in > pending tasks….how many pending? > > Later, > Dean > > From: Jay Svc mailto:jaytechg...@gmail.com>> > Reply-To: &quo
Re: Insert v/s Update performance
Hi Aaron, Thank you for your input. I have been monitoring my GC activities and looking at my Heap, it shows pretty linear activities, without any spikes. When I look at CPU it shows higher utilization while during writes alone. I also expect hevy read traffic. When I tried compaction_throughput_* parameter, I obsered that higher number here in my case gets better CPU utilization and keeps pending compactions pretty low. How this parameter works? I have 3 nodes and 2 core each CPU and I have higher writes. So usually for high *update* and high read situation what parameter we should consider for tuning? Thanks, Jay On Wed, Mar 27, 2013 at 9:55 PM, aaron morton wrote: > * Check for GC activity in the logs > * check the volume the commit log is on to see it it's over utilised. > * check if the dropped messages correlate to compaction, look at the > compaction_* settings in yaml and consider reducing the throughput. > > Like Dean says if you have existing data it will result in more > compaction. You may be able to get a lot of writes through in a clean new > cluster, but it also has to work when compaction and repair are running. > > Cheers > > - > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 27/03/2013, at 1:43 PM, Jay Svc wrote: > > Thanks Dean again! > > My use case is high number of reads and writes out of that I am just > focusing on write now. I thought LCS is a suitable for my situation. I > tried simillar on STCS and results are same. > > I ran nodetool for tpstats and MutationStage pending are very high. At the > same time the SSTable count and Pending Compaction are high too during my > updates. > > Please find the snapshot of my syslog. > > INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line > 116) OpsCenter.rollups864000,0 > INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264) > Writing Memtable-InventoryPrice@1051586614(11438914/129587272 > serialized/live bytes, 404320 ops) > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java > (line 658) 2701 MUTATION messages dropped in last 5000ms > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line > 57) Pool NameActive Pending Blocked > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line > 72) ReadStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line > 72) RequestResponseStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line > 72) ReadRepairStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line > 72) MutationStage32 19967 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line > 72) ReplicateOnWriteStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line > 72) GossipStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line > 72) AntiEntropyStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line > 72) MigrationStage0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line > 72) StreamStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,629 StatusLogger.java (line > 72) MemtablePostFlusher 1 1 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line > 72) FlushWriter 1 1 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line > 72) MiscStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,673 StatusLogger.java (line > 72) commitlog_archiver0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line > 72) InternalResponseStage 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line > 72) HintedHandoff 0 0 0 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,674 StatusLogger.java (line > 77) CompactionManager 127 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,675 StatusLogger.java (line > 89) MessagingServicen/a 0,22 > INFO [ScheduledTasks:1] 2013-03-26 15:05:53,724 StatusLogger.java (line > 99) Cache Type Size > Cap
Re: Insert v/s Update performance
Hi Aaron, I will try to slow down the compaction. Since this is our DEV environment the core problem remains as we just have 2 cores. I will get back in case I see any different situation. Thank you for your inputs. Thanks, Jay On Sun, Mar 31, 2013 at 5:35 AM, aaron morton wrote: > > How this parameter works? I have 3 nodes and 2 core each CPU and I have > higher writes. > It slows down the rate that compaction reads from disk. It reads at bit > then has to take a break and wait until it can read again. > With only 2 cores you will be running into issues when compaction or > repair do their work. > > > So usually for high update and high read situation what parameter we > should consider for tuning? > In this case I think the issue is only having 2 cores. There are > background processing like compaction and repair that have to run when you > system is running. > > Slowing down compaction will reduce it's impact. > > Cheers > > - > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 30/03/2013, at 12:58 AM, Jay Svc wrote: > > > Hi Aaron, > > > > Thank you for your input. I have been monitoring my GC activities and > looking at my Heap, it shows pretty linear activities, without any spikes. > > > > When I look at CPU it shows higher utilization while during writes > alone. I also expect hevy read traffic. > > > > When I tried compaction_throughput_* parameter, I obsered that higher > number here in my case gets better CPU utilization and keeps pending > compactions pretty low. How this parameter works? I have 3 nodes and 2 core > each CPU and I have higher writes. > > > > So usually for high update and high read situation what parameter we > should consider for tuning? > > > > Thanks, > > Jay > > > > > > > > > > > > On Wed, Mar 27, 2013 at 9:55 PM, aaron morton > wrote: > > * Check for GC activity in the logs > > * check the volume the commit log is on to see it it's over utilised. > > * check if the dropped messages correlate to compaction, look at the > compaction_* settings in yaml and consider reducing the throughput. > > > > Like Dean says if you have existing data it will result in more > compaction. You may be able to get a lot of writes through in a clean new > cluster, but it also has to work when compaction and repair are running. > > > > Cheers > > > > - > > Aaron Morton > > Freelance Cassandra Consultant > > New Zealand > > > > @aaronmorton > > http://www.thelastpickle.com > > > > On 27/03/2013, at 1:43 PM, Jay Svc wrote: > > > >> Thanks Dean again! > >> > >> My use case is high number of reads and writes out of that I am just > focusing on write now. I thought LCS is a suitable for my situation. I > tried simillar on STCS and results are same. > >> > >> I ran nodetool for tpstats and MutationStage pending are very high. At > the same time the SSTable count and Pending Compaction are high too during > my updates. > >> > >> Please find the snapshot of my syslog. > >> > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:48,560 StatusLogger.java (line > 116) OpsCenter.rollups864000,0 > >> INFO [FlushWriter:55] 2013-03-26 15:05:48,608 Memtable.java (line 264) > Writing Memtable-InventoryPrice@1051586614(11438914/129587272 > serialized/live bytes, 404320 ops) > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,561 MessagingService.java > (line 658) 2701 MUTATION messages dropped in last 5000ms > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,562 StatusLogger.java (line > 57) Pool NameActive Pending Blocked > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,563 StatusLogger.java (line > 72) ReadStage 0 0 0 > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,568 StatusLogger.java (line > 72) RequestResponseStage 0 0 0 > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line > 72) ReadRepairStage 0 0 0 > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,627 StatusLogger.java (line > 72) MutationStage32 19967 0 > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line > 72) ReplicateOnWriteStage 0 0 0 > >> INFO [ScheduledTasks:1] 2013-03-26 15:05:53,628 StatusLogger.java (line > 72) GossipStage 0 0
Does Memtable resides in Heap?
Hi Team, I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My question is the size of Memtable does it contribute to heap size? or they are part of off-heap? Does bigger Memtable would have impact on GC and overall memory management? I am using DSE 3.0 / Cassandra 1.1.9. Thanks, Jay
Re: Does Memtable resides in Heap?
Thanks Vitor, So as per recommendation its only efficient when heap size is below 8GB. How about when we have more RAM, does that rest of the RAM can be left for OS to make use? How about the bloom filter and index samples, are they part of off-heap? Thank you for your response. Regards, Jay On Thu, Apr 11, 2013 at 10:35 PM, Viktor Jevdokimov < viktor.jevdoki...@adform.com> wrote: > Memtables resides in heap, write rate impacts GC, more writes - more > frequent and longer ParNew GC pauses. > > > From: Jay Svc [mailto:jaytechg...@gmail.com] > Sent: Friday, April 12, 2013 01:03 > To: user@cassandra.apache.org > Subject: Does Memtable resides in Heap? > > Hi Team, > > I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My > question is the size of Memtable does it contribute to heap size? or they > are part of off-heap? > > Does bigger Memtable would have impact on GC and overall memory management? > > I am using DSE 3.0 / Cassandra 1.1.9. > > Thanks, > Jay > > Best regards / Pagarbiai > > Viktor Jevdokimov > Senior Developer > > Email: viktor.jevdoki...@adform.com > Phone: +370 5 212 3063 > Fax: +370 5 261 0453 > > J. Jasinskio 16C, > LT-01112 Vilnius, > Lithuania > > > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. >
Re: Does Memtable resides in Heap?
Thanks Edward.! On Fri, Apr 12, 2013 at 9:46 AM, Edward Capriolo wrote: > This issue describes the design of the arena allocation of memtabes. > https://issues.apache.org/jira/browse/CASSANDRA-2252 > > > On Fri, Apr 12, 2013 at 1:35 AM, Viktor Jevdokimov < > viktor.jevdoki...@adform.com> wrote: > >> Memtables resides in heap, write rate impacts GC, more writes - more >> frequent and longer ParNew GC pauses. >> >> >> From: Jay Svc [mailto:jaytechg...@gmail.com] >> Sent: Friday, April 12, 2013 01:03 >> To: user@cassandra.apache.org >> Subject: Does Memtable resides in Heap? >> >> Hi Team, >> >> I have got this 8GB of RAM out of that 4GB allocated to Java Heap. My >> question is the size of Memtable does it contribute to heap size? or they >> are part of off-heap? >> >> Does bigger Memtable would have impact on GC and overall memory >> management? >> >> I am using DSE 3.0 / Cassandra 1.1.9. >> >> Thanks, >> Jay >> >> Best regards / Pagarbiai >> >> Viktor Jevdokimov >> Senior Developer >> >> Email: viktor.jevdoki...@adform.com >> Phone: +370 5 212 3063 >> Fax: +370 5 261 0453 >> >> J. Jasinskio 16C, >> LT-01112 Vilnius, >> Lithuania >> >> >> >> Disclaimer: The information contained in this message and attachments is >> intended solely for the attention and use of the named addressee and may be >> confidential. If you are not the intended recipient, you are reminded that >> the information remains the property of the sender. You must not use, >> disclose, distribute, copy, print or rely on this e-mail. If you have >> received this message in error, please contact the sender immediately and >> irrevocably delete this message and any copies. >> > >
How to make compaction run faster?
Hi Team, I have a high write traffic to my Cassandra cluster. I experience a very high number of pending compactions. As I expect higher writes, The pending compactions keep increasing. Even when I stop my writes it takes several hours to finishing pending compactions. My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. How can I increase the compaction rate so it will run bit faster to match my write speed? Your inputs are appreciated. Thanks, Jay
Re: How to make compaction run faster?
Hi Edward, Thank you for response. I have tried the following - 1. I have tried various compaction throughput ranging from 16M to 1G. CPU continued to be low and memory between 40% to 50%. I still see compaction more backing. 2. Does concurrent compactors take effect with Leveled compaction? (yaml says this parameter has no effect with LCS) 3. I have tried multi-threaded compaction as well I do not see change in handling the compaction. 4. I have 24 core CPU, have RAID 10 disks and commitlog on SSD and 48GB or RAM with 8GB Heap. Thanks, Jay On Wed, Apr 17, 2013 at 3:26 PM, Edward Capriolo wrote: > three things: > 1) compaction throughput is fairly low (yaml nodetool) > 2) concurrent compactions is fairly low (yaml) > 3) multithreaded compaction might be off in your version > > Try raising these things. Otherwise consider option 4. > > 4)$$$ RAID,RAM > > On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc wrote: > >> Hi Team, >> >> >> >> I have a high write traffic to my Cassandra cluster. I experience a very >> high number of pending compactions. As I expect higher writes, The pending >> compactions keep increasing. Even when I stop my writes it takes several >> hours to finishing pending compactions. >> >> >> My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below >> 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. >> >> >> How can I increase the compaction rate so it will run bit faster to match >> my write speed? >> >> >> Your inputs are appreciated. >> >> >> Thanks, >> >> Jay >> >> >
Re: How to make compaction run faster?
Hi Alexis, Thank you for your response. My commit log is on SSD. which shows me 30 to 40 ms of disk latency. When I ran iostat; I see "await" 26ms to 30 ms for my commit log disk. My CPU is less than 18% used. How I reduce the disk latency for my commit log disk. They are SSDs. Thank you in advance, Jay On Wed, Apr 17, 2013 at 3:58 PM, Alexis Rodríguez < arodrig...@inconcertcc.com> wrote: > :D > > Jay, check if your disk(s) utilization allows you to change the > configuration the way Edward suggest. iostat -xkcd 1 will show you how much > of your disk(s) are in use. > > > > > On Wed, Apr 17, 2013 at 5:26 PM, Edward Capriolo wrote: > >> three things: >> 1) compaction throughput is fairly low (yaml nodetool) >> 2) concurrent compactions is fairly low (yaml) >> 3) multithreaded compaction might be off in your version >> >> Try raising these things. Otherwise consider option 4. >> >> 4)$$$ RAID,RAM> >> >> On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc wrote: >> >>> Hi Team, >>> >>> >>> >>> I have a high write traffic to my Cassandra cluster. I experience a very >>> high number of pending compactions. As I expect higher writes, The pending >>> compactions keep increasing. Even when I stop my writes it takes several >>> hours to finishing pending compactions. >>> >>> >>> My CF is configured with LCS, with sstable_size_mb=20M. My CPU is below >>> 20%, JVM memory usage is between 45%-55%. I am using Cassandra 1.1.9. >>> >>> >>> How can I increase the compaction rate so it will run bit faster to >>> match my write speed? >>> >>> >>> Your inputs are appreciated. >>> >>> >>> Thanks, >>> >>> Jay >>> >>> >> >
Re: How to make compaction run faster?
Hi Alexis, Yes compaction happens on data files - 1. What my disk latency is high for SSDs which been only used for commit log 2. Why my compaction is not catching up with my write traffic in spite of low CPU, low memory and low JVM usage. I am adding more details to this thread. Thanks, Jayant K Kenjale On Wed, Apr 17, 2013 at 6:58 PM, Alexis Rodríguez < arodrig...@inconcertcc.com> wrote: > Jay, > > I believe that compaction occurs on the data directories and not in the > commitlog. > > http://wiki.apache.org/cassandra/MemtableSSTable > > > > > On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc wrote: > >> Hi Alexis, >> >> Thank you for your response. >> >> My commit log is on SSD. which shows me 30 to 40 ms of disk latency. >> >> When I ran iostat; I see "await" 26ms to 30 ms for my commit log disk. My >> CPU is less than 18% used. >> >> How I reduce the disk latency for my commit log disk. They are SSDs. >> >> Thank you in advance, >> Jay >> >> >> On Wed, Apr 17, 2013 at 3:58 PM, Alexis Rodríguez < >> arodrig...@inconcertcc.com> wrote: >> >>> :D >>> >>> Jay, check if your disk(s) utilization allows you to change the >>> configuration the way Edward suggest. iostat -xkcd 1 will show you how much >>> of your disk(s) are in use. >>> >>> >>> >>> >>> On Wed, Apr 17, 2013 at 5:26 PM, Edward Capriolo >>> wrote: >>> >>>> three things: >>>> 1) compaction throughput is fairly low (yaml nodetool) >>>> 2) concurrent compactions is fairly low (yaml) >>>> 3) multithreaded compaction might be off in your version >>>> >>>> Try raising these things. Otherwise consider option 4. >>>> >>>> 4)$$$ RAID,RAM>>> >>>> >>>> On Wed, Apr 17, 2013 at 4:01 PM, Jay Svc wrote: >>>> >>>>> Hi Team, >>>>> >>>>> >>>>> >>>>> I have a high write traffic to my Cassandra cluster. I experience a >>>>> very high number of pending compactions. As I expect higher writes, The >>>>> pending compactions keep increasing. Even when I stop my writes it takes >>>>> several hours to finishing pending compactions. >>>>> >>>>> >>>>> My CF is configured with LCS, with sstable_size_mb=20M. My CPU is >>>>> below 20%, JVM memory usage is between 45%-55%. I am using Cassandra >>>>> 1.1.9. >>>>> >>>>> >>>>> How can I increase the compaction rate so it will run bit faster to >>>>> match my write speed? >>>>> >>>>> >>>>> Your inputs are appreciated. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Jay >>>>> >>>>> >>>> >>> >> >
Re: How to make compaction run faster?
Hi Aaron, Alexis, Thanks for reply, Please find some more details below. *Core problems:* Compaction is taking longer time to finish. So it will affect my reads. I have more CPU and memory, want to utilize that to speed up the compaction process. *Parameters used:* 1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) 2. Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) 3. Concurrent write: 196 (tried from 32 to 296) 4. Concurrent compactors: 72 (tried disabling to making it 172) 5. Multithreaded compaction: true (tried both true and false) 6. Compaction strategy: LCS (tried STCS as well) 7. Memtable total space in mb: 4096 MB (tried default and some other params too) Note: I have tried almost all permutation combination of these parameters. *Observations: * I ran test for 1.15 hrs with writes at the rate of 21000 records/sec(total 60GB data during 1.15 hrs). After I stopped the test compaction took additional 1.30 hrs to finish compaction, that reduced the SSTable count from 170 to 17. CPU(24 cores): almost 80% idle during the run JVM: 48G RAM, 8G Heap, (3G to 5G heap used) Pending Writes: sometimes high spikes for small amount of time otherwise pretty flat Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog disks. Please find the iostat of some of 3 different boxes in my cluster. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.11 0.59 0.01 0.00 97.09 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 0.32<24.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32> dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.20 1.12 0.52 0.01 0.00 97.14 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. -bash-4.1$ iostat -xkcd Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013 _x86_64_ (24 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.15 1.13 0.52 0.01 0.00 97.19 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.02 429.97 9.28 7.29 1176.81 1749.00 353.12 0.07 4.10 0.55 0.91 sdb 0.00 173.65 0.08 3.09 10.50 706.96 452.25 0.09 27.23 0.99 0.31 dm-0 0.00 0.00 0.04 0.79 0.82 3.16 9.61 0.00 1.54 0.27 0.02 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.68 0.63 0.00 dm-2 0.00 0.00 0.08 176.74 10.50 706.96 8.12 3.46 19.53 0.02 0.31 dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.85 0.83 0.00 dm-4 0.00 0.00 9.26 436.46 1175.98 1745.84 13.11 0.03 0.03 0.02 0.89 Thanks, Jay On Thu, Apr 18, 2013 at 2:50 AM, aaron morton wrote: > > I believe that compaction occurs on the data directories and not in the > commitlog. > Yes, compaction only works on the data files. > > > When I ran iostat; I see "await" 26ms to 30 ms for my commit log disk. > My CPU is less than 18% used. > > > > How I reduce the disk latency for my commit log disk. They are SSDs. > That does not sound right. Can you include the output from iostat for the > commit log and data volumes. Also some information on how many writes you > are processing the the size of rows as well. > > Cheers > > - > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 18/04/2013, at 11:58 AM, Alexis Rodríguez > wrote: > > > Jay, > > > > I believe that compaction occurs on the data directories and not in the > commitlog. > > > > http://wiki.apache.org/cassandra/MemtableSSTable > > > > > > > > > > On Wed, Apr 17, 2013 at 7:58 PM, Jay Svc wrote: > > Hi Alexis, > > > > Thank you for your response. > > > > My commit log is on SSD. which shows me 30 to 40 ms of disk latency. > > > > When I ran iostat; I see "await" 26ms to 30 ms for my commit log disk. > My CPU is less than 18%
Re: How to make compaction run faster?
Looks like formatting is bit messed up. Please let me know if you want the same in clean format. Thanks, Jay On Thu, Apr 18, 2013 at 12:05 PM, Jay Svc wrote: > Hi Aaron, Alexis, > > Thanks for reply, Please find some more details below. > > *Core problems:* Compaction is taking longer time to finish. So it will > affect my reads. I have more CPU and memory, want to utilize that to speed > up the compaction process. > *Parameters used:* > >1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) >2. Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) >3. Concurrent write: 196 (tried from 32 to 296) >4. Concurrent compactors: 72 (tried disabling to making it 172) >5. Multithreaded compaction: true (tried both true and false) >6. Compaction strategy: LCS (tried STCS as well) >7. Memtable total space in mb: 4096 MB (tried default and some other >params too) > > Note: I have tried almost all permutation combination of these parameters. > *Observations: * > I ran test for 1.15 hrs with writes at the rate of 21000 records/sec(total > 60GB data during 1.15 hrs). After I stopped the test > compaction took additional 1.30 hrs to finish compaction, that reduced the > SSTable count from 170 to 17. > CPU(24 cores): almost 80% idle during the run > JVM: 48G RAM, 8G Heap, (3G to 5G heap used) > Pending Writes: sometimes high spikes for small amount of time otherwise > pretty flat > Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog > disks. > Please find the iostat of some of 3 different boxes in my cluster. > -bash-4.1$ iostat -xkcd > Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ > (24 CPU) > avg-cpu: %user %nice %system %iowait %steal %idle > 1.20 1.11 0.59 0.01 0.00 97.09 > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm > %util > sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 > sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 > 0.32<24.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32> > dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 > dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 > dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 > dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 > -bash-4.1$ iostat -xkcd > Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ > (24 CPU) > avg-cpu: %user %nice %system %iowait %steal %idle > 1.20 1.12 0.52 0.01 0.00 97.14 > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc > sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. > sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. > dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. > dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. > dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. > dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. > -bash-4.1$ iostat -xkcd > Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013 _x86_64_ > (24 CPU) > avg-cpu: %user %nice %system %iowait %steal %idle > 1.15 1.13 0.52 0.01 0.00 97.19 > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm > %util > sda 0.02 429.97 9.28 7.29 1176.81 1749.00 353.12 0.07 4.10 0.55 0.91 > sdb 0.00 173.65 0.08 3.09 10.50 706.96 452.25 0.09 27.23 0.99 0.31 > dm-0 0.00 0.00 0.04 0.79 0.82 3.16 9.61 0.00 1.54 0.27 0.02 > dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.68 0.63 0.00 > dm-2 0.00 0.00 0.08 176.74 10.50 706.96 8.12 3.46 19.53 0.02 0.31 > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.85 0.83 0.00 > dm-4 0.00 0.00 9.26 436.46 1175.98 1745.84 13.11 0.03 0.03 0.02 0.89 > Thanks, > Jay > > > On Thu, Apr 18, 2013 at 2:50 AM, aaron morton wrote: > >> > I believe that compaction occurs on the data directories and not in the >> commitlog. >> Yes, compaction only works on the data files. >> >> > When I ran iostat; I see "await" 26ms to 30 ms for my commit log disk. >> My CPU is less than 18% used. >> > >> > How I reduce the disk latency for my commit log disk. They are SSDs. >> That does not sound right. Can you include the output from iostat for the >> commit log and data volumes. Also some information on how many writes you >> are processing the the size of rows as well. >> >> Cheers >> >> - >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >&
Re: How to make compaction run faster?
By the way the compaction and commit log disk latency, these are two seperate problems I see. The important one is compaction problem, How I can speed that up? Thanks, Jay On Thu, Apr 18, 2013 at 12:07 PM, Jay Svc wrote: > Looks like formatting is bit messed up. Please let me know if you want the > same in clean format. > > Thanks, > Jay > > > On Thu, Apr 18, 2013 at 12:05 PM, Jay Svc wrote: > >> Hi Aaron, Alexis, >> >> Thanks for reply, Please find some more details below. >> >> *Core problems:* Compaction is taking longer time to finish. So it will >> affect my reads. I have more CPU and memory, want to utilize that to speed >> up the compaction process. >> *Parameters used:* >> >>1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) >>2. Compaction throughput mb per sec: 250MB (tried from 16MB to 640MB) >>3. Concurrent write: 196 (tried from 32 to 296) >>4. Concurrent compactors: 72 (tried disabling to making it 172) >>5. Multithreaded compaction: true (tried both true and false) >>6. Compaction strategy: LCS (tried STCS as well) >>7. Memtable total space in mb: 4096 MB (tried default and some other >>params too) >> >> Note: I have tried almost all permutation combination of these parameters. >> *Observations: * >> I ran test for 1.15 hrs with writes at the rate of 21000 >> records/sec(total 60GB data during 1.15 hrs). After I stopped the test >> compaction took additional 1.30 hrs to finish compaction, that reduced >> the SSTable count from 170 to 17. >> CPU(24 cores): almost 80% idle during the run >> JVM: 48G RAM, 8G Heap, (3G to 5G heap used) >> Pending Writes: sometimes high spikes for small amount of time otherwise >> pretty flat >> Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog >> disks. >> Please find the iostat of some of 3 different boxes in my cluster. >> -bash-4.1$ iostat -xkcd >> Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 _x86_64_ >> (24 CPU) >> avg-cpu: %user %nice %system %iowait %steal %idle >> 1.20 1.11 0.59 0.01 0.00 97.09 >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm >> %util >> sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 >> sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 >> 0.32<24.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32> >> dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 >> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 >> dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 >> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 >> dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 >> -bash-4.1$ iostat -xkcd >> Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 _x86_64_ >> (24 CPU) >> avg-cpu: %user %nice %system %iowait %steal %idle >> 1.20 1.12 0.52 0.01 0.00 97.14 >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc >> sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. >> sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. >> dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. >> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. >> dm-2 0.00 0.00 0.08 175.93 10.52 703.74 8.12 3.13 17.78 0. >> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 1.14 0. >> dm-4 0.00 0.00 9.19 427.55 1166.91 1710.21 13.18 4.67 10.65 0. >> -bash-4.1$ iostat -xkcd >> Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-1) 04/18/2013 _x86_64_ >> (24 CPU) >> avg-cpu: %user %nice %system %iowait %steal %idle >> 1.15 1.13 0.52 0.01 0.00 97.19 >> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm >> %util >> sda 0.02 429.97 9.28 7.29 1176.81 1749.00 353.12 0.07 4.10 0.55 0.91 >> sdb 0.00 173.65 0.08 3.09 10.50 706.96 452.25 0.09 27.23 0.99 0.31 >> dm-0 0.00 0.00 0.04 0.79 0.82 3.16 9.61 0.00 1.54 0.27 0.02 >> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.68 0.63 0.00 >> dm-2 0.00 0.00 0.08 176.74 10.50 706.96 8.12 3.46 19.53 0.02 0.31 >> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.85 0.83 0.00 >> dm-4 0.00 0.00 9.26 436.46 1175.98 1745.84 13.11 0.03 0.03 0.02 0.89 >> Thanks, >> Jay >> >> >> On Thu, Apr 18, 2013 at 2:50 AM, aaron morton wrote: >> >>> > I believe that compaction occurs on the data directories and not in >>> the commitlog. >>> Yes, compaction only works on the data files. >>> >>> > When I ran iostat; I see "await" 26ms to 30 ms fo
Re: How to make compaction run faster?
Hi Alexis, Thank you so much for inputs. Let me try those suggested options. I will also look forward to see if you have any more suggestions on our Compaction. Thanks, Jay On Thu, Apr 18, 2013 at 1:03 PM, Alexis Rodríguez < arodrig...@inconcertcc.com> wrote: > Jay, > > await, according to iostat's man page it is the time of a request to the > disk to get served. You may try changing the io scheduler. I've read that > noop it's recommended for SSDs, you can check here http://goo.gl/XMiIA > > Regarding compaction, a week ago we had serious problems with compaction > in a test machine, solved by changing from openjdk 1.6 to sun-jdk 1.6. > > > > > On Thu, Apr 18, 2013 at 2:08 PM, Jay Svc wrote: > >> By the way the compaction and commit log disk latency, these are two >> seperate problems I see. >> >> The important one is compaction problem, How I can speed that up? >> >> Thanks, >> Jay >> >> >> On Thu, Apr 18, 2013 at 12:07 PM, Jay Svc wrote: >> >>> Looks like formatting is bit messed up. Please let me know if you want >>> the same in clean format. >>> >>> Thanks, >>> Jay >>> >>> >>> On Thu, Apr 18, 2013 at 12:05 PM, Jay Svc wrote: >>> >>>> Hi Aaron, Alexis, >>>> >>>> Thanks for reply, Please find some more details below. >>>> >>>> *Core problems:* Compaction is taking longer time to finish. So it >>>> will affect my reads. I have more CPU and memory, want to utilize that to >>>> speed up the compaction process. >>>> *Parameters used:* >>>> >>>>1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) >>>>2. Compaction throughput mb per sec: 250MB (tried from 16MB to >>>>640MB) >>>>3. Concurrent write: 196 (tried from 32 to 296) >>>>4. Concurrent compactors: 72 (tried disabling to making it 172) >>>>5. Multithreaded compaction: true (tried both true and false) >>>>6. Compaction strategy: LCS (tried STCS as well) >>>>7. Memtable total space in mb: 4096 MB (tried default and some >>>>other params too) >>>> >>>> Note: I have tried almost all permutation combination of these >>>> parameters. >>>> *Observations: * >>>> I ran test for 1.15 hrs with writes at the rate of 21000 >>>> records/sec(total 60GB data during 1.15 hrs). After I stopped the test >>>> compaction took additional 1.30 hrs to finish compaction, that reduced >>>> the SSTable count from 170 to 17. >>>> CPU(24 cores): almost 80% idle during the run >>>> JVM: 48G RAM, 8G Heap, (3G to 5G heap used) >>>> Pending Writes: sometimes high spikes for small amount of time >>>> otherwise pretty flat >>>> Aaron, Please find the iostat below: the sdb and dm-2 are the >>>> commitlog disks. >>>> Please find the iostat of some of 3 different boxes in my cluster. >>>> -bash-4.1$ iostat -xkcd >>>> Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 >>>> _x86_64_ (24 CPU) >>>> avg-cpu: %user %nice %system %iowait %steal %idle >>>> 1.20 1.11 0.59 0.01 0.00 97.09 >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz awaitsvctm >>>> %util >>>> sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 >>>> sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 >>>> 0.32<24.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32> >>>> dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 >>>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 >>>> dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 >>>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 >>>> dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 >>>> -bash-4.1$ iostat -xkcd >>>> Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 >>>> _x86_64_ (24 CPU) >>>> avg-cpu: %user %nice %system %iowait %steal %idle >>>> 1.20 1.12 0.52 0.01 0.00 97.14 >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc >>>> sda 0.01 421.17 9.22 7.43 1167.81 1714.38 346.10 0.07 3.99 0. >>>> sdb 0.00 172.68 0.08 3.26 10.52 703.74 427.79 0.08 25.01 0. >>>> dm-0 0.00 0.00 0.04 1.04 0.89 4.16 9.34 0.00 2.58 0. >>>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.77 0. >&g
Re: How to make compaction run faster?
Thanks Aaron, Please find answers to your questions. 1. I started test with default parameters the compaction is backing up. So went for various options. 2. The data is on RAID10. 3. As I watched Disk latency on DSE Opscenter as well as on iostat the await is always 35 to 40 ms for longer period of time during the test. (which probably gives me high write latency on client side) Do you think this could contribute to slowing down the compaction? probably not..! So Aaron, I am trying to understand - You are suggesting to go back to STCS and increase the compaction_throughput step by step to see if compaction catch up with write traffic? Thank you for your inputs. Regards, Jay On Thu, Apr 18, 2013 at 1:52 PM, aaron morton wrote: > > Parameters used: > > • SSTable size: 500MB (tried various sizes from 20MB to 1GB) > > • Compaction throughput mb per sec: 250MB (tried from 16MB to > 640MB) > > • Concurrent write: 196 (tried from 32 to 296) > > • Concurrent compactors: 72 (tried disabling to making it 172) > > • Multithreaded compaction: true (tried both true and false) > > • Compaction strategy: LCS (tried STCS as well) > > • Memtable total space in mb: 4096 MB (tried default and some > other params too) > I would restore to default settings before I did anything else. > > > Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog > disks. > > Please find the iostat of some of 3 different boxes in my cluster. > > What is the data on ? > It's important to call iostat with a period and watch the await / queue > size of time. Not just view a snapshot. > > I would go back to STS with default settings, and ramp up the throughput > until compaction cannot keep up. Then increase the throughout and see how > that works. Then increase throughput again and see what happens. > > Cheers > > > - > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 19/04/2013, at 5:05 AM, Jay Svc wrote: > > > Hi Aaron, Alexis, > > > > Thanks for reply, Please find some more details below. > > > > Core problems: Compaction is taking longer time to finish. So it will > affect my reads. I have more CPU and memory, want to utilize that to speed > up the compaction process. > > Parameters used: > > • SSTable size: 500MB (tried various sizes from 20MB to 1GB) > > • Compaction throughput mb per sec: 250MB (tried from 16MB to > 640MB) > > • Concurrent write: 196 (tried from 32 to 296) > > • Concurrent compactors: 72 (tried disabling to making it 172) > > • Multithreaded compaction: true (tried both true and false) > > • Compaction strategy: LCS (tried STCS as well) > > • Memtable total space in mb: 4096 MB (tried default and some > other params too) > > Note: I have tried almost all permutation combination of these > parameters. > > Observations: > > I ran test for 1.15 hrs with writes at the rate of 21000 > records/sec(total 60GB data during 1.15 hrs). After I stopped the test > > compaction took additional 1.30 hrs to finish compaction, that reduced > the SSTable count from 170 to 17. > > CPU(24 cores): almost 80% idle during the run > > JVM: 48G RAM, 8G Heap, (3G to 5G heap used) > > Pending Writes: sometimes high spikes for small amount of time otherwise > pretty flat > > Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog > disks. > > Please find the iostat of some of 3 different boxes in my cluster. > > -bash-4.1$ iostat -xkcd > > Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 > _x86_64_ (24 CPU) > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.20 1.11 0.59 0.01 0.00 97.09 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm > %util > > sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 > > sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 0.32 > > dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 > > dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 > > dm-2 0.00 0.00 0.08 175.72 10.76 702.89 8.12 3.26 18.49 0.02 0.32 > > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 7.97 0.00 0.83 0.62 0.00 > > dm-4 0.00 0.00 8.99 422.89 1141.87 1691.55 13.12 4.64 10.71 0.02 0.90 > > -bash-4.1$ iostat -xkcd > > Linux 2.6.32-358.2.1.el6.x86_64 (ndc-epod014-dl380-1) 04/18/2013 > _x86_64_ (24 CPU) > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.20 1.12 0.52 0.01 0.00 97.14 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svc
Re: How to make compaction run faster?
Hi Wei, Thank you for your reply. Yes, I observed that all the concurrent_compactors and multithreaded_compaction has no effect on LCS. I also tried with large SSTable size it helped keeping the SSTable count low so keeping the pending compaction low. But in spite I have more CPU, I am not able to utilize it to make compaction faster. Compaction takes few hours to complete. By the way are you using DSE 3.0+ or community edition? How can we use Cassandra 1.2. Its not supported by DSE yet. Thanks, Jayant K Kenjale On Thu, Apr 18, 2013 at 1:25 PM, Wei Zhu wrote: > We have tried very hard to speed up lcs on 1.1.6 with no luck. It seems to > be single threaded and not much parallelism you can achieve. 1.2 does come > with parallel lcs which should help. > One more thing to try is to enlarge the sstable size which will reduce the > number of SSTable. It *might* help the lcs. > > > -Wei > -- > *From: *"Alexis Rodríguez" > *To: *user@cassandra.apache.org > *Sent: *Thursday, April 18, 2013 11:03:13 AM > *Subject: *Re: How to make compaction run faster? > > > Jay, > > await, according to iostat's man page it is the time of a request to the > disk to get served. You may try changing the io scheduler. I've read that > noop it's recommended for SSDs, you can check here http://goo.gl/XMiIA > > Regarding compaction, a week ago we had serious problems with compaction > in a test machine, solved by changing from openjdk 1.6 to sun-jdk 1.6. > > > > > On Thu, Apr 18, 2013 at 2:08 PM, Jay Svc wrote: > >> By the way the compaction and commit log disk latency, these are two >> seperate problems I see. >> >> The important one is compaction problem, How I can speed that up? >> >> Thanks, >> Jay >> >> >> On Thu, Apr 18, 2013 at 12:07 PM, Jay Svc wrote: >> >>> Looks like formatting is bit messed up. Please let me know if you want >>> the same in clean format. >>> >>> Thanks, >>> Jay >>> >>> >>> On Thu, Apr 18, 2013 at 12:05 PM, Jay Svc wrote: >>> >>>> Hi Aaron, Alexis, >>>> >>>> Thanks for reply, Please find some more details below. >>>> >>>> *Core problems:* Compaction is taking longer time to finish. So it >>>> will affect my reads. I have more CPU and memory, want to utilize that to >>>> speed up the compaction process. >>>> *Parameters used:* >>>> >>>>1. SSTable size: 500MB (tried various sizes from 20MB to 1GB) >>>>2. Compaction throughput mb per sec: 250MB (tried from 16MB to >>>>640MB) >>>>3. Concurrent write: 196 (tried from 32 to 296) >>>>4. Concurrent compactors: 72 (tried disabling to making it 172) >>>>5. Multithreaded compaction: true (tried both true and false) >>>>6. Compaction strategy: LCS (tried STCS as well) >>>>7. Memtable total space in mb: 4096 MB (tried default and some >>>>other params too) >>>> >>>> Note: I have tried almost all permutation combination of these >>>> parameters. >>>> *Observations: * >>>> I ran test for 1.15 hrs with writes at the rate of 21000 >>>> records/sec(total 60GB data during 1.15 hrs). After I stopped the test >>>> compaction took additional 1.30 hrs to finish compaction, that reduced >>>> the SSTable count from 170 to 17. >>>> CPU(24 cores): almost 80% idle during the run >>>> JVM: 48G RAM, 8G Heap, (3G to 5G heap used) >>>> Pending Writes: sometimes high spikes for small amount of time >>>> otherwise pretty flat >>>> Aaron, Please find the iostat below: the sdb and dm-2 are the >>>> commitlog disks. >>>> Please find the iostat of some of 3 different boxes in my cluster. >>>> -bash-4.1$ iostat -xkcd >>>> Linux 2.6.32-358.2.1.el6.x86_64 (edc-epod014-dl380-3) 04/18/2013 >>>> _x86_64_ (24 CPU) >>>> avg-cpu: %user %nice %system %iowait %steal %idle >>>> 1.20 1.11 0.59 0.01 0.00 97.09 >>>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz awaitsvctm >>>> %util >>>> sda 0.03 416.56 9.00 7.08 1142.49 1694.55 352.88 0.07 4.08 0.57 0.92 >>>> sdb 0.00 172.38 0.08 3.34 10.76 702.89 416.96 0.09 24.84 0.94 >>>> 0.32<24.84%C2%A0%C2%A0%200.94%C2%A0%C2%A0%200.32> >>>> dm-0 0.00 0.00 0.03 0.75 0.62 3.00 9.24 0.00 1.45 0.33 0.03 >>>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.74 0.68 0.00 >>>> dm-2
Re: How to make compaction run faster?
Thanks Aaron, The parameters I tried above are set one at a time, based on what I observed, the problem at the core is that "can compaction catch up with write speed". I have gone up to 30,000 to 35000 writes per second. I do not see number of writes a much issue either. I see compaction is not catching up with a write speed is an issue, in spite of I have more CPU and memory. Because over the period of time, we will see growing number of pending compactions, as write continues, that will degrade my read performance. Do you think STCS is compaction strategy to speed up compaction? What is a good approach when we have greater number of reads and writes, so compaction catch up with the write speed? Thank you in advance. Jay On Sun, Apr 21, 2013 at 1:43 PM, aaron morton wrote: > You are suggesting to go back to STCS and increase the > compaction_throughput step by step to see if compaction catch up with write > traffic? > > As a personal approach, when so many config settings are changed it > becomes impossible to understand cause and effect. So I try to return to a > known base line and then make changes. > > As I watched Disk latency on DSE Opscenter as well as on iostat the await > is always 35 to 40 ms for longer period of time during the test. > > You previously said this was the await on the commit log. > What is the queue size ? > > The problem sounds like IO is not keeping up, moving to STS will reduce > the IO. Levelled Compaction is designed to reduce the number os SSTables in > a read, not to do compaction faster. > > At some point you may be writing too fast for the nodes. I'm not sure if > you have discussed the level of writes going through the system. Get > something that works and then make one change at a time until it does not. > You should then be able to say "The system can handle X writes of Y size > per second, but after that compaction cannot keep up." > > Cheers > > - > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 19/04/2013, at 7:16 AM, Jay Svc wrote: > > Thanks Aaron, > > Please find answers to your questions. > > 1. I started test with default parameters the compaction is backing up. So > went for various options. > 2. The data is on RAID10. > 3. As I watched Disk latency on DSE Opscenter as well as on iostat the > await is always 35 to 40 ms for longer period of time during the test. > (which probably gives me high write latency on client side) Do you think > this could contribute to slowing down the compaction? probably not..! > > So Aaron, I am trying to understand - > You are suggesting to go back to STCS and increase the > compaction_throughput step by step to see if compaction catch up with write > traffic? > > Thank you for your inputs. > > Regards, > Jay > > > On Thu, Apr 18, 2013 at 1:52 PM, aaron morton wrote: > >> > Parameters used: >> > • SSTable size: 500MB (tried various sizes from 20MB to 1GB) >> > • Compaction throughput mb per sec: 250MB (tried from 16MB to >> 640MB) >> > • Concurrent write: 196 (tried from 32 to 296) >> > • Concurrent compactors: 72 (tried disabling to making it 172) >> > • Multithreaded compaction: true (tried both true and false) >> > • Compaction strategy: LCS (tried STCS as well) >> > • Memtable total space in mb: 4096 MB (tried default and some >> other params too) >> I would restore to default settings before I did anything else. >> >> > Aaron, Please find the iostat below: the sdb and dm-2 are the commitlog >> disks. >> > Please find the iostat of some of 3 different boxes in my cluster. >> >> What is the data on ? >> It's important to call iostat with a period and watch the await / queue >> size of time. Not just view a snapshot. >> >> I would go back to STS with default settings, and ramp up the throughput >> until compaction cannot keep up. Then increase the throughout and see how >> that works. Then increase throughput again and see what happens. >> >> Cheers >> >> >> - >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 19/04/2013, at 5:05 AM, Jay Svc wrote: >> >> > Hi Aaron, Alexis, >> > >> > Thanks for reply, Please find some more details below. >> > >> > Core problems: Compaction is taking longer time to finish. So it will >> affect my reads. I have more CPU and memory, want to utilize that to speed &g
Commitlog files not getting deleted
Hi Users, In our cluster, the commit log is getting filled up as write progresses. It is noticed that once the commit log is flushed to SSTable the commit log files are not removed/deleted. The result of that the commit log volume is getting filled with commit log files. Any reason why commit log is not cleared even after successful flush. Thanks, Jayant K Kenjale
Re: Commitlog files not getting deleted
its DSE 3.1 Cassandra 2.1 On Thu, Aug 22, 2013 at 10:28 AM, Robert Coli wrote: > On Thu, Aug 22, 2013 at 10:24 AM, Jay Svc wrote: > >> In our cluster, the commit log is getting filled up as write progresses. >> It is noticed that once the commit log is flushed to SSTable the commit log >> files are not removed/deleted. The result of that the commit log volume is >> getting filled with commit log files. >> > > What version of Cassandra? There are some older versions with bugs like > this, but haven't heard this symptom in a while... > > =Rob >
