Re: Upgredesstables doing 4K reads
Hi, Sorry no one get back to you yet. Do you still have the issue? It's unclear to me what produces this yet. A few ideas though: We are quite pedantic about OS settings. All nodes got same settings > and C* configuration. Considering this hypothesis, I hope that's 100% true. 2 nodes behaving badly out of 6, makes me think of an unbalanced cluster. Do you use RF=2 there ? do you have wide rows or unbalanced data (partition keys not well distributes)? Could you check and paste the output from nodetool cfstats and nodetool cfhistograms on the most impacting tables ? Could those nodes have hardware issues of some kind ? C*heers, --- Alain Rodriguez - al...@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2016-06-02 13:43 GMT+02:00 Jacek Luczak : > Hi, > > I've got a 6 node C* cluster (all nodes are equal both in OS and HW > setup, they are DL380 Gen9 with Smart Array RAID 50,3 on SAS 15K HDDs) > which has been recently updated from 2.2.5 to 3.5. As part of the > update I've done the upgradesstables. > > On 4 nodes the average request size issued to the block dev was never > higher than 8 (that maps to 4K reads) while on remaining 2 nodes it > was basically always maxed 512 (256K reads). > > Nodes doing 4K reads were pumping max 2K read IOPs while the 2 nodes > never went up above 30 IOPs. > > We are quite pedantic about OS settings. All nodes got same settings > and C* configuration. On all nodes block dev got noop scheduler set > and read ahead aligned with strip size. > > During heavy read workloads we've also noticed that those 4 nodes can > swing up to 10K IOPs to get data from storage, the 2 are much below. > > What can cause such difference? > > -Jacek >
Upgrade from 3.0.6 to 3.7.
Hi, We are short before going in prod with our cassandra cluster. Now I wonder if this maybe (while still not fully in prod) a good moment to switch from the 3.0.x to the new tick-tock versions. On planet cassandra the tick-tock article mentions: “…We do recognize that it will take some time for tick-tock releases to deliver production-level stability, which is why we will continue to deliver 2.2.y and 3.0.y bugfix releases. (But if we do demonstrate that tick-tock can deliver the stability we want, there will be no need for a 4.0.y bugfix series, only 4.x tick-tock.)…" But the article is from June 2015 so I guess it might not be that valid anymore. What are your experiences with the newer versions ? Is there anything to keep in mind before the upgrade from 3.0.x ? BR, Bieniu
Re: Upgrade from 3.0.6 to 3.7.
Hi, If I were you I would stick on 3.0.x. I haven't experienced 3.x to be production ready. We went to prod with 3.5 and I wish we hadn't. /Oskar > On 23 juni 2016, at 10:56, Bienek, Marcin wrote: > > Hi, > > We are short before going in prod with our cassandra cluster. Now I wonder if > this maybe (while still not fully in prod) a good moment to switch from the > 3.0.x to the new tick-tock versions. > On planet cassandra the tick-tock article mentions: > > “…We do recognize that it will take some time for tick-tock releases to > deliver production-level stability, which is why we will continue to deliver > 2.2.y and 3.0.y bugfix releases. (But if we do demonstrate that tick-tock > can deliver the stability we want, there will be no need for a 4.0.y bugfix > series, only 4.x tick-tock.)…" > > But the article is from June 2015 so I guess it might not be that valid > anymore. What are your experiences with the newer versions ? > Is there anything to keep in mind before the upgrade from 3.0.x ? > > BR, > Bieniu
Ring connection timeouts with 2.2.6
Hi, We have a 12 node 2.2.6 ring running in AWS, single DC with RF=3, that is sitting at <25% CPU, doing mostly writes, and not showing any particular long GC times/pauses. By all observed metrics the ring is healthy and performing well. However, we are noticing a pretty consistent number of connection timeouts coming from the messaging service between various pairs of nodes in the ring. The "Connection.TotalTimeouts" meter metric show 100k's of timeouts per minute, usually between two pairs of nodes for several hours at a time. It seems to occur for several hours at a time, then may stop or move to other pairs of nodes in the ring. The metric "Connection.SmallMessageDroppedTasks." will also grow for one pair of the nodes in the TotalTimeouts metric. Looking at the debug log typically shows a large number of messages like the following on one of the nodes: StorageProxy.java:1033 - Skipped writing hint for /172.26.33.177 (ttl 0) We have cross node timeouts enabled, but ntp is running on all nodes and no node appears to have time drift. The network appears to be fine between nodes, with iperf tests showing that we have a lot of headroom. Any thoughts on what to look for? Can we increase thread count/pool sizes for the messaging service? Thanks, Mike -- Mike Heffner Librato, Inc.
Question about sequential repair vs parallel
Cassandra 2.1.12 In the moment of a repair -pr sequential, we are experimenting an exponential increase of number of sstables. For a table lcs. If someone of you guys kowns if theoritically speaking a sequential repair doing more sstable streaming among replicas than a parallel repair? Best regards Jean Carlo "The best way to predict the future is to invent it" Alan Kay
Re: Question about sequential repair vs parallel
Yes, because you keep a snapshot in the meanwhile if I remember correctly. Regards, Stefano On Thu, Jun 23, 2016 at 4:22 PM, Jean Carlo wrote: > Cassandra 2.1.12 > > In the moment of a repair -pr sequential, we are experimenting an > exponential increase of number of sstables. For a table lcs. > > If someone of you guys kowns if theoritically speaking a sequential repair > doing more sstable streaming among replicas than a parallel repair? > > > > Best regards > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay >
Question about hector api documentation
Hello, I'm finding hector java api doc. I searched though google but couldn't find hector api doc. This link is broken also. https://hector-client.github.io/hector/build/html/content/api.html# Can I know the way to get the doc? Thanks. Sungju.
Re: Question about hector api documentation
The very first line README tells the story THIS PROJECT IS NO LONGER ACTIVE But you should be able to generate doc from source code. Regards, Noorul Sungju Hong writes: > Hello, > > I'm finding hector java api doc. > > I searched though google but couldn't find hector api doc. > > This link is broken also. > https://hector-client.github.io/hector/build/html/content/api.html# > > Can I know the way to get the doc? > > Thanks. > > Sungju.
