Hi Edward, This could have been a valid case here but if hotspots indeed existed then along with really high disk io , the node should have been doing proportionate high network io as well. - higher queries per second as well.
But from the output shared by Benjamin that doesnt appear to be the case and things look balanced. Regards, On Sun, Feb 19, 2017 at 7:47 PM, Edward Capriolo <edlinuxg...@gmail.com> wrote: > > > On Sat, Feb 18, 2017 at 3:35 PM, Benjamin Roth <benjamin.r...@jaumo.com> > wrote: > >> We are talking about a read IO increase of over 2000% with 512 tokens >> compared to 256 tokens. 100% increase would be linear which would be >> perfect. 200% would even okay, taking the RAM/Load ratio for caching into >> account. But > 20x the read IO is really incredible. >> The nodes are configured with puppet, they share the same roles and no >> manual "optimizations" are applied. So I can't imagine, a different >> configuration is responsible for it. >> >> 2017-02-18 21:28 GMT+01:00 Benjamin Roth <benjamin.r...@jaumo.com>: >> >>> This is status of the largest KS of these both nodes: >>> UN 10.23.71.10 437.91 GiB 512 49.1% >>> 2679c3fa-347e-4845-bfc1-c4d0bc906576 RAC1 >>> UN 10.23.71.9 246.99 GiB 256 28.3% >>> 2804ef8a-26c8-4d21-9e12-01e8b6644c2f RAC1 >>> >>> So roughly as expected. >>> >>> 2017-02-17 23:07 GMT+01:00 kurt greaves <k...@instaclustr.com>: >>> >>>> what's the Owns % for the relevant keyspace from nodetool status? >>>> >>> >>> >>> >>> -- >>> Benjamin Roth >>> Prokurist >>> >>> Jaumo GmbH · www.jaumo.com >>> Wehrstraße 46 · 73035 Göppingen · Germany >>> Phone +49 7161 304880-6 <07161%203048806> · Fax +49 7161 304880-1 >>> <07161%203048801> >>> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >>> >> >> >> >> -- >> Benjamin Roth >> Prokurist >> >> Jaumo GmbH · www.jaumo.com >> Wehrstraße 46 · 73035 Göppingen · Germany >> Phone +49 7161 304880-6 <+49%207161%203048806> · Fax +49 7161 304880-1 >> <+49%207161%203048801> >> AG Ulm · HRB 731058 · Managing Director: Jens Kammerer >> > > When I read articles like this: > > http://www.doanduyhai.com/blog/?p=1930 > > And see the word hot-spot. > > "Another performance consideration worth mentioning is hot-spot. Similar > to manual denormalization, if your view partition key is chosen poorly, > you’ll end up with hot spots in your cluster. A simple example with our > *user* table is to create a materialized > > *view user_by_gender"It leads me to ask a question back: What can you say > about hotspots in your data? Even if your nodes had the identical number of > tokens this autho seems to suggesting that you still could have hotspots. > Maybe the issue is you have a hotspot 2x hotspots, or your application has > a hotspot that would be present even with perfect token balancing.* > >
