On Mon, Feb 13, 2012 at 7:48 PM, Peter Schuller <peter.schul...@infidyne.com > wrote:
> > Yep - I've been looking at these - I don't see anything in iostat/dstat > etc > > that point strongly to a problem. There is quite a bit of I/O load, but > it > > looks roughly uniform on slow and fast instances of the queries. The last > > compaction ran 4 days ago - which was before I started seeing variable > > performance > > [snip] > > > I now why it is slow - it's clearly I/O bound. I am trying to hunt down > why > > it is sometimes much faster even though I have (tried) to replicate the > > same conditions > > What does clearly I/O bound mean, and what is "quite a bit" of I/O > load? the servers spending >50% of the time in io-wait > In general, if you have queries that come in at some rate that > is determined by outside sources (rather than by the time the last > query took to execute), That's an interesting approach - is that likely to give close to optimal performance ? > you will typically either get more queries > than your cluster can take, or fewer. If fewer, there is a > non-trivially sized grey area where overall I/O throughput needed is > lower than that available, but the closer you are to capacity the more > often requests have to wait for other I/O to complete, for purely > statistical reasons. > > If you're running close to maximum capacity, it would be expected that > the variation in query latency is high. > That may well explain it - I'll have to think about what that means for our use case as load will be extremely bursty > > That said, if you're seeing consistently bad latencies for a while > where you sometimes see consistently good latencies, that sounds > different but would hopefully be observable somehow. > > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com) > -- *Franc Carter* | Systems architect | Sirca Ltd <marc.zianideferra...@sirca.org.au> franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
