Local storage is more the just the norm. Unless you have a very good reason you should not be using NFS.
Edward On Fri, Jul 20, 2012 at 4:55 AM, aaron morton <aa...@thelastpickle.com> wrote: > 45 minutes for 90GB is high. > > The odd one out here is using NFS, local storage is the norm. > > I would look into the NFS first, low network IO and low CPU would suggest it > is waiting on disk IO. The simple thing would be to try starting from local > disk and see how much faster it is. Or look at the await time in iostat. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 19/07/2012, at 4:54 PM, Ben Kaehne wrote: > > Good evening, > > I am interested in improving the startup time of our cassandra cluster. > > We have a 3 node cluster (replication factor of 3) in which our application > requires quorum reads and writes to function. > > Each machine is well specced with 24gig of ram, 10 cores, jna enabled etc. > > On each server our keyspace files are so far around 90 Gb (stored on NFS > although I am not seeing signs that we have much network io). This size will > grow in future. > > Our startup time for 1 server at the moment is greater then half an hour (45 > minutes to 50 minutes even) which is putting a risk factor on the resiliance > of our service. I have tried version 1.09 to latest 1.12. > > I do not see too much system utilization while starting either. > > I gazed apon an article suggesting increased speed in 1.2 although when I > set it up, it did not seem to be any faster at all (if not slower). > > I was observing what was happening during startup and I noticed (via > strace), cassandra was doing lots of 8 byte reads from: > > > /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1871-CompressionInfo.db > > /var/lib/cassandra/data/XXXXXX/YYYYYY/XXXXXXX-YYYYYYY-hc-1874-CompressionInfo.db > > Also... Is there someone I can change the 8 byte reads to something greater? > 8 byte reads across NFS is terribly inefficient (and I am guessing the cause > of our terribly slow startup times). > > Regards, > > -- > -Ben > >
