I performed this kernel upgrade (to 3.19.30) over the weekend on my cluster, and my before / after benchmarks were very close to each other, about 500MB/s each.
On Tue, Oct 6, 2015 at 3:15 PM, Nick Fisk <n...@fisk.me.uk> wrote: > I'm wondering if you are hitting the "bug" with the readahead changes? > > I know the changes to limit readahead to 2MB was introduced in 3.15, but I > don't know if it was back ported into 3.13 or not. I have a feeling this > may > also limit maximum request size to 2MB as well. > > If you look in iostat do you see different request sizes between the two > kernels? > > There is a 4.2 kernel with the readahead change reverted, it might be worth > testing it. > > > http://gitbuilder.ceph.com/kernel-deb-precise-x86_64-basic/ref/ra-bring-back > / > > > > > -----Original Message----- > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > > MailingLists - EWS > > Sent: 06 October 2015 18:12 > > To: ceph-users@lists.ceph.com > > Subject: Re: [ceph-users] Poor Read Performance with Ubuntu 14.04 LTS > > 3.19.0-30 Kernel > > > > > Hi, > > > > > > Very interesting! Did you upgrade the kernel on both the OSDs and > > > clients > > or > > > just some of them? I remember there were some kernel performance > > > regressions a little while back. You might try running perf during > > > your > > tests > > > and look for differences. Also, iperf might be worth trying to see if > > it's a > > > network regression. > > > > > > I also have a script that compares output from sysctl which might be > > > worth trying to see if any defaults changes. > > > > > > https://github.com/ceph/cbt/blob/master/tools/compare_sysctl.py > > > > > > basically just save systctl -a with both kernels and pass them as > > arguments to > > > the python script. > > > > > > Mark > > > > Mark, > > > > The testing was done with 3.19 on the client with 3.13 on the OSD nodes > > using "rados bench -p bench 50 seq" with an initial "rados bench -p bench > 50 > > write --no-cleanup". We suspected the network as well and tested with > iperf > > as one of our first steps and saw expected speeds (9.9Gb/s as we are > using > > bonded X540-T2 interfaces) on both kernels. As an added data point, we > > have no problem with write performance to the same pool with the same > > kernel configuration (~1GB/s). We also checked the values of > > read_ahead_kb of the block devices but both were shown to be the default > > of 128 (we have since changed these to 4096 in our configuration, but the > > results were seen with the default of 128). > > > > We are in the process of rebuilding the entire cluster to use 3.13 and a > > completely fresh installation of Ceph to make sure nothing else is at > play > > here. > > > > We did check a few things in iostat and collectl, but we didn't see any > read IO > > against the OSDs, so I am leaning towards something further up the stack. > > > > Just a little more background on the cluster configuration: > > > > Specific pool created just for benchmarking, using 512 pgs and pgps and 2 > > replicas. Using 3 OSD nodes (also handling MON duties) with 8 SATA 7.2K > > RPM OSDs and 2 NVMe journals (4 OSD to 1 Journal ratio). 1 x Hex core > CPUs > > with 32GB of RAM per OSD node. > > > > Tom > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
