On Thu, Jun 12, 2014 at 5:02 PM, David <da...@visions.se> wrote: > Thanks Mark! > > Well, our workload has more IOs and quite low throughput, perhaps 10MB/s -> > 100MB/s. It’s a quite mixed workload, but mostly small files (http / mail / > sql). > During the recovery we had ranged between 600-1000MB/s throughput. > > So the only way to currently ”fix” this is to have enough IO to handle both > recovery and client IOs? > What’s the easiest/best way to add more IOs to a current cluster if you don’t > want to scale? Add more RAM to OSD servers or add a SSD backed r/w cache tier? >
RAM usable only as read cache, SSD holds both types of operations. Dealing with a lot of small operations is very hard because the way cluster behaves changes dramatically with scale or with involved caching methods, therefore workloads which worked very reliable on certain number of OSDs may choke on 5 times higher count, so there almost nothing I can suggest to you except try-n-check variant. > Kind Regards, > > David Majchrzak > > > 12 jun 2014 kl. 14:42 skrev Mark Nelson <mark.nel...@inktank.com>: > >> On 06/12/2014 03:44 AM, David wrote: >>> Hi, >>> >>> We have 5 OSD servers, with 10 OSDs each (journals on enterprise SSDs). >>> >>> We lost an OSD and the cluster started to backfill the data to the rest of >>> the OSDs - during which the latency skyrocketed on some OSDs and connected >>> clients experienced massive IO wait. >>> >>> I’m trying to rectify the situation now and from what I can tell, these are >>> the settings that might help. >>> >>> osd client op priority >>> osd recovery op priority >>> osd max backfills >>> osd recovery max active >>> >>> 1. Does a high priority value mean it has higher priority? (if the other >>> one has lower value) Or does a priority of 1 mean highest priority? >>> 2. I’m running with default on these settings. Does anyone else have any >>> experience changing those? >> >> We did some investigation into this a little while back. I suspect you'll >> see some benefit by reducing backfill/recovery priority and max concurrent >> operations, but you have to be careful. We found that the higher the number >> of concurrent client IOs (past the saturation point), the greater relative >> proportion of throughput is used by client IO. That makes it hard to nail >> down specific priority and concurrency settings. If your workload requires >> high throughput and low latency with few client IOs (ie below the saturation >> point), you may need to overly favour client IO. If you are over-saturating >> the cluster with many concurrent IOs, you may want to give client IO less >> priority. If you overly favor client IO when over-saturating the cluster, >> recovery can take much much longer and client throughput may actually be >> lower in aggregate. Obviously this isn't ideal, but seems to be what's >> going on right now. >> >> Mark >> >>> >>> Kind Regards, >>> David Majchrzak >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
