Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Hello, in a nutshell, I can confirm that write amplification, see inline. On Mon, 20 Oct 2014 10:43:51 -0500 Mark Nelson wrote: > On 10/20/2014 09:28 AM, Mark Wu wrote: > > > > > > 2014-10-20 21:04 GMT+08:00 Mark Nelson > >: > > > > On 10/20/2014 06:27 AM, M

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

On 10/20/2014 09:28 AM, Mark Wu wrote: 2014-10-20 21:04 GMT+08:00 Mark Nelson mailto:mark.nel...@inktank.com>>: On 10/20/2014 06:27 AM, Mark Wu wrote: Test result Update: Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS SSD Disk I

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

2014-10-20 21:04 GMT+08:00 Mark Nelson : > On 10/20/2014 06:27 AM, Mark Wu wrote: > >> Test result Update: >> >> >> Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS >> SSD Disk IOPS SSD Disk Utilization >> >> 7 14k 45k 9800+ >> 90%

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

On 10/20/2014 06:27 AM, Mark Wu wrote: Test result Update: Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS SSD Disk IOPS SSD Disk Utilization 7 14k 45k 9800+90% 8 21k 50k

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Test result Update: Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS SSD Disk IOPS SSD Disk Utilization 7 14k45k 9800+90% 8

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

ager" mailto:daniel.schwa...@dtnet.de>> Cc: ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> Envoyé: Jeudi 16 Octobre 2014 19:19:17 Objet: Re: [ceph-users] Performance doesn't scale well on a full ssd cluster. Thanks for the detailed information

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

; Envoyé: Jeudi 16 Octobre 2014 19:19:17 > Objet: Re: [ceph-users] Performance doesn't scale well on a full ssd > cluster. > > > > Thanks for the detailed information. but I am already using fio with rbd > engine. Almost 4 volumes can reach the peak. > 2014 年 10 月 17 日 上午 1

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Decrease the rbd debug level from 5 to 0, and almost all the debugging and logging are disabled. It doesn't help. ceph tell osd.* injectargs '--debug_rbd 0\/0' ceph tell osd.* injectargs '--debug_objectcacher 0\/0' ceph tell osd.* injectargs '--debug_rbd_replay 0\/0' 2014-10-17 8:45 GMT+08:00 Sh

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

De: "Mark Wu" À: "Gregory Farnum" Cc: "ceph-users" Envoyé: Vendredi 17 Octobre 2014 10:52:44 Objet: Re: [ceph-users] Performance doesn't scale well on a full ssd cluster. >> I assume you added more clients and checked that it didn't scale past &g

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

>> I assume you added more clients and checked that it didn't scale past >> that? Yes, correct. >> You might look through the list archives; there are a number of discussions about how and how far you can scale SSD-backed cluster performance. I have look at those discussions before, particular the

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

; À: "Daniel Schwager" Cc: ceph-users@lists.ceph.com Envoyé: Jeudi 16 Octobre 2014 19:19:17 Objet: Re: [ceph-users] Performance doesn't scale well on a full ssd cluster. Thanks for the detailed information. but I am already using fio with rbd engine. Almost 4 volumes can rea

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Hello (Greg in particular), On Thu, 16 Oct 2014 10:06:58 -0700 Gregory Farnum wrote: > [Re-added the list.] > > I assume you added more clients and checked that it didn't scale past > that? You might look through the list archives; there are a number of > discussions about how and how far you c

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

We do observe the same issue on our 12 SSD setup, disable the all log maybe helpful. Cheers, xinxin From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Wu Sent: Friday, October 17, 2014 12:18 AM To: ceph-users@lists.ceph.com Subject: [ceph-users] Performance doesn't sca

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Mark, please read this: https://www.mail-archive.com/ceph-users@lists.ceph.com/msg12486.html On 16 Oct 2014, at 19:19, Mark Wu wrote: > > Thanks for the detailed information. but I am already using fio with rbd > engine. Almost 4 volumes can reach the peak. > > 2014 年 10 月 17 日 上午 1:03于 wud.

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

[Re-added the list.] I assume you added more clients and checked that it didn't scale past that? You might look through the list archives; there are a number of discussions about how and how far you can scale SSD-backed cluster performance. Just scanning through the config options you set, you mig

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

forgot to cc the list -- 转发的邮件 -- 发件人："Mark Wu" 日期：2014 年 10 月 17 日 上午 12:51 主题：Re: [ceph-users] Performance doesn't scale well on a full ssd cluster. 收件人："Gregory Farnum" 抄送： Thanks for the reply. I am not using single client. Writing 5 rbd volumes on

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

Thanks for the detailed information. but I am already using fio with rbd engine. Almost 4 volumes can reach the peak. 2014 年 10 月 17 日 上午 1:03于 wud...@gmail.com写道： Thanks for the detailed information. but I am already using fio with rbd engine. Almost 4 volumes can reach the peak. 2014 年 10 月 17 日

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

If you're running a single client to drive these tests, that's your bottleneck. Try running multiple clients and aggregating their numbers. -Greg On Thursday, October 16, 2014, Mark Wu wrote: > Hi list, > > During my test, I found ceph doesn't scale as I expected on a 30 osds > cluster. > The fo