x27;20,38', '22,30', '26,37', '26,38', '27,33', '27,34', '27,36',
> '28,32', '28,34', '28,36', '28,37', '3,18', '3,27', '3,29', '3,37', '4,10
2015-01-07 15:48:34.429997 7fc0e9bfd700 0 log [WRN] : slow request 60.742016
seconds old, received at 2015-01-07 15:47:33.687935: osd_op(client.92886.0:4711
benchmark_data_tvsaq1_29431_object4710 [write 0~4194304] 3.1639422f ack+ondisk+
write e1464) v4 currently waiting for subops from 22,36
____________
5-01-07
> 15:48:34.429997 7fc0e9bfd700 0 log [WRN] : slow request 60.742016
> seconds old, received at 2015-01-07 15:47:33.687935:
> osd_op(client.92886.0:4711 benchmark_data_tvsaq1_29431_object4710 [write
> 0~4194304] 3.1639422f ack+ondisk+ write e1464) v4 currently waiting for
> s
Hi,
I just ran this test and found my system is not better. But I use
commodity hardware. The only difference is latency. You should look at
it.
Total time run: 62.412381
Total writes made: 919
Write size: 4194304
Bandwidth (MB/sec): 58.899
Stddev Bandwidth:
On Tue, 6 Jan 2015 12:07:26 AM Sanders, Bill wrote:
> 14 and 18 happened to show up during that run, but its certainly not only
> those OSD's. It seems to vary each run. Just from the runs I've done
> today I've seen the following pairs of OSD's:
Could your osd nodes be paging? I know from watc
On Mon, 5 Jan 2015 22:36:29 + Sanders, Bill wrote:
> Hi Ceph Users,
>
> We've got a Ceph cluster we've built, and we're experiencing issues with
> slow or hung IO's, even running 'rados bench' on the OSD cluster.
> Things start out great, ~600 MB/s, then rapidly drops off as the test
> waits
'14,39', '16,20', '16,27', '18,38', '19,30', '19,31',
'19,39', '20,38', '22,30', '26,37', '26,38', '27,33', '27,34', '27,36',
'28,32', '28,34
Hi BIll,
From your log excerpt, it looks like your slow requests are happening on OSDs
14 and 18. Is it always these two OSDs?
If you don't have a long recovery time (e.g., the cluster is just full of test
data), maybe you could try setting OSDs 14 and 18 out and re-benching?
Alternatively I s
delgado.com]
Sent: Monday, January 05, 2015 3:47 PM
To: Sanders, Bill
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Slow/Hung IOs
Hi,
I just ran this test and found my system is not better. But I use commodity
hardware. The only difference is latency. You should look at it.
Total ti
