Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-21 Thread Anton Dmitriev
Sent: Wednesday, May 10, 2017 10:14 AM To: Piotr Nowosielski mailto:piotr.nowosiel...@allegrogroup.com>>; ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] All OSD fails after few r

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-11 Thread David Turner
lestore merge and >>> >> split". >>> >> >>> >> Some explain: >>> >> The OSD, after reaching a certain number of files in the directory >>> >> (it depends of 'filestore merge threshold' and 'filestore split >&

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-11 Thread Anton Dmitriev
m <mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] All OSD fails after few requests to RGW When I created cluster, I made a mistake in configuration, and set split parameter to 32 and merge to 40, so 32*40*16 = 20480 files per folder.

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-11 Thread David Turner
ot been >> migrated. >> crushmap settings? Weight of OSD? >> >> One thing is certain - you will not find any information about the split >> process in the logs ... >> >> pn >> >> -Original Message----- >> From: Anton Dmitriev [

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread Anton Dmitriev
r.nowosiel...@allegrogroup.com>>; ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> Subject: Re: [ceph-users] All OSD fails after few requests to RGW When I created cluster, I made a mistake in configuration, and set split parameter to 32 and merge t

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread David Turner
, May 10, 2017 10:14 AM > To: Piotr Nowosielski ; > ceph-users@lists.ceph.com > Subject: Re: [ceph-users] All OSD fails after few requests to RGW > > When I created cluster, I made a mistake in configuration, and set split > parameter to 32 and merge to 40, so 32*40*16 = 20480 files pe

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread Piotr Nowosielski
: Wednesday, May 10, 2017 10:14 AM To: Piotr Nowosielski ; ceph-users@lists.ceph.com Subject: Re: [ceph-users] All OSD fails after few requests to RGW When I created cluster, I made a mistake in configuration, and set split parameter to 32 and merge to 40, so 32*40*16 = 20480 files per folder. After that

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread Anton Dmitriev
h-users@lists.ceph.com Subject: Re: [ceph-users] All OSD fails after few requests to RGW How did you solved it? Set new split/merge thresholds, and manually applied it by ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-${osd_num} --journal-path /var/lib/ceph/osd/ceph-${osd_num}/jo

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread Piotr Nowosielski
rastruktury 5 Grupa Allegro sp. z o.o. Tel: +48 512 08 55 92 -Original Message- From: Anton Dmitriev [mailto:t...@enumnet.ru] Sent: Wednesday, May 10, 2017 9:19 AM To: Piotr Nowosielski ; ceph-users@lists.ceph.com Subject: Re: [ceph-users] All OSD fails after few requests to RGW How di

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread Anton Dmitriev
.@lists.ceph.com] On Behalf Of Anton Dmitriev Sent: Wednesday, May 10, 2017 8:14 AM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] All OSD fails after few requests to RGW Hi! I increased pg_num and pgp_num for pool default.rgw.buckets.data from 2048 to 4096, and it seems that situation beca

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-10 Thread Piotr Nowosielski
m w wysokości 33 976 500,00 zł, posiadająca numer identyfikacji podatkowej NIP: 5272525995. -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Anton Dmitriev Sent: Wednesday, May 10, 2017 8:14 AM To: ceph-users@lists.ceph.com Subject: Re: [ceph-

Re: [ceph-users] All OSD fails after few requests to RGW

2017-05-09 Thread Anton Dmitriev
Hi! I increased pg_num and pgp_num for pool default.rgw.buckets.data from 2048 to 4096, and it seems that situation became a bit better, cluster dies after 20-30 PUTs, not after 1. Could someone please give me some recommendations how to rescue the cluster? On 27.04.2017 09:59, Anton Dmitri

[ceph-users] All OSD fails after few requests to RGW

2017-04-27 Thread Anton Dmitriev
Cluster was going well for a long time, but on the previous week osds start to fail. We use cluster like image storage for Opennebula with small load and like object storage with high load. Sometimes disks of some osds utlized by 100 %, iostat shows avgqu-sz over 1000, while reading or writing a