Re: [ceph-users] Minimize data lost with PG incomplete

# ceph -s cluster 29a91870-2ed2-40dc-969e-07b22f37928b health HEALTH_ERR clock skew detected on mon.loki04 155 pgs are stuck inactive for more than 300 seconds 7 pgs backfill_toofull 1028 pgs backfill_wait 48 pgs backfilling

Re: [ceph-users] Minimize data lost with PG incomplete

I am not sure about "incomplete" part out of my head, but you can try setting min_size to 1 for pools toreactivate some PG, if they are down/inactive due to missing replicas. On 17-01-31 10:24, José M. Martín wrote: # ceph -s cluster 29a91870-2ed2-40dc-969e-07b22f37928b health HEALT

Re: [ceph-users] Bluestore: v11.2.0 peering not happening when OSD is down

Hi Greg, Now we could see the same problem exists for kraken-filestore also. Attached the requested osdmap and crushmap. OSD.1 was stopped in this following procedure and OSD map for a PG is displayed. ceph osd dump | grep cdvr_ec 2017-01-31 08:39:44.827079 7f323d66c700 -1 WARNING: the following

[ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

Hi all, our cluster is currently performing a big expansion and is in recovery mode (we doubled in size and osd# from 600 TB to 1,2 TB). Now we get the following message from our monitor nodes: mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail Reading [0] it says that it is

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

> Op 31 januari 2017 om 10:22 schreef Martin Palma : > > > Hi all, > > our cluster is currently performing a big expansion and is in recovery > mode (we doubled in size and osd# from 600 TB to 1,2 TB). > Yes, that is to be expected. When not all PGs are active+clean the MONs will not trim th

Re: [ceph-users] Minimize data lost with PG incomplete

Already min_size = 1 Thanks, Jose M. Martín El 31/01/17 a las 09:44, Henrik Korkuc escribió: > I am not sure about "incomplete" part out of my head, but you can try > setting min_size to 1 for pools toreactivate some PG, if they are > down/inactive due to missing replicas. > > On 17-01-31 10:24,

Re: [ceph-users] Minimize data lost with PG incomplete

Hi José, Too late, but you could have updated the CRUSHmap *before* moving the disks. Something like: “ceph osd crush set osd.0 0.90329 root=default rack=sala2.2 host=loki05” would move the osd.0 to loki05 and would trigger the appropriate PG movements before any physical move. Then the physic

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

Hi Wido, thank you for the clarification. We will wait until recovery is over we have plenty of space on the mons :-) Best, Martin On Tue, Jan 31, 2017 at 10:35 AM, Wido den Hollander wrote: > >> Op 31 januari 2017 om 10:22 schreef Martin Palma : >> >> >> Hi all, >> >> our cluster is currently

Re: [ceph-users] Minimize data lost with PG incomplete

Thanks. I just realized I keep some of the original OSD. If it contains some of the incomplete PGs , would be possible to add then into the new disks? Maybe following this steps? http://ceph.com/community/incomplete-pgs-oh-my/ El 31/01/17 a las 10:44, Maxime Guyot escribió: > Hi José, > > Too late

Re: [ceph-users] Minimize data lost with PG incomplete

Any idea of how could I recover files from the filesystem mount? Doing a cp, it hungs when find a damaged file/folder. I would be happy getting no damaged files Thanks El 31/01/17 a las 11:19, José M. Martín escribió: > Thanks. > I just realized I keep some of the original OSD. If it contains som

[ceph-users] rsync service download.ceph.com partially broken

Hi cephers, since some time i get errors while rsyncing from the ceph download server: download.ceph.com: rsync: send_files failed to open "/debian-jewel/db/lockfile" (in ceph): Permission denied (13) "/debian-jewel/pool/main/c/ceph/.ceph-fuse-dbg_10.1.0-1~bpo80+1_amd64.deb.h0JvHM" (in ceph):

Re: [ceph-users] Bluestore: v11.2.0 peering not happening when OSD is down

Hi Greg, Following are the test outcomes on EC profile ( n = k + m) 1. Kraken filestore and bluetore with m=1 , recovery does not start . 2. Jewel filestore and bluestore with m=1 , recovery happens . 3. Kraken bluestore all default configuration and m=1, no recovery. 4.

Re: [ceph-users] [Ceph-mirrors] rsync service download.ceph.com partially broken

> Op 31 januari 2017 om 13:46 schreef Björn Lässig : > > > Hi cephers, > > since some time i get errors while rsyncing from the ceph download server: > > download.ceph.com: > > rsync: send_files failed to open "/debian-jewel/db/lockfile" (in ceph): > Permission denied (13) > "/debian-jewel/

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

If you do have a large enough drive on all of your mons (and always intend to do so) you can increase the mon store warning threshold in the config file so that it no longer warns at 15360 MB. [cid:image5e6ea5.JPG@20abe996.44926dad]

[ceph-users] Unique object IDs and crush on object striping

Hi, Two quickies: 1) How does Ceph handle unique object IDs without any central information about the object names? 2) How CRUSH is used in case of splitting an object in stripes? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://li

[ceph-users] Import Ceph RBD snapshot

Hello, I just wonder if there is a way to import a Ceph RBD snapshot that I have previously exported, but without recover the current image state ? Thanks in advance.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.c

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

On 01/31/2017 03:35 PM, David Turner wrote: If you do have a large enough drive on all of your mons (and always intend to do so) you can increase the mon store warning threshold in the config file so that it no longer warns at 15360 MB. And if you so decide to go that route, please be aware tha

Re: [ceph-users] Bluestore: v11.2.0 peering not happening when OSD is down

Hi Greg, the problem is in kraken, when a pool is created with EC profile , min_size equals erasure size. For 3+1 profile , following is the pool status , pool 2 'cdvr_ec' erasure size 4 min_size 4 crush_ruleset 1 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 234 flags hashpspool str

Re: [ceph-users] Unique object IDs and crush on object striping

On Tue, Jan 31, 2017 at 7:42 AM, Ukko wrote: > Hi, > > Two quickies: > > 1) How does Ceph handle unique object IDs without any > central information about the object names? > That's where CRUSH comes in. It maps an object name to a unique placement group ID based on the available placement group

[ceph-users] No space left on device on directory with > 1000000 files

I'm running into a problem on a really large directory of over a million files (don't ask, my users are clueless). Anyway, I'm trying to to use Ceph as backup storage for their filesystem. As I rsync the directory, it started giving me a "No space left on device" for this directory, even though

Re: [ceph-users] Bluestore: v11.2.0 peering not happening when OSD is down

On Tue, Jan 31, 2017 at 9:06 AM, Muthusamy Muthiah wrote: > Hi Greg, > > the problem is in kraken, when a pool is created with EC profile , min_size > equals erasure size. > > For 3+1 profile , following is the pool status , > pool 2 'cdvr_ec' erasure size 4 min_size 4 crush_ruleset 1 object_hash

Re: [ceph-users] No space left on device on directory with > 1000000 files

On Tue, Jan 31, 2017 at 6:29 PM, Jorge Garcia wrote: > I'm running into a problem on a really large directory of over a million > files (don't ask, my users are clueless). Anyway, I'm trying to to use Ceph > as backup storage for their filesystem. As I rsync the directory, it started > giving me a

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

On Wed, Feb 1, 2017 at 1:51 AM, Joao Eduardo Luis wrote: > On 01/31/2017 03:35 PM, David Turner wrote: >> >> If you do have a large enough drive on all of your mons (and always >> intend to do so) you can increase the mon store warning threshold in the >> config file so that it no longer warns at

Re: [ceph-users] Bluestore: v11.2.0 peering not happening when OSD is down

On Wed, Feb 1, 2017 at 3:38 AM, Gregory Farnum wrote: > On Tue, Jan 31, 2017 at 9:06 AM, Muthusamy Muthiah > wrote: >> Hi Greg, >> >> the problem is in kraken, when a pool is created with EC profile , min_size >> equals erasure size. >> >> For 3+1 profile , following is the pool status , >> pool

Re: [ceph-users] mon.mon01 store is getting too big! 18119 MB >= 15360 MB -- 94% avail

On 01/31/2017 07:12 PM, Shinobu Kinjo wrote: On Wed, Feb 1, 2017 at 1:51 AM, Joao Eduardo Luis wrote: On 01/31/2017 03:35 PM, David Turner wrote: If you do have a large enough drive on all of your mons (and always intend to do so) you can increase the mon store warning threshold in the config

Re: [ceph-users] Bluestore: v11.2.0 peering not happening when OSD is down

Hi Greg, Thanks for the info and hope this will be solved in the upcoming minor updates of kraken. Regarding k+1 , I will take your feedback to our architect team and to increase this to k+2 and revert back the pool to normal state. Thanks, Muthu On 1 February 2017 at 02:01, Shinobu Kinjo wrote

Re: [ceph-users] Minimize data lost with PG incomplete

Hi José If you have some of the original OSDs (not zapped or erased) then you might be able to just re-add them to your cluster and have a happy cluster. If you attempt the ceph_objectstore_tool –op export & import make sure to do it on a temporary OSD of weight 0 as recommended in the link prov