Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-26 Thread Kostis Fardelas
I will catch up with the issues in the ML and hopefully with the code. Yes the 2 nodes are very different from the other 2, we are in the middle of restructuring this cluster thus the irregularity. Thanks a lot Dan On 26 July 2016 at 15:25, Dan van der Ster wrote: > Cool, glad that worked. You'

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-26 Thread Dan van der Ster
Cool, glad that worked. You'll have to read backwards in the ML to find this discussed -- though it is rarely needed, therefore rarely discussed. For code, it's used in src/crush/mapper.c. Most clusters, irrespective of size, work with 50 tries. Clusters that need more than 50 tries usually have s

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-26 Thread Kostis Fardelas
Hello Dan, I increased choose_local_tries to 75 and the misplaced objects reduced to 286. One more increase to 100 to get 141 misplaced objects and one more to 125 for the cluster to fully recover! I also verified that I can now down + out an OSD and the cluster will also fully recover. My problem

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-26 Thread Dan van der Ster
Hi, Starting from the beginning... If a 3-replica PG gets stuck with only 2 replicas after changing tunables, it's probably a case where choose_total_tries is too low for your cluster configuration. Try increasing choose_total_tries from 50 to 75. -- Dan On Fri, Jul 22, 2016 at 4:17 PM, Kosti

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-26 Thread Kostis Fardelas
leaf_descend_once": 1, "chooseleaf_vary_r": 1, "straw_calc_version": 1, "allowed_bucket_algs": 22, "profile": "unknown", "optimal_tunables": 0, "legacy_tunables": 0, &qu

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-25 Thread Brad Hubbard
is always better to use powers >>> of 2) and see if the recover completes.. >>> >>> Cheers >>> G. >>> ____ >>> From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Kostis >>> Fardelas [dan

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-25 Thread Kostis Fardelas
behalf of Kostis >> Fardelas [dante1...@gmail.com] >> Sent: 23 July 2016 16:32 >> To: Brad Hubbard >> Cc: ceph-users >> Subject: Re: [ceph-users] Recovery stuck after adjusting to recent tunables >> >> Hi Brad, >> >> pool 0 'data' r

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-24 Thread Kostis Fardelas
ph-users-boun...@lists.ceph.com] on behalf of Kostis > Fardelas [dante1...@gmail.com] > Sent: 23 July 2016 16:32 > To: Brad Hubbard > Cc: ceph-users > Subject: Re: [ceph-users] Recovery stuck after adjusting to recent tunables > > Hi Brad, > > pool 0 'data'

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-23 Thread Goncalo Borges
2) and see if the recover completes.. Cheers G. From: ceph-users [ceph-users-boun...@lists.ceph.com] on behalf of Kostis Fardelas [dante1...@gmail.com] Sent: 23 July 2016 16:32 To: Brad Hubbard Cc: ceph-users Subject: Re: [ceph-users] Recovery stuck after

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-22 Thread Kostis Fardelas
Hi Brad, pool 0 'data' replicated size 2 min_size 1 crush_ruleset 3 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 119047 crash_replay_interval 45 stripe_width 0 pool 1 'metadata' replicated size 2 min_size 1 crush_ruleset 3 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 1190

Re: [ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-22 Thread Brad Hubbard
On Sat, Jul 23, 2016 at 12:17 AM, Kostis Fardelas wrote: > Hello, > being in latest Hammer, I think I hit a bug with more recent than > legacy tunables. > > Being in legacy tunables for a while, I decided to experiment with > "better" tunables. So first I went from argonaut profile to bobtail > an

[ceph-users] Recovery stuck after adjusting to recent tunables

2016-07-22 Thread Kostis Fardelas
Hello, being in latest Hammer, I think I hit a bug with more recent than legacy tunables. Being in legacy tunables for a while, I decided to experiment with "better" tunables. So first I went from argonaut profile to bobtail and then to firefly. However, I decided to make the changes on chooseleaf