Does anyone else have any suggestions here? I am increasingly concerned
about my config if other folks aren't seeing this.
I could change to a manual crushmap but otherwise have no need to.
I emailed the Ceph-dev list but have not had a response yet.
Best Regards
Wade
On Fri, Jan 8, 2016 at 11:12 AM Wade Holler <wade.hol...@gmail.com> wrote:
> It is not set in the conf file. So why do I still have this behavior ?
>
> On Fri, Jan 8, 2016 at 11:08 AM hnuzhoulin <hnuzhoul...@gmail.com> wrote:
>
>> Yeah,this setting can not see in asok config.
>> You just set it in ceph.conf and restart mon and osd service(sorry I
>> forget if these restart is necessary)
>>
>> what I use this config is when I changed crushmap manually,and I do not
>> want the service init script to rebuild crushmap as default way.
>>
>> maybe this is not siut for your problem.just have a try.
>>
>> 在 Fri, 08 Jan 2016 21:51:32 +0800，Wade Holler <wade.hol...@gmail.com> 写道:
>>
>> That is not set as far as I can tell. Actually it is strange that I
>> don't see that setting at all.
>>
>> [root@cpn00001 ~]# ceph daemon osd.0 config show | grep update | grep
>> crush
>>
>> [root@cpn00001 ~]# grep update /etc/ceph/ceph.conf
>>
>> [root@cpn00001 ~]#
>>
>> On Fri, Jan 8, 2016 at 1:50 AM Mart van Santen <m...@greenhost.nl> wrote:
>>
>>>
>>>
>>> Hi,
>>>
>>> Do you have by any chance disabled automatic crushmap updates in your
>>> ceph config?
>>>
>>> osd crush update on start = false
>>>
>>> If this is the case, and you move disks around hosts, they won't update
>>> their position/host in the crushmap, even if the crushmap does not reflect
>>> reality.
>>>
>>> Regards,
>>>
>>> Mart
>>>
>>>
>>>
>>>
>>>
>>> On 01/08/2016 02:16 AM, Wade Holler wrote:
>>>
>>> Sure. Apologies for all the text: We have 12 Nodes for OSDs, 15 OSDs
>>> per node, but I will only include a sample:
>>>
>>> ceph osd tree | head -35
>>>
>>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>>
>>> -1 130.98450 root default
>>>
>>> -2 5.82153 host cpn00001
>>>
>>> 4 0.72769 osd.4 up 1.00000 1.00000
>>>
>>> 14 0.72769 osd.14 up 1.00000 1.00000
>>>
>>> 3 0.72769 osd.3 up 1.00000 1.00000
>>>
>>> 24 0.72769 osd.24 up 1.00000 1.00000
>>>
>>> 5 0.72769 osd.5 up 1.00000 1.00000
>>>
>>> 2 0.72769 osd.2 up 1.00000 1.00000
>>>
>>> 17 0.72769 osd.17 up 1.00000 1.00000
>>>
>>> 69 0.72769 osd.69 up 1.00000 1.00000
>>>
>>> -3 6.54922 host cpn00003
>>>
>>> 7 0.72769 osd.7 up 1.00000 1.00000
>>>
>>> 8 0.72769 osd.8 up 1.00000 1.00000
>>>
>>> 9 0.72769 osd.9 up 1.00000 1.00000
>>>
>>> 0 0.72769 osd.0 up 1.00000 1.00000
>>>
>>> 28 0.72769 osd.28 up 1.00000 1.00000
>>>
>>> 10 0.72769 osd.10 up 1.00000 1.00000
>>>
>>> 1 0.72769 osd.1 up 1.00000 1.00000
>>>
>>> 6 0.72769 osd.6 up 1.00000 1.00000
>>>
>>> 29 0.72769 osd.29 up 1.00000 1.00000
>>>
>>> -4 2.91077 host cpn00004
>>>
>>>
>>> Compared with the actual processes that are running:
>>>
>>>
>>> [root@cpx00001 ~]# ssh cpn00001 ps -ef | grep ceph\-osd
>>>
>>> ceph 92638 1 26 16:19 ? 01:00:55 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
>>>
>>> ceph 92667 1 20 16:19 ? 00:48:04 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 0 --setuser ceph --setgroup ceph
>>>
>>> ceph 92673 1 18 16:19 ? 00:42:48 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
>>>
>>> ceph 92681 1 19 16:19 ? 00:45:52 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 7 --setuser ceph --setgroup ceph
>>>
>>> ceph 92701 1 15 16:19 ? 00:36:05 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 12 --setuser ceph --setgroup ceph
>>>
>>> ceph 92748 1 14 16:19 ? 00:34:07 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 10 --setuser ceph --setgroup ceph
>>>
>>> ceph 92756 1 16 16:19 ? 00:38:40 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 9 --setuser ceph --setgroup ceph
>>>
>>> ceph 92758 1 17 16:19 ? 00:39:28 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 13 --setuser ceph --setgroup ceph
>>>
>>> ceph 92777 1 19 16:19 ? 00:46:17 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
>>>
>>> ceph 92988 1 18 16:19 ? 00:42:47 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 5 --setuser ceph --setgroup ceph
>>>
>>> ceph 93058 1 18 16:19 ? 00:43:18 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 11 --setuser ceph --setgroup ceph
>>>
>>> ceph 93078 1 17 16:19 ? 00:41:38 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 14 --setuser ceph --setgroup ceph
>>>
>>> ceph 93127 1 15 16:19 ? 00:36:29 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 4 --setuser ceph --setgroup ceph
>>>
>>> ceph 93130 1 17 16:19 ? 00:40:44 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
>>>
>>> ceph 93173 1 21 16:19 ? 00:49:37 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 3 --setuser ceph --setgroup ceph
>>>
>>> [root@cpx00001 ~]# ssh cpn00003 ps -ef | grep ceph\-osd
>>>
>>> ceph 82454 1 18 16:19 ? 00:43:58 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 25 --setuser ceph --setgroup ceph
>>>
>>> ceph 82464 1 24 16:19 ? 00:55:40 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 21 --setuser ceph --setgroup ceph
>>>
>>> ceph 82473 1 21 16:19 ? 00:50:14 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 17 --setuser ceph --setgroup ceph
>>>
>>> ceph 82612 1 19 16:19 ? 00:45:25 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 22 --setuser ceph --setgroup ceph
>>>
>>> ceph 82629 1 20 16:19 ? 00:48:38 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 16 --setuser ceph --setgroup ceph
>>>
>>> ceph 82651 1 16 16:19 ? 00:39:24 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 20 --setuser ceph --setgroup ceph
>>>
>>> ceph 82687 1 17 16:19 ? 00:40:31 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 18 --setuser ceph --setgroup ceph
>>>
>>> ceph 82697 1 26 16:19 ? 01:02:12 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 23 --setuser ceph --setgroup ceph
>>>
>>> ceph 82719 1 20 16:19 ? 00:47:15 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 15 --setuser ceph --setgroup ceph
>>>
>>> ceph 82722 1 14 16:19 ? 00:33:41 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 28 --setuser ceph --setgroup ceph
>>>
>>> ceph 82725 1 14 16:19 ? 00:33:16 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 26 --setuser ceph --setgroup ceph
>>>
>>> ceph 82743 1 14 16:19 ? 00:34:17 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 29 --setuser ceph --setgroup ceph
>>>
>>> ceph 82769 1 19 16:19 ? 00:46:00 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 19 --setuser ceph --setgroup ceph
>>>
>>> ceph 82816 1 13 16:19 ? 00:30:26 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 27 --setuser ceph --setgroup ceph
>>>
>>> ceph 82828 1 27 16:19 ? 01:04:38 /usr/bin/ceph-osd
>>> -f --cluster ceph --id 24 --setuser ceph --setgroup ceph
>>>
>>> [root@cpx00001 ~]#
>>>
>>>
>>> Looks like the crushmap is bad also:
>>>
>>> (Cluster appears to be operating ok but this really concerns me.)
>>>
>>> # begin crush map
>>>
>>> tunable choose_local_tries 0
>>>
>>> tunable choose_local_fallback_tries 0
>>>
>>> tunable choose_total_tries 50
>>>
>>> tunable chooseleaf_descend_once 1
>>>
>>> tunable straw_calc_version 1
>>>
>>>
>>> # devices
>>>
>>> device 0 osd.0
>>>
>>> device 1 osd.1
>>>
>>> device 2 osd.2
>>>
>>> device 3 osd.3
>>>
>>> device 4 osd.4
>>>
>>> device 5 osd.5
>>>
>>> device 6 osd.6
>>>
>>> device 7 osd.7
>>>
>>> device 8 osd.8
>>>
>>> device 9 osd.9
>>>
>>> device 10 osd.10
>>>
>>> device 11 osd.11
>>>
>>> device 12 osd.12
>>>
>>> device 13 osd.13
>>>
>>> device 14 osd.14
>>>
>>> device 15 osd.15
>>>
>>> device 16 osd.16
>>>
>>> device 17 osd.17
>>>
>>> device 18 osd.18
>>>
>>> device 19 osd.19
>>>
>>> device 20 osd.20
>>>
>>> device 21 osd.21
>>>
>>> device 22 osd.22
>>>
>>> device 23 osd.23
>>>
>>> device 24 osd.24
>>>
>>> device 25 osd.25
>>>
>>> device 26 osd.26
>>>
>>> device 27 osd.27
>>>
>>> device 28 osd.28
>>>
>>> device 29 osd.29
>>>
>>> device 30 osd.30
>>>
>>> device 31 osd.31
>>>
>>> device 32 osd.32
>>>
>>> device 33 osd.33
>>>
>>> device 34 osd.34
>>>
>>> device 35 osd.35
>>>
>>> device 36 osd.36
>>>
>>> device 37 osd.37
>>>
>>> ...
>>>
>>>
>>> # types
>>>
>>> type 0 osd
>>>
>>> type 1 host
>>>
>>> type 2 chassis
>>>
>>> type 3 rack
>>>
>>> type 4 row
>>>
>>> type 5 pdu
>>>
>>> type 6 pod
>>>
>>> type 7 room
>>>
>>> type 8 datacenter
>>>
>>> type 9 region
>>>
>>> type 10 root
>>>
>>>
>>> # buckets
>>>
>>> host cpn00001 {
>>>
>>> id -2 # do not change unnecessarily
>>>
>>> # weight 5.822
>>>
>>> alg straw
>>>
>>> hash 0 # rjenkins1
>>>
>>> item osd.4 weight 0.728
>>>
>>> item osd.14 weight 0.728
>>>
>>> item osd.3 weight 0.728
>>>
>>> item osd.24 weight 0.728
>>>
>>> item osd.5 weight 0.728
>>>
>>> item osd.2 weight 0.728
>>>
>>> item osd.17 weight 0.728
>>>
>>> item osd.69 weight 0.728
>>>
>>> }
>>>
>>> host cpn00003 {
>>>
>>> id -3 # do not change unnecessarily
>>>
>>> # weight 6.549
>>>
>>> alg straw
>>>
>>> hash 0 # rjenkins1
>>>
>>> item osd.7 weight 0.728
>>>
>>> item osd.8 weight 0.728
>>>
>>> item osd.9 weight 0.728
>>>
>>> item osd.0 weight 0.728
>>>
>>> item osd.28 weight 0.728
>>>
>>> item osd.10 weight 0.728
>>>
>>> item osd.1 weight 0.728
>>>
>>> item osd.6 weight 0.728
>>>
>>> item osd.29 weight 0.728
>>>
>>> }
>>>
>>> host cpn00004 {....
>>>
>>>
>>>
>>> Thank you for your review !!!!!
>>>
>>> Wade
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jan 7, 2016 at 6:03 PM Shinobu Kinjo <ski...@redhat.com> wrote:
>>>
>>>> Can you share the output with us?
>>>>
>>>> Rgds,
>>>> Shinobu
>>>>
>>>> ----- Original Message -----
>>>> From: "Wade Holler" <wade.hol...@gmail.com>
>>>> To: "ceph-users" <ceph-users@lists.ceph.com>
>>>> Sent: Friday, January 8, 2016 7:29:07 AM
>>>> Subject: [ceph-users] ceph osd tree output
>>>>
>>>> Sometimes my ceph osd tree output is wrong. Ie. Wrong osds on the wrong
>>>> hosts ?
>>>>
>>>> Anyone else have this issue?
>>>>
>>>> I have seen this at Infernalis and Jewell.
>>>>
>>>> Thanks
>>>> Wade
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing
>>> listceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>>
>> --
>> 使用Opera的电子邮件客户端：http://www.opera.com/mail/
>>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
