Deployment method: ceph-deploy
Centos 7.2, systemctl
Infernalis. This also happened when I was testing @ Jewell.
I am restarting ( or stop/start ) the ceph old processes ( after they die
or something ), with:
systemctl stop|start|restart ceph.target
Is there another way that it is more appropriate ?
Thank you ahead of time for your help!
Best Regards,
Wade
On Mon, Jan 11, 2016 at 5:43 PM John Spray <jsp...@redhat.com> wrote:
> On Mon, Jan 11, 2016 at 10:32 PM, Wade Holler <wade.hol...@gmail.com>
> wrote:
> > Does anyone else have any suggestions here? I am increasingly concerned
> > about my config if other folks aren't seeing this.
> >
> > I could change to a manual crushmap but otherwise have no need to.
>
> What did you use to deploy ceph? What init system are you using to
> start the ceph daemons? Also, usual questions, what distro, what ceph
> version?
>
> The script that updates the OSD host locations in the crush map is
> meant to be invoked by the init system before the ceph-osd process,
> but if you're starting it in some unconventional way then that might
> not be happening.
>
> John
>
> >
> > I emailed the Ceph-dev list but have not had a response yet.
> >
> > Best Regards
> > Wade
> >
> > On Fri, Jan 8, 2016 at 11:12 AM Wade Holler <wade.hol...@gmail.com>
> wrote:
> >>
> >> It is not set in the conf file. So why do I still have this behavior ?
> >>
> >> On Fri, Jan 8, 2016 at 11:08 AM hnuzhoulin <hnuzhoul...@gmail.com>
> wrote:
> >>>
> >>> Yeah,this setting can not see in asok config.
> >>> You just set it in ceph.conf and restart mon and osd service(sorry I
> >>> forget if these restart is necessary)
> >>>
> >>> what I use this config is when I changed crushmap manually,and I do not
> >>> want the service init script to rebuild crushmap as default way.
> >>>
> >>> maybe this is not siut for your problem.just have a try.
> >>>
> >>> 在 Fri, 08 Jan 2016 21:51:32 +0800，Wade Holler <wade.hol...@gmail.com>
> 写道:
> >>>
> >>> That is not set as far as I can tell. Actually it is strange that I
> >>> don't see that setting at all.
> >>>
> >>> [root@cpn00001 ~]# ceph daemon osd.0 config show | grep update | grep
> >>> crush
> >>>
> >>> [root@cpn00001 ~]# grep update /etc/ceph/ceph.conf
> >>>
> >>> [root@cpn00001 ~]#
> >>>
> >>>
> >>> On Fri, Jan 8, 2016 at 1:50 AM Mart van Santen <m...@greenhost.nl>
> wrote:
> >>>>
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> Do you have by any chance disabled automatic crushmap updates in your
> >>>> ceph config?
> >>>>
> >>>> osd crush update on start = false
> >>>>
> >>>> If this is the case, and you move disks around hosts, they won't
> update
> >>>> their position/host in the crushmap, even if the crushmap does not
> reflect
> >>>> reality.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Mart
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 01/08/2016 02:16 AM, Wade Holler wrote:
> >>>>
> >>>> Sure. Apologies for all the text: We have 12 Nodes for OSDs, 15 OSDs
> >>>> per node, but I will only include a sample:
> >>>>
> >>>> ceph osd tree | head -35
> >>>>
> >>>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
> >>>>
> >>>> -1 130.98450 root default
> >>>>
> >>>> -2 5.82153 host cpn00001
> >>>>
> >>>> 4 0.72769 osd.4 up 1.00000 1.00000
> >>>>
> >>>> 14 0.72769 osd.14 up 1.00000 1.00000
> >>>>
> >>>> 3 0.72769 osd.3 up 1.00000 1.00000
> >>>>
> >>>> 24 0.72769 osd.24 up 1.00000 1.00000
> >>>>
> >>>> 5 0.72769 osd.5 up 1.00000 1.00000
> >>>>
> >>>> 2 0.72769 osd.2 up 1.00000 1.00000
> >>>>
> >>>> 17 0.72769 osd.17 up 1.00000 1.00000
> >>>>
> >>>> 69 0.72769 osd.69 up 1.00000 1.00000
> >>>>
> >>>> -3 6.54922 host cpn00003
> >>>>
> >>>> 7 0.72769 osd.7 up 1.00000 1.00000
> >>>>
> >>>> 8 0.72769 osd.8 up 1.00000 1.00000
> >>>>
> >>>> 9 0.72769 osd.9 up 1.00000 1.00000
> >>>>
> >>>> 0 0.72769 osd.0 up 1.00000 1.00000
> >>>>
> >>>> 28 0.72769 osd.28 up 1.00000 1.00000
> >>>>
> >>>> 10 0.72769 osd.10 up 1.00000 1.00000
> >>>>
> >>>> 1 0.72769 osd.1 up 1.00000 1.00000
> >>>>
> >>>> 6 0.72769 osd.6 up 1.00000 1.00000
> >>>>
> >>>> 29 0.72769 osd.29 up 1.00000 1.00000
> >>>>
> >>>> -4 2.91077 host cpn00004
> >>>>
> >>>>
> >>>> Compared with the actual processes that are running:
> >>>>
> >>>>
> >>>> [root@cpx00001 ~]# ssh cpn00001 ps -ef | grep ceph\-osd
> >>>>
> >>>> ceph 92638 1 26 16:19 ? 01:00:55 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 6 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92667 1 20 16:19 ? 00:48:04 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 0 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92673 1 18 16:19 ? 00:42:48 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 8 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92681 1 19 16:19 ? 00:45:52 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 7 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92701 1 15 16:19 ? 00:36:05 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 12 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92748 1 14 16:19 ? 00:34:07 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 10 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92756 1 16 16:19 ? 00:38:40 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 9 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92758 1 17 16:19 ? 00:39:28 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 13 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92777 1 19 16:19 ? 00:46:17 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 1 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 92988 1 18 16:19 ? 00:42:47 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 5 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 93058 1 18 16:19 ? 00:43:18 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 11 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 93078 1 17 16:19 ? 00:41:38 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 14 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 93127 1 15 16:19 ? 00:36:29 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 4 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 93130 1 17 16:19 ? 00:40:44 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 2 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 93173 1 21 16:19 ? 00:49:37 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 3 --setuser ceph --setgroup ceph
> >>>>
> >>>> [root@cpx00001 ~]# ssh cpn00003 ps -ef | grep ceph\-osd
> >>>>
> >>>> ceph 82454 1 18 16:19 ? 00:43:58 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 25 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82464 1 24 16:19 ? 00:55:40 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 21 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82473 1 21 16:19 ? 00:50:14 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 17 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82612 1 19 16:19 ? 00:45:25 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 22 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82629 1 20 16:19 ? 00:48:38 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 16 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82651 1 16 16:19 ? 00:39:24 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 20 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82687 1 17 16:19 ? 00:40:31 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 18 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82697 1 26 16:19 ? 01:02:12 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 23 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82719 1 20 16:19 ? 00:47:15 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 15 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82722 1 14 16:19 ? 00:33:41 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 28 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82725 1 14 16:19 ? 00:33:16 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 26 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82743 1 14 16:19 ? 00:34:17 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 29 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82769 1 19 16:19 ? 00:46:00 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 19 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82816 1 13 16:19 ? 00:30:26 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 27 --setuser ceph --setgroup ceph
> >>>>
> >>>> ceph 82828 1 27 16:19 ? 01:04:38 /usr/bin/ceph-osd
> -f
> >>>> --cluster ceph --id 24 --setuser ceph --setgroup ceph
> >>>>
> >>>> [root@cpx00001 ~]#
> >>>>
> >>>>
> >>>> Looks like the crushmap is bad also:
> >>>>
> >>>> (Cluster appears to be operating ok but this really concerns me.)
> >>>>
> >>>> # begin crush map
> >>>>
> >>>> tunable choose_local_tries 0
> >>>>
> >>>> tunable choose_local_fallback_tries 0
> >>>>
> >>>> tunable choose_total_tries 50
> >>>>
> >>>> tunable chooseleaf_descend_once 1
> >>>>
> >>>> tunable straw_calc_version 1
> >>>>
> >>>>
> >>>> # devices
> >>>>
> >>>> device 0 osd.0
> >>>>
> >>>> device 1 osd.1
> >>>>
> >>>> device 2 osd.2
> >>>>
> >>>> device 3 osd.3
> >>>>
> >>>> device 4 osd.4
> >>>>
> >>>> device 5 osd.5
> >>>>
> >>>> device 6 osd.6
> >>>>
> >>>> device 7 osd.7
> >>>>
> >>>> device 8 osd.8
> >>>>
> >>>> device 9 osd.9
> >>>>
> >>>> device 10 osd.10
> >>>>
> >>>> device 11 osd.11
> >>>>
> >>>> device 12 osd.12
> >>>>
> >>>> device 13 osd.13
> >>>>
> >>>> device 14 osd.14
> >>>>
> >>>> device 15 osd.15
> >>>>
> >>>> device 16 osd.16
> >>>>
> >>>> device 17 osd.17
> >>>>
> >>>> device 18 osd.18
> >>>>
> >>>> device 19 osd.19
> >>>>
> >>>> device 20 osd.20
> >>>>
> >>>> device 21 osd.21
> >>>>
> >>>> device 22 osd.22
> >>>>
> >>>> device 23 osd.23
> >>>>
> >>>> device 24 osd.24
> >>>>
> >>>> device 25 osd.25
> >>>>
> >>>> device 26 osd.26
> >>>>
> >>>> device 27 osd.27
> >>>>
> >>>> device 28 osd.28
> >>>>
> >>>> device 29 osd.29
> >>>>
> >>>> device 30 osd.30
> >>>>
> >>>> device 31 osd.31
> >>>>
> >>>> device 32 osd.32
> >>>>
> >>>> device 33 osd.33
> >>>>
> >>>> device 34 osd.34
> >>>>
> >>>> device 35 osd.35
> >>>>
> >>>> device 36 osd.36
> >>>>
> >>>> device 37 osd.37
> >>>>
> >>>> ...
> >>>>
> >>>>
> >>>> # types
> >>>>
> >>>> type 0 osd
> >>>>
> >>>> type 1 host
> >>>>
> >>>> type 2 chassis
> >>>>
> >>>> type 3 rack
> >>>>
> >>>> type 4 row
> >>>>
> >>>> type 5 pdu
> >>>>
> >>>> type 6 pod
> >>>>
> >>>> type 7 room
> >>>>
> >>>> type 8 datacenter
> >>>>
> >>>> type 9 region
> >>>>
> >>>> type 10 root
> >>>>
> >>>>
> >>>> # buckets
> >>>>
> >>>> host cpn00001 {
> >>>>
> >>>> id -2 # do not change unnecessarily
> >>>>
> >>>> # weight 5.822
> >>>>
> >>>> alg straw
> >>>>
> >>>> hash 0 # rjenkins1
> >>>>
> >>>> item osd.4 weight 0.728
> >>>>
> >>>> item osd.14 weight 0.728
> >>>>
> >>>> item osd.3 weight 0.728
> >>>>
> >>>> item osd.24 weight 0.728
> >>>>
> >>>> item osd.5 weight 0.728
> >>>>
> >>>> item osd.2 weight 0.728
> >>>>
> >>>> item osd.17 weight 0.728
> >>>>
> >>>> item osd.69 weight 0.728
> >>>>
> >>>> }
> >>>>
> >>>> host cpn00003 {
> >>>>
> >>>> id -3 # do not change unnecessarily
> >>>>
> >>>> # weight 6.549
> >>>>
> >>>> alg straw
> >>>>
> >>>> hash 0 # rjenkins1
> >>>>
> >>>> item osd.7 weight 0.728
> >>>>
> >>>> item osd.8 weight 0.728
> >>>>
> >>>> item osd.9 weight 0.728
> >>>>
> >>>> item osd.0 weight 0.728
> >>>>
> >>>> item osd.28 weight 0.728
> >>>>
> >>>> item osd.10 weight 0.728
> >>>>
> >>>> item osd.1 weight 0.728
> >>>>
> >>>> item osd.6 weight 0.728
> >>>>
> >>>> item osd.29 weight 0.728
> >>>>
> >>>> }
> >>>>
> >>>> host cpn00004 {....
> >>>>
> >>>>
> >>>>
> >>>> Thank you for your review !!!!!
> >>>>
> >>>> Wade
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jan 7, 2016 at 6:03 PM Shinobu Kinjo <ski...@redhat.com>
> wrote:
> >>>>>
> >>>>> Can you share the output with us?
> >>>>>
> >>>>> Rgds,
> >>>>> Shinobu
> >>>>>
> >>>>> ----- Original Message -----
> >>>>> From: "Wade Holler" <wade.hol...@gmail.com>
> >>>>> To: "ceph-users" <ceph-users@lists.ceph.com>
> >>>>> Sent: Friday, January 8, 2016 7:29:07 AM
> >>>>> Subject: [ceph-users] ceph osd tree output
> >>>>>
> >>>>> Sometimes my ceph osd tree output is wrong. Ie. Wrong osds on the
> wrong
> >>>>> hosts ?
> >>>>>
> >>>>> Anyone else have this issue?
> >>>>>
> >>>>> I have seen this at Infernalis and Jewell.
> >>>>>
> >>>>> Thanks
> >>>>> Wade
> >>>>>
> >>>>> _______________________________________________
> >>>>> ceph-users mailing list
> >>>>> ceph-users@lists.ceph.com
> >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list
> >>>> ceph-users@lists.ceph.com
> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list
> >>>> ceph-users@lists.ceph.com
> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> 使用Opera的电子邮件客户端：http://www.opera.com/mail/
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
