Hi,
We have the same issue on CentOS 7.2.1511 and Ceph 10.2.2 : sometimes
ceph-disk@ services are not started and OSD daemons are failed.
With udev in debug mode, we observe that udev triggers are fired but failed
because /var (on LVM) is not ready.
In ceph-disk, the setup_statedir function is called systematically but it
is not useful for the first call of main_trigger that starting the systemd
service ceph-disk@{dev}.
I'm not sure but a solution here could be to call setup_statedir and
setup_sysconfigdir only when necessary and to start ceph-disk@{dev}
services after the local-fs.target.
11019:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err)
'Traceback (most recent call last):'
11046:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) ' File
"/usr/sbin/ceph-disk", line 9, in <module>'
11074:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) '
load_entry_point('ceph-disk==1.0.0', 'console_scripts',
'ceph-disk')()'
11099:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) ' File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4994, in
run'
11125:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) '
main(sys.argv[1:])'
11150:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) ' File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4936, in
main'
11175:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) '
setup_statedir(args.statedir)'
11199:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) ' File
"/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4396, in
setup_statedir'
11223:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) '
os.mkdir(STATEDIR)'
11246:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1'(err) 'OSError:
[Errno 2] No such file or directory: '/var/lib/ceph''
11271:Aug 25 15:58:42 cephnode03 systemd-udevd[984]:
'/usr/sbin/ceph-disk --log-stdout -v trigger /dev/sdf1' [1103] exit
with return code 1
Regards,
Antoine.
2016-06-24 17:24 GMT+02:00 SCHAER Frederic <frederic.sch...@cea.fr>:
> Hi,
>
> I'm facing the same thing after I reinstalled a node directly in jewel...
>
> Reading : http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/31917
> I can confirm that running : "udevadm trigger -c add -s block " fires the
> udev rules and gets ceph-osd up.
>
> Thing is : I now have reinstalled boxes (CentOS 7.2.1511 ) which do not
> fire udev rules at boot, and get no /dev/disk/by-parttypeuuid - and I fear
> there is none also just after installing the ceph RPMs since the udev rules
> did not pre-exist -, and other exact same boxes (same setup, same hardware,
> same paritions) which were upgraded from previous ceph versions, which do
> seem to work correctly - or so I think.
> All with rootfs on LVM...
>
> I'll try to compare the 2 kinds of hosts to see if I can find something
> usefull ...
>
> Regards
>
>
> -----Message d'origine-----
> De : ceph-users [mailto:ceph-users-boun...@lists.ceph.com] De la part de
> stephane.d...@orange.com
> Envoyé : vendredi 24 juin 2016 12:10
> À : Loic Dachary <l...@dachary.org>
> Cc : ceph-users <ceph-users@lists.ceph.com>
> Objet : Re: [ceph-users] osds udev rules not triggered on reboot (jewel,
> jessie)
>
> Hi Loïc,
>
> Sorry for the delay. Well, it's a vanillia Centos iso image downloaded
> from centos.org mirror:
> [root@hulk-stg ~]# cat /etc/redhat-release
> CentOS Linux release 7.2.1511 (Core)
>
> This issue happens after Ceph upgrade from hammer, I haven't tested with
> distro starting with a fresh Ceph install
>
> Thanks,
>
> Stéphane
>
> -----Original Message-----
> From: Loic Dachary [mailto:l...@dachary.org]
> Sent: Tuesday, June 21, 2016 14:48
> To: DAVY Stephane OBS/OCB
> Cc: ceph-users
> Subject: Re: [ceph-users] osds udev rules not triggered on reboot (jewel,
> jessie)
>
>
>
> On 16/06/2016 18:01, stephane.d...@orange.com wrote:
> > Hi,
> >
> > Same issue with Centos 7, I also put back this file in /etc/udev/rules.d.
>
> Hi Stephane,
>
> Could you please detail which version of CentOS 7 you are using ? I tried
> to reproduce the problem with CentOS 7.2 as found on the CentOS cloud
> images repository ( http://cloud.centos.org/centos/7/images/CentOS-7-x86_
> 64-GenericCloud-1511.qcow2 ) but it "works for me".
>
> Thanks !
>
> >
> > -----Original Message-----
> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> > Of Alexandre DERUMIER
> > Sent: Thursday, June 16, 2016 17:53
> > To: Karsten Heymann; Loris Cuoghi
> > Cc: Loic Dachary; ceph-users
> > Subject: Re: [ceph-users] osds udev rules not triggered on reboot
> > (jewel, jessie)
> >
> > Hi,
> >
> > I have the same problem with osd disks not mounted at boot on jessie
> > with ceph jewel
> >
> > workaround is to re-add 60-ceph-partuuid-workaround.rules file to udev
> >
> > http://tracker.ceph.com/issues/16351
> >
> >
> > ----- Mail original -----
> > De: "aderumier" <aderum...@odiso.com>
> > À: "Karsten Heymann" <karsten.heym...@gmail.com>, "Loris Cuoghi"
> > <l...@stella-telecom.fr>
> > Cc: "Loic Dachary" <l...@dachary.org>, "ceph-users"
> > <ceph-users@lists.ceph.com>
> > Envoyé: Jeudi 28 Avril 2016 07:42:04
> > Objet: Re: [ceph-users] osds udev rules not triggered on reboot (jewel,
> jessie)
> >
> > Hi,
> > they are missing target files in debian packages
> >
> > http://tracker.ceph.com/issues/15573
> > https://github.com/ceph/ceph/pull/8700
> >
> > I have also done some other trackers about packaging bug
> >
> > jewel: debian package: wrong /etc/default/ceph/ceph location
> > http://tracker.ceph.com/issues/15587
> >
> > debian/ubuntu : TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES not specified in
> > /etc/default/cep
> > http://tracker.ceph.com/issues/15588
> >
> > jewel: debian package: init.d script bug
> > http://tracker.ceph.com/issues/15585
> >
> >
> > @CC loic dachary, maybe he could help to speed up packaging fixes
> >
> > ----- Mail original -----
> > De: "Karsten Heymann" <karsten.heym...@gmail.com>
> > À: "Loris Cuoghi" <l...@stella-telecom.fr>
> > Cc: "ceph-users" <ceph-users@lists.ceph.com>
> > Envoyé: Mercredi 27 Avril 2016 15:20:29
> > Objet: Re: [ceph-users] osds udev rules not triggered on reboot
> > (jewel, jessie)
> >
> > 2016-04-27 15:18 GMT+02:00 Loris Cuoghi <l...@stella-telecom.fr>:
> >> Le 27/04/2016 14:45, Karsten Heymann a écrit :
> >>> one workaround I found was to add
> >>>
> >>> [Install]
> >>> WantedBy=ceph-osd.target
> >>>
> >>> to /lib/systemd/system/ceph-disk@.service and then manually enable
> >>> my disks with
> >>>
> >>> # systemctl enable ceph-disk\@dev-sdi1 # systemctl start
> >>> ceph-disk\@dev-sdi1
> >>>
> >>> That way they at least are started at boot time.
> >
> >> Great! But only if the disks keep their device names, right ?
> >
> > Exactly. It's just a little workaround until the real issue is fixed.
> >
> > +Karsten
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> > ______________________________________________________________________
> > ___________________________________________________
> >
> > Ce message et ses pieces jointes peuvent contenir des informations
> > confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
> > exploites ou copies sans autorisation. Si vous avez recu ce message
> > par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que
> les pieces jointes. Les messages electroniques etant susceptibles
> d'alteration, Orange decline toute responsabilite si ce message a ete
> altere, deforme ou falsifie. Merci.
> >
> > This message and its attachments may contain confidential or
> > privileged information that may be protected by law; they should not be
> distributed, used or copied without authorisation.
> > If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> > As emails may be altered, Orange is not liable for messages that have
> been modified, changed or falsified.
> > Thank you.
> >
>
> --
> Loïc Dachary, Artisan Logiciel Libre
>
> ____________________________________________________________
> _____________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and
> delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
