Sometimes I’ve had similar problems that can by fixed by running fsck against the rbd device in bare meta oob via rbd-nbd. I’ve been thinking it’s related to trim/discard and some sort of disk geometry mismatch.
> On Apr 30, 2018, at 1:22 PM, Jonathan Proulx <j...@csail.mit.edu> wrote: > > > In Proulx's Corollary to Murphy's Law, just after hitting send I tried > something that "worked". > > I noticed the volume shared nothing with the image it was based on > so tried "flattening" it just to try something. > > Oddly that worked, that or just having waited in power off state for > an hour wile I was at lunch. > > Still have no theory on why it broke or how that could be a fix...if > anyone else does please do tell :) > > Thanks, > -JOn > > On Mon, Apr 30, 2018 at 12:58:16PM -0400, Jonathan Proulx wrote: > :Hi All, > : > :I have a VM with ephemeral root on RBD spewing I/O erros on boot after > :hypervisor crash. I've (unfortunately) seen a lot of hypervisors go > :down badly with lots of VMs on them and this is a new one on me. > : > :I can 'rbd export' the volume and I get a clean filesystem. > : > :version details > : > :OpenStack: Mitaka > :Host OS: Ubuntu 16.04 > :Ceph: Luminous (12.2.4) > : > :after booting to initrd VM shows: > : > :end_request: I/O error, dev vda, sector <lots of sectors> > : > :Tried hard reboot, tried rescue (in which case vdb shows same > :issue) tried migrating to different hypervisor and all have consistent > :failure. > : > :I do have writeback caching enable on the crashed hypervisor so I can > :imaging filesystem corruption, but not this type of I/O error. > : > :Also if the rbd volume doesn't seem to be dammaged since I could dump > :it to an iamge and see correct partioning and filesystems. > : > :Anyone seen this before? I have the bits since the export worked but > :concerned about possibility of recurrence. > : > :Thanks, > :-Jon > : > :-- > > -- > > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
