> > > > Now I'm trying to clear the stale PGs. I've tried removing the OSD from the > > crush maps, the OSD lists etc, without any luck. > > Note that this means that you destroyed all copies of those 3 PGs, which > means this experiment lost data. > > You can make ceph recreate the PGs (empty!) with > > ceph pg force_create_pg <pgid> >
Just out of curiosity, in the rbd case, what does the OS get when it tries to access data from a missing pg? I assume something like 'unrecoverable read error' on read, and some other sort of error on write. And what about when the pg is recreated? Ideally, each 'sector' of the rbd device would still initially be flagged to report as 'unrecoverable read' on read, but then a write would clear that flag, in the same way as a physical disk would, but that would require a lot of extra metadata. This would allow a filesystem with such capabilities to best recover from the data loss. James _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
