> You can try... > unmount the filesystem and the run a force e2fsck on all OSTs and MDT. > > e2fsck -f -p /dev/ost... > > Regargs.
Thank you Mahmoud. The problem is that e2fsck in the MDT runs very very slowly…10 inodes per second for more than 100 million inodes. >> Are there a lot of inodes moved to lost+found by the fsck, which contribute >> to the occupied quota now? Thank you Martin. I can’t finish the e2fsck in the MDT. > Correct, LFSCK will repair the OST object ownership, but does not *directly* > repair quota > files. That said, if an object is owned by the wrong owner for some reason, > changing the > ownership _should_ transfer the quota usage properly to the new owner, so in > that regard > the quota usage should be indirectly repaired by an LFSCK run. Thank you Andreas. I can confirm that lfsck layout does not repair the wrong quotes. I’m running lfsck namespace in the MDT to try other way. > Note the "tune2fs -O ^quota" to repair the quota accounting is a bit of a > "cargo cult" behaviour. > > Running "e2fsck -fp" is the proper way to repair the quota files, since it > not only recalculates the quota accounting using the same code as "tune2fs -O > quota" does, but it also ensures that the files themselves are valid in the > first place. They should take about the same time, except in the case your > filesystem is corrupted, in which case you'd want e2fsck to repair the > filesystem anyway. I ran this in the lfsck MDT and lfsck OSTs. After do that the quotes for all users were more corrupted than before (for example for a user with 27 TB the quota reports 900 MB). Then I ran the e2fsck in the lfsck OSTs. I tried to do the same in the lfsck MDT but it ran very very slow….and I canceled it. I suppose that the correct way to repair quota is to run the e2fsck in the MDT, but the problem is the system downtime. I copy the lfsck stats at the end of this message: > lctl get_param -n mdd.lustre-MDT0000.lfsck_layout > name: lfsck_layout > magic: 0xb17371b9 > version: 2 > status: partial > flags: > param: > last_completed_time: 1555393553 > time_since_last_completed: 50098 seconds > latest_start_time: 1555339300 > time_since_latest_start: 104351 seconds > last_checkpoint_time: 1555393553 > time_since_last_checkpoint: 50098 seconds > latest_start_position: 77 > last_checkpoint_position: 102983584 > first_failure_position: 21342296 > success_count: 1 > repaired_dangling: 2470024 > repaired_unmatched_pair: 7487028 > repaired_multiple_referenced: 0 > repaired_orphan: 0 > repaired_inconsistent_owner: 18950288 > repaired_others: 0 > skipped: 0 > failed_phase1: 1622485 > failed_phase2: 0 > checked_phase1: 119083557 > checked_phase2: 0 > run_time_phase1: 31777 seconds > run_time_phase2: 22423 seconds > average_speed_phase1: 3747 items/sec > average_speed_phase2: 0 objs/sec > real-time_speed_phase1: N/A > real-time_speed_phase2: N/A > current_position: N/A > lctl lfsck_query -t namespace -M lustre-MDT0000 > namespace_mdts_init: 0 > namespace_mdts_scanning-phase1: 0 > namespace_mdts_scanning-phase2: 1 > namespace_mdts_completed: 0 > namespace_mdts_failed: 0 > namespace_mdts_stopped: 0 > namespace_mdts_paused: 0 > namespace_mdts_crashed: 0 > namespace_mdts_partial: 0 > namespace_mdts_co-failed: 0 > namespace_mdts_co-stopped: 0 > namespace_mdts_co-paused: 0 > namespace_mdts_unknown: 0 > namespace_osts_init: 0 > namespace_osts_scanning-phase1: 0 > namespace_osts_scanning-phase2: 0 > namespace_osts_completed: 0 > namespace_osts_failed: 0 > namespace_osts_stopped: 0 > namespace_osts_paused: 0 > namespace_osts_crashed: 0 > namespace_osts_partial: 0 > namespace_osts_co-failed: 0 > namespace_osts_co-stopped: 0 > namespace_osts_co-paused: 0 > namespace_osts_unknown: 0 > namespace_repaired: 4550034 ============================================= Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 ============================================= > El 16 abr 2019, a las 20:43, Martin Hecht <[email protected]> escribió: > > Are there a lot of inodes moved to lost+found by the fsck, which contribute > to the occupied quota now? > > ----- Ursprüngliche Mail ----- > Von: Fernando Pérez <[email protected]> > An: [email protected] > Gesendet: Tue, 16 Apr 2019 16:24:13 +0200 (CEST) > Betreff: Re: [lustre-discuss] lfsck repair quota > > Thank you Rick. > > I followed these steps for the ldiskfs OSTs and MDT, but the quotes for all > users is more corrupted than before. > > I tried to run e2fsck in ldiskfs OSTs MDT, but the problem was the MDT e2fsck > ran very slow ( 10 inodes per second for more than 100 million inodes). > > According to the lustre wiki I though that the lfsck could repair corrupted > quotes: > > http://wiki.lustre.org/Lustre_Quota_Troubleshooting > > Regards. > > ============================================ > Fernando Pérez > Institut de Ciències del Mar (CSIC) > Departament Oceanografía Física i Tecnològica > Passeig Marítim de la Barceloneta,37-49 > 08003 Barcelona > Phone: (+34) 93 230 96 35 > ============================================ > >> El 16 abr 2019, a las 15:34, Mohr Jr, Richard Frank (Rick Mohr) >> <[email protected]> escribió: >> >> >>> On Apr 15, 2019, at 10:54 AM, Fernando Perez <[email protected]> wrote: >>> >>> Could anyone confirm me that the correct way to repair wrong quotes in a >>> ldiskfs mdt is lctl lfsck_start -t layout -A? >> >> As far as I know, lfsck doesn’t repair quota info. It only fixes internal >> consistency within Lustre. >> >> Whenever I have had to repair quotas, I just follow the procedure you did >> (unmount everything, run “tune2fs -O ^quota <dev>”, run “tune2fs -O quota >> <dev>”, and then remount). But all my systems used ldiskfs, so I don’t know >> if the ZFS OSTs introduce any sort of complication. (Actually, I am not >> even sure if/how you can regenerate quota info for ZFS.) >> >> -- >> Rick Mohr >> Senior HPC System Administrator >> National Institute for Computational Sciences >> http://www.nics.tennessee.edu >> > _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
