Auerbach, Steven wrote: > [10/Aug/2021:09:03:52.832686801 -0400] - NOTICE - dblayer_start - Detected > Disorderly Shutdown last time Directory Server was running, recovering > database. > [10/Aug/2021:09:03:53.307038716 -0400] - ERR - libdb - BDB2506 file > /var/lib/dirsrv/slapd-FBOG-LOCAL/cldb/21741a1f-b31a11ea-ac83c7bf-de3c3622_5eded6dc000000600000.db > has LSN 1859/5569522, past end of log at 1859/5527979 > [10/Aug/2021:09:03:53.309248835 -0400] - ERR - libdb - BDB2507 Commonly > caused by moving a database from one database environment > [10/Aug/2021:09:03:53.310844909 -0400] - ERR - libdb - BDB2508 to another > without clearing the database LSNs, or by removing all of > [10/Aug/2021:09:03:53.312311253 -0400] - ERR - libdb - BDB2509 the log files > from a database environment > [10/Aug/2021:09:03:53.313770893 -0400] - ERR - libdb - BDB1521 Recovery > function for LSN 1859 5496332 failed > [10/Aug/2021:09:03:53.315181085 -0400] - ERR - libdb - BDB0061 PANIC: Invalid > argument > [10/Aug/2021:09:03:53.327435763 -0400] - ERR - libdb - BDB1546 unable to join > the environment > [10/Aug/2021:09:03:53.343830873 -0400] - CRIT - dblayer_start - Database > Recovery Process FAILED. The database is not recoverable. err=-30973: BDB0087 > DB_RUNRECOVERY: Fatal error, run database recovery > [10/Aug/2021:09:03:53.345786469 -0400] - CRIT - dblayer_start - Please make > sure there is enough disk space for dbcache (1610612736 bytes) and db region > files > [10/Aug/2021:09:03:53.347245636 -0400] - ERR - ldbm_back_start - Failed to > init database, err=-30973 BDB0087 DB_RUNRECOVERY: Fatal error, run database > recovery > [10/Aug/2021:09:03:53.349104988 -0400] - ERR - plugin_dependency_startall - > Failed to start database plugin ldbm database > [10/Aug/2021:09:03:53.350954638 -0400] - ERR - schema-compat-plugin - > scheduled schema-compat-plugin tree scan in about 5 seconds after the server > startup! > [10/Aug/2021:09:03:53.353877687 -0400] - WARN - > ldbm_instance_add_instance_entry_callback - ldbm instance userRoot already > exists > [10/Aug/2021:09:03:53.355345539 -0400] - ERR - > ldbm_config_read_instance_entries - Failed to add instance entry > cn=userRoot,cn=ldbm database,cn=plugins,cn=config > [10/Aug/2021:09:03:53.356791214 -0400] - ERR - ldbm_config_load_dse_info - > failed to read instance entries > [10/Aug/2021:09:03:53.358068888 -0400] - ERR - ldbm_back_start - Loading > database configuration failed > [10/Aug/2021:09:03:53.359235194 -0400] - ERR - plugin_dependency_startall - > Failed to start database plugin ldbm database > [10/Aug/2021:09:03:53.360000476 -0400] - ERR - plugin_dependency_startall - > Failed to resolve plugin dependencies > [10/Aug/2021:09:03:53.360703493 -0400] - ERR - plugin_dependency_startall - > betxnpreoperation plugin 7-bit check is not started > [10/Aug/2021:09:03:53.361576474 -0400] - ERR - plugin_dependency_startall - > preoperation plugin Account Usability Plugin is not started > [10/Aug/2021:09:03:53.362552803 -0400] - ERR - plugin_dependency_startall - > accesscontrol plugin ACL Plugin is not started > [10/Aug/2021:09:03:53.363610744 -0400] - ERR - plugin_dependency_startall - > preoperation plugin ACL preoperation is not started > [10/Aug/2021:09:03:53.364277146 -0400] - ERR - plugin_dependency_startall - > betxnpreoperation plugin Auto Membership Plugin is not started > [10/Aug/2021:09:03:53.365004305 -0400] - ERR - plugin_dependency_startall - > preoperation plugin caacl name uniqueness is not started > [10/Aug/2021:09:03:53.365741513 -0400] - ERR - plugin_dependency_startall - > preoperation plugin certificate store issuer/serial uniqueness is not started > ....more things not started in the log. > > There are 39 GB available on root filesystem so that should meet the " make > sure there is enough disk space for dbcache (1610612736 bytes) and db region > files" recommendation > If database recovery fails (Database Recovery Process FAILED. The database is > not recoverable. err=-30973: BDB0087 DB_RUNRECOVERY: Fatal error, run > database recovery), what do we do?
I'd try db_recover first. Change to the database directory in /var/lib/dirsrv/slapd-FBOG-LOCAL/db Then run: # db_recover -c -f -v -c catastrophic recovery -f progress -v verbose rob > > -Steven > > -----Original Message----- > From: Rob Crittenden <rcrit...@redhat.com> > Sent: Tuesday, August 10, 2021 9:19 AM > To: FreeIPA users list <freeipa-users@lists.fedorahosted.org> > Cc: Shirley Schaeffer <shirley_schaef...@nwrdc.fsu.edu>; Simpson, Brett > <brett_simp...@nwrdc.fsu.edu>; Auerbach, Steven <steven.auerb...@flbog.edu> > Subject: Re: [Freeipa-users] Hard Crash of Server Corrupted IPA > > Auerbach, Steven via FreeIPA-users wrote: >> A storage subsystem failure below our virtualization layer caused a >> hard crash of our 2^nd IPA Master. It will not start back up. >> >> $ Systemctl status –l ipa >> >> ● ipa.service - Identity, Policy, Audit >> >> Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; >> vendor >> preset: disabled) >> >> Active: failed (Result: exit-code) since Fri 2021-08-06 15:47:24 >> EDT; >> 3 days ago >> >> Process: 1554 ExecStart=/usr/sbin/ipactl start (code=exited, >> status=1/FAILURE) >> >> Main PID: 1554 (code=exited, status=1/FAILURE) >> >> >> >> Aug 06 15:46:46 ipa04.fbog.local systemd[1]: Starting Identity, >> Policy, Audit... >> >> Aug 06 15:47:24 ipa04.fbog.local ipactl[1554]: Failed to start >> Directory >> Service: Command '/bin/systemctl start dirsrv@FBOG-LOCAL.service' >> returned non-zero exit status 1 >> >> Aug 06 15:47:24 ipa04.fbog.local ipactl[1554]: Starting Directory >> Service >> >> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: ipa.service: main process >> exited, code=exited, status=1/FAILURE >> >> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: Failed to start Identity, >> Policy, Audit. >> >> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: Unit ipa.service entered >> failed state. >> >> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: ipa.service failed. >> >> >> >> Multiple OS restarts do not clear this. There must be a pid file >> somewhere to delete. Not sure where to look in documentation or a >> meaningful search expression for researching the web. >> >> Help? > > You need to look in the 389-ds error log for details, > /var/log/dirsrv/slapd-FBOG-LOCAL/errors > > rob > _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
