"First-time poster here - we're a small ops team running a Galera cluster in
production and ran into what we believe is a significant, reproducible InnoDB
purge bug during a recovery effort. Details below."
Subject: Re: MDEV-33315 - InnoDB purge coordinator permanently frozen on
10.11.18 (Galera), strong reproduction evidence, unable to access JIRA (CAPTCHA)
Hi all,
We are experiencing what appears to be the same issue reported in MDEV-33315
("InnoDB history length and undo tablespace files keep growing"), which was
closed as Incomplete due to insufficient reproduction data. We believe we now
have significantly stronger evidence and would like to either get that issue
reopened or start a new discussion here, since we are currently unable to log
into jira.mariadb.org (the CAPTCHA is not rendering/completing for us, tried
multiple browsers).
If anyone here has the ability to reopen MDEV-33315 or can point us to an
alternative way to attach this report, we'd appreciate it.
--- Summary ---
After an unclean shutdown/crash on a Galera cluster node, the InnoDB purge
coordinator gets stuck at a fixed transaction ID boundary and makes zero
progress across multiple full process restarts (each including a fresh crash
recovery) - bit-for-bit identical, even after hours of active runtime. No
visible blocking transaction, no lock conflict, no error log entry explains
this.
--- Environment ---
- MariaDB 10.11.18-MariaDB-deb12-log (also reproduced on 10.11.15; upgrading
did not help)
- Galera 26.4.27 (rf032a460)
- Debian 12
- innodb_undo_tablespaces=0 (undo lives in the shared system tablespace,
ibdata1)
- innodb_buffer_pool_size=49152M
--- Key evidence ---
1) Purge boundary frozen across 3 full restarts + crash recovery over ~5 hours:
Snapshot 1:
Trx id counter 5466324699
Purge done for trx's n:o < 5463787581
History list length 1096233
Snapshot 2 (~5h later, 3 restarts incl. kill -9 + crash recovery):
Trx id counter 5466324749
Purge done for trx's n:o < 5463787581 <- identical
History list length 1096250
2) Backlog grows unbounded under continued production activity (after a ~1-20GB
SQL dump import, 291 tables, thousands of new transactions):
Trx id counter 5466326793 (+2094 new transactions)
Purge done for trx's n:o < 5463787581 <- still identical, nothing purged
History list length 1097123
3) The frozen state is physically persisted: a second cluster node received the
data via Galera SST (mariabackup, physical file copy) and shows the exact same
frozen purge boundary post-transfer - pointing to a persistent issue in the
undo tablespace metadata rather than a transient in-memory state of one process.
4) Ruled out: open/blocking transactions (INNODB_TRX empty), prepared XA
transactions (XA RECOVER empty), innodb_force_recovery=0, innodb_read_only=OFF,
storage/filesystem errors (dmesg/journalctl clean, all error status counters at
0), purge thread/batch tuning (innodb_purge_threads=8, innodb_max_purge_lag=0,
innodb_purge_batch_size=1000 - no effect). A forced innodb_fast_shutdown=0
shutdown ran for several hours with continuous CPU activity but no visible
reduction in History list length before being aborted.
5) A related MDL side-effect: a background thread identified as
thread/innodb/thread_pool_thread (performance_schema.threads, TYPE: BACKGROUND)
held a permanent SHARED MDL on a system table, blocking a DROP TABLE
indefinitely with no timeout - consistent with the purge coordinator holding
locks while not making progress.
We have full SHOW ENGINE INNODB STATUS snapshots, gdb thread dumps (partially
symbolized, we ran into a build-ID mismatch installing dbgsym after the fact),
and error logs available and are happy to share the complete report or run
additional diagnostics on our reproduction environment (we still have it
running on a clone).
Thanks in advance for any pointers,
Domenik Falk
_______________________________________________
discuss mailing list -- [email protected]
To unsubscribe send an email to [email protected]
