Your message dated Tue, 16 Sep 2025 09:14:05 -0700
with message-id
<caou6tabxjobqt0vfaq8ugzwbpevic08uyoqfr99qehaotkr...@mail.gmail.com>
and subject line Re: Bug#1101340: mariadb-server-core:
innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
has caused the Debian Bug report #1101340,
regarding mariadb-server-core: innodb_fatal_semaphore_wait_threshold was
exceeded for dict_sys.latch.
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1101340: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1101340
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: mariadb-server-core
Version: 1:10.11.11-0+deb12u1
Severity: critical
Tags: upstream
Justification: causes serious data loss
X-Debbugs-Cc: [email protected]
Dear Maintainer,
*** Reporter, please consider answering these questions, where appropriate ***
* What led up to the situation?
Upgraded to mariadb (1:10.11.11-0+deb12u1) bookworm;
after that so did these crashes start to happen.
Previews version of mariadb in debian bookworm worked fine.
* What exactly did you do (or not do) that was effective (or
ineffective)?
No workaround so far.
This Mariadb crash happens on multiple servers with different hardware.
Uppstream repport:
https://jira.mariadb.org/projects/MDEV/issues/MDEV-35424?filter=allopenissues
The maraidb settings for the servers in this report.
thread_stack = 192K
thread_cache_size = 8
max_connections = 600
query_cache_size = 0
query_cache_type = 0
max_heap_table_size = 128M
tmp_table_size = 128M
read_buffer_size = 512K
innodb_buffer_pool_size = 12G
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 4M
innodb_flush_method = O_DIRECT
innodb_log_file_size = 1024M
innodb_log_files_in_group = 2
join_buffer_size = 2M
table_definition_cache = 10000
* What was the outcome of this action?
Crash of mariadb on first web9 server.
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61628 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61462 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61447 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61602 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61351 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:20:17 web9 mariadbd[3888776]: 2025-03-24 7:20:17 61342 [Warning]
InnoDB: A long wait (313 seconds) was observed for dict_sys.latch
Mar 24 07:40:12 web9 mariadbd[3888776]: 2025-03-24 7:40:12 0 [ERROR] [FATAL]
InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 24 07:40:12 web9 mariadbd[3888776]: 250324 7:40:12 [ERROR]
/usr/sbin/mariadbd got signal 6 ;
Mar 24 07:40:12 web9 mariadbd[3888776]: Sorry, we probably made a mistake, and
this is a bug.
Mar 24 07:40:12 web9 mariadbd[3888776]: Your assistance in bug reporting will
enable us to fix this for the next release.
Mar 24 07:40:12 web9 mariadbd[3888776]: To report this bug, see
https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 24 07:40:12 web9 mariadbd[3888776]: a bug on https://jira.mariadb.org/.
Mar 24 07:40:12 web9 mariadbd[3888776]: Please include the information from the
server start above, to the end of the
Mar 24 07:40:12 web9 mariadbd[3888776]: information below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Server version:
10.11.11-MariaDB-0+deb12u1 source revision:
e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 24 07:40:12 web9 mariadbd[3888776]: The information page at
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 24 07:40:12 web9 mariadbd[3888776]: contains instructions to obtain a
better version of the backtrace below.
Mar 24 07:40:12 web9 mariadbd[3888776]: Following these instructions will help
MariaDB developers provide a fix quicker.
Mar 24 07:40:12 web9 mariadbd[3888776]: Attempting backtrace. Include this in
the bug report.
Mar 24 07:40:12 web9 mariadbd[3888776]: (note: Retrieving this information may
fail)
Mar 24 07:40:12 web9 mariadbd[3888776]: Thread pointer: 0x0
Mar 24 07:40:12 web9 mariadbd[3888776]: stack_bottom = 0x0 thread_stack 0x30000
Mar 24 07:40:13 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5576c94642ee]
Mar 24 07:40:13 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x5576c8f42e29]
Mar 24 07:40:14 web9 mariadbd[3888776]: 2025-03-24 7:40:14 0 [Note] InnoDB:
Memory pressure event freed 771915 pages
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7f52b305b050]
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x8aeec)[0x7f52b30a9eec]
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f52b305afb2]
Mar 24 07:40:14 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f52b3045472]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(+0x6c323b)[0x5576c8b5f23b]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(+0x6bb7fa)[0x5576c8b577fa]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic13timer_generic7executeEPv+0x38)[0x5576c93fa378]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(_ZN5tpool4task7executeEv+0x2f)[0x5576c93fac6f]
Mar 24 07:40:15 web9 mariadbd[3888776]:
/usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x4f)[0x5576c93f967f]
Mar 24 07:40:16 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xd44a3)[0x7f52b32d44a3]
Mar 24 07:40:17 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7f52b30a81f5]
Mar 24 07:40:17 web9 mariadbd[3888776]:
/lib/x86_64-linux-gnu/libc.so.6(+0x10989c)[0x7f52b312889c]
Mar 24 07:40:17 web9 mariadbd[3888776]: Writing a core file...
Mar 24 07:40:17 web9 mariadbd[3888776]: Working directory at /var/lib/mysql
Mar 24 07:40:17 web9 mariadbd[3888776]: Resource Limits (excludes unlimited
resources):
Mar 24 07:40:17 web9 mariadbd[3888776]: Limit Soft Limit
Hard Limit Units
Mar 24 07:40:17 web9 mariadbd[3888776]: Max stack size 8388608
unlimited bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max core file size 0
unlimited bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max processes 256997
256997 processes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max open files 32768
32768 files
Mar 24 07:40:17 web9 mariadbd[3888776]: Max locked memory 524288
524288 bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max pending signals 256997
256997 signals
Mar 24 07:40:17 web9 mariadbd[3888776]: Max msgqueue size 819200
819200 bytes
Mar 24 07:40:17 web9 mariadbd[3888776]: Max nice priority 0
0
Mar 24 07:40:17 web9 mariadbd[3888776]: Max realtime priority 0
0
Mar 24 07:40:17 web9 mariadbd[3888776]: Core pattern: core
Mar 24 07:40:17 web9 mariadbd[3888776]: Kernel version: Linux version
6.1.0-32-amd64 ([email protected]) (gcc-12 (Debian 12.2.0-14)
12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian
6.1.129-1 (2025-03-06)
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Main process exited,
code=killed, status=6/ABRT
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 24 07:40:18 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU
time.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Scheduled restart job,
restart counter is at 1.
Mar 24 07:40:24 web9 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11
database server.
Mar 24 07:40:24 web9 systemd[1]: mariadb.service: Consumed 18h 2min 38.872s CPU
time.
Crash of mariadb on second web8 server.
Mar 22 10:55:13 web8 mariadbd[3951656]: 2025-03-22 10:55:13 0 [ERROR] [FATAL]
InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:13 web8 mariadbd[3951656]: 250322 10:55:13 [ERROR]
/usr/sbin/mariadbd got signal 6 ;
Mar 22 10:55:13 web8 mariadbd[3951656]: Sorry, we probably made a mistake, and
this is a bug.
Mar 22 10:55:13 web8 mariadbd[3951656]: Your assistance in bug reporting will
enable us to fix this for the next release.
Mar 22 10:55:13 web8 mariadbd[3951656]: To report this bug, see
https://mariadb.com/kb/en/reporting-bugs about how to report
Mar 22 10:55:13 web8 mariadbd[3951656]: a bug on https://jira.mariadb.org/.
Mar 22 10:55:13 web8 mariadbd[3951656]: Please include the information from the
server start above, to the end of the
Mar 22 10:55:13 web8 mariadbd[3951656]: information below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Server version:
10.11.11-MariaDB-0+deb12u1 source revision:
e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4
Mar 22 10:55:13 web8 mariadbd[3951656]: The information page at
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/
Mar 22 10:55:13 web8 mariadbd[3951656]: contains instructions to obtain a
better version of the backtrace below.
Mar 22 10:55:13 web8 mariadbd[3951656]: Following these instructions will help
MariaDB developers provide a fix quicker.
Mar 22 10:55:13 web8 mariadbd[3951656]: Attempting backtrace. Include this in
the bug report.
Mar 22 10:55:13 web8 mariadbd[3951656]: (note: Retrieving this information may
fail)
Mar 22 10:55:13 web8 mariadbd[3951656]: Thread pointer: 0x0
Mar 22 10:55:13 web8 mariadbd[3951656]: stack_bottom = 0x0 thread_stack 0x30000
Mar 22 10:55:16 web8 mariadbd[3951656]: 2025-03-22 10:55:16 0 [Note] InnoDB:
Memory pressure event freed 387350 pages
Mar 22 10:55:21 web8 mariadbd[3951656]:
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ed836892ee]
Mar 22 10:55:21 web8 mariadbd[3951656]:
/usr/sbin/mariadbd(handle_fatal_signal+0x229)[0x55ed83167e29]
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 0 [Note] InnoDB:
Memory pressure event freed 90 pages
Mar 22 10:55:25 web8 mariadbd[3951656]: 2025-03-22 10:55:25 418105 [ERROR]
[FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for
dict_sys.latch. Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Main process exited,
code=killed, status=6/ABRT
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Failed with result 'signal'.
Mar 22 10:55:27 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s
CPU time.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Scheduled restart job,
restart counter is at 1.
Mar 22 10:55:32 web8 systemd[1]: Stopped mariadb.service - MariaDB 10.11.11
database server.
Mar 22 10:55:32 web8 systemd[1]: mariadb.service: Consumed 1d 20h 49min 7.743s
CPU time.
Mar 22 10:55:32 web8 systemd[1]: Starting mariadb.service - MariaDB 10.11.11
database server...
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] Starting
MariaDB 10.11.11-MariaDB-0+deb12u1 source revision
e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid
Xmmv8P9Fsl7fwijTPqiP1aMaxa8= as process 200075
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Compressed tables use zlib 1.2.13
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Number of transaction pools: 1
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Using crc32 + pclmulqdq instructions
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Using liburing
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Initializing buffer pool, total size = 6.000GiB, chunk size = 96.000MiB
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Initialized memory pressure event listener
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Completed initialization of buffer pool
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
File system buffers for log disabled (block size=512 bytes)
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB:
Starting crash recovery from checkpoint LSN=2376864813498
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: End
of log at LSN=2376872752547
Mar 22 10:55:32 web8 mariadbd[200075]: 2025-03-22 10:55:32 0 [Note] InnoDB: To
recover: 1923 pages
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: 128
rollback segments are active.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
Removed temporary tablespace data file: "./ibtmp1"
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full;
Please wait ...
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
File './ibtmp1' size is now 12.000MiB.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB: log
sequence number 2376872752547; transaction id 7237064055
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] InnoDB:
Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Plugin
'FEEDBACK' is disabled.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Warning] You need
to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds
work.
Mar 22 10:55:33 web8 mariadbd[200075]: 2025-03-22 10:55:33 0 [Note] Server
socket created on IP: '127.0.0.1'.
Mar 22 10:55:34 web8 mariadbd[200075]: 2025-03-22 10:55:34 0 [Note]
/usr/sbin/mariadbd: ready for connections.
Mar 22 10:55:34 web8 mariadbd[200075]: Version: '10.11.11-MariaDB-0+deb12u1'
socket: '/run/mysqld/mysqld.sock' port: 3306 Debian 12
Mar 22 10:55:34 web8 systemd[1]: Started mariadb.service - MariaDB 10.11.11
database server.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200167]: Upgrading MySQL tables if
necessary.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200170]: /usr/bin/mysql_upgrade:
the '--basedir' option is always ignored
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200190]: Checking for insecure
root accounts.
Mar 22 10:55:34 web8 /etc/mysql/debian-start[200196]: Triggering myisam-recover
for all MyISAM tables and aria-recover for all Aria tables
Mar 22 10:55:38 web8 mariadbd[200075]: 2025-03-22 10:55:38 0 [Note] InnoDB:
Memory pressure event freed 62433 pages
Mar 22 10:55:39 web8 mariadbd[200075]: 2025-03-22 10:55:39 0 [Note] InnoDB:
Buffer pool(s) load completed at 250322 10:55:39
* What outcome did you expect instead?
That Mariadb did not crash.
*** End of the template - remove these template lines ***
-- System Information:
Debian Release: 12.10
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.1.0-32-amd64 (SMP w/64 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server-core depends on:
ii libc6 2.36-9+deb12u10
ii libcrypt1 1:4.4.33-2
ii libgcc-s1 12.2.0-14
ii libnuma1 2.0.16-1
ii libpcre2-8-0 10.42-1
ii libssl3 3.0.15-1~deb12u1
ii libstdc++6 12.2.0-14
ii libsystemd0 252.36-1~deb12u1
ii liburing2 2.3-3
ii mariadb-common 1:10.11.11-0+deb12u1
ii zlib1g 1:1.2.13.dfsg-1
mariadb-server-core recommends no packages.
mariadb-server-core suggests no packages.
-- no debconf information
--- End Message ---
--- Begin Message ---
Roger that
--- End Message ---
