Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-27 Thread Pavel Begunkov
On 1/27/25 16:38, Xan Charbonnet wrote: The MariaDB developers are wondering whether another corruption bug, MDEV-35334 ( https://jira.mariadb.org/browse/MDEV-35334 ) might be related. The symptom was described as: the first 1 byte of a .ibd file is changed from 0 to 1, or the first 4 bytes ar

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-27 Thread Pavel Begunkov
On 1/26/25 22:48, Xan Charbonnet wrote: Since applying the final patch on Friday, I have seen no problems with either the backup snapshot or catching up with replication.  It sure seems like things are all fixed.  I haven't yet tried it on our production Galera cluster, but I expect to on Mond

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-27 Thread Xan Charbonnet
The MariaDB developers are wondering whether another corruption bug, MDEV-35334 ( https://jira.mariadb.org/browse/MDEV-35334 ) might be related. The symptom was described as: the first 1 byte of a .ibd file is changed from 0 to 1, or the first 4 bytes are changed from 0 0 0 0 to 1 0 0 0. Is i

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-26 Thread Xan Charbonnet
Since applying the final patch on Friday, I have seen no problems with either the backup snapshot or catching up with replication. It sure seems like things are all fixed. I haven't yet tried it on our production Galera cluster, but I expect to on Monday. Here are Debian packages containing

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-24 Thread Jens Axboe
On 1/24/25 1:33 PM, Salvatore Bonaccorso wrote: > Hi Pavel, > > On Fri, Jan 24, 2025 at 06:40:51PM +, Pavel Begunkov wrote: >> On 1/24/25 16:30, Xan Charbonnet wrote: >>> On 1/24/25 04:33, Pavel Begunkov wrote: Thanks for narrowing it down. Xan, can you try this change please? Waiter

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-24 Thread Salvatore Bonaccorso
Hi Pavel, On Fri, Jan 24, 2025 at 06:40:51PM +, Pavel Begunkov wrote: > On 1/24/25 16:30, Xan Charbonnet wrote: > > On 1/24/25 04:33, Pavel Begunkov wrote: > > > Thanks for narrowing it down. Xan, can you try this change please? > > > Waiters can miss wake ups without it, seems to match the de

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-24 Thread Pavel Begunkov
On 1/24/25 16:30, Xan Charbonnet wrote: On 1/24/25 04:33, Pavel Begunkov wrote: Thanks for narrowing it down. Xan, can you try this change please? Waiters can miss wake ups without it, seems to match the description. diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9b58ba4616d40..e5

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-24 Thread Xan Charbonnet
On 1/24/25 04:33, Pavel Begunkov wrote: Thanks for narrowing it down. Xan, can you try this change please? Waiters can miss wake ups without it, seems to match the description. diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9b58ba4616d40..e5a8ee944ef59 100644 --- a/io_uring/io_urin

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-24 Thread Pavel Begunkov
On 1/24/25 05:24, Salvatore Bonaccorso wrote: HI Pavel, hi Jens, On Thu, Jan 23, 2025 at 11:20:40PM +, Pavel Begunkov wrote: On 1/23/25 20:49, Salvatore Bonaccorso wrote: Hi Xan, On Thu, Jan 23, 2025 at 02:31:34PM -0600, Xan Charbonnet wrote: I rented a Linode and have been trying to loa

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Salvatore Bonaccorso
HI Pavel, hi Jens, On Thu, Jan 23, 2025 at 11:20:40PM +, Pavel Begunkov wrote: > On 1/23/25 20:49, Salvatore Bonaccorso wrote: > > Hi Xan, > > > > On Thu, Jan 23, 2025 at 02:31:34PM -0600, Xan Charbonnet wrote: > > > I rented a Linode and have been trying to load it down with sysbench > > > a

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Xan Charbonnet
On 1/23/25 20:49, Salvatore Bonaccorso wrote: Additionally please try with 6.1.120 and revert this commit 3ab9326f93ec ("io_uring: wake up optimisations") (which landed in 6.1.120). If that solves the problem maybe we miss some prequisites in the 6.1.y series here? I hope I did all this rig

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Pavel Begunkov
On 1/23/25 20:49, Salvatore Bonaccorso wrote: Hi Xan, On Thu, Jan 23, 2025 at 02:31:34PM -0600, Xan Charbonnet wrote: I rented a Linode and have been trying to load it down with sysbench activity while doing a mariabackup and a mysqldump, also while spinning up the CPU with zstd benchmarks. So

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Salvatore Bonaccorso
Hi Xan, On Thu, Jan 23, 2025 at 02:31:34PM -0600, Xan Charbonnet wrote: > I rented a Linode and have been trying to load it down with sysbench > activity while doing a mariabackup and a mysqldump, also while spinning up > the CPU with zstd benchmarks. So far I've had no luck triggering the fault.

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Xan Charbonnet
I rented a Linode and have been trying to load it down with sysbench activity while doing a mariabackup and a mysqldump, also while spinning up the CPU with zstd benchmarks. So far I've had no luck triggering the fault. I've also been doing some kernel compilation. I followed this guide: htt

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Jens Axboe
On 1/23/25 1:05 PM, Salvatore Bonaccorso wrote: > Hi all, > > On Wed, Jan 22, 2025 at 08:49:13PM +0100, Salvatore Bonaccorso wrote: >> Control: forwarded -1 >> https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886 >> Hi, >> >> On Tue, Jan 21, 2025 at 08:06:18PM +0100, Bernhard Schmidt wrote: >

Bug#1093243: Upgrade to 6.1.123 kernel causes mariadb hangs

2025-01-23 Thread Salvatore Bonaccorso
Hi all, On Wed, Jan 22, 2025 at 08:49:13PM +0100, Salvatore Bonaccorso wrote: > Control: forwarded -1 https://jira.mariadb.org/projects/MDEV/issues/MDEV-35886 > Hi, > > On Tue, Jan 21, 2025 at 08:06:18PM +0100, Bernhard Schmidt wrote: > > Control: affects -1 src:mariadb > > Control: tags -1 + con