Update: drive firmware is not the cause.
We flashed all 12 KIOXIA drives of one host from GPK4 to GPK6 (latest vendor
firmware, HPE component GPK6(B)) and
booted 6.8.0-138 without iommu=pt. The first IO_PAGE_FAULT and checksum failure
came 10 minutes after boot:
15:26:13.169815 kernel: nvme 0000:82:00.0: AMD-Vi: Event logged
[IO_PAGE_FAULT domain=0x002b address=0xfbf2a800 flags=0x0020]
15:26:13.169931 ceph-osd: bluestore(...) _verify_csum bad crc32c/0x1000 ...
During the same time the host on 6.8.0-138 + iommu=pt stayed clean (0
IO_PAGE_FAULTs, 0 checksum failures after 7 h+),
as did the hosts on 6.8.0-83.
So far:
- 6.8.0-65 .. 6.8.0-124: good
- 6.8.0-138 and 6.8.0-147 (proposed): bad
- 6.8.0-138 + iommu=pt: good
- drive firmware GPK4 or GPK6, cold power cycle: no difference
** Description changed:
- Description:
After upgrading from 6.8.0-124 (and older) to 6.8.0-138.138~22.04.1, hosts
with KIOXIA CM6 NVMe drives on AMD EPYC
get reads that return wrong data. Every bad read comes with an AMD-Vi
IO_PAGE_FAULT from the NVMe controller at the
same moment. A re-read of the same block returns the correct data; data on
disk is fine.
We see the bad reads through Ceph BlueStore, which checksums every 4 KiB
block on read and retries on mismatch
(health warning BLUESTORE_SPURIOUS_READ_ERRORS). On one host during one boot,
all 36 checksum failures match
36 IO_PAGE_FAULTs to the second (on 6.8.0-147: 19 and 19):
- 14:55:21.455819 kernel: nvme 0000:82:00.0: AMD-Vi: Event logged
[IO_PAGE_FAULT domain=0x002b address=0x91390a00 flags=0x0020]
- 14:55:21.456114 ceph-osd: bluestore(...) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x3000, got 0x5ae22152, expected 0xc8e6ccd7
+ 14:55:21.455819 kernel: nvme 0000:82:00.0: AMD-Vi: Event logged
[IO_PAGE_FAULT domain=0x002b address=0x91390a00 flags=0x0020]
+ 14:55:21.456114 ceph-osd: bluestore(...) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x3000, got 0x5ae22152, expected 0xc8e6ccd7
All faults have flags=0x0020 (a write to a not-present page, i.e. the device
writing read data into host memory), and the
fault addresses are 512-byte aligned. No NVMe timeouts, resets or errors are
logged. The data that comes back is not
zeroes (the checksum of an all-zero block, 0x6706be76, never appears).
Without application-level checksums this is
silent data corruption.
With iommu=pt the same kernel shows no faults and no bad reads. So the device
DMA targets the right buffer, but the
IOMMU has no mapping for it at that moment: the mapping is set up too late or
torn down too early.
Hardware / software:
- HPE ProLiant DL345 Gen11, AMD EPYC 9334 32-Core, BIOS 1.58 and 2.86
- 12x KIOXIA KCM6FRUL3T84 (CM6, PCI 1e0f:0007), firmware GPK4, native nvme
driver, 64 I/O queues per drive
- Ubuntu 22.04.5, linux-hwe-6.8; kernel cmdline without IOMMU options
- dmesg: "iommu: Default domain type: Translated", "DMA domain TLB
invalidation policy: lazy mode",
- "AMD-Vi: Interrupt remapping enabled"
+ "AMD-Vi: Interrupt remapping enabled"
Kernel A/B (failures = checksum mismatches on read; same workload, ~100-175
reads/s per host):
- 6.8.0-65 up to 6.8.0-124: 0 failures (about 130 hosts, 14 days)
- 6.8.0-138: 8-26 failures per host per day. Cluster 1: one host rebooted
from -85 into -138 had 0 failures before
- and failures every day for 33 days after; its 3 neighbours stayed on -85
with 0.
- Cluster 2: all 4 hosts failing on -138; one host booted back into -83: 0
failures since.
+ and failures every day for 33 days after; its 3 neighbours stayed on -85
with 0.
+ Cluster 2: all 4 hosts failing on -138; one host booted back into -83: 0
failures since.
- 6.8.0-147.147~22.04.1 (jammy-proposed, includes the CVE-2026-68329 fix
"iommu/amd: Wait for completion instead of
- returning early in iommu_completion_wait()"): still failing, ~2.2 per hour.
+ returning early in iommu_completion_wait()"): still failing, ~2.2 per hour.
- 6.8.0-138 + iommu=pt: 0 failures and 0 IO_PAGE_FAULTs in 6.5 h on a host
that had ~1.8 failures per hour on -138
- without it (about 12 expected), while a neighbour on -147 failed during the
same time.
+ without it (about 12 expected), while a neighbour on -147 failed during the
same time.
- Same drives behind a SCSI/tri-mode controller (shown as sda, not nvme) on
-138: 0 failures (13 hosts, up to 57 days).
- Lightly loaded host (~3 reads/s) on -138: 0 failures in 45 days; the
trigger seems to need concurrent I/O.
- Firmware (GPK4), BIOS and hardware were the same on good and bad kernels; a
cold power cycle on -138 did not help.
So the regression entered between 6.8.0-125 and 6.8.0-138, is still in -147,
and sits in the DMA mapping path through the
AMD IOMMU in translated (lazy) mode. Candidates from the changelog in that
range include
"iommu/amd: serialize sequence allocation under concurrent TLB
invalidations", the nvme-pci queue changes
("cap queue creation to used queues", "ensure we're polling a polled queue",
"Fix race bug in nvme_poll_irqdisable()")
and "block: add pgmap check to biovec_phys_mergeable". We have not bisected
yet; we can test intermediate kernels
(-130, -131, -134, -136, -137) or test kernels if that helps.
Workaround: iommu=pt on the kernel cmdline (or stay on <= -124).
The noble linux 6.8.0-138 source is presumably affected as well (not
tested).
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172387
Title:
Regression in 6.8.0-138: AMD-Vi IO_PAGE_FAULT on NVMe read DMA returns
wrong data (still in -147, iommu=pt avoids it)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-hwe-6.8/+bug/2172387/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
