On Wed, 23 Sep 2026, Mikulas Patocka wrote:
> I did some benchmarking some times ago - and all the SSDs that I tested
> had worse IOPS for sub-4k writes than for 4k-aligned writes. This means
> that the SSDs have 4k sectors internally and do read-modify-write cycle
> for sub-4k writes.
>
> The SSDs have remapping table that maps logical blocks to locations in the
> flash chips - and RAM is expensive, so I doubt that any SSD vendor would
> put 8 times more RAM on the SSD in order to map 512-byte sectors
> individually.
I accept the measurement and what you conclude from it about the mapping
table. A drive that is slower for sub-4k writes is doing
read-modify-write, and a 4k mapping granularity is the obvious reason.
What I do not think follows is that a 4k write is therefore atomic
across a power failure. Those are two different properties:
- the mapping granularity determines how much has to be rewritten for a
partial update, which is what the IOPS measurement shows;
- atomicity across power loss depends on whether the commit is
all-or-nothing - whether the NAND program can be interrupted part way
through a page, and whether the mapping update itself survives.
Without power-loss protection there is nothing holding either up, and
consumer drives do not have it.
That is what AWUPF and NAWUPF exist to express, and it is why the value
is advertised separately rather than derived from the geometry.
The kernel takes that position too, and more firmly than I had been
arguing. nvme_configure_atomic_write() accepts only NAWUPF, explicitly
ignores controller-level AWUPF, and where neither is advertised falls
back to a single logical block. Then in nvme_update_disk_info():
/*
* Linux filesystems assume writing a single physical block is
* an atomic operation. Hence limit the physical block size to the
* value of the Atomic Write Unit Power Fail parameter.
*/
lim->logical_block_size = bs;
lim->physical_block_size = min(phys_bs, atomic_bs);
So the block layer will not even report a 4096-byte physical block
unless the drive advertised that it can write 4096 bytes atomically,
specifically so that nothing above it draws the conclusion we are being
asked to draw here. On the drive in front of me that yields
physical_block_size = 512 - the kernel declining to infer atomicity from
geometry, rather than the drive being unusual.
The empirical work agrees: the FAST'13 power-fault study found shorn
writes - partial writes inside a single page - along with metadata
corruption and worse, on most of the consumer devices tested. Whatever
the mapping granularity was on those drives, 4k writes were not atomic
on them.
So my difficulty is not that 4096 is too small a limit. It is that 4096
is not a boundary between safe and unsafe: below it there is no
guarantee either, unless the drive advertises one, and few do.
What this patch does, and does not do
-------------------------------------
It raises the maximum sector size. It does not change what any cipher
mode does when a sector is torn, and it does not make a torn sector
possible where it was not possible before - a 4096-byte sector on a
drive that advertises no atomic write unit already spans eight of them.
What a tear costs, in full, so that nothing here is hidden:
XTS, ECB each cipher block is independent, so the sector decrypts
to a mixture of old and new plaintext - what a torn write
gives on an unencrypted device
CBC P_i = D(C_i) ^ C_(i-1), so exactly one block, the one
whose predecessor is on the other side of the tear,
decrypts to garbage; the rest is old or new data
AEAD authentication fails and the read returns an error rather
than data
diffusers the whole sector decrypts to garbage; in practice only
reachable by writing a table by hand, since BITLK uses
512 or 4096
A sector whose write was not atomic is lost data in all of these. The
filesystem above cannot rely on a partially written block whatever comes
back from it. The cipher mode decides what the loss looks like - stale
data, one corrupt block, or an I/O error - not whether the data
survived, because it did not.
So I would rather document it than refuse configurations, in
Documentation/admin-guide/device-mapper/dm-crypt.rst:
An encryption sector larger than the unit the underlying device
writes atomically can be torn by a power failure, leaving part of
the sector written and part not. Unless the device advertises an
atomic write unit, that unit is a single logical block, so this is
already possible at 4096 bytes; a larger sector widens the window.
With XTS and ECB the torn sector decrypts to a mixture of old and
new data, as a torn write does on an unencrypted device. With
chaining modes the block at the tear also decrypts to garbage, and
with the wide-block diffusers the whole sector does. Use a large
sector only where losing a sector to a power failure is acceptable.
Reworded however you like, and I will put it in v3.
If enforcement is wanted later, the version worth having is a comparison
against what the device advertises rather than against a constant:
refuse a sector larger than queue_atomic_write_unit_max_bytes() for
modes that cannot absorb a tear. That cannot land as it stands, because
with NAWUPF unset it would also refuse 4096 and break existing tables,
so it needs a deprecation path rather than a one-line check. You enabled
DM_TARGET_ATOMIC_WRITES for dm-crypt last year, so the plumbing is
there, and I am happy to work on it separately.
Hardcoding 4096 seems to me to give up the guarantee the block layer is
careful not to give, while looking like it provides one - and to do it
for a combination almost nobody uses, since cryptsetup has defaulted to
xts-plain64 for plain mode, LUKS1 and LUKS2 for years.
I realise I am pushing back on a maintainer here, and I will not keep
doing it indefinitely. But I would rather make the case once, properly,
than change the patch while believing the line is drawn in the wrong
place.
v3 will carry the documentation above and the two changes I already owe
this thread: dropping the dm-verity comparison, which does not apply to
a writable target as Milan pointed out, and dropping QCE as the stated
motivation, which Eric was right about.
Thanks,
Itai
