Re: [dm-devel] [PATCH v3 6/7] dm: Inline __dm_mq_kick_requeue_list()

2023-05-23 Thread Christoph Hellwig
Looks good: Reviewed-by: Christoph Hellwig -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH 23/24] mtd: block2mtd: don't call early_lookup_bdev after the system is running

2023-05-23 Thread Christoph Hellwig
early_lookup_bdev is supposed to only be called from the early boot code, but mdtblock_early_get_bdev is called as a general fallback when lookup_bdev fails, which is problematic because early_lookup_bdev bypasses all normal path based permission checking, and might cause problems with certain cont

[dm-devel] [PATCH 24/24] block: mark early_lookup_bdev as __init

2023-05-23 Thread Christoph Hellwig
early_lookup_bdev is now only used during the early boot code as it should, so mark it __init to not waste run time memory on it. Signed-off-by: Christoph Hellwig --- block/early-lookup.c | 19 +-- include/linux/blkdev.h | 2 +- 2 files changed, 10 insertions(+), 11 deletions(

[dm-devel] [PATCH 15/24] block: move the code to do early boot lookup of block devices to block/

2023-05-23 Thread Christoph Hellwig
Create a new block/early-lookup.c to keep the early block device lookup code instead of having this code sit with the early mount code. Signed-off-by: Christoph Hellwig --- .../admin-guide/kernel-parameters.txt | 2 +- block/Makefile| 2 +- block/early

[dm-devel] [PATCH 07/24] init: refactor mount_root

2023-05-23 Thread Christoph Hellwig
Provide stubs for all the lower level mount helpers, and just switch on ROOT_DEV in the main function. Signed-off-by: Christoph Hellwig --- init/do_mounts.c | 104 +-- 1 file changed, 56 insertions(+), 48 deletions(-) diff --git a/init/do_mounts.c b/i

[dm-devel] [PATCH 17/24] dm-snap: simplify the origin_dev == cow_dev check in snapshot_ctr

2023-05-23 Thread Christoph Hellwig
Use the block_device acquired in dm_get_device for the check instead of doing an extra lookup. Signed-off-by: Christoph Hellwig --- drivers/md/dm-snap.c | 14 +- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c index 9c49f53760

[dm-devel] [PATCH 08/24] init: pass root_device_name explicitly

2023-05-23 Thread Christoph Hellwig
Instead of declaring root_device_name as a global variable pass it as an argument to the funtions using it. Signed-off-by: Christoph Hellwig --- init/do_mounts.c| 29 - init/do_mounts.h| 14 +++--- init/do_mounts_initrd.c | 11 ++- 3 fi

[dm-devel] [PATCH 20/24] dm: only call early_lookup_bdev from early boot context

2023-05-23 Thread Christoph Hellwig
early_lookup_bdev is supposed to only be called from the early boot code, but dm_get_device calls it as a general fallback when lookup_bdev fails, which is problematic because early_lookup_bdev bypasses all normal path based permission checking, and might cause problems with certain container envir

[dm-devel] [PATCH 02/24] PM: hibernate: factor out a helper to find the resume device

2023-05-23 Thread Christoph Hellwig
Split the logic to find the resume device out software_resume and into a separate helper to start unwindig the convoluted goto logic. Signed-off-by: Christoph Hellwig --- kernel/power/hibernate.c | 72 +--- 1 file changed, 37 insertions(+), 35 deletions(-) di

[dm-devel] [PATCH 12/24] init: move the nfs/cifs/ram special cases out of name_to_dev_t

2023-05-23 Thread Christoph Hellwig
The nfs/cifs/ram special cass only need to be parsed once, and only in the boot code. Move them out of name_to_dev_t and into prepare_namespace. Signed-off-by: Christoph Hellwig --- init/do_mounts.c | 14 ++ 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/init/do_mount

[dm-devel] [PATCH 18/24] dm: open code dm_get_dev_t in dm_init_init

2023-05-23 Thread Christoph Hellwig
dm_init_init is called from early boot code, and thus lookup_bdev will never succeed. Just open code that call to early_lookup_bdev instead. Signed-off-by: Christoph Hellwig --- drivers/md/dm-init.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/md/dm-init.c b/dr

[dm-devel] [PATCH 13/24] init: improve the name_to_dev_t interface

2023-05-23 Thread Christoph Hellwig
name_to_dev_t has a very misleading name, that doesn't make clear it should only be used by the early init code, and also has a bad calling convention that doesn't allow returning different kinds of errors. Rename it to early_lookup_bdev to make the use case clear, and return an errno, where -EINV

[dm-devel] [PATCH 05/24] init: remove pointless Root_* values

2023-05-23 Thread Christoph Hellwig
Remove all unused defines, and just use the expanded versions for the scsi disk majors. I've decided to keep Root_RAM0 even if it could be expanded as there is a lot of special casing for it in the init code. Signed-off-by: Christoph Hellwig --- arch/alpha/kernel/setup.c | 2 +- a

[dm-devel] [PATCH 22/24] mtd: block2mtd: factor the early block device open logic into a helper

2023-05-23 Thread Christoph Hellwig
Simply add_device a bit by splitting out the cumbersome early boot logic into a separate helper. Signed-off-by: Christoph Hellwig --- drivers/mtd/devices/block2mtd.c | 53 +++-- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/drivers/mtd/devices/block2

[dm-devel] [PATCH 09/24] init: don't remove the /dev/ prefix from error messages

2023-05-23 Thread Christoph Hellwig
Remove the code that drops the /dev/ prefix from root_device_name, which is only used for error messages when mounting the root device fails. Signed-off-by: Christoph Hellwig --- init/do_mounts.c | 17 ++--- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/init/do_mount

[dm-devel] [PATCH 19/24] dm: remove dm_get_dev_t

2023-05-23 Thread Christoph Hellwig
Open code dm_get_dev_t in the only remaining caller, and propagate the exact error code from lookup_bdev and early_lookup_bdev. Signed-off-by: Christoph Hellwig --- drivers/md/dm-table.c | 20 include/linux/device-mapper.h | 2 -- 2 files changed, 4 insertions(+), 1

[dm-devel] [PATCH 11/24] init: factor the root_wait logic in prepare_namespace into a helper

2023-05-23 Thread Christoph Hellwig
The root_wait logic is a bit obsfucated right now. Expand it and move it into a helper. Signed-off-by: Christoph Hellwig --- init/do_mounts.c | 32 ++-- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/init/do_mounts.c b/init/do_mounts.c index be6d1473

[dm-devel] [PATCH 01/24] driver core: return bool from driver_probe_done

2023-05-23 Thread Christoph Hellwig
bool is the most sensible return value for a yes/no return. Also add __init as this funtion is only called from the early boot code. Signed-off-by: Christoph Hellwig --- drivers/base/dd.c | 6 ++ include/linux/device/driver.h | 2 +- init/do_mounts.c | 2 +- 3 files

[dm-devel] [PATCH 21/24] PM: hibernate: don't use early_lookup_bdev in resume_store

2023-05-23 Thread Christoph Hellwig
resume_store is a sysfs attribute written during normal kernel runtime, and it should not use the early_lookup_bdev API that bypasses all normal path based permission checking, and might cause problems with certain container environments renaming devices. Switch to lookup_bdev, which does a normal

[dm-devel] [PATCH 06/24] init: rename mount_block_root to mount_root_generic

2023-05-23 Thread Christoph Hellwig
mount_block_root is also used to mount non-block file systems, so give it a better name. Signed-off-by: Christoph Hellwig --- init/do_mounts.c| 6 +++--- init/do_mounts.h| 2 +- init/do_mounts_initrd.c | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/init/do

[dm-devel] [PATCH 10/24] init: handle ubi/mtd root mounting like all other root types

2023-05-23 Thread Christoph Hellwig
Assign a Root_Generic magic value for UBI/MTD root and handle the root mounting in mount_root like all other root types. Besides making the code more clear this also means that UBI/MTD root can be used together with an initrd (not that anyone should care). Also factor parsing of the root name int

[dm-devel] fix the name_to_dev_t mess

2023-05-23 Thread Christoph Hellwig
Hi all, this series tries to sort out accumulated mess around the name_to_dev_t function. This function is intended to allow looking up the dev_t of a block device based on a name string before the root file systems is mounted and thus the normal path based lookup is available. Unfortunately a f

[dm-devel] [PATCH 04/24] PM: hibernate: move finding the resume device out of software_resume

2023-05-23 Thread Christoph Hellwig
software_resume can be called either from an init call in the boot code, or from sysfs once the system has finished booting, and the two invocation methods this can't race with each other. For the latter case we did just parse the suspend device manually, while the former might not have one. Spli

[dm-devel] [PATCH 14/24] init: clear root_wait on all invalid root= strings

2023-05-23 Thread Christoph Hellwig
Instead of only clearing root_wait in devt_from_partuuid when the UUID format was invalid, do that in parse_root_device for all strings that failed to parse. Signed-off-by: Christoph Hellwig --- init/do_mounts.c | 18 +++--- 1 file changed, 11 insertions(+), 7 deletions(-) diff --gi

[dm-devel] [PATCH 16/24] block: move more code to early-lookup.c

2023-05-23 Thread Christoph Hellwig
blk_lookup_devt is only used by code in early-lookup.c, so move it there. printk_all_partitions and it's helper bdevt_str are only used by the early init code in init/do_mounts.c, so they should go there as well. Signed-off-by: Christoph Hellwig --- block/early-lookup.c | 92 +

[dm-devel] [PATCH 03/24] PM: hibernate: remove the global snapshot_test variable

2023-05-23 Thread Christoph Hellwig
Passing call dependent variable in global variables is a huge antipattern. Fix it up. Signed-off-by: Christoph Hellwig --- kernel/power/hibernate.c | 17 ++--- kernel/power/power.h | 3 +-- kernel/power/swap.c | 2 +- 3 files changed, 8 insertions(+), 14 deletions(-) dif

Re: [dm-devel] [PATCH v7 0/5] Introduce provisioning primitives

2023-05-23 Thread Brian Foster
On Mon, May 22, 2023 at 02:27:57PM -0400, Mike Snitzer wrote: > On Fri, May 19 2023 at 7:07P -0400, > Dave Chinner wrote: > > > On Fri, May 19, 2023 at 10:41:31AM -0400, Mike Snitzer wrote: > > > On Fri, May 19 2023 at 12:09P -0400, > > > Christoph Hellwig wrote: > > > > > > > FYI, I really do

Re: [dm-devel] [PATCH v7 0/5] Introduce provisioning primitives

2023-05-23 Thread Mike Snitzer
On Tue, May 23 2023 at 10:05P -0400, Brian Foster wrote: > On Mon, May 22, 2023 at 02:27:57PM -0400, Mike Snitzer wrote: > > On Fri, May 19 2023 at 7:07P -0400, > > Dave Chinner wrote: > > ... > > > e.g. If the device takes a snapshot, it needs to reprovision the > > > potential COW ranges tha

Re: [dm-devel] [PATCH 01/24] driver core: return bool from driver_probe_done

2023-05-23 Thread Greg Kroah-Hartman
On Tue, May 23, 2023 at 09:45:12AM +0200, Christoph Hellwig wrote: > bool is the most sensible return value for a yes/no return. Also > add __init as this funtion is only called from the early boot code. > > Signed-off-by: Christoph Hellwig > --- > drivers/base/dd.c | 6 ++ > in

Re: [dm-devel] [PATCH 19/24] dm: remove dm_get_dev_t

2023-05-23 Thread Mike Snitzer
On Tue, May 23 2023 at 3:45P -0400, Christoph Hellwig wrote: > Open code dm_get_dev_t in the only remaining caller, and propagate the > exact error code from lookup_bdev and early_lookup_bdev. > > Signed-off-by: Christoph Hellwig > --- > drivers/md/dm-table.c | 20

Re: [dm-devel] [PATCH 17/24] dm-snap: simplify the origin_dev == cow_dev check in snapshot_ctr

2023-05-23 Thread Mike Snitzer
On Tue, May 23 2023 at 3:45P -0400, Christoph Hellwig wrote: > Use the block_device acquired in dm_get_device for the check instead > of doing an extra lookup. > > Signed-off-by: Christoph Hellwig Reviewed-by: Mike Snitzer -- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.

Re: [dm-devel] [PATCH 18/24] dm: open code dm_get_dev_t in dm_init_init

2023-05-23 Thread Mike Snitzer
On Tue, May 23 2023 at 3:45P -0400, Christoph Hellwig wrote: > dm_init_init is called from early boot code, and thus lookup_bdev > will never succeed. Just open code that call to early_lookup_bdev > instead. > > Signed-off-by: Christoph Hellwig Reviewed-by: Mike Snitzer -- dm-devel mailing

Re: [dm-devel] [PATCH 20/24] dm: only call early_lookup_bdev from early boot context

2023-05-23 Thread Mike Snitzer
On Tue, May 23 2023 at 3:45P -0400, Christoph Hellwig wrote: > early_lookup_bdev is supposed to only be called from the early boot > code, but dm_get_device calls it as a general fallback when lookup_bdev > fails, which is problematic because early_lookup_bdev bypasses all normal > path based pe

Re: [dm-devel] dm overlaybd: targets mapping OverlayBD image

2023-05-23 Thread Mike Snitzer
On Fri, May 19 2023 at 6:27P -0400, Du Rui wrote: > OverlayBD is a novel layering block-level image format, which is design > for container, secure container and applicable to virtual machine, > published in USENIX ATC '20 > https://www.usenix.org/system/files/atc20-li-huiba.pdf > > OverlayBD a

Re: [dm-devel] dm crypt: fix sleep-in-atomic-context bug in kcryptd_crypt_tasklet

2023-05-23 Thread Mike Snitzer
On Tue, May 23 2023 at 3:58P -0400, Duoming Zhou wrote: > In order to improve the IO performance of the dm-crypt > implementation, the commit 39d42fa96ba1 ("dm crypt: > add flags to optionally bypass kcryptd workqueues") > adds tasklet to do the crypto operations. > > The tasklet callback funct

[dm-devel] [PATCH v2 00/39] Add the dm-vdo deduplication and compression device mapper target.

2023-05-23 Thread J. corwin Coburn
The dm-vdo target provides inline deduplication, compression, zero-block elimination, and thin provisioning. A dm-vdo target can be backed by up to 256TB of storage, and can present a logical size of up to 4PB. This target was originally developed at Permabit Technology Corp. starting in 2009. It w

[dm-devel] [PATCH v2 02/39] Add the MurmurHash3 fast hashing algorithm.

2023-05-23 Thread J. corwin Coburn
MurmurHash3 is a fast, non-cryptographic, 128-bit hash. It was originally written by Austin Appleby and placed in the public domain. This version has been modified to produce the same result on both big endian and little endian processors, making it suitable for use in portable persistent data. Si

[dm-devel] [PATCH v2 01/39] Add documentation for dm-vdo.

2023-05-23 Thread J. corwin Coburn
This adds the admin-guide documentation for dm-vdo. vdo.rst is the guide to using dm-vdo. vdo-design is an overview of the design of dm-vdo. Signed-off-by: J. corwin Coburn --- .../admin-guide/device-mapper/vdo-design.rst | 390 ++ .../admin-guide/device-mapper/vdo.rst

[dm-devel] [PATCH v2 03/39] Add memory allocation utilities.

2023-05-23 Thread J. corwin Coburn
This patch adds standardized allocation macros and memory tracking tools to track and report any allocated memory that is not freed. This makes it easier to ensure that the vdo target does not leak memory. This patch also adds utilities for controlling whether certain threads are allowed to alloca

[dm-devel] [PATCH v2 05/39] Add vdo type declarations, constants, and simple data structures.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/constants.c| 15 + drivers/md/dm-vdo/constants.h| 102 +++ drivers/md/dm-vdo/release-versions.h | 20 ++ drivers/md/dm-vdo/status-codes.c | 126 + drivers/md/dm-vdo/status-codes.h | 112 drivers/m

[dm-devel] [PATCH v2 04/39] Add basic logging and support utilities.

2023-05-23 Thread J. corwin Coburn
Add various support utilities for the vdo target and deduplication index, including logging utilities, string and time management, and index-specific error codes. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/errors.c | 316 +++ drivers/md/dm-vdo/errors.

[dm-devel] [PATCH v2 06/39] Add thread and synchronization utilities.

2023-05-23 Thread J. corwin Coburn
This patch adds utilities for managing and using named threads, as well as several locking and sychronization utilities. These utilities help dm-vdo minimize thread transitions nad manage cross-thread interations. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/thread-cond-var.c | 46

[dm-devel] [PATCH v2 09/39] Add deduplication configuration structures.

2023-05-23 Thread J. corwin Coburn
Add structures which record the configuration of various deduplication index parameters. This also includes facilities for saving and loading the configuration and validating its integrity. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/config.c | 389 +++

[dm-devel] [PATCH v2 07/39] Add specialized request queueing functionality.

2023-05-23 Thread J. corwin Coburn
This patch adds funnel_queue, a mostly lock-free multi-producer, single-consumer queue. It also adds the request queue used by the dm-vdo deduplication index, and the work_queue used by the dm-vdo data store. Both of these are built on top of funnel queue and are intended to support the dispatching

[dm-devel] [PATCH v2 10/39] Add deduplication index storage interface.

2023-05-23 Thread J. corwin Coburn
This patch adds infrastructure for managing reads and writes to the underlying storage layer for the deduplication index. The deduplication index uses dm-bufio for all of its reads and writes, so part of this infrastructure is managing the various dm-bufio clients required. It also adds the buffere

[dm-devel] [PATCH v2 11/39] Implement the delta index.

2023-05-23 Thread J. corwin Coburn
The delta index is a space and memory efficient alternative to a hashtable. Instead of storing the entire key for each entry, the entries are sorted by key and only the difference between adjacent keys (the delta) is stored. If the keys are evenly distributed, the size of the deltas follows an expo

[dm-devel] [PATCH v2 18/39] Add vio, the request object for vdo metadata.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/vio.c | 525 drivers/md/dm-vdo/vio.h | 221 + 2 files changed, 746 insertions(+) create mode 100644 drivers/md/dm-vdo/vio.c create mode 100644 drivers/md/dm-vdo/vio.h diff --git a/dri

[dm-devel] [PATCH v2 17/39] Add administrative state and scheduling for vdo.

2023-05-23 Thread J. corwin Coburn
This patch adds the admin_state structures which are used to track the states of individual vdo components for handling of operations like suspend and resume. It also adds the action manager which is used to schedule and manage cross-thread administrative and internal operations. Signed-off-by: J.

[dm-devel] [PATCH v2 19/39] Add data_vio, the request object which services incoming bios.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/data-vio.c | 2076 ++ drivers/md/dm-vdo/data-vio.h | 683 +++ 2 files changed, 2759 insertions(+) create mode 100644 drivers/md/dm-vdo/data-vio.c create mode 100644 drivers/md/dm-vdo/data-vio.h diff

[dm-devel] [PATCH v2 23/39] Add use of the deduplication index in hash zones.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/dedupe.c | 622 + drivers/md/dm-vdo/dedupe.h | 26 ++ 2 files changed, 648 insertions(+) diff --git a/drivers/md/dm-vdo/dedupe.c b/drivers/md/dm-vdo/dedupe.c index 18c7509ef90..6ffecabd772 100644 --- a/dri

[dm-devel] [PATCH v2 21/39] Add the vdo io_submitter.

2023-05-23 Thread J. corwin Coburn
The io_submitter handles bio submission from vdo data store to the storage below. It will merge bios when possible. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/io-submitter.c | 483 +++ drivers/md/dm-vdo/io-submitter.h | 52 2 files changed, 535 insert

[dm-devel] [PATCH v2 24/39] Add the compressed block bin packer.

2023-05-23 Thread J. corwin Coburn
When blocks do not deduplicate, vdo will attempt to compress them. Up to 14 compressed blocks may be packed into a single data block (this limitation is imposed by the block map). The packer implements a simple best-fit packing algorithm and also manages the formatting and writing of compressed blo

[dm-devel] [PATCH v2 22/39] Add hash locks and hash zones.

2023-05-23 Thread J. corwin Coburn
In order to deduplicate concurrent writes of the same data (to different locations), data_vios which are writing the same data are grouped together in a "hash lock," named for and keyed by the hash of the data being written. Each hash lock is assigned to a hash zone based on a portion of its hash.

[dm-devel] [PATCH v2 25/39] Add vdo_slab.

2023-05-23 Thread J. corwin Coburn
Most of the vdo volume belongs to the slab depot. The depot contains a collection of slabs. The slabs can be up to 32GB, and are divided into three sections. Most of a slab consists of a linear sequence of 4K blocks. These blocks are used either to store data, or to hold portions of the block map (

[dm-devel] [PATCH v2 20/39] Add flush support to vdo.

2023-05-23 Thread J. corwin Coburn
This patch adds support for handling incoming flush and/or FUA bios. Each such bio is assigned to a struct vdo_flush. These are allocated as needed, but there is always one kept in reserve in case allocations fail. In the event of an allocation failure, bios may need to wait for an outstanding flus

[dm-devel] [PATCH v2 28/39] Add the slab depot itself.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/slab-depot.c | 964 + drivers/md/dm-vdo/slab-depot.h | 121 + 2 files changed, 1085 insertions(+) diff --git a/drivers/md/dm-vdo/slab-depot.c b/drivers/md/dm-vdo/slab-depot.c index 47707497eb5..0c3d66578e0

[dm-devel] [PATCH v2 32/39] Add repair (crash recovery and read-only rebuild) of damaged vdos.

2023-05-23 Thread J. corwin Coburn
When a vdo is restarted after a crash, it will automatically attempt to recover from its journals. If a vdo encounters an unrecoverable error, it will enter read-only mode. This mode indicates that some previously acknowledged data may have been lost. The vdo may be instructed to rebuild as best i

[dm-devel] [PATCH v2 37/39] Add vdo debugging support.

2023-05-23 Thread J. corwin Coburn
Add support for dumping detailed vdo state to the kernel log via a dmsetup message. The dump code is not thread-safe and is generally intended for use only when the vdo is hung. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/dump.c | 288 +++ drivers/md

[dm-devel] [PATCH v2 35/39] Add statistics tracking.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/message-stats.c | 1222 + drivers/md/dm-vdo/message-stats.h | 13 + drivers/md/dm-vdo/statistics.h| 279 +++ 3 files changed, 1514 insertions(+) create mode 100644 drivers/md/dm-vdo/message-stats.c crea

[dm-devel] [PATCH v2 39/39] Enable configuration and building of dm-vdo.

2023-05-23 Thread J. corwin Coburn
This adds dm-vdo to the drivers/md Kconfig and Makefile. Signed-off-by: J. corwin Coburn --- drivers/md/Kconfig | 16 drivers/md/Makefile | 2 ++ 2 files changed, 18 insertions(+) diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig index b0a22e99bad..9fa9dec1029 100644 --- a

[dm-devel] [PATCH v2 29/39] Add the vdo block map.

2023-05-23 Thread J. corwin Coburn
The block map contains the logical to physical mapping. It can be thought of as an array with one entry per logical address. Each entry is 5 bytes: 36 bits contain the physical block number which holds the data for the given logical address, and the remaining 4 bits are used to indicate the nature

[dm-devel] [PATCH v2 08/39] Add basic data structures.

2023-05-23 Thread J. corwin Coburn
This patch adds two hash maps, one keyed by integers, the other by pointers, and also a priority heap. The integer map is used for locking of logical and physical addresses. The pointer map is used for managing concurrent writes of the same data, ensuring that those writes are deduplicated. The pri

[dm-devel] [PATCH v2 26/39] Add the slab summary.

2023-05-23 Thread J. corwin Coburn
The slab depot maintains an additional small data structure, the "slab summary," which is used to reduce the amount of work needed to come back online after a crash. The slab summary maintains an entry for each slab indicating whether or not the slab has ever been used, whether it is clean (i.e. al

[dm-devel] [PATCH v2 34/39] Add the on-disk formats and marshalling of vdo structures.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/encodings.c | 1523 + drivers/md/dm-vdo/encodings.h | 1307 2 files changed, 2830 insertions(+) create mode 100644 drivers/md/dm-vdo/encodings.c create mode 100644 drivers/md/dm-vd

[dm-devel] [PATCH v2 12/39] Implement the volume index.

2023-05-23 Thread J. corwin Coburn
The volume index is a large delta index that maps each record name to the chapter which contains the newest record for that name. The volume index can contain several million records and is stored entirely in memory while the index is operating, accounting for the majority of the deduplication inde

[dm-devel] [PATCH v2 31/39] Add the vdo recovery journal.

2023-05-23 Thread J. corwin Coburn
The recovery journal is used to amortize updates across the block map and slab depot. Each write request causes an entry to be made in the journal. Entries are either "data remappings" or "block map remappings." For a data remapping, the journal records the logical address affected and its old and

[dm-devel] [PATCH v2 27/39] Add the block allocators and physical zones.

2023-05-23 Thread J. corwin Coburn
Each slab is independent of every other. They are assigned to "physical zones" in round-robin fashion. If there are P physical zones, then slab n is assigned to zone n mod P. The set of slabs in each physical zone is managed by a block allocator. Signed-off-by: J. corwin Coburn --- drivers/md/dm

[dm-devel] [PATCH v2 15/39] Implement top-level deduplication index.

2023-05-23 Thread J. corwin Coburn
The top-level deduplication index brings all the earlier components together. The top-level index creates the separate zone structures that enable the index to handle several requests in parallel, handles dispatching requests to the right zones and components, and coordinates metadata to ensure tha

[dm-devel] [PATCH v2 16/39] Implement external deduplication index interface.

2023-05-23 Thread J. corwin Coburn
The deduplication index interface for index clients includes the deduplication request and index session structures. This is the interface that the rest of the vdo target uses to make requests, receive responses, and collect statistics. This patch also adds sysfs nodes for inspecting various index

[dm-devel] [PATCH v2 30/39] Implement the vdo block map page cache.

2023-05-23 Thread J. corwin Coburn
The set of leaf pages of the block map tree is too large to fit in memory, so each block map zone maintains a cache of leaf pages. This patch adds the implementation of that cache. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/block-map.c | 1230 + drivers

[dm-devel] [PATCH v2 36/39] Add sysfs support for setting vdo parameters and fetching statistics.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/pool-sysfs-stats.c | 2063 ++ drivers/md/dm-vdo/pool-sysfs.c | 193 +++ drivers/md/dm-vdo/pool-sysfs.h | 19 + drivers/md/dm-vdo/sysfs.c| 84 ++ 4 files changed, 2359 insertions(+) create m

[dm-devel] [PATCH v2 13/39] Implement the open chapter and chapter indexes.

2023-05-23 Thread J. corwin Coburn
Deduplication records are stored in groups called chapters. New records are collected in a structure called the open chapter, which is optimized for adding, removing, and sorting records. When a chapter fills, it is packed into a read-only structure called a closed chapter, which is optimized for

[dm-devel] [PATCH v2 33/39] Add the vdo structure itself.

2023-05-23 Thread J. corwin Coburn
Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo/vdo.c | 1846 +++ drivers/md/dm-vdo/vdo.h | 381 2 files changed, 2227 insertions(+) create mode 100644 drivers/md/dm-vdo/vdo.c create mode 100644 drivers/md/dm-vdo/vdo.h diff --git a/drivers/md

[dm-devel] [PATCH v2 38/39] Add dm-vdo-target.c

2023-05-23 Thread J. corwin Coburn
This adds the dm-vdo target. Signed-off-by: J. corwin Coburn --- drivers/md/dm-vdo-target.c | 2983 1 file changed, 2983 insertions(+) create mode 100644 drivers/md/dm-vdo-target.c diff --git a/drivers/md/dm-vdo-target.c b/drivers/md/dm-vdo-target.c new fil

[dm-devel] [PATCH v2 14/39] Implement the chapter volume store.

2023-05-23 Thread J. corwin Coburn
The volume store structures manage the reading and writing of chapter pages. When a chapter is closed, it is packed into a read-only structure, split across several pages, and written to storage. The volume store also contains a cache and specialized queues that sort and batch requests by the page

Re: [dm-devel] [PATCH v2 02/39] Add the MurmurHash3 fast hashing algorithm.

2023-05-23 Thread Eric Biggers
On Tue, May 23, 2023 at 05:45:02PM -0400, J. corwin Coburn wrote: > MurmurHash3 is a fast, non-cryptographic, 128-bit hash. It was originally > written by Austin Appleby and placed in the public domain. This version has > been modified to produce the same result on both big endian and little > endi

Re: [dm-devel] [PATCH v2 02/39] Add the MurmurHash3 fast hashing algorithm.

2023-05-23 Thread corwin
On 5/23/23 6:06 PM, Eric Biggers wrote: On Tue, May 23, 2023 at 05:45:02PM -0400, J. corwin Coburn wrote: MurmurHash3 is a fast, non-cryptographic, 128-bit hash. It was originally written by Austin Appleby and placed in the public domain. This version has been modified to produce the same result

Re: [dm-devel] [PATCH v2 03/39] Add memory allocation utilities.

2023-05-23 Thread Eric Biggers
On Tue, May 23, 2023 at 05:45:03PM -0400, J. corwin Coburn wrote: > diff --git a/drivers/md/dm-vdo/memory-alloc.c > b/drivers/md/dm-vdo/memory-alloc.c > new file mode 100644 > index 000..00b992e96bd > --- /dev/null > +++ b/drivers/md/dm-vdo/memory-alloc.c > @@ -0,0 +1,447 @@ > +// SPDX-Lic

Re: [dm-devel] [PATCH v2 02/39] Add the MurmurHash3 fast hashing algorithm.

2023-05-23 Thread Eric Biggers
On Tue, May 23, 2023 at 06:13:08PM -0400, corwin wrote: > On 5/23/23 6:06 PM, Eric Biggers wrote: > > On Tue, May 23, 2023 at 05:45:02PM -0400, J. corwin Coburn wrote: > > > MurmurHash3 is a fast, non-cryptographic, 128-bit hash. It was originally > > > written by Austin Appleby and placed in the p

Re: [dm-devel] [PATCH v2 00/39] Add the dm-vdo deduplication and compression device mapper target.

2023-05-23 Thread Eric Biggers
On Tue, May 23, 2023 at 05:45:00PM -0400, J. corwin Coburn wrote: > The dm-vdo target provides inline deduplication, compression, zero-block > elimination, and thin provisioning. A dm-vdo target can be backed by up to > 256TB of storage, and can present a logical size of up to 4PB. This target > wa

Re: [dm-devel] [PATCH v2 02/39] Add the MurmurHash3 fast hashing algorithm.

2023-05-23 Thread Eric Biggers
On Tue, May 23, 2023 at 10:25:01PM +, Eric Biggers wrote: > On Tue, May 23, 2023 at 06:13:08PM -0400, corwin wrote: > > On 5/23/23 6:06 PM, Eric Biggers wrote: > > > On Tue, May 23, 2023 at 05:45:02PM -0400, J. corwin Coburn wrote: > > > > MurmurHash3 is a fast, non-cryptographic, 128-bit hash.

Re: [dm-devel] [PATCH v7 0/5] Introduce provisioning primitives

2023-05-23 Thread Dave Chinner
On Tue, May 23, 2023 at 11:26:18AM -0400, Mike Snitzer wrote: > On Tue, May 23 2023 at 10:05P -0400, Brian Foster wrote: > > On Mon, May 22, 2023 at 02:27:57PM -0400, Mike Snitzer wrote: > > ... since I also happen to think there is a potentially interesting > > development path to make this sort

Re: [dm-devel] [PATCH v2 02/39] Add the MurmurHash3 fast hashing algorithm.

2023-05-23 Thread corwin
On 5/23/23 7:06 PM, Eric Biggers wrote: > On Tue, May 23, 2023 at 10:25:01PM +, Eric Biggers wrote: >> On Tue, May 23, 2023 at 06:13:08PM -0400, corwin wrote: >>> On 5/23/23 6:06 PM, Eric Biggers wrote: On Tue, May 23, 2023 at 05:45:02PM -0400, J. corwin Coburn wrote: > MurmurHash3 is

Re: [dm-devel] [PATCH 03/24] PM: hibernate: remove the global snapshot_test variable

2023-05-23 Thread Rafael J. Wysocki
On Tue, May 23, 2023 at 9:45 AM Christoph Hellwig wrote: > > Passing call dependent variable in global variables is a huge > antipattern. Fix it up. > > Signed-off-by: Christoph Hellwig Acked-by: Rafael J. Wysocki > --- > kernel/power/hibernate.c | 17 ++--- > kernel/power/power.

Re: [dm-devel] [PATCH 21/24] PM: hibernate: don't use early_lookup_bdev in resume_store

2023-05-23 Thread Rafael J. Wysocki
On Tue, May 23, 2023 at 9:46 AM Christoph Hellwig wrote: > > resume_store is a sysfs attribute written during normal kernel runtime, > and it should not use the early_lookup_bdev API that bypasses all normal > path based permission checking, and might cause problems with certain > container enviro

Re: [dm-devel] [PATCH 02/24] PM: hibernate: factor out a helper to find the resume device

2023-05-23 Thread Rafael J. Wysocki
On Tue, May 23, 2023 at 9:45 AM Christoph Hellwig wrote: > > Split the logic to find the resume device out software_resume and into > a separate helper to start unwindig the convoluted goto logic. > > Signed-off-by: Christoph Hellwig Acked-by: Rafael J. Wysocki > --- > kernel/power/hibernate.

Re: [dm-devel] [PATCH v11 1/9] block: Introduce queue limits for copy-offload support

2023-05-23 Thread Nitesh Shetty
On Mon, May 22, 2023 at 08:45:44PM +0900, Damien Le Moal wrote: > On 5/22/23 19:41, Nitesh Shetty wrote: > > Add device limits as sysfs entries, > > - copy_offload (RW) > > - copy_max_bytes (RW) > > - copy_max_bytes_hw (RO) > > > > Above limits help to split the copy payloa

[dm-devel] [PATCH] dm crypt: fix sleep-in-atomic-context bug in kcryptd_crypt_tasklet

2023-05-23 Thread Duoming Zhou
In order to improve the IO performance of the dm-crypt implementation, the commit 39d42fa96ba1 ("dm crypt: add flags to optionally bypass kcryptd workqueues") adds tasklet to do the crypto operations. The tasklet callback function kcryptd_crypt_tasklet() calls kcryptd_crypt() which is an original

Re: [dm-devel] [PATCH 23/24] mtd: block2mtd: don't call early_lookup_bdev after the system is running

2023-05-23 Thread Miquel Raynal
Hi Christoph, h...@lst.de wrote on Tue, 23 May 2023 09:45:34 +0200: > early_lookup_bdev is supposed to only be called from the early boot > code, but mdtblock_early_get_bdev is called as a general fallback when > lookup_bdev fails, which is problematic because early_lookup_bdev > bypasses all nor

Re: [dm-devel] [PATCH 04/24] PM: hibernate: move finding the resume device out of software_resume

2023-05-23 Thread Rafael J. Wysocki
On Tue, May 23, 2023 at 9:45 AM Christoph Hellwig wrote: > > software_resume can be called either from an init call in the boot code, > or from sysfs once the system has finished booting, and the two > invocation methods this can't race with each other. > > For the latter case we did just parse th

Re: [dm-devel] dm overlaybd: targets mapping OverlayBD image

2023-05-23 Thread Gao Xiang
Hi Mike, On 2023/5/24 10:28, Mike Snitzer wrote: On Fri, May 19 2023 at 6:27P -0400, Du Rui wrote: OverlayBD is a novel layering block-level image format, which is design for container, secure container and applicable to virtual machine, published in USENIX ATC '20 https://www.usenix.org/sys

Re: [dm-devel] [PATCH 22/24] mtd: block2mtd: factor the early block device open logic into a helper

2023-05-23 Thread Miquel Raynal
Hi Christoph, h...@lst.de wrote on Tue, 23 May 2023 09:45:33 +0200: > Simply add_device a bit by splitting out the cumbersome early boot logic I guess you meant "Simplify..." Otherwise lgtm so, Reviewed-by: Miquel Raynal > into a separate helper. > > Signed-off-by: Christoph Hellwig > ---

Re: [dm-devel] [PATCH v2 06/39] Add thread and synchronization utilities.

2023-05-23 Thread kernel test robot
Hi corwin, kernel test robot noticed the following build warnings: [auto build test WARNING on device-mapper-dm/for-next] [also build test WARNING on song-md/md-next linus/master v6.4-rc3 next-20230523] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting

Re: [dm-devel] [PATCH 15/24] block: move the code to do early boot lookup of block devices to block/

2023-05-23 Thread Randy Dunlap
On 5/23/23 00:45, Christoph Hellwig wrote: > diff --git a/Documentation/admin-guide/kernel-parameters.txt > b/Documentation/admin-guide/kernel-parameters.txt > index f08b83e62c6222..3f8cf6dc7de887 100644 > --- a/Documentation/admin-guide/kernel-parameters.txt > +++ b/Documentation/admin-guide/k

Re: [dm-devel] [PATCH 15/24] block: move the code to do early boot lookup of block devices to block/

2023-05-23 Thread Randy Dunlap
On 5/23/23 21:58, Randy Dunlap wrote: > > > On 5/23/23 00:45, Christoph Hellwig wrote: >> diff --git a/Documentation/admin-guide/kernel-parameters.txt >> b/Documentation/admin-guide/kernel-parameters.txt >> index f08b83e62c6222..3f8cf6dc7de887 100644 >> --- a/Documentation/admin-guide/kernel-

Re: [dm-devel] [PATCH 19/24] dm: remove dm_get_dev_t

2023-05-23 Thread Christoph Hellwig
On Tue, May 23, 2023 at 12:49:16PM -0400, Mike Snitzer wrote: > > - dev = dm_get_dev_t(path); > > - if (!dev) > > + r = lookup_bdev(path, &dev); > > + if (r) > > + r = early_lookup_bdev(path, &dev); > > + if (r) > >

Re: [dm-devel] [PATCH 15/24] block: move the code to do early boot lookup of block devices to block/

2023-05-23 Thread Christoph Hellwig
On Tue, May 23, 2023 at 09:59:50PM -0700, Randy Dunlap wrote: > >>root= [KNL] Root filesystem > >> - See early_lookup_bdev comment in init/do_mounts.c. > >> + See early_lookup_bdev comment in block/early-lookup.c > > > > Patch 13 does this: > > > >