On Mon, Jan 28, 2019 at 04:24:37PM -0500, Dennis Zhou wrote: > As mentioned above, a requirement that differs zstd from zlib is that > higher levels of compression require more memory. To manage this, each > compression level has its own queue of workspaces. A global LRU is used > to help with reclaim. To guarantee forward progress, a max level > workspace is preallocated and hidden from the LRU. > > When getting a workspace, it uses a bitmap to identify the levels that > are populated and scans up. If it finds a workspace that is greater than > it, it uses it, but does not update the last_used time and the > corresponding place in the LRU. This provides a mechanism to decrease > memory utilization as we only keep around workspaces that are sized > appropriately for the in use compression levels. > > By knowing which compression levels have available workspaces, we can > recycle rather than always create new workspaces as well as take > advantage of the preallocated max level for forward progress. If we hit > memory pressure, we sleep on the max level workspace. We continue to > rescan in case we can use a smaller workspace, but eventually should be > able to obtain the max level workspace or allocate one again should > memory pressure subside. The memory requirement for decompression is the > same as level 1, and therefore can use any of available workspace. > > The number of workspaces is bound by an upper limit of the workqueue's > limit which currently is 2 (percpu limit). Second, a reclaim timer is > used to free inactive/improperly sized workspaces. The reclaim timer is > set to 67s to avoid colliding with transaction commit (every 30s) and > attempts to reclaim any unused workspace older than 45s.
> --- a/fs/btrfs/zstd.c > +++ b/fs/btrfs/zstd.c > #define ZSTD_BTRFS_MAX_WINDOWLOG 17 > #define ZSTD_BTRFS_MAX_INPUT (1 << ZSTD_BTRFS_MAX_WINDOWLOG) > #define ZSTD_BTRFS_DEFAULT_LEVEL 3 > +#define ZSTD_BTRFS_MAX_LEVEL 15 > +#define ZSTD_BTRFS_RECLAIM_NS (45 * NSEC_PER_SEC) > +/* 67s to avoid clashing with transaction commit (every 30s) */ > +#define ZSTD_BTRFS_RECLAIM_JIFFIES (67 * HZ) I think you can copy the relevant part of changelog here to explain the logic of the levels and reclaim as it's scattered over several functions. The description would give a nice overview in one place.
