Hi Xiang,
On 2021/2/22 12:44, Gao Xiang wrote:
Hi Jianan,
On Thu, Feb 18, 2021 at 08:00:49PM +0800, Huang Jianan via Linux-erofs wrote:
From: huangjianan <huangjia...@oppo.com>
lz4 uses LZ4_DISTANCE_MAX to record history preservation. When
using rolling decompression, a block with a higher compression
ratio will cause a larger memory allocation (up to 64k). It may
cause a large resource burden in extreme cases on devices with
small memory and a large number of concurrent IOs. So appropriately
reducing this value can improve performance.
Decreasing this value will reduce the compression ratio (except
when input_size <LZ4_DISTANCE_MAX). But considering that erofs
currently only supports 4k output, reducing this value will not
significantly reduce the compression benefits.
Signed-off-by: Huang Jianan <huangjia...@oppo.com>
Signed-off-by: Guo Weichao <guoweic...@oppo.com>
---
fs/erofs/decompressor.c | 13 +++++++++----
fs/erofs/erofs_fs.h | 3 ++-
fs/erofs/internal.h | 3 +++
fs/erofs/super.c | 3 +++
4 files changed, 17 insertions(+), 5 deletions(-)
diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c
index 1cb1ffd10569..94ae56b3ff71 100644
--- a/fs/erofs/decompressor.c
+++ b/fs/erofs/decompressor.c
@@ -36,22 +36,27 @@ static int z_erofs_lz4_prepare_destpages(struct
z_erofs_decompress_req *rq,
struct page *availables[LZ4_MAX_DISTANCE_PAGES] = { NULL };
unsigned long bounced[DIV_ROUND_UP(LZ4_MAX_DISTANCE_PAGES,
BITS_PER_LONG)] = { 0 };
+ unsigned int lz4_distance_pages = LZ4_MAX_DISTANCE_PAGES;
void *kaddr = NULL;
unsigned int i, j, top;
+ if (EROFS_SB(rq->sb)->compr_alg)
+ lz4_distance_pages = DIV_ROUND_UP(EROFS_SB(rq->sb)->compr_alg,
+ PAGE_SIZE) + 1;
+
Thanks for your patch, I agree that will reduce runtime memory
footpoint. and keep max sliding window ondisk in bytes (rather
than in blocks) is better., but could we calculate lz4_distance_pages
ahead when reading super_block?
Thanks for suggestion, i will update it soon.
Also, in the next cycle, I'd like to introduce a bitmap for available
algorithms (maximum 16-bit) for the next LZMA algorithm, and for each
available algorithm introduces an on-disk variable-array like below:
bitmap(16-bit) 2 1 0
... LZMA LZ4
__le16 compr_opt_off; /* get the opt array start offset
(I think also in 4-byte) */
compr alg 0 (lz4) __le16 alg_opt_size;
/* next opt off = roundup(off + alg_opt_size, 4); */
__le16 lz4_max_distance;
/* 4-byte aligned */
compr alg x (if available) u8 alg_opt_size;
...
...
When reading sb, first, it scans the whole bitmap, and get all the
available algorithms in the image at once. And then read such compr
opts one-by-one.
Do you have some interest and extra time to implement it? :) That
makes me work less since I'm debugging mbpcluster compression now...
Sounds good, I will try to do this part of the work.
Thanks,
Jianan
Thanks,
Gao Xiang
