W dniu 23.12.2023 o 09:51, Miroslav Lachman pisze:
On 23/12/2023 08:18, Xin Li wrote:
Hi,
Inspired by D42961, I propose that we move forward with disabling the
compression by default in newsyslog, as implemented in
https://reviews.freebsd.org/D43169
Historically, newsyslog has compressed rotated log files to save disk
space. This approach was valuable in the early days where storage
space was limited. However, the landscape has changed
significantly. Modern file systems, such as ZFS, now offer native
compression capabilities. Additionally, the widespread availability
of larger hard drives has diminished the necessity for additional
compression. Notably, the need to decompress log files for pattern
searches poses a significant inconvenience, further questioning the
utility of this legacy feature.
In commit 906748d208d3, flags J, X, Y, Z can now indicate that a log
file is eligible for compression rather than directly enforcing it.
It allows for a more flexible approach, wherein the actual
compression method can be set to "none" or specified as one among
bzip2, gzip, xz, or zstd.
Therefore I would propose that we change the default compression
setting to "none" in FreeBSD 15.0. This change reflects our
adaptation to the evolving technological environment and user needs.
It also aligns with the broader initiative to modernize our systems
while maintaining flexibility and efficiency.
I look forward to your thoughts and feedback on this proposal.
Thank you for this change and future MFCs. Log files and an urge to
process them differ a lot between setups. I guess some people would back
completely disabling compression, and some would be against this
proposal. There are also small VMs with limited storage formatted with
UFS where current defaults: "-c legacy" and using bzip2 (J from
newsyslog.conf) to compress logs seems to be perfectly suited.
I don't think anything needs to be changed on newsyslog. Those who
want to disable compression can do so in the "default" newsyslog.conf
file. Why force this change in the newsyslog code?
I also don't think that the log file should not be compressed even on
a compressed filesystem. Compressed log files can be handled by tools
like zcat, zless, zgrep, etc. without first decompressing the log file.
When newsyslog has to deal only with small/medium files and logs are
just stored, if processed then very rarely, the compression method is
not very relevant, but the situation changes if you have to deal with
larger log files, which have to be processed after rotation.
While testing this change, I have done a very simple benchmark with the
largest log file which I have to rotate daily. It looks to me that
having a compressed log file sometimes makes sense even if you have to
process these logs in place. Please take a look at the result of running
zcat over the compressed file and the timings for the original,
uncompressed one. All the files are stored on ZFS lz4 compressed dataset
lying on mirrored, not so fast, SSD drives.
[r-b] /tmp# /usr/bin/time zcat flowd.0.zst > /dev/null
3.22 real 3.03 user 0.18 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.zst.orig > /dev/null
3.40 real 3.24 user 0.15 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.gz > /dev/null
6.58 real 6.46 user 0.12 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.xz > /dev/null
18.15 real 16.97 user 1.18 sys
[r-b] /tmp# /usr/bin/time zcat flowd.0.bz2 > /dev/null
45.47 real 45.33 user 0.12 sys
[r-b] /tmp# /usr/bin/time cat flowd.0 > /dev/null
3.35 real 0.25 user 3.10 sys
[r-b] /tmp# ls -l flowd.0*
-rw------- 1 root wheel 3823756280 Dec 23 09:56 flowd.0
-rw------- 1 root wheel 195036235 Dec 23 09:56 flowd.0.bz2
-rw------- 1 root wheel 305250854 Dec 23 09:56 flowd.0.gz
-rw------- 1 root wheel 180642412 Dec 23 09:56 flowd.0.xz
-rw------- 1 root wheel 252470186 Dec 23 09:56 flowd.0.zst (zstd with
--long --adapt -T0)
-rw------- 1 root wheel 297184160 Dec 23 09:56 flowd.0.zst.orig (pre
906748d208d3 zstd options)
Text log files can still grow to large sizes, and if you have a daily
backup job over a long distance network, if you use a protocol without
own compression, it is still better to have compressed log files than
uncompressed on a compressed filesystem. To me, it's the same as
compressing large database dumps and not relying on filesystem
compression.
YMMV, but I really don't see any benefit of changing the newsyslog
code, just change defaults in newsyslog.conf.
There is a clear benefit for me, users who don't want their logs to be
compressed, will have just to alter one line in /etc/crontab:
# Rotate log files every hour, if necessary.
0 * * * * root newsyslog -c none
instead of changing /etc/newsyslog.conf line by line.
Thank you for the new newsyslog(8) -c option providing this feature !
With kind regards
--
Marek Zarychta
