I'd like to weigh in with some perf stats on that ever-changing 15GB file, but I can't:
With TMPDIR same drive as Svn's root dir: sometime 6mins, sometimes 14 mins for the PUT. With TMPDIR different drive as Svn's root dir: sometime 6mins, sometimes 14 mins for the PUT. I can't prove or disprove the benefit of that env var setting. I have compression-level = 0 at the moment, and have filled the drive to the 3.4TB level (now full) which isn't pertinent to the question posed, I guess. With such oscillation on times for back to back PUTs on that v large resource, when nothing else is changing, I can't come back with meaningful stats after a build of Svn rom HEAD or trunk/ :-( Great work Evgeny. Is the implementation toggled somehow in the build, or another property? I looked through the diffs but couldn't see anything that stuck out. - Paul On Fri, Jul 14, 2017 at 7:38 AM, Johan Corveleyn <jcor...@gmail.com> wrote: > On Fri, Jul 14, 2017 at 1:18 PM, Evgeny Kotkov > <evgeny.kot...@visualsvn.com> wrote: > > Paul Hammant <p...@hammant.org> writes: > > > >> 1. compression-exempt-suffixes = mp3,mp4,jpeg > >> > >> 2. deltification-exempt-suffixes = mp3,mp4,jpeg > >> > >> Regardless of the setting of 'compression-level', #1 above two mean > certain > >> things can skip the compression attempt. It must give up at a certain > point > >> right? > > > > Hi everyone, > > > > To improve the situation with slow commits of large binary and, possibly, > > incompressible files I committed a patch (http://svn.apache.org/r1801940 > ) > > that adds initial support for LZ4 compression in the backend. While > still > > providing a decent compression ratio, LZ4 offers much faster compression > > even than zlib with level=1, and can skip incompressible data chunks. > > (Presumably, LZ4 is used for on-the-fly compression in different file > > systems for these reasons.) > > > > I have seen significant (up to 3 times) speed improvement for svn import > > and commit with large binary files. Sometimes, using LZ4 compression can > > even outperform a configuration with disabled compression, if the file is > > at least somehow compressible, as it reduces the time required to write > > the data to disk. > > > > Currently, LZ4 compression is enabled if the fsfs.conf file specifies > > compression-level=1, and all other levels still use zlib for compression. > > Right now, the support for LZ4 is only implemented for the file:// > protocol, > > but support/negotiation for other protocols can be added later. > > Nice! > > But how about deltification? Has anyone tried / benchmarked the effect > of turning off deltification (with or without compression), to see > what the effect would be on the commit time? Like I suggested in this > thread yesterday (i.e. set max-deltification-walk to 0 in fsfs.conf -- > or perhaps play with both max-deltification-walk and > max-linear-deltification) ... > > -- > Johan >
