Re: [RFC] Using LZ4 compression by default

Stefan Sperling writes: >> With all that in mind, I propose that we do (2). Any objections? > > I want (2) as well. Thanks for doing this work, and for very clearly > and transparently describing the tradeoffs and our options. Johan Corveleyn writes: >> With all that in mind, I propose that w

Re: [RFC] Using LZ4 compression by default

Op 19 aug. 2017 11:40 schreef "Stefan Sperling" : On Fri, Aug 18, 2017 at 10:58:13PM +0300, Evgeny Kotkov wrote: > With all that in mind, I propose that we do (2). Any objections? I want (2) as well. Thanks for doing this work, and for very clearly and transparently describing the tradeoffs and

Re: [RFC] Using LZ4 compression by default

On Fri, Aug 18, 2017 at 10:58:13PM +0300, Evgeny Kotkov wrote: > With all that in mind, I propose that we do (2). Any objections? I want (2) as well. Thanks for doing this work, and for very clearly and transparently describing the tradeoffs and our options.

Re: [RFC] Using LZ4 compression by default

ousands of times a day on a typical team. However, this is more than a fraction! Great quantitative metrics Evgeny. On Fri, Aug 18, 2017 at 2:58 PM, Evgeny Kotkov wrote: > Evgeny Kotkov writes: > > > (B) For the on-disk data, we start using LZ4 compression by default > >

Re: [RFC] Using LZ4 compression by default

Evgeny Kotkov writes: > (B) For the on-disk data, we start using LZ4 compression by default > (in format 8 repositories). > > The reasoning behind this is that currently, zlib compression is a > hotspot that can limit the performance of both read and write >

Re: [RFC] Using LZ4 compression by default

lot of_ unnecessary compression work, and that visibly slows things down. The assumption here is that low-latency connections should most likely have enough bandwidth to cover the difference between compression ratios of LZ4 and zlib-5, and allow us to use the much faster compression algorithm. (N

Re: [RFC] Using LZ4 compression by default

ce in aggregate I don't understand why the priority (biased defaults) would not be to increase the speed of every operation. ? On Fri, Aug 4, 2017 at 9:30 AM, Nathan Hartman wrote: > On Aug 2, 2017, at 2:59 PM, Evgeny Kotkov > wrote: > > With the recently added support for LZ4 compr

Re: [RFC] Using LZ4 compression by default

On Aug 2, 2017, at 2:59 PM, Evgeny Kotkov wrote: > With the recently added support for LZ4 compression (r1801940 et al), > we now have an option of using it by default for the on-disk data and > over the wire. > [...] > The amount of the required additional space is proportion

Re: [RFC] Using LZ4 compression by default

Paul Hammant writes: > Wouldn't the svn client just speculatively specify a HTTP "Accepts" header > with requests up to the server? You'd be able to do back/forwards > compatibility with that, and not have to change any other wire spec ? I think that this is close to how things currently work,

Re: [RFC] Using LZ4 compression by default

proxy is used for remote work, where we > would want to keep the slower zlib algorithm. However, if the access > to the local server happens through a proxy, we won't be currently able > to use the faster compression algorithm. (Perhaps, we could improve on > this in the future

Re: [RFC] Using LZ4 compression by default

Wouldn't the svn client just speculatively specify a HTTP "Accepts" header with requests up to the server? You'd be able to do back/forwards compatibility with that, and not have to change any other wire spec ?

Re: [RFC] Using LZ4 compression by default

Branko Čibej writes: >> - Using LZ4 over the wire requires both endpoints to advertise that they >>know how to deal with the new svndiff2 format that allows LZ4 compression. > > "Allows" or "requires"? I expect that anything in svndiff2 that&#

Re: [RFC] Using LZ4 compression by default

On 02.08.2017 20:59, Evgeny Kotkov wrote: [...] > - Using LZ4 over the wire requires both endpoints to advertise that they >know how to deal with the new svndiff2 format that allows LZ4 compression. "Allows" or "requires"? I expect that anything in svndiff2

[RFC] Using LZ4 compression by default

Hi all, With the recently added support for LZ4 compression (r1801940 et al), we now have an option of using it by default for the on-disk data and over the wire. For those who haven't been following this topic, here's a quick recap: - Currently, our default compression algorit

Re: LZ4 compression

ith > LZ4 will be used in the repository on-disk data and as the wire format for > http://, if the corresponding compression level is set to 1. > > Speaking of only using it with compression level 1, that's not a starting > point. While LZ4 offers superior speeds, it is no

Re: LZ4 compression

Evgeny wrote: Speaking of only using it with compression level 1, that's not a starting > point. While LZ4 offers superior speeds, it is not a substitute for any > zlib compression level > 1, including our current default of 5, as the > latter gives better compression ratio. &

Re: LZ4 compression

Julian Foad wrote on Mon, 17 Jul 2017 16:09 +0100: > Stefan Sperling wrote: > > On Mon, Jul 17, 2017 at 02:54:59PM +0200, Evgeny Kotkov wrote: > > Is there any reason to wait before rolling a 1.10 alpha release? > > danielsh made suggestions about the config option, in reply to the > commit email

Re: LZ4 compression

and support for LZ4 in mod_dav_svn and in ra_serf. Except for the PUT requests, [...] It would be nice to see the same negotiation added for svnserve protocol, if anyone is willing to do that. Speaking of only using it with compression level 1, that's not a starting point. While LZ4 o

Re: LZ4 compression

t; http://, if the corresponding compression level is set to 1. > > Speaking of only using it with compression level 1, that's not a starting > point. While LZ4 offers superior speeds, it is not a substitute for any > zlib compression level > 1, including our current default of 5, as

Re: LZ4 compression

t for the PUT requests, which require a couple of tweaks to the negotiation scheme, with this change svndiff2 with LZ4 will be used in the repository on-disk data and as the wire format for http://, if the corresponding compression level is set to 1. Speaking of only using it with compression le

LZ4 compression

Evgeny Kotkov wrote (in the 'Proposal: new fsfs.conf properties' thread): To improve the situation with slow commits of large binary and, possibly, incompressible files I committed a patch (http://svn.apache.org/r1801940) that adds initial support for LZ4 compression in the backe

Re: about the compression level in svndiff

Hello, On 15/04/13 23:14, Bo Chen wrote: > SVN uses zlib to compress the new data in delta. By default, the > compression level in the current implementation is 5, right? I am > wondering what are differences by choosing different compression > level? Is is a trade-off between comp

about the compression level in svndiff

SVN uses zlib to compress the new data in delta. By default, the compression level in the current implementation is 5, right? I am wondering what are differences by choosing different compression level? Is is a trade-off between compression rate and computational overhead? Thanks. Bo

Re: [serf-dev] Re: Double compression over HTTPS

On Thu, Nov 15, 2012 at 1:01 AM, Greg Stein wrote: > > On Nov 14, 2012 1:44 AM, "Lieven Govaerts" wrote: >>... > > >> Implemented in serf r1692, with an API for applications to enable SSL >> compression. Which to be clear, I don't propose svn to use.

Re: [serf-dev] Re: Double compression over HTTPS

On Nov 14, 2012 1:44 AM, "Lieven Govaerts" wrote: >... > Implemented in serf r1692, with an API for applications to enable SSL > compression. Which to be clear, I don't propose svn to use. This API would go into a 1.2 and 2.0, but if nobody is asking for it, then I'

Re: Double compression over HTTPS

the initial handshake >> "Client Hello" and "Server Hello" where client and server agree on the >> compression mechanism to be used. >> >> If the data being sent or received is already compressed, OpenSSL will >> compress it a second time. This can

Re: [serf-dev] Re: Double compression over HTTPS

On Tue, Nov 13, 2012 at 2:41 AM, Lieven Govaerts wrote: > The proposed approach here is to disable SSL compression completely, > so in terms of the above that leaves us with scenario 2. > > Serf doesn't have an option currently to disable SSL compression from > the client si

Re: [serf-dev] Re: Double compression over HTTPS

>> "Client Hello" and "Server Hello" where client and server agree on the >> compression mechanism to be used. >> >> If the data being sent or received is already compressed, OpenSSL will >> compress it a second time. This can happen when already com

Re: Double compression over HTTPS

uot; where client and server agree on the > compression mechanism to be used. > > If the data being sent or received is already compressed, OpenSSL will > compress it a second time. This can happen when already compressed binaries > like .gif or .zip are sent, or when the server uses gzip

Re: [serf-dev] Double compression over HTTPS

On Fri, Oct 5, 2012 at 1:51 PM, Lieven Govaerts wrote: > Hi, > [...] > > I have tested 4 different scenario's: > 1. As-is setup, OpenSSL compression enabled + gzip encoding enabled. (double > compression) > 2. OpenSSL compression disabled + gzip encoding enabled. (co

Re: [serf-dev] Double compression over HTTPS

hake > > "Client Hello" and "Server Hello" where client and server agree on the > > compression mechanism to be used. > > > > If the data being sent or received is already compressed, OpenSSL will > > compress it a second time. This can happen w

Re: [serf-dev] Double compression over HTTPS

On Fri, Oct 5, 2012 at 1:51 PM, Lieven Govaerts wrote: > Hi, > > when OpenSSL is built with zlib, it will automatically compress all data > sent over an SSL connection. You can see this in the initial handshake > "Client Hello" and "Server Hello" where client an

Re: Double compression over HTTPS

On Fri, Oct 5, 2012 at 12:15 PM, Bert Huijben wrote: > +1 on this. > > > > Can you see if zlib is being used (after negotiation) before requesting > compression from the Server? > It's possible, openssl has functions SSL_get_current_compression and SSL_get_current_

RE: Double compression over HTTPS

+1 on this. Can you see if zlib is being used (after negotiation) before requesting compression from the Server? Just checking if the feature is available in openssl doesn't tell if the feature is compiled in at the other side. Using 1m9 (=69 seconds) and 109 seconds in the same

Double compression over HTTPS

Hi, when OpenSSL is built with zlib, it will automatically compress all data sent over an SSL connection. You can see this in the initial handshake "Client Hello" and "Server Hello" where client and server agree on the compression mechanism to be used. If the data being

Re: svn commit: r1072282 - in /subversion/branches/integrate-compression-level/subversion: include/svn_delta.h libsvn_delta/svndiff.c svnserve/main.c svnserve/server.h

ault compression level and introduce boundary defines. * subversion/include/svn_delta.h (SVN_DEFAULT_COMPRESSSION_LEVEL): rename from SVNDIFF1_COMPRESS_LEVEL s/COMPRESSSION/COMPRESSION/ (not only in the log message, also in the code; you're quite consistent with this alternative spelling ;-))

Re: svn commit: r1072282 - in /subversion/branches/integrate-compression-level/subversion: include/svn_delta.h libsvn_delta/svndiff.c svnserve/main.c svnserve/server.h

On Sat, Feb 19, 2011 at 10:31 AM, wrote: > Author: stefan2 > Date: Sat Feb 19 09:31:01 2011 > New Revision: 1072282 > > URL: http://svn.apache.org/viewvc?rev=1072282&view=rev > Log: > Public definitions should be prefixed by SVN_. Thus, rename the default > com

Re: svn commit: r1068651 - in /subversion/branches/integrate-compression-level: ./ subversion/include/ subversion/libsvn_delta/ subversion/libsvn_ra_svn/ subversion/svnserve/ subversion/tests/libsvn_d

rev=1068651&view=rev > Log: > Introduce svn_txdelta_to_svndiff3() that introduces an additional ZIP > compression level parameter. Do all the plumbing to set that value on > the svnserve command line. For compression level 0, deactivate compression > entirely (instead of us

Re: compression

On Wed, Jun 30, 2010 at 12:01 PM, Edward Ned Harvey wrote: > Actually, I'm surprised what I'm learning now.  Although it matters if the > server is busy, that's not the root cause of the problem.  Also, changing > the compression level makes a difference, but it's

RE: compression

nless this is a bug. We're > running svn 1.5.7 via svnserve, built from source on centos 5.1 x86_64. If > anyone cares, I'll happily move to the users list. You may want to try 1.6/1.7 too. (not in production; just to see if the issues are still there) > But for the sake of con

RE: compression

7;m learning now. Although it matters if the server is busy, that's not the root cause of the problem. Also, changing the compression level makes a difference, but it's not the difference we were hoping for. I finally found a rock-solid and reproducible test case, as follows: .

RE: compression

copied uncompressed in 1min. > > > > How do you know how long the commits would have taken with compression > > disabled? > > Without svn, I "cp" or "cat > /dev/null" the file (after reboot, with cold > cache.) So I see how long it takes to do

RE: compression

> From: Daniel Shahaf [mailto:d...@daniel.shahaf.name] > > Edward Ned Harvey wrote on Tue, 29 Jun 2010 at 07:15 -: > > Svnserve is using standard zlib (not a parallel implementation) so > > You can disable all (most?) compression by not advertising the > svndiff

Re: compression

Edward Ned Harvey wrote on Tue, 29 Jun 2010 at 07:15 -: > Svnserve is using standard zlib (not a parallel implementation) so You can disable all (most?) compression by not advertising the svndiff1 capability in svnserve/serve.c. > Some people are seeing 20min commit times on change

compression

Compression can be good, or bad for performance. Whenever somebody has large binary files managed by svn, and all the access is on a high speed LAN, then compression is usually bad ... I'm trying to help a company right now, where that is the situation. Performance is *crazy* bad because