Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-10 Thread Otto Wyss
>>> > gzip --compress-like=old-foo foo > >gzip creates a dictionary (that gets realy large) of strings that are >used and encodes references to them. At the start the dictionary is >empty, so the first char is pretty much unencoded and inserted into >the dictionary. The next char is encoded usi

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-10 Thread Andrew Lenharth
> > No, this won't work with very many compression algorithms. Most > > algorithms update their dictionaries/probability tables dynamically based > > on input. There isn't just one static table that could be used for > > another file, since the table is automatically updated after every (or > > n

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-09 Thread Goswin Brederlow
> " " == Otto Wyss <[EMAIL PROTECTED]> writes: >> > gzip --compress-like=old-foo foo > > where foo will be >> compressed as old-foo was or as aquivalent as > possible. Gzip >> does not need to know anything about foo except how it > was >> compressed. The switch "--compress-lik

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-09 Thread Goswin Brederlow
> " " == Otto Wyss <[EMAIL PROTECTED]> writes: >> > gzip --compress-like=old-foo foo >> >> AFAIK thats NOT possible with gzip. Same with bzip2. >> > Why not. gzip creates a dictionary (that gets realy large) of strings that are used and encodes references to them. At th

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-09 Thread Otto Wyss
> > gzip --compress-like=old-foo foo > > AFAIK thats NOT possible with gzip. Same with bzip2. > Why not. > I wish it where that simple. > I'm not saying it's simple, I'm saying it's possible. I'm not a compression speciallist but from the theory there is nothing which prevents this

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-09 Thread Otto Wyss
> > gzip --compress-like=old-foo foo > > > > where foo will be compressed as old-foo was or as aquivalent as > > possible. Gzip does not need to know anything about foo except how it > > was compressed. The switch "--compress-like" could be added to any > > compression algorithmus (bzip?)

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-09 Thread Martijn van Oosterhout
Goswin Brederlow wrote: > >> gzip --rsyncable, aloready implemented, ask Rusty Russell. > > > The --rsyncable switch might yield the same result (I haven't > > checked it sofar) but will need some internal knowledge how to > > determine the old compression. > > As far as I unde

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Goswin Brederlow
> " " == John O Sullivan <[EMAIL PROTECTED]> writes: > There was a few discussions on the rsync mailing lists about > how to handle compressed files, specifically .debs I'd like to > see some way of handling it better, but I don't think it'll > happen at the rsync end. Reas

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread John O Sullivan
There was a few discussions on the rsync mailing lists about how to handle compressed files, specifically .debs I'd like to see some way of handling it better, but I don't think it'll happen at the rsync end. Reasons include higher server cpu load to (de)compress every file that is transferred and

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Goswin Brederlow
> " " == Andrew Lenharth <[EMAIL PROTECTED]> writes: > What is better and easier is to ensure that the compression is > deturministic (gzip by default is not, bzip2 seems to be), so > that rsync can decompress, rsync, compress, and get the exact > file back on the other sid

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Goswin Brederlow
> " " == Otto Wyss <[EMAIL PROTECTED]> writes: >>> So why not solve the compression problem at the root? Why not >>> try to change the compression in a way so it does produce a >>> compressed > result >>> with the same (or similar) difference rate as the source? >> Ar

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Andrew Lenharth
> No, I want rsync not even to be mentioned. All I want is something > similar to > > gzip --compress-like=old-foo foo > > where foo will be compressed as old-foo was or as aquivalent as > possible. Gzip does not need to know anything about foo except how it > was compressed. The switch "

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Matt Zimmerman
On Mon, Jan 08, 2001 at 05:28:56PM +0100, Otto Wyss wrote: > >> So why not solve the compression problem at the root? Why not try to > >> change the compression in a way so it does produce a compressed > result > >> with the same (or similar) difference rate as the source? > > > >Are you going to

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Otto Wyss
>> So why not solve the compression problem at the root? Why not try to >> change the compression in a way so it does produce a compressed result >> with the same (or similar) difference rate as the source? > >Are you going to hack at *every* different kind of file format that you >might ever want

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Peter Eckersley
On Mon, Jan 08, 2001 at 11:58:26PM +1100, Peter Eckersley wrote: > On Mon, Jan 08, 2001 at 08:27:53AM +1100, Sam Couter wrote: > > Otto Wyss <[EMAIL PROTECTED]> wrote: > > > > > > So why not solve the compression problem at the root? Why not try to > > > change the compression in a way so it does

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Peter Eckersley
On Mon, Jan 08, 2001 at 08:27:53AM +1100, Sam Couter wrote: > Otto Wyss <[EMAIL PROTECTED]> wrote: > > > > So why not solve the compression problem at the root? Why not try to > > change the compression in a way so it does produce a compressed result > > with the same (or similar) difference rate

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-08 Thread Goswin Brederlow
> " " == Jason Gunthorpe <[EMAIL PROTECTED]> writes: > On 7 Jan 2001, Bdale Garbee wrote: >> > gzip --rsyncable, aloready implemented, ask Rusty Russell. >> >> I have a copy of Rusty's patch, but have not applied it since I >> don't like diverging Debian packages from up

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Wichert Akkerman
Previously Bdale Garbee wrote: > Wichert, have you or Rusty or anyone taken this up with the gzip upstream > maintainer? I'm not sure; I'll meet Rusty next week at linux.conf.au, I'll ask him. Wichert. -- / Generally uninteres

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Wichert Akkerman
Previously Jason Gunthorpe wrote: > Has anyone checked out what the size hit is, and how well ryncing debs > like this performs in actual use? Rusty has, the size difference is amazingly minimal. Wichert. -- / Generally unint

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Bdale Garbee
[EMAIL PROTECTED] (Matt Zimmerman) writes: > As you know, it's been eons since the last upstream gzip release. On advice of the current FSF upstream, we moved to 1.3 in November 2000. I think it is entirely reasonable to talk to upstream about this before contemplating forking. Bdale

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Matt Zimmerman
On Sun, Jan 07, 2001 at 08:16:08PM -0700, Bdale Garbee wrote: > [EMAIL PROTECTED] (Wichert Akkerman) writes: > > > gzip --rsyncable, aloready implemented, ask Rusty Russell. > > I have a copy of Rusty's patch, but have not applied it since I don't like > diverging Debian packages from upstream t

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Jason Gunthorpe
On 7 Jan 2001, Bdale Garbee wrote: > > gzip --rsyncable, aloready implemented, ask Rusty Russell. > > I have a copy of Rusty's patch, but have not applied it since I don't like > diverging Debian packages from upstream this way. Wichert, have you or Rusty > or anyone taken this up with the gzip

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Bdale Garbee
[EMAIL PROTECTED] (Wichert Akkerman) writes: > gzip --rsyncable, aloready implemented, ask Rusty Russell. I have a copy of Rusty's patch, but have not applied it since I don't like diverging Debian packages from upstream this way. Wichert, have you or Rusty or anyone taken this up with the gzip

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Goswin Brederlow
> " " == Otto Wyss <[EMAIL PROTECTED]> writes: > It's commonly agreed that compression does prevent rsync from > profit of older versions of packages when synchronizing Debian > mirrors. All the discussion about fixing rsync to solve this, > even trough a deb-plugin is IMHO

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Sam Couter
Otto Wyss <[EMAIL PROTECTED]> wrote: > > So why not solve the compression problem at the root? Why not try to > change the compression in a way so it does produce a compressed result > with the same (or similar) difference rate as the source? Are you going to hack at *every* different kind of fi

Re: Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Wichert Akkerman
Previously Otto Wyss wrote: > So why not solve the compression problem at the root? Why not try to > change the compression in a way so it does produce a compressed result > with the same (or similar) difference rate as the source? gzip --rsyncable, aloready implemented, ask Rusty Russell. Wiche

Solving the compression dilema when rsync-ing Debian versions

2001-01-07 Thread Otto Wyss
It's commonly agreed that compression does prevent rsync from profit of older versions of packages when synchronizing Debian mirrors. All the discussion about fixing rsync to solve this, even trough a deb-plugin is IMHO not the right way. Rsync's task is to synchronize files without knowing what's