Re: Steering committee, please, consider using lzip instead of xz
On Thu, Jun 8, 2017 at 2:10 AM, Matias Fonzo wrote: > On Wed, 7 Jun 2017 17:18:49 -0600 > Jeff Law wrote: > >> On 06/07/2017 02:25 PM, Antonio Diaz Diaz wrote: >> > Dear GCC steering committee, >> > >> > This has been recently asked in this list[1], but in case you have >> > missed it because of a subject line not explicit enough, I would >> > like to appeal to you directly. >> > >> > [1] http://gcc.gnu.org/ml/gcc/2017-06/msg9.html >> > >> > Since 2017-05-24 weekly snapshots use xz compression instead of >> > bzip2. I suppose this means that release tarballs will also use xz >> > at some point. >> > >> > If this is the case, I politely request you to consider using lzip >> > instead of xz. I have spent a lot of time during the last 9 years >> > developing lzip and studying the xz format, and based on this >> > experience I consider that lzip is a better choice than xz, now and >> > in the long term. >> > >> > I have been developing software since the early 80s, and I am a GNU >> > maintainer since 2003. You are all experienced developers. All I >> > ask is that you read carefully the following references and then >> > consider lzip and xz based on their technical merits. >> > >> > http://www.nongnu.org/lzip/xz_inadequate.html >> > http://www.nongnu.org/lzip/lzip_benchmark.html#xz1 >> > >> > Also note that 'lzip -9' produces a tarball a 1% smaller than xz, in >> > spite of lzip using half the RAM to compress and requiring half the >> > RAM to decompress than xz. >> > >> > -rw-r--r-- 1 58765134 2017-06-07 09:13 gcc-8-20170604.tar.lz >> > -rw-r--r-- 1 59367680 2017-06-07 09:13 gcc-8-20170604.tar.xz >> Given the breadth that xz already has in the distribution space, I'd >> have a hard time supporting moving to lz over xz. >> >> We've got far more important items to tackle than this. But if you >> want me to bring it up formally with the SC I can. >> >> jeff >> >> ps. And just to be clear, I actually don't like xz and I'm always >> annoyed when I run into something delivered in xz format. But xz >> support at the distro level is pretty ubiquitous at this point. > > All the important distributions have the lzip tool, besides GNU tar > has the support for it. If not, it is easy to add lzip for decompress > the source. This does not affect the preference of the distribution > for X format in the package redistribution. While openSUSE has it, SLES does not. tar support seems to be via calling the external lzip tool (failing if that is not available). The decision to use xz was largely due to availability. If we use lzip and 80% of the users have to fall back to the .gz tarball that would be pointless. Richard.
Re: Steering committee, please, consider using lzip instead of xz
Richard Biener wrote: While openSUSE has it, SLES does not. tar support seems to be via calling the external lzip tool (failing if that is not available). You mean SUSE Linux Enterprise Server is using a format that does not guarantee safe interoperability among implementations[1] to "power mission-critical workloads"!? [1] http://www.nongnu.org/lzip/xz_inadequate.html#fragmented The decision to use xz was largely due to availability. If we use lzip and 80% of the users have to fall back to the .gz tarball that would be pointless. For the case of SLES[2] I would say that avoiding xz is a must. IMHO SLES users are better served by falling back to the .gz tarball until the next major version of SLES is released than by suffering xz forever. [2] http://www.suse.com/products/server/ "SUSE Linux Enterprise Server 12 is a highly reliable, scalable and secure server operating system". Best regards, Antonio.
Re: Steering committee, please, consider using lzip instead of xz
Jeff Law wrote: We've got far more important items to tackle than this. But if you want me to bring it up formally with the SC I can. ps. And just to be clear, I actually don't like xz and I'm always annoyed when I run into something delivered in xz format. But xz support at the distro level is pretty ubiquitous at this point. Then please, I beg you to bring this up formally with the SC and do your best to get it considered. As Alexandre Oliva likes to quote, "you must be the change you wish to see in the world" (Gandhi). Gzip was once ubiquituous in distro packages and it was replaced. But this time distros won't lead the change because they can work around the main defects of xz. As you can read in section 2.2 of http://www.nongnu.org/lzip/xz_inadequate.html#fragmented "Distributing software in xz format can only be guaranteed to be safe if the distributor controls the decompressor run by the user (or can force the use of external means of integrity checking)". Distros control the package manager, which can even verify package signatures by default. For them xz, or even lzma-alone, is good enough. The only way for distros to change is that a significant number of upstream projects change first. This is why upstream projects willing and able to compare lzip and xz based on their technical merits are required to lead the way. The level of knowledge, experience and ethical commitment of GNU maintainers is above the mean of free software developers, and as a result xz adoption in GNU projects is lower than outside GNU. As you can see at [1], only a 10% of GNU projects are currently using xz but not lzip. A number of GNU projects (including those I maintain) are distributing tarballs in lzip format only. I always expected that GCC would not adopt xz precisely because of the high level of knowledge and experience of its main developers. [1] http://www.nongnu.org/lzip/lzip_benchmark.html#xz1 Thank you very much, Antonio.
Re: Steering committee, please, consider using lzip instead of xz
On Thu, Jun 08, 2017 at 11:27:30AM +0200, Antonio Diaz Diaz wrote: > Gzip was once ubiquituous in distro packages and it was replaced. But this > time distros won't lead the change because they can work around the main > defects of xz. As you can read in section 2.2 of > http://www.nongnu.org/lzip/xz_inadequate.html#fragmented You keep referencing the marketing pages of one of the formats comparing to other formats, that can be hardly considered unbiased. Most of the compression formats have similar kind of pages, usually biased as well. > "Distributing software in xz format can only be guaranteed to be safe if the > distributor controls the decompressor run by the user (or can force the use > of external means of integrity checking)". > > Distros control the package manager, which can even verify package > signatures by default. For them xz, or even lzma-alone, is good enough. The > only way for distros to change is that a significant number of upstream > projects change first. This is why upstream projects willing and able to > compare lzip and xz based on their technical merits are required to lead the > way. For integrity checking, gcc provides the md5.sum, sha512.sum files on gcc.gnu.org and gpg signatures on ftp.gnu.org. The choice of xz is that it is used very widely these days, which is not the case of lzip. Jakub
Re: Steering committee, please, consider using lzip instead of xz
On Thu, Jun 8, 2017 at 10:56 AM, Antonio Diaz Diaz wrote: > Richard Biener wrote: >> >> While openSUSE has it, SLES does not. tar support seems to be >> via calling the external lzip tool (failing if that is not available). > > > You mean SUSE Linux Enterprise Server is using a format that does not > guarantee safe interoperability among implementations[1] to "power > mission-critical workloads"!? SLES ships with and supports xz. It doesn't ship lzip. Source rpms may have their tarballs compressed with xz and it looks like rotated logfiles get compressed with xz. > [1] http://www.nongnu.org/lzip/xz_inadequate.html#fragmented > > >> The decision to use xz was largely due to availability. If we use lzip >> and 80% of the users have to fall back to the .gz tarball that would >> be pointless. > > > For the case of SLES[2] I would say that avoiding xz is a must. IMHO SLES > users are better served by falling back to the .gz tarball until the next > major version of SLES is released than by suffering xz forever. Not shipping xz to users is likely not an option given its wide-spread use. Not using xz compression but sth else itself might, but I do not have a strong opinion here. Richard. > [2] http://www.suse.com/products/server/ > "SUSE Linux Enterprise Server 12 is a highly reliable, scalable and secure > server operating system". > > > Best regards, > Antonio.
Re: Getting spurious FAILS in testsuite?
On 05.06.2017 18:25, Jim Wilson wrote: On 06/01/2017 05:59 AM, Georg-Johann Lay wrote: Hi, when I am running the gcc testsuite in $builddir/gcc then $ make check-gcc RUNTESTFLAGS='ubsan.exp' comes up with spurious fails. This was discussed before, and the suspicion was that it was a linux kernel bug. There were multiple kernel fixes pointed at, it wasn't clear which one was required to fix it. I have Ubuntu 16.04 LTS on my laptop, and I see the problem. I can't run the ubsan testsuites with -j factor greater than one and get reproducible results. There may also be other ways to trigger the problem. See for instance the thread https://gcc.gnu.org/ml/gcc/2016-07/msg00117.html The first message in the thread from Andrew Pinski mentions that the log output is corrupted from apparent buffer overflow. Jim I have "Ubuntu 16.04.2 LTS". Asking this at DejaGNU's, I got the following pointer: https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00034.html AFAIU there is a problem separating stdout and stderr? Johann
Re: Getting spurious FAILS in testsuite?
On 8 June 2017 at 11:57, Georg-Johann Lay wrote: > On 05.06.2017 18:25, Jim Wilson wrote: >> >> On 06/01/2017 05:59 AM, Georg-Johann Lay wrote: >>> >>> Hi, when I am running the gcc testsuite in $builddir/gcc then >>> $ make check-gcc RUNTESTFLAGS='ubsan.exp' >>> comes up with spurious fails. >> >> >> This was discussed before, and the suspicion was that it was a linux >> kernel bug. There were multiple kernel fixes pointed at, it wasn't >> clear which one was required to fix it. >> >> I have Ubuntu 16.04 LTS on my laptop, and I see the problem. I can't >> run the ubsan testsuites with -j factor greater than one and get >> reproducible results. There may also be other ways to trigger the >> problem. >> >> See for instance the thread >> https://gcc.gnu.org/ml/gcc/2016-07/msg00117.html >> The first message in the thread from Andrew Pinski mentions that the log >> output is corrupted from apparent buffer overflow. >> >> Jim > > > > I have "Ubuntu 16.04.2 LTS". > > Asking this at DejaGNU's, I got the following pointer: > > https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00034.html > > AFAIU there is a problem separating stdout and stderr? > Be careful, I'm not a dejagnu maintainer/developer :-) I just meant to say I had "similar" problems, but according to this thread, I'm not the only one :( > Johann > > >
Re: Steering committee, please, consider using lzip instead of xz
Hi, On Thu, 8 Jun 2017, Jakub Jelinek wrote: > For integrity checking, gcc provides the md5.sum, sha512.sum files on > gcc.gnu.org and gpg signatures on ftp.gnu.org. The choice of xz is that > it is used very widely these days, which is not the case of lzip. And given http://lists.gnu.org/archive/html/coreutils/2017-01/msg9.html and especially the ensuing thread thereafter doesn't exactly increase my confidence in lzip taking off. Ciao, Michael.
Re: Doxygen for GCC
On 04/28/2017 02:32 PM, Martin Liška wrote: > I'm planning to periodically build it and rsync to a public website. > I guess, sometimes it can be handy. Done that (it's weekly updated): http://gcc.opensuse.org/gcc-doxygen/ Martin
Tmjj
🏄🏾♀️🎇⌨️💽🗜🖨🌃 StXXLx Sent 📸🌃from my iPhone
Re: Steering committee, please, consider using lzip instead of xz
On Thu, 8 Jun 2017 11:42:48 +0200 Jakub Jelinek wrote: > On Thu, Jun 08, 2017 at 11:27:30AM +0200, Antonio Diaz Diaz wrote: > > Gzip was once ubiquituous in distro packages and it was replaced. > > But this time distros won't lead the change because they can work > > around the main defects of xz. As you can read in section 2.2 of > > http://www.nongnu.org/lzip/xz_inadequate.html#fragmented > > You keep referencing the marketing pages of one of the formats > comparing to other formats, that can be hardly considered unbiased. > Most of the compression formats have similar kind of pages, usually > biased as well. > > > "Distributing software in xz format can only be guaranteed to be > > safe if the distributor controls the decompressor run by the user > > (or can force the use of external means of integrity checking)". > > > > Distros control the package manager, which can even verify package > > signatures by default. For them xz, or even lzma-alone, is good > > enough. The only way for distros to change is that a significant > > number of upstream projects change first. This is why upstream > > projects willing and able to compare lzip and xz based on their > > technical merits are required to lead the way. > > For integrity checking, gcc provides the md5.sum, sha512.sum files on > gcc.gnu.org and gpg signatures on ftp.gnu.org. The choice of xz is > that it is used very widely these days, which is not the case of lzip. > This works well as a complement, but this seems to be a mere excuse to palliate the defects of the compressor, in this case of xz. It would be different if the signatures are accompanied with a well-designed compressor (like lzip). pgpj49ucJA8qn.pgp Description: OpenPGP digital signature
Re: Steering committee, please, consider using lzip instead of xz
Jakub Jelinek wrote: http://www.nongnu.org/lzip/xz_inadequate.html#fragmented You keep referencing the marketing pages of one of the formats comparing to other formats, that can be hardly considered unbiased. Most of the compression formats have similar kind of pages, usually biased as well. The above article is intended to be a serious (and as unbiased as possible) analysis of the xz format. AFAIK, this is the only analysis of the xz format ever written. I can't find anything similar in the xz site[1], the bzip2 site[2], nor in any of the two gzip sites[3][4]. All the formats document their file format and usually offer some kind of benchmark, but that is all. If you know of any such analysis, specially of xz or lzip, please share it. [1] http://tukaani.org/xz/ [2] http://www.bzip.org/ [3] http://www.gnu.org/software/gzip/ [4] http://www.gzip.org/ BTW, writing such an in-dept analysis is a lot of work, and it hurts when someone describes it as "marketing" without verifying it first. If anybody finds any error or inaccuracy in the above article I'll be happy to correct it. Thanks. The choice of xz is that it is used very widely these days, which is not the case of lzip. IMHO It is pretty obvious that whatever format is chosen to distribute the code of a major project (GCC, coreutils, linux) will become widely used, and in the meanwhile the users without support for the new format can fall back to the .gz tarball, which is not so much larger than the current bzip2 tarball. Best regards, Antonio.
Re: Steering committee, please, consider using lzip instead of xz
On 06/08/2017 03:27 AM, Antonio Diaz Diaz wrote: > Jeff Law wrote: >> We've got far more important items to tackle than this. But if you >> want me to bring it up formally with the SC I can. >> >> ps. And just to be clear, I actually don't like xz and I'm always >> annoyed when I run into something delivered in xz format. But xz >> support at the distro level is pretty ubiquitous at this point. > > Then please, I beg you to bring this up formally with the SC and do your > best to get it considered. As Alexandre Oliva likes to quote, "you must > be the change you wish to see in the world" (Gandhi). It's been raised with the SC. Please do not continue to push this discussion here. Jeff
Re: Getting spurious FAILS in testsuite?
On 06/08/2017 04:24 AM, Christophe Lyon wrote: > On 8 June 2017 at 11:57, Georg-Johann Lay wrote: >> On 05.06.2017 18:25, Jim Wilson wrote: >>> >>> On 06/01/2017 05:59 AM, Georg-Johann Lay wrote: Hi, when I am running the gcc testsuite in $builddir/gcc then $ make check-gcc RUNTESTFLAGS='ubsan.exp' comes up with spurious fails. >>> >>> >>> This was discussed before, and the suspicion was that it was a linux >>> kernel bug. There were multiple kernel fixes pointed at, it wasn't >>> clear which one was required to fix it. >>> >>> I have Ubuntu 16.04 LTS on my laptop, and I see the problem. I can't >>> run the ubsan testsuites with -j factor greater than one and get >>> reproducible results. There may also be other ways to trigger the >>> problem. >>> >>> See for instance the thread >>> https://gcc.gnu.org/ml/gcc/2016-07/msg00117.html >>> The first message in the thread from Andrew Pinski mentions that the log >>> output is corrupted from apparent buffer overflow. >>> >>> Jim >> >> >> >> I have "Ubuntu 16.04.2 LTS". >> >> Asking this at DejaGNU's, I got the following pointer: >> >> https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00034.html >> >> AFAIU there is a problem separating stdout and stderr? >> > > Be careful, I'm not a dejagnu maintainer/developer :-) > I just meant to say I had "similar" problems, but according to this > thread, I'm not the only one :( There was most definitely a linux kernel bug that affected the behavior of "expect" used by dejagnu in the past. THe gcc.gnu.org reference to a message from Pinski is the right one -- it identifies the problematical change in the kernel that mucked up expect's behavior. In the thread you'll find a reference to: https://bugzilla.kernel.org/show_bug.cgi?id=96311 This has long since been fixed. BUt I have no idea what version of hte kernel is in Ubuntu and whether or not it is subject to this problem. jeff
Re: Getting spurious FAILS in testsuite?
On Thu, Jun 8, 2017 at 2:25 PM, Jeff Law wrote: > On 06/08/2017 04:24 AM, Christophe Lyon wrote: >> On 8 June 2017 at 11:57, Georg-Johann Lay wrote: >>> On 05.06.2017 18:25, Jim Wilson wrote: On 06/01/2017 05:59 AM, Georg-Johann Lay wrote: > > Hi, when I am running the gcc testsuite in $builddir/gcc then > $ make check-gcc RUNTESTFLAGS='ubsan.exp' > comes up with spurious fails. This was discussed before, and the suspicion was that it was a linux kernel bug. There were multiple kernel fixes pointed at, it wasn't clear which one was required to fix it. I have Ubuntu 16.04 LTS on my laptop, and I see the problem. I can't run the ubsan testsuites with -j factor greater than one and get reproducible results. There may also be other ways to trigger the problem. See for instance the thread https://gcc.gnu.org/ml/gcc/2016-07/msg00117.html The first message in the thread from Andrew Pinski mentions that the log output is corrupted from apparent buffer overflow. Jim >>> >>> >>> >>> I have "Ubuntu 16.04.2 LTS". >>> >>> Asking this at DejaGNU's, I got the following pointer: >>> >>> https://lists.gnu.org/archive/html/dejagnu/2016-03/msg00034.html >>> >>> AFAIU there is a problem separating stdout and stderr? >>> >> >> Be careful, I'm not a dejagnu maintainer/developer :-) >> I just meant to say I had "similar" problems, but according to this >> thread, I'm not the only one :( > There was most definitely a linux kernel bug that affected the behavior > of "expect" used by dejagnu in the past. > > THe gcc.gnu.org reference to a message from Pinski is the right one -- > it identifies the problematical change in the kernel that mucked up > expect's behavior. > > In the thread you'll find a reference to: > > https://bugzilla.kernel.org/show_bug.cgi?id=96311 > > This has long since been fixed. BUt I have no idea what version of hte > kernel is in Ubuntu and whether or not it is subject to this problem. I think 4.10 or 4.11 has the full fix. There was still some bugs even in 4.8 (which was the one included with Ubuntu 1604). I only say this is because I have a 4.8 kernel which sees the problem but a 4.11 kernel does not. The behavior I see with a non fixed kernel is that the guailty tests will not run at all. With the fixed kernel, it will run. Thanks, Andrew > > jeff
gcc-7-20170608 is now available
Snapshot gcc-7-20170608 is now available on ftp://gcc.gnu.org/pub/gcc/snapshots/7-20170608/ and on various mirrors, see http://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 7 SVN branch with the following options: svn://gcc.gnu.org/svn/gcc/branches/gcc-7-branch revision 249041 You'll find: gcc-7-20170608.tar.xzComplete GCC SHA256=8862da0f8a2a4d29637624233501b745a3cba3d8cd3d78a4de30e11b64c358e2 SHA1=2407ae281ccc7bd093712f7ad947b965b7c8e8f6 Diffs from 7-20170601 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-7 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
Re: Doxygen for GCC
On 6/8/17, Martin Liška wrote: > On 04/28/2017 02:32 PM, Martin Liška wrote: >> I'm planning to periodically build it and rsync to a public website. >> I guess, sometimes it can be handy. > > Done that (it's weekly updated): > > http://gcc.opensuse.org/gcc-doxygen/ > > Martin > Clicking on the first link on the sidebar, it looks like doxygen needs to be taught how to handle gcc's kind of md files: http://gcc.opensuse.org/gcc-doxygen/md_gcc_common.html Has there been a bug raised with doxygen for this?
