Richard Biener <richard.guent...@gmail.com> writes:
> On Tue, Sep 13, 2016 at 6:01 PM, Joseph Myers <jos...@codesourcery.com> wrote:
>> On Tue, 13 Sep 2016, Moritz Klammler wrote:
>>
>>> I have made an actual diff now, containing also the checksum files.
>>> I
>>
>> I don't think checksums of lots of miscellaneous files should be
>> included, just the checksums for those files the current script will
>> actually use.
>
> I generally like the script but agree with Joseph here. We should be
> able to verify (upon committing changes to the script) to verify the
> sums by performing ./contrib/download_prerequesites [--md5] which
> means only including those that can be verified that way.
>
> Richard.
Thanks both of you for your feedback. I generally agree that it would
be cleaner to only include those checksums that are actually needed. On
the other hand, it means an increased maintenance burden each time the
version of the dependency is changed. In order to mitigate this and
eventually get the best of both worlds, I have added another
(intentionally undocumented) option `--strip-sums` to the script that
will strip the checksum file to include only the packages it will
download. A maintainer can run the script with this option to both
strip the checksum file and immediately test the new configuration. A
comment in the script next to where the version numbers are set reminds
you to do that.
If there were also GPG signatures of the checksum files on the server, I
could further modify the script to even download (and verify) the new
checksum files automatically, assuming that the maintainer who applies
the update will have the GPG key of the person who maintains the FTP
directory. (Might well be the same person anyway, I don't know.)
By the way: as mentioned in the bug tracker, there also is the
`./contrib/download_ecj` script but I believe that it will become
obsolete with the upcoming deletion of GCJ so we can ignore it for now.
Is this right?
* contrib/download_prerequisites: Verify integrity of downloaded
packages and added more command-line options.
* contrib/prerequisites.sha512: New.
* contrib/prerequisites.md5: New.
Index: contrib/download_prerequisites
===================================================================
--- contrib/download_prerequisites (revision 240137)
+++ contrib/download_prerequisites (working copy)
@@ -1,60 +1,240 @@
-#! /bin/sh
+#! /bin/sh -eu
+#! -*- coding:utf-8; mode:shell-script; -*-
-# Download some prerequisites needed by gcc.
-# Run this from the top level of the gcc source tree and the gcc
-# build will do the right thing.
+# Download some prerequisites needed by GCC.
+# Run this from the top level of the GCC source tree and the GCC build will do
+# the right thing. Run it with the `--help` option for more information.
#
-# (C) 2010-2016 Free Software Foundation
+# (C) 2016 Free Software Foundation
#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see http://www.gnu.org/licenses/.
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see http://www.gnu.org/licenses/.
-# If you want to disable Graphite loop optimizations while building GCC,
-# DO NOT set GRAPHITE_LOOP_OPT as yes so that the isl package will not
-# be downloaded.
-GRAPHITE_LOOP_OPT=yes
-if [ ! -e gcc/BASE-VER ] ; then
- echo "You must run this script in the top level GCC source directory."
- exit 1
-fi
+program='download_prerequisites'
+version='(unversioned)'
-# Necessary to build GCC.
-MPFR=mpfr-3.1.4
-GMP=gmp-6.1.0
-MPC=mpc-1.0.3
+# MAINTAINERS: If you update the package versions below, please
+# replace the files `contrib/prerequisites.sha512` and
+# `contrib/prerequisites.md5` with a new authoritative copy of the
+# checksum files and then run this script with the (intentionally
+# undocumented) `--strip-sums` option. This will strip any unneeded
+# entries from the checksum file and immediately self-test the new
+# configuration. You have to do this twice, once with the `--sha512`
+# (or none) and once with the `--md5` option in order to strip the
+# checksum files for both hash algorithms.
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPFR.tar.bz2 || exit 1
-tar xjf $MPFR.tar.bz2 || exit 1
-if test -L mpfr; then rm -f mpfr; fi
-ln -sf $MPFR mpfr || exit 1
+gmp='gmp-6.1.0.tar.bz2'
+mpfr='mpfr-3.1.4.tar.bz2'
+mpc='mpc-1.0.3.tar.gz'
+isl='isl-0.16.1.tar.bz2'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$GMP.tar.bz2 || exit 1
-tar xjf $GMP.tar.bz2 || exit 1
-if test -L gmp; then rm -f gmp; fi
-ln -sf $GMP gmp || exit 1
+base_url='ftp://gcc.gnu.org/pub/gcc/infrastructure/'
-wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$MPC.tar.gz || exit 1
-tar xzf $MPC.tar.gz || exit 1
-if test -L mpc; then rm -f mpc; fi
-ln -sf $MPC mpc || exit 1
+echo_archives() {
+ echo "${gmp}"
+ echo "${mpfr}"
+ echo "${mpc}"
+ if [ ${graphite} -gt 0 ]; then echo "${isl}"; fi
+}
-# Necessary to build GCC with the Graphite loop optimizations.
-if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
- ISL=isl-0.16.1
+graphite=1
+verify=1
+force=0
+chksum='sha512'
+directory='.'
+stripsums=0
- wget ftp://gcc.gnu.org/pub/gcc/infrastructure/$ISL.tar.bz2 || exit 1
- tar xjf $ISL.tar.bz2 || exit 1
- if test -L isl; then rm -f isl; fi
- ln -sf $ISL isl || exit 1
+helptext="usage: ${program} [OPTION...]
+
+Downloads some prerequisites needed by GCC. Run this from the top level of the
+GCC source tree and the GCC build will do the right thing.
+
+The following options are available:
+
+ --directory=DIR download and unpack packages into DIR instead of '.'
+ --force download again overwriting existing packages
+ --no-force do not download existing packages again (default)
+ --isl download ISL, needed for Graphite loop optimizations (default)
+ --graphite same as --isl
+ --no-isl don't download ISL
+ --no-graphite same as --no-isl
+ --verify verify package integrity after download (default)
+ --no-verify don't verify package integrity
+ --sha512 use SHA512 checksum to verify package integrity (default)
+ --md5 use MD5 checksum to verify package integrity
+ --help show this text and exit
+ --version show version information and exit
+"
+
+versiontext="${program} ${version}
+Copyright (C) 2016 Free Software Foundation, Inc.
+This is free software; see the source for copying conditions. There is NO
+warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
+
+die() {
+ echo "error: $@" >&2
+ exit 1
+}
+
+for arg in "$@"
+do
+ case "${arg}" in
+ --help)
+ echo "${helptext}"
+ exit
+ ;;
+ --version)
+ echo "${versiontext}"
+ exit
+ ;;
+ esac
+done
+unset arg
+
+argnext=
+for arg in "$@"
+do
+ if [ "x${argnext}" = x ]
+ then
+ case "${arg}" in
+ --directory)
+ argnext='directory'
+ ;;
+ --directory=*)
+ directory="${arg#--directory=}"
+ ;;
+ --force)
+ force=1
+ ;;
+ --no-force)
+ force=0
+ ;;
+ --isl|--graphite)
+ graphite=1
+ ;;
+ --no-isl|--no-graphite)
+ graphite=0
+ ;;
+ --verify)
+ verify=1
+ ;;
+ --no-verify)
+ verify=0
+ ;;
+ --sha512)
+ chksum='sha512'
+ verify=1
+ ;;
+ --md5)
+ chksum='md5'
+ verify=1
+ ;;
+ --strip-sums)
+ stripsums=1
+ ;;
+ -*)
+ die "unknown option: ${arg}"
+ ;;
+ *)
+ die "too many arguments"
+ ;;
+ esac
+ else
+ case "${arg}" in
+ -*)
+ die "Missing argument for option --${argnext}"
+ ;;
+ esac
+ case "${argnext}" in
+ directory)
+ directory="${arg}"
+ ;;
+ *)
+ die "The impossible has happened"
+ ;;
+ esac
+ argnext=
+ fi
+done
+[ "x${argnext}" = x ] || die "Missing argument for option --${argnext}"
+unset arg argnext
+
+[ -e ./gcc/BASE-VER ] \
+ || die "You must run this script in the top-level GCC source directory"
+
+[ -d "${directory}" ] \
+ || die "No such directory: ${directory}"
+
+if [ ${stripsums} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ stripsumfile="${chksumfile}.stripped"
+ rm -f "${stripsumfile}"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" >> "${stripsumfile}" \
+ || die "Cannot find checksum for '${ar}' in file '${chksumfile}'"
+ done
+ mv "${stripsumfile}" "${chksumfile}" \
+ || die "Cannot overwrite '${chksumfile}' with '${stripsumfile}'"
+ unset ar chksumfile stripsumfile
fi
+
+for ar in $(echo_archives)
+do
+ if [ ${force} -gt 0 ]; then rm -f "${directory}/${ar}"; fi
+ [ -e "${directory}/${ar}" ] \
+ || wget --no-verbose -O "${directory}/${ar}" "${base_url}${ar}" \
+ || die "Cannot download ${ar} from ${base_url}"
+done
+unset ar
+
+if [ ${verify} -gt 0 ]
+then
+ chksumfile="contrib/prerequisites.${chksum}"
+ [ -r "${chksumfile}" ] || die "No checksums available"
+ for ar in $(echo_archives)
+ do
+ grep "${ar}" "${chksumfile}" \
+ | ( cd "${directory}" && "${chksum}sum" --check ) \
+ || die "Cannot verify integrity of possibly corrupted file ${ar}"
+ done
+ unset chksumfile
+fi
+unset ar
+
+for ar in $(echo_archives)
+do
+ package="${ar%.tar*}"
+ if [ ${force} -gt 0 ]; then rm -rf "${directory}/${package}"; fi
+ [ -e "${directory}/${package}" ] \
+ || ( cd "${directory}" && tar -xf "${ar}" ) \
+ || die "Cannot extract package from ${ar}"
+ unset package
+done
+unset ar
+
+for ar in $(echo_archives)
+do
+ target="${directory}/${ar%.tar*}/"
+ linkname="${ar%-*}"
+ if [ ${force} -gt 0 ]; then rm -f "${linkname}"; fi
+ [ -e "${linkname}" ] \
+ || ln -s "${target}" "${linkname}" \
+ || die "Cannot create symbolic link ${linkname} --> ${target}"
+ unset target linkname
+done
+unset ar
+
+echo "All prerequisites downloaded successfully."
Index: contrib/prerequisites.md5
===================================================================
--- contrib/prerequisites.md5 (nonexistent)
+++ contrib/prerequisites.md5 (working copy)
@@ -0,0 +1,4 @@
+86ee6e54ebfc4a90b643a65e402c4048 gmp-6.1.0.tar.bz2
+b8a2f6b0e68bef46e53da2ac439e1cf4 mpfr-3.1.4.tar.bz2
+d6a1d5f8ddea3abd2cc3e98f58352d26 mpc-1.0.3.tar.gz
+ac1f25a0677912952718a51f5bc20f32 isl-0.16.1.tar.bz2
Index: contrib/prerequisites.sha512
===================================================================
--- contrib/prerequisites.sha512 (nonexistent)
+++ contrib/prerequisites.sha512 (working copy)
@@ -0,0 +1,4 @@
+3c82aeab9c1596d4da8afac2eec38e429e84f3211e1a572cf8fd2b546493c44c039b922a1133eaaa48bd7f3e11dbe795a384e21ed95cbe3ecc58d7ac02246117 gmp-6.1.0.tar.bz2
+51066066ff2c12ed2198605ecf68846b0c96b548adafa5b80e0c786d0df488411a5e8973358fce7192dc977ad4e68414cf14500e3c39746de62465eb145bb819 mpfr-3.1.4.tar.bz2
+0028b76df130720c1fad7de937a0d041224806ce5ef76589f19c7b49d956071a683e2f20d154c192a231e69756b19e48208f2889b0c13950ceb7b3cfaf059a43 mpc-1.0.3.tar.gz
+c188667a84dc5bdddb4ab7c35f89c91bf15a8171f4fcaf41301cf285fb7328846d9a367c096012fec4cc69d244f0bc9e95d84c09ec097394cd4093076f2a041b isl-0.16.1.tar.bz2
