Package: release.debian.org
Severity: normal
User: release.debian....@packages.debian.org
Usertags: unblock
Hi,
ccache 3.3.3 (which is in testing) has a serious bug which leads to
potentially
bad content of dependency files when compiling identical source code but
with
different source paths.
Debian bug: #853834
Upstream bug: https://github.com/ccache/ccache/issues/134
ccache 3.3.4 fixes this and two other minor bugs. No other functionality
was touched between 3.3.3 and 3.3.4. Please see the attached diff (changes
to auto-generated files have been trimmed).
I think that it would be a good idea to let ccache/3.3.4-1 into testing.
unblock ccache/3.3.4-1
-- Joel
diff -Nru ccache-3.3.3/debian/changelog ccache-3.3.4/debian/changelog
--- ccache-3.3.3/debian/changelog 2016-10-28 00:00:15.000000000 +0200
+++ ccache-3.3.4/debian/changelog 2017-02-17 23:02:40.000000000 +0100
@@ -1,3 +1,18 @@
+ccache (3.3.4-1) unstable; urgency=high
+
+ * New upstream release 3.3.4, which contains the following changes:
+ * Fixes a serious regression in ccache 3.3 related to potentially bad
+ content of dependency files when compiling identical source code but
+ with different source paths (closes: #853834).
+ * Trivial bug fix: Corrects which statistics counter to update for
+ -optf/--options-file failure.
+ * Trivial bug fixes: Fixes undefined behavior warnings found by
+ -fsanitize=undefined.
+ * Documentation update: Adds description of statistics counters to the
+ man page.
+
+ -- Joel Rosdahl <j...@debian.org> Fri, 17 Feb 2017 23:02:40 +0100
+
ccache (3.3.3-1) unstable; urgency=medium
* New upstream release 3.3.3
diff -Nru ccache-3.3.3/AUTHORS.txt ccache-3.3.4/AUTHORS.txt
--- ccache-3.3.3/AUTHORS.txt 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/AUTHORS.txt 2017-02-17 22:28:53.000000000 +0100
@@ -7,7 +7,7 @@
ccache is a collective work with contributions from many people, including:
* Alfred Landrum <alfred.land...@riverbed.com>
-* Anders Björklund <and...@itension.se>
+* Anders Björklund <and...@psqr.se>
* Andrea Bittau <a.bit...@cs.ucl.ac.uk>
* Andrew P Boie <andrew.p.b...@intel.com>
* Andrew Stubbs <a...@codesourcery.com>
@@ -56,6 +56,7 @@
* Patrick von Reth <vonr...@kde.org>
* Paul Griffith <pa...@cse.yorku.ca>
* Pavel Boldin <pbol...@cloudlinux.com>
+* Philippe Proulx <eeppelitel...@gmail.com>
* RW <fbs...@mlists.homeunix.com>
* Ramiro Polla <ramiro.po...@gmail.com>
* Robin H. Johnson <robb...@gentoo.org>
diff -Nru ccache-3.3.3/LICENSE.txt ccache-3.3.4/LICENSE.txt
--- ccache-3.3.3/LICENSE.txt 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/LICENSE.txt 2017-02-17 22:28:53.000000000 +0100
@@ -38,7 +38,7 @@
-------------------------------------------------------------------------------
Copyright (C) 2002-2007 Andrew Tridgell
- Copyright (C) 2009-2016 Joel Rosdahl
+ Copyright (C) 2009-2017 Joel Rosdahl
-------------------------------------------------------------------------------
diff -Nru ccache-3.3.3/MANUAL.txt ccache-3.3.4/MANUAL.txt
--- ccache-3.3.3/MANUAL.txt 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/MANUAL.txt 2017-02-17 22:28:53.000000000 +0100
@@ -163,10 +163,11 @@
When run as a compiler, ccache usually just takes the same command line options
as the compiler you are using. The only exception to this is the option
*--ccache-skip*. That option can be used to tell ccache to avoid interpreting
-the next option in any way and to pass it along to the compiler as-is. *Note*:
-*--ccache-skip* currently only tells ccache not to interpret the next option as
-a special compiler option -- the option will still be included in the direct
-mode hash.
+the next option in any way and to pass it along to the compiler as-is.
+
+NOTE: *--ccache-skip* currently only tells ccache not to interpret the next
+option as a special compiler option -- the option will still be included in the
+direct mode hash.
The reason this can be important is that ccache does need to parse the command
line and determine what is an input filename and what is a compiler option, as
@@ -297,8 +298,14 @@
separator. Examples:
+
--
-* +%compiler% -v+
-* +%compiler% -dumpmachine; %compiler% -dumpversion+
+
+----
+%compiler% -v
+----
+
+----
+%compiler% -dumpmachine; %compiler% -dumpversion
+----
You should make sure that the specified command is as fast as possible since it
will be run once for each ccache invocation.
@@ -374,15 +381,15 @@
when generating debug info (compiler option *-g* with variations).
Exception: The CWD will not be included in the hash if *base_dir* is set
(and matches the CWD) and the compiler option *-fdebug-prefix-map* is used.
-
- The reason for including the CWD in the hash by default is to prevent a
- problem with the storage of the current working directory in the debug info
- of an object file, which can lead ccache to return a cached object file
- that has the working directory in the debug info set incorrectly.
-
- You can disable this setting to get cache hits when compiling the same
- source code in different directories if you don't mind that CWD in the
- debug info might be incorrect.
++
+The reason for including the CWD in the hash by default is to prevent a problem
+with the storage of the current working directory in the debug info of an
+object file, which can lead ccache to return a cached object file that has the
+working directory in the debug info set incorrectly.
++
+You can disable this setting to get cache hits when compiling the same source
+code in different directories if you don't mind that CWD in the debug info
+might be incorrect.
*ignore_headers_in_manifest* (*CCACHE_IGNOREHEADERS*)::
@@ -462,13 +469,13 @@
code (see <<_the_preprocessor_mode,THE PREPROCESSOR MODE>>) and then on a
cache miss run the compiler on the source code to get hold of the object
file. This is the default.
-
- If false, ccache will first run preprocessor to preprocess the source code
- and then on a cache miss run the compiler on the _preprocessed source code_
- instead of the original source code. This makes cache misses slightly
- faster since the source code only has to be preprocessed once. The downside
- is that some compilers won't produce the same result (for instance
- diagnostics warnings) when compiling preprocessed source code.
++
+If false, ccache will first run preprocessor to preprocess the source code and
+then on a cache miss run the compiler on the _preprocessed source code_ instead
+of the original source code. This makes cache misses slightly faster since the
+source code only has to be preprocessed once. The downside is that some
+compilers won't produce the same result (for instance diagnostics warnings)
+when compiling preprocessed source code.
*sloppiness* (*CCACHE_SLOPPINESS*)::
@@ -560,6 +567,116 @@
the compression level with *compression_level*.
+Cache statistics
+----------------
+
+*ccache -s/--show-stats* can show the following statistics:
+
+[options="header",cols="30%,70%"]
+|==============================================================================
+|Name | Description
+| autoconf compile/link |
+Uncachable compilation or linking by an autoconf test.
+
+| bad compiler arguments |
+Malformed compiler argument, e.g. missing a value for an option that requires
+an argument or failure to read a file specified by an option argument.
+
+| cache file missing |
+A file was unexpectedly missing from the cache. This only happens in rare
+situations, e.g. if one ccache instance is about to get a file from the cache
+while another instance removed the file as part of cache cleanup.
+
+| cache hit (direct) |
+A result was successfully found using <<_the_direct_mode,the direct mode>>.
+
+| cache hit (preprocessed) |
+A result was successfully found using <<_the_preprocessor_mode,the preprocessor
+mode>>.
+
+| cache miss |
+No result was found.
+
+| cache size |
+Current size of the cache.
+
+| called for link |
+The compiler was called for linking, not compiling.
+
+| called for preprocessing |
+The compiler was called for preprocessing, not compiling.
+
+| can't use precompiled header |
+Preconditions for using <<_precompiled_headers,precompiled headers>> were not
+fulfilled.
+
+| ccache internal error |
+Unexpected failure, e.g. due to problems reading/writing the cache.
+
+| cleanups performed |
+Number of cleanups performed, either implicitly due to the cache size limit
+being reached or due to explicit *ccache -c/--cleanup* calls.
+
+| compile failed |
+The compilation failed. No result stored in the cache.
+
+| compiler check failed |
+A compiler check program specified by *compiler_check* (*CCACHE_COMPILERCHECK*)
+failed.
+
+| compiler produced empty output |
+The compiler's output file (typically an object file) was empty after
+compilation.
+
+| compiler produced no output |
+The compiler's output file (typically an object file) was missing after
+compilation.
+
+| compiler produced stdout |
+The compiler wrote data to standard output. This is something that compilers
+normally never do, so ccache is not designed to store such output in the cache.
+
+| couldn't find the compiler |
+The compiler to execute could not be found.
+
+| error hashing extra file |
+Failure reading a file specified by *extra_files_to_hash*
+(*CCACHE_EXTRAFILES*).
+
+| files in cache |
+Current number of files in the cache.
+
+| multiple source files |
+The compiler was called to compile multiple source files in one go. This is not
+supported by ccache.
+
+| no input file |
+No input file was specified to the compiler.
+
+| output to a non-regular file |
+The output path specified with *-o* is not a file (e.g. a directory or a device
+node).
+
+| output to stdout |
+The compiler was instructed to write its output to standard output using *-o
+-*. This is not supported by ccache.
+
+| preprocessor error |
+Preprocessing the source code using the compiler's *-E* option failed.
+
+| unsupported code directive |
+Code like the assembler ``.incbin'' directive was found. This is not supported
+by ccache.
+
+| unsupported compiler option |
+A compiler option not supported by ccache was found.
+
+| unsupported source language |
+A source language e.g. specified with *-x* was unsupported by ccache.
+
+|==============================================================================
+
+
How ccache works
----------------
@@ -741,8 +858,13 @@
(and that you trust all users of the shared cache).
* Make sure that the setgid bit is set on all directories in the cache. This
tells the filesystem to inherit group ownership for new directories. The
- command ``find $CCACHE_DIR -type d | xargs chmod g+s'' might be useful for
- this.
+ following command might be useful for this:
++
+--
+----
+find $CCACHE_DIR -type d | xargs chmod g+s
+----
+--
The reason to avoid the hard link mode is that the hard links cause unwanted
side effects, as all links to a cached file share the file's modification
diff -Nru ccache-3.3.3/NEWS.txt ccache-3.3.4/NEWS.txt
--- ccache-3.3.3/NEWS.txt 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/NEWS.txt 2017-02-17 22:28:53.000000000 +0100
@@ -1,6 +1,30 @@
ccache news
===========
+ccache 3.3.4
+------------
+Release date: 2017-02-17
+
+New features and improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Documented the different cache statistics counters.
+
+
+Bug fixes
+~~~~~~~~~
+
+- Fixed a regression in ccache 3.3 related to potentially bad content of
+ dependency files when compiling identical source code but with different
+ source paths. This was only partially fixed in 3.3.2 and reverts the new
+ ``Names of included files are no longer included in the hash of the
+ compiler's preprocessed output'' feature in 3.3.
+
+- Corrected statistics counter for `-optf`/`--options-file` failure.
+
+- Fixed undefined behavior warnings in ccache found by `-fsanitize=undefined`.
+
+
ccache 3.3.3
------------
Release date: 2016-10-26
diff -Nru ccache-3.3.3/ccache.c ccache-3.3.4/ccache.c
--- ccache-3.3.3/ccache.c 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/ccache.c 2017-02-17 22:28:53.000000000 +0100
@@ -1,7 +1,7 @@
// ccache -- a fast C/C++ compiler cache
//
// Copyright (C) 2002-2007 Andrew Tridgell
-// Copyright (C) 2009-2016 Joel Rosdahl
+// Copyright (C) 2009-2017 Joel Rosdahl
//
// This program is free software; you can redistribute it and/or modify it
// under the terms of the GNU General Public License as published by the Free
@@ -37,7 +37,7 @@
MYNAME " version %s\n"
"\n"
"Copyright (C) 2002-2007 Andrew Tridgell\n"
- "Copyright (C) 2009-2016 Joel Rosdahl\n"
+ "Copyright (C) 2009-2017 Joel Rosdahl\n"
"\n"
"This program is free software; you can redistribute it and/or modify it
under\n"
"the terms of the GNU General Public License as published by the Free
Software\n"
@@ -557,7 +557,6 @@
}
}
- // Let's hash the include file.
if (!(conf->sloppiness & SLOPPY_INCLUDE_FILE_MTIME)
&& st.st_mtime >= time_of_compilation) {
cc_log("Include file %s too new", path);
@@ -570,6 +569,7 @@
goto failure;
}
+ // Let's hash the include file content.
struct mdfour fhash;
hash_start(&fhash);
@@ -813,8 +813,28 @@
has_absolute_include_headers =
is_absolute_path(inc_path);
}
inc_path = make_relative_path(inc_path);
+
+ bool should_hash_inc_path = true;
+ if (!conf->hash_dir) {
+ char *cwd = gnu_getcwd();
+ if (str_startswith(inc_path, cwd) &&
str_endswith(inc_path, "//")) {
+ // When compiling with -g or similar,
GCC adds the absolute path to
+ // CWD like this:
+ //
+ // # 1 "CWD//"
+ //
+ // If the user has opted out of
including the CWD in the hash, don't
+ // hash it. See also how
debug_prefix_map is handled.
+ should_hash_inc_path = false;
+ }
+ free(cwd);
+ }
+ if (should_hash_inc_path) {
+ hash_string(hash, inc_path);
+ }
+
remember_include_file(inc_path, hash, system);
- p = q;
+ p = q; // Everything of interest between p and q has
been hashed now.
} else if (q[0] == '.' && q[1] == 'i' && q[2] == 'n' && q[3] ==
'c'
&& q[4] == 'b' && q[5] == 'i' && q[6] == 'n') {
// An assembler .incbin statement (which could be part
of inline
@@ -1106,7 +1126,9 @@
if (tmp_cov) {
tmp_unlink(tmp_cov);
}
- tmp_unlink(tmp_dwo);
+ if (tmp_dwo) {
+ tmp_unlink(tmp_dwo);
+ }
failed();
}
if (st.st_size != 0) {
@@ -1117,7 +1139,9 @@
if (tmp_cov) {
tmp_unlink(tmp_cov);
}
- tmp_unlink(tmp_dwo);
+ if (tmp_dwo) {
+ tmp_unlink(tmp_dwo);
+ }
failed();
}
tmp_unlink(tmp_stdout);
@@ -1178,7 +1202,9 @@
if (tmp_cov) {
tmp_unlink(tmp_cov);
}
- tmp_unlink(tmp_dwo);
+ if (tmp_dwo) {
+ tmp_unlink(tmp_dwo);
+ }
failed();
}
@@ -1545,13 +1571,6 @@
}
}
- // Possibly hash input file location to avoid false positive cache hits
since
- // the dependency file includes the source file path.
- if (generating_dependencies) {
- hash_delimiter(hash, "inputfile");
- hash_string(hash, input_file);
- }
-
// Possibly hash the coverage data file path.
if (generating_coverage && profile_arcs) {
char *dir = dirname(output_obj);
@@ -2145,7 +2164,7 @@
if (str_eq(argv[i], "-optf") || str_eq(argv[i],
"--options-file")) {
if (i > argc) {
cc_log("Expected argument after
-optf/--options-file");
- stats_update(STATS_UNSUPPORTED_OPTION);
+ stats_update(STATS_ARGS);
result = false;
goto out;
}
diff -Nru ccache-3.3.3/configure.ac ccache-3.3.4/configure.ac
--- ccache-3.3.3/configure.ac 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/configure.ac 2017-02-17 22:28:53.000000000 +0100
@@ -20,6 +20,8 @@
AC_SUBST(include_dev_mk)
AC_SUBST(test_suites)
+m4_include(m4/feature_macros.m4)
+
dnl Checks for programs.
AC_PROG_CC_C99
if test "$ac_cv_prog_cc_c99" = no; then
@@ -34,8 +36,6 @@
AC_MSG_ERROR(cannot find ar)
fi
-m4_include(m4/feature_macros.m4)
-
# If GCC, turn on warnings.
if test "x$GCC" = "xyes"; then
CFLAGS="$CFLAGS -Wall -W"
diff -Nru ccache-3.3.3/manifest.c ccache-3.3.4/manifest.c
--- ccache-3.3.3/manifest.c 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/manifest.c 2017-02-17 22:28:53.000000000 +0100
@@ -165,15 +165,16 @@
#define READ_INT(size, var) \
do { \
- (var) = 0; \
+ uint64_t u_ = 0; \
for (size_t i_ = 0; i_ < (size); i_++) { \
int ch_ = gzgetc(f); \
if (ch_ == EOF) { \
goto error; \
} \
- (var) <<= 8; \
- (var) |= ch_ & 0xFF; \
+ u_ <<= 8; \
+ u_ |= ch_ & 0xFF; \
} \
+ (var) = u_; \
} while (false)
#define READ_STR(var) \
@@ -289,10 +290,11 @@
#define WRITE_INT(size, var) \
do { \
+ uint64_t u_ = (var); \
uint8_t ch_; \
size_t i_; \
for (i_ = 0; i_ < (size); i_++) { \
- ch_ = ((var) >> (8 * ((size) - i_ - 1))); \
+ ch_ = (u_ >> (8 * ((size) - i_ - 1))); \
if (gzputc(f, ch_) == EOF) { \
goto error; \
} \
diff -Nru ccache-3.3.3/murmurhashneutral2.c ccache-3.3.4/murmurhashneutral2.c
--- ccache-3.3.3/murmurhashneutral2.c 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/murmurhashneutral2.c 2017-02-17 22:28:53.000000000 +0100
@@ -13,9 +13,9 @@
while (len >= 4) {
unsigned int k = data[0];
- k |= data[1] << 8;
- k |= data[2] << 16;
- k |= data[3] << 24;
+ k |= ((unsigned int) data[1]) << 8;
+ k |= ((unsigned int) data[2]) << 16;
+ k |= ((unsigned int) data[3]) << 24;
k *= m;
k ^= k >> r;
@@ -30,9 +30,9 @@
switch (len)
{
- case 3: h ^= data[2] << 16;
- case 2: h ^= data[1] << 8;
- case 1: h ^= data[0];
+ case 3: h ^= ((unsigned int) data[2]) << 16;
+ case 2: h ^= ((unsigned int) data[1]) << 8;
+ case 1: h ^= ((unsigned int) data[0]);
h *= m;
};
diff -Nru ccache-3.3.3/test.sh ccache-3.3.4/test.sh
--- ccache-3.3.3/test.sh 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/test.sh 2017-02-17 22:28:53.000000000 +0100
@@ -3,7 +3,7 @@
# A simple test suite for ccache.
#
# Copyright (C) 2002-2007 Andrew Tridgell
-# Copyright (C) 2009-2016 Joel Rosdahl
+# Copyright (C) 2009-2017 Joel Rosdahl
#
# This program is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
@@ -976,20 +976,6 @@
expect_stat 'unsupported compiler option' 1
# -------------------------------------------------------------------------
- TEST "-MMD for different source files"
-
- mkdir a b
- touch a/source.c b/source.c
- $CCACHE_COMPILE -MMD -c a/source.c
- expect_file_content source.d "source.o: a/source.c"
-
- $CCACHE_COMPILE -MMD -c b/source.c
- expect_file_content source.d "source.o: b/source.c"
-
- $CCACHE_COMPILE -MMD -c a/source.c
- expect_file_content source.d "source.o: a/source.c"
-
- # -------------------------------------------------------------------------
TEST "-Wp,-P"
# Check that -Wp,-P disables ccache. (-P removes preprocessor information
@@ -1483,7 +1469,7 @@
# -------------------------------------------------------------------------
TEST "CCACHE_NODIRECT"
- $CCACHE_COMPILE -c test.c
+ CCACHE_NODIRECT=1 $CCACHE_COMPILE -c test.c
expect_stat 'cache hit (direct)' 0
expect_stat 'cache hit (preprocessed)' 0
expect_stat 'cache miss' 1
@@ -1554,6 +1540,37 @@
expect_stat 'files in cache' 12
# -------------------------------------------------------------------------
+ TEST "-MMD for different source files"
+
+ mkdir a b
+ touch a/source.c b/source.c
+ backdate a/source.h b/source.h
+ $CCACHE_COMPILE -MMD -c a/source.c
+ expect_file_content source.d "source.o: a/source.c"
+
+ $CCACHE_COMPILE -MMD -c b/source.c
+ expect_file_content source.d "source.o: b/source.c"
+
+ $CCACHE_COMPILE -MMD -c a/source.c
+ expect_file_content source.d "source.o: a/source.c"
+
+ # -------------------------------------------------------------------------
+ TEST "-MMD for different include file paths"
+
+ mkdir a b
+ touch a/source.h b/source.h
+ backdate a/source.h b/source.h
+ echo '#include <source.h>' >source.c
+ $CCACHE_COMPILE -MMD -Ia -c source.c
+ expect_file_content source.d "source.o: source.c a/source.h"
+
+ $CCACHE_COMPILE -MMD -Ib -c source.c
+ expect_file_content source.d "source.o: source.c b/source.h"
+
+ $CCACHE_COMPILE -MMD -Ia -c source.c
+ expect_file_content source.d "source.o: source.c a/source.h"
+
+ # -------------------------------------------------------------------------
TEST "-Wp,-MD"
$CCACHE_COMPILE -c -Wp,-MD,other.d test.c
@@ -1887,8 +1904,8 @@
$CCACHE_COMPILE -c `pwd`/file.c
expect_stat 'cache hit (direct)' 1
- expect_stat 'cache hit (preprocessed)' 1
- expect_stat 'cache miss' 1
+ expect_stat 'cache hit (preprocessed)' 0
+ expect_stat 'cache miss' 2
# -------------------------------------------------------------------------
TEST "__FILE__ in include file disables direct mode"
@@ -1916,8 +1933,8 @@
$CCACHE_COMPILE -c `pwd`/file2_h.c
expect_stat 'cache hit (direct)' 1
- expect_stat 'cache hit (preprocessed)' 1
- expect_stat 'cache miss' 1
+ expect_stat 'cache hit (preprocessed)' 0
+ expect_stat 'cache miss' 2
# -------------------------------------------------------------------------
TEST "__FILE__ in source file ignored if sloppy"
@@ -2123,13 +2140,13 @@
CPATH=subdir2 $CCACHE_COMPILE -c foo.c
expect_stat 'cache hit (direct)' 1
- expect_stat 'cache hit (preprocessed)' 1
- expect_stat 'cache miss' 1
+ expect_stat 'cache hit (preprocessed)' 0
+ expect_stat 'cache miss' 2 # subdir2 is part of the preprocessor output
CPATH=subdir2 $CCACHE_COMPILE -c foo.c
expect_stat 'cache hit (direct)' 2
- expect_stat 'cache hit (preprocessed)' 1
- expect_stat 'cache miss' 1
+ expect_stat 'cache hit (preprocessed)' 0
+ expect_stat 'cache miss' 2
# -------------------------------------------------------------------------
TEST "Comment in strings"
@@ -2278,8 +2295,8 @@
# CCACHE_BASEDIR="" is the default:
$CCACHE_COMPILE -I`pwd`/include -c src/test.c
expect_stat 'cache hit (direct)' 0
- expect_stat 'cache hit (preprocessed)' 1
- expect_stat 'cache miss' 1
+ expect_stat 'cache hit (preprocessed)' 0
+ expect_stat 'cache miss' 2
# -------------------------------------------------------------------------
TEST "Path normalization"
diff -Nru ccache-3.3.3/version.c ccache-3.3.4/version.c
--- ccache-3.3.3/version.c 2016-10-26 23:31:19.000000000 +0200
+++ ccache-3.3.4/version.c 2017-02-17 22:28:53.000000000 +0100
@@ -1 +1 @@
-const char CCACHE_VERSION[] = "3.3.3";
+const char CCACHE_VERSION[] = "3.3.4";
