Re: volatile semantics

[D. Hugh Redelmeier]

Sorry for the very late response.  It is actually triggered by the
bugzilla entry
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=22278

The motivating example, abstracted from a misbehaving part of X, is:
void test (char *addr) {
*((volatile char *) addr);
}
In this case, the fetch ("access") is required for the hardware to
behave.

| From: Dale Johannesen 
| Date: Tue, 3 May 2005 10:04:36 -0700
| 
| On May 3, 2005, at 7:41 AM, Nathan Sidwell wrote:
| > Mike Stump wrote:
| > > int avail;
| > > int main() {
| > >   while (*(volatile int *)&avail == 0)
| > > continue;
| > >   return 0;
| > > }
| > > Ok, so, the question is, should gcc produce code that infinitely  loops,
| > > or should it be obligated to actually fetch from memory?   Hint, 3.3
| > > fetched.
| > 
| > I beleive the compiler is so licensed. [5.1.2.3/2] talks about accessing
| > a volatile object.  If the compiled can determine the actual object
| > being accessed through a series of pointer and volatile cast conversions,
| > then I see nothing in the std saying it must behave as-if the object
| > were volatile when it is not.

That turns out not to be the case.  See the chain of reasoning later
in this message.

| This is correct; the standard consistently talks about the type of the object,
| not the type of the lvalue, when describing volatile.
|
| However, as a QOI issue, I believe the compiler should treat the reference as
| volatile if either the object or the lvalue is volatile.  That is obviously
| the
| user's intent.

It appears to me that standards conformance requires the access to be
performed.  So GCC4 has a bug.  One that is subtle, so the damage it
is doing may not be evident.  The worst kind.

When I asked Henry Spencer to look at this, he gave a convincing
argument from chapter and verse of C99.  I will quote it here.  I
added the identical text to the bugzilla entry (comment #43), so there
is no need to read the rest of this message if you read that comment.

 start of Henry's comment

There is little room for compiler writers to maneuver here, unless they
have announced their intentions in advance in their documentation. 
Reading C99 carefully:

6.5.3.2:  applying `*' to a pointer of type `T *' which points to an
object yields an lvalue of type `T' designating that object.  So the
lvalue in the assignment has a volatile-qualified type. 

6.3.2.1:  when an object is said to have a particular type, the type is
specified by the lvalue used to designate the object.  So the lvalue
having a volatile-qualified type *means* that the object it designates has
a volatile-qualified type; "has type X" and "is designated by an lvalue of
type X" are synonymous (!).

6.7.3:  any expression referring to an object of volatile-qualified
type must be evaluated strictly by the rules of the abstract machine,
although precisely what constitutes an "access" to the object is
implementation-defined.  (Note, "implementation-defined" behavior is
required to be well-defined and *documented*.)  So if the reference in
question is an "access", it must occur where the abstract machine says
it should.

5.1.2.3:  the abstract machine evaluates all expressions as specified by
semantics and all side effects must occur, side effects including
"accessing a volatile object"; implementations are allowed to skip part
of expression evaluation only if they can deduce that no needed side
effects (notably "accessing a volatile object") are therefore skipped. 
So if the reference is an "access", it must occur, period.

I see no useful wiggle room in the difference between "access" and
"accessing", or the difference between "volatile object" and "object of
volatile-qualified type".  These appear to me to be minor accidents of
inconsistent terminology, not useful to a sane implementer.

6.7.3 says that to refer to an object "defined with" volatile-qualified
type "through use of an lvalue" with non-volatile-qualified type yields
undefined behavior.  However, the reference here uses a volatile-qualified
lvalue, so this is not relevant.  A pointer value is not an lvalue; there
is no lvalue present until the `*' operator is applied. 

Surprising though it might seem, I see no express or implied permission to
distinguish based on whether the object in question was *defined* with a
volatile-qualified type.  There are places in the standard where how an
object is defined is significant, e.g. the rules for `const' and the part
of 6.7.3 noted in the previous paragraph, but none of them is part of the
chain of reasoning above.

The only loophole is the definition of "access".  If GCC wishes to claim
standard conformance, GCC is required to document its definition.  I'm not
aware of any mention of this in the GCC documentation, although I haven't
dug hard for it. 

I see no room for a reasonable definition of "access" which retains some
useful meaning for `volatile' and doesn't cover the reference in question. 
(I agree that y

Re: volatile semantics

[D. Hugh Redelmeier]

[I'm just a tourist here.  I don't subscribe to the gcc list.  I don't
hack on gcc itself.  I'm just posting because this bug hits me and
didn't seem to be analyzed correctly.  I have participated in the C
standardization process for perhaps 20 years.  Now that I look at the
GCC list archives, I see a more of misunderstanding of the standard
than I'd like.]

| From: Daniel Berlin <[EMAIL PROTECTED]>

[quoting Henry's comment]

| > 6.5.3.2:  applying `*' to a pointer of type `T *' which points to an
| > object yields an lvalue of type `T' designating that object.  So the
| > lvalue in the assignment has a volatile-qualified type. 
| > 
| > 6.3.2.1:  when an object is said to have a particular type, the type is
| > specified by the lvalue used to designate the object.  So the lvalue
| > having a volatile-qualified type *means* that the object it designates has
| > a volatile-qualified type; "has type X" and "is designated by an lvalue of
| 
| How does this reasoning not apply to *((char *)a) = 5 where a was
| originally of a const qualified type?

Did you read Henry's comment in full?  It specifically addresses
"const".

Surprising though it might seem, I see no express or implied permission to
distinguish based on whether the object in question was *defined* with a
volatile-qualified type.  There are places in the standard where how an
object is defined is significant, e.g. the rules for `const' and the part
of 6.7.3 noted in the previous paragraph, but none of them is part of the
chain of reasoning above.

| Or do you think you can only *add* qualifiers, and not remove them?

No, that was not said.

| Because if you allow casting away, then you can't ever trust const to be
| true either, just like we apparently can't trust the user saying "this
| is not volatile" (which they are doing by not declaring the original
| object volatile).

No, that is not correct reasoning.

"const" says: this lvalue will only be used for reading.  It does not
say that the underlying object will not be changed.  More is implied
if the object in question is *defined* with a const attribute.  For
example:
const int foo = 0xF00;

| There is no point in type qualifiers if they can be simply changed at
| will.

This has an element of truth to it.  But they do have points:

- const means that a program cannot accidentally change something
  through a const-qualified thingee (lvalue, pointer, whatever)

- volatile tells the compiler "hands off" when such an lvalue is used.

- let's not talk about "restrict"

|  Do not lie about your objects, and you will not be screwed over.

Funny that you should be paraphrasing one of Henry's famous aphorisms
"If you lie to the compiler, it will get its revenge.".  Yes, this is
the same Henry Spencer.

Re: volatile semantics

[D. Hugh Redelmeier]

| From: Gabriel Dos Reis <[EMAIL PROTECTED]>

| The way I see it is that people who designed and wrote the standard
| offer their view and interpretation of of they wrote and some people
| are determined to offer a different interpretation so that they can
| claim they are well-founded to apply  their transformations.

I don't know exactly what you are referring to here.  That's OK, I
think.

The standard should stand alone.  It should be able to be interpreted
without "insider knowledge".

The standard is quite complicated and intricate.  It helps to already
know it when trying to read it.  (On the other hand, that same
phenomenon makes it hard for the authors to see its ambiguities.)

Generations have worked on improving the standards.  Not all of the
changes were made by people who understood what went before.

Having said all that, I think that the standard gives no authority to
do the optimization that GCC4 is doing to the code sample I gave.  I
included Henry's carefully justified-by-scripture argument.  If anyone
disagrees, PLEASE give a careful argument why, siting the Standard.

If you wish to answer other questions about the standard, I recommend
that the first step would be to read the standard.  Arguing by opinion
isn't getting us too far.

PS: thanks for not just ignoring my report.

Re: volatile semantics

[D. Hugh Redelmeier]

| From: Gabriel Dos Reis <[EMAIL PROTECTED]>

|  After many exchanges via private mails and
| looking at the various reports related to this issue, it has become
| clear to me that the interpretations offered to justify why GCC is
| behaving the way it does seem to go beyond what can be inferred.

OK.

Is there a consensus on this?  If not, how can a consensus be reached?

If so, how can we get a fix?

I think that is urgent.  This bug is causing X to misbehave and the
current workarounds might be harmful.  Who knows what other
manifestations might be lurking?

As I said, I'm not a GCC hacker.  Who is the likely maintainer to fix
this?  Does he or she agree that this needs to be done?  Urgently?

Re: volatile semantics

[D. Hugh Redelmeier]

| From: Gabriel Dos Reis <[EMAIL PROTECTED]>

| Daniel Berlin <[EMAIL PROTECTED]> writes:
| 
| [...]
| 
| | > I think that is urgent. 
| | No offense, but everyone thinks the problems that affect them are the
| | most urgent.
| 
| miscompilation of KDE was declared urgent; I hope bug affecting code
| semantics for X is not just "request for enhancement".

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=161242#c120
[Comment 120, if your browser doesn't get to the right spot]

Here is the text, written by Olivier Baudron:

There are 100+ casts into volatile in xorg-x11 and some of them
are in libvgahw.  All of these are "miscompiled" by gcc4 -O2. The
patch in -43 (one volatile fixed in libvgahw) was enough for my
G400.  Unfortunatly, other hardware are depending upon volatile
side effects found elsewhere in libvgahw or even elsewhere in
xorg. Thus reported scenarii a,b,c,d are easily understood.

For now, we have fixed: 1 bug out of 100+ in 1 package out of
1000+.  I *really* suggest gcc4 be compatible with gcc3 in a near
future.

I have not verified these numbers, but they do make the bug seem
serious.


Just as I'm not a GCC hacker, I'm not an X hacker nor a kernel hacker.
What follows is idle speculation.

My feeling (i.e. it is not certain knowledge) is that a lot of X
driver code may represent scar tissue from battles with intransigent
hardware.  There may not be explicit specs that the code is following.
This may make migrating the code to another approach to volatile
somewhat destabilizing.

Another feeling I have is that 100+ casts to volatile are an
indication that the code should be refactored.  A long-term process
that should not be attempted in an emergency.

If GCC4 causes this much problem with X, I wonder what GCC4 will do to
the Linux kernel.  I understand that Linus generally prefers older
GCCs to newer ones.  It would be great if his preference were only
superstition.

Re: volatile semantics

[D. Hugh Redelmeier]

| From: Gabriel Dos Reis <[EMAIL PROTECTED]>
| 
| "D. Hugh Redelmeier" <[EMAIL PROTECTED]> writes:
| 
| | | From: Gabriel Dos Reis <[EMAIL PROTECTED]>
| | 
| | |  After many exchanges via private mails and
| | | looking at the various reports related to this issue, it has become
| | | clear to me that the interpretations offered to justify why GCC is
| | | behaving the way it does seem to go beyond what can be inferred.
| | 
| | OK.
| | 
| | Is there a consensus on this?
| 
| JSM, please chime in.

Maybe Joseph isn't seeing these messages.  I've added him to the CC list.  For 
that reason, I have not trimmed my quotations of your message.

Who wants off the CC list?  No fair if you are not subscribed to the
gcc list :-)

I've just now subscribed, so you can drop me from the CC lists.

| | If not, how can a consensus be reached?
| 
| try to explain those who read the standard to you their interpretation
| does not match the intent and the letter?  Sorry :-)

If I understand what you are saying, I guess the step before is to
have them construct an argument, quoting the standard, that supports
the buggy behaviour (he who frames the question wins :-).

| More seriously, at this point Daniel Berlin has indicated that he gives
| "less of a crap about volatile and optimizing volatile than [he does]
| const, restrict, etc".  So that is, I hope, a path to a saner behaviour
| from the code transformation part of the compiler.
| 
| | If so, how can we get a fix?
| | 
| | I think that is urgent.  This bug is causing X to misbehave and the
| | current workarounds might be harmful.  Who knows what other
| | manifestations might be lurking?
| | 
| | As I said, I'm not a GCC hacker.  Who is the likely maintainer to fix
| | this?  Does he or she agree that this needs to be done?  Urgently?
| 
| Joseph S. Myers (jsm at polyomino.org.uk)

Ah, polyominoes.  That is one area where I will claim some expertise.  For 
about 25 years I had enumerated more polyominoes than anyone else.

| and Richard Henderson (rth
| at redhat.com)

I've also added Richard to the CC list.

| are the C front-end maintainers.  The following is
| from the last message I received from Joseph (on access through volatile
| lvalue expression): 
| 
|   # Using always the qualification of the lvalue is reasonable semantics to 
|   # specify (I don't know about to implement).  It's still a matter of making 
|   # a choice and documenting and implementing it.
| 
| The issue is compounded by the fact that the code transformation part
| (the "optimizer") is maintained by a different set of people and it takes
| coordination between both worlds to arrive to a useful semantics. 
| 
| At this point we need:
|   (1) agreement from C and C++ maintainers on access through volatile
|   lvalue 

I don't know C++ well enough to say whether the analogous optimization
is wrong for C++.

|   (2) agreement with the middle-end maintainers not to "optimize"
|   volatile lvalue expressions
|   (3) document the behaviour.
| 
| The most important for you and X is to have (2) done, i.e. Richard
| Henderson, Roger Sayle, Daniel Berlin and others.
| 
| -- Gaby

As far as (3) is concerned, I think that the combination of the gcc
info node on Volatiles is pretty clear, when read in the context of
Henry's reading of the C standard.

It would be a good idea for everyone to read that node.

To read that node, invoke info:
$ info gcc
Then type:
gVolatiles
to go to the node "6.2 When is a Volatile Object Accessed?"

Re: volatile semantics

[D. Hugh Redelmeier]

| From: Daniel Berlin <[EMAIL PROTECTED]>

| On Sat, 2005-07-16 at 21:36 -0400, D. Hugh Redelmeier wrote:
| > | From: Gabriel Dos Reis <[EMAIL PROTECTED]>
| > 
| > |  After many exchanges via private mails and
| > | looking at the various reports related to this issue, it has become
| > | clear to me that the interpretations offered to justify why GCC is
| > | behaving the way it does seem to go beyond what can be inferred.
| > 
| > OK.
| > 
| > Is there a consensus on this?  If not, how can a consensus be reached?
| > 
| I'll pass on this, since I've said my piece, and i don't care about
| volatile much.

I would very much like you to restate your objections with careful
reference to the C Standard.  I really want the correct analysis more
than I want my analysis.

|  However, if you come after const or restrict I'll bite
| back.

What exactly do you think const says that you can find useful for
optimization?  I don't think that it is helpful (except on actual
definitions).  But I haven't looked closely at this issue.

| Personally, I think a DR should be filed to clarify this, instead of all
| this argument and opinion. 

I would like you to explain where you think that the current standard
is ambiguous on this matter.  Without ambiguity or error, a DR is not
appropriate.

| > If so, how can we get a fix?
| Usually by asking nicely and pressuring people. 
| Or waiting long enough for someone to get around to it.
| Or paying someone to fix it :)

Those are good answers.

| > I think that is urgent. 
| No offense, but everyone thinks the problems that affect them are the
| most urgent.

Yeah.  I've given a few arguments for urgency.

| >  This bug is causing X to misbehave and the
| > current workarounds might be harmful.  Who knows what other
| > manifestations might be lurking?
| 
| Whoever is testing distributions compiled with mainline :)

Testing can show the presence of bugs but not their absence.  But you
knew that.

| > As I said, I'm not a GCC hacker.  Who is the likely maintainer to fix
| > this?
| Anyone can fix it, however, who can review the fix depends on what it
| touches.
| 
| >  Does he or she agree that this needs to be done? 
| > Urgently?
| 
| This is actually probably pretty unlikely.  There are few bugs most
| people consider urgent, and i'd venture this is not one of them.  It
| would probably be fixed by release time.
| 
| In that spirit, here is a patch against mainline that fixes your bug (a
| similar patch to the same function should work on 4.0)

Thanks!

| Someone else can go through the process of testing and getting this
| reviewed, i'm currently swamped (IE i have no plans to try to submit
| this to gcc-patches).

Re: Pointers in comparison expressions

[D. Hugh Redelmeier]

| From: Vincent Lefevre <[EMAIL PROTECTED]>
| To: gcc@gcc.gnu.org
| Subject: Re: Pointers in comparison expressions
| 
| On 2005-07-17 12:55:38 -0400, Paul Koning wrote:
| > Are you sayinvg that a-b is not always "guaranteed to work" when a
| > and b point to elements of the same array? That sounds wrong; can
| > you given an example or standards text that supports this?
| 
|6.5.6  Additive operators
| [...]
|[#9]  When  two pointers are subtracted, both shall point to
|elements of the same array object,  or  one  past  the  last
|element of the array object; the result is the difference of
|the subscripts of the two array elements.  The size  of  the
|result  is  implementation-defined,  and  its type (a signed
|integer type) is ptrdiff_t defined in the  header.
|If  the  result  is  not  representable in an object of that
|type, the behavior is undefined.  In  other  words,  if  the
|expressions  P and Q point to, respectively, the i-th and j-
|th elements of an array object, the expression  (P)-(Q)  has
|the  value  i-j provided the value fits in an object of type
|ptrdiff_t.  [...]
| 
| See the sentence "If the result..." and the last few words of the
| next sentence.

This is true.  And an abomination.  But I will explain a bit more
where this came from.

On, for example, the PDP-11, size_t would be unsigned int (16 bits).
ptrdiff_t would be signed int because making it long feels wrong -- it
is too expensive and would only be more correct in an extremely rare
case.  That case can only come up when dealing with a char array
larger than 32K.  Since the maximum addressable data space was 64K, no
real program would do this (except to make fools of us).

So the committee felt that it was important to allow ptrdiff_t to be
"too small".  I suggested that ptrdiff_t should be required to be at
least as large as the signed variant of size_t, but they did not
accept this.  So a legal implementation may make ptrdiff_t as small as
signed char (recent versions of the standard mandate that PTRDIFF_MAX
must be at least 64K-1; this negates the original reason for the
laxity).

This same logic can now be transferred to 32-bit machines.  ptrdiff_t
on most 32-bit machines is 32 bits, so a char array larger than 2G
could allow for overflowing pointer subtraction.

Since GCC knows the range of values that can be represented in
ptrdiff_t, it knows whether the subtraction might overflow.

Re: volatile semantics

[D. Hugh Redelmeier]

| From: Paul Schlie <[EMAIL PROTECTED]>

| > |  void foo(void) {
| > |int *x = 4;
| > | *x = 3;

| The point I was attempting to make, was that just because a specified
| statement's effective behavior/side-effects are not well defined, it doesn't
| mean that it's clearly specified semantic actions may be summarily ignored.

The Standard specifically allows an implementation a lot of choices
when dealing with "undefined behavior".  Quoting n1124.pdf:

3.4.3

undefined behavior

behavior, upon use of a nonportable or erroneous program construct
or of erroneous for which this International Standard imposes no
requirements

NOTE Possible undefined behavior ranges from ignoring the
situation completely with unpredictable results, to behaving
during translation or program execution in a documented manner
characteristic environment (with or without the issuance of a
diagnostic message), to terminating a translation execution (with
the issuance of a diagnostic message).

EXAMPLE An example of undefined behavior is the behavior on
integer overflow.

I think (but am not 100% sure) that this example falls into the
category of "undefined behavior".  Certainly the earlier example does.

So the implementation is free to do pretty much anything at all to the
program as a whole (for example, refuse to compile it) just because a
part of it invokes "undefined behavior".  Even though the NOTE does
not call this out, the implementation may do completely unreasonable
things -- the committee felt that it could not actually require
anything of an implementation when dealing with "undefined behavior".
Hence the wording "imposes no requirements" (but note that the sense
of that phrase in context is subtle).  The effect is no way prevented
from changing the behaviour of things that are well defined (the
undefinedness bleeds through to the whole program).

Not all Bad Things are categorized as "undefined behavior".

Let's think about your concrete example.  That assignment might (on a
reasonable implementation) curdle an interrupt vector.  Which might
have almost any effect on the execution of the program.  Including
apparently retroactive effects.  Certainly effects that cannot be
explained in the abstraction that is the C Standard +
implementation-supplied documentation mandated by the Standard.  No
wonder that the Standard throws up its hands.

[Sorry to be so long winded.  I think that being more concise would
enable more misunderstanding.]

UTF-8 quotation marks in diagnostics

[D. Hugh Redelmeier]

Several of us don't want UTF-8 quotation marks in diagnostics in our 
environment (Jove subshells).  We'd like a way to turn them off.  We don't 
think that they are a bad idea but they are bad in our environment.



English-language diagnostic messages will now use Unicode
quotation marks in UTF-8 locales. (Non-English messages
already used the quotes appropriate for the language in
previous releases.) If your terminal does not support UTF-8
but you are using a UTF-8 locale (such locales are the default
on many GNU/Linux systems) then you should set LC_CTYPE=C in
the environment to disable that locale. Programs that parse
diagnostics and expect plain ASCII English-language messages
should set LC_ALL=C. See Markus Kuhn's explanation of Unicode
quotation marks for more information.

This suggests that LC_CTYPE=C would do what we want: go back to ` and
' instead of 342\200\230 and \342\200\231.

I find that a little confusing and scary.  I would expect that setting
LC_CTYPE=C would have the affect of changing the lexing done by the C
compiler.  For one thing, valid characters in strings would be
different.  This we don't want.

gcc(1) says:

The LC_CTYPE environment variable specifies character
classification.  GCC uses it to determine the character
boundaries in a string; this is needed for some multibyte
encodings that contain quote and escape characters that are
otherwise interpreted as a string end or escape.

The LC_MESSAGES environment variable specifies the language to
use in diagnostic messages.


An experiment on my Fedora 20 system shows:

- LANG=en_CA.UTF-8 [correct]

- LC_CTYPE isn't set by default

- setting LC_CTYPE to C gets rid of the UTF-8 quotes in GCC diagnostics.
  That's surprising because the manpage doesn't say that it affects diagnostics.

- setting LC_MESSAGES to C DOES NOT get rid of the UTF-8 quotes in GCC 
diagnostics
  That's surprising because the manpage does say that it affects diagnostics.
  I hope that it only affect compile-time diagnostics.

That sure sounds like we should NOT set LC_CTYPE=C because of bad
side-effects: it changes how the program is lexed.  And the
documentation gives no basis for thinking that it would suppress those
UTF-8 quotes in messages (even though testing shows that this works).

That sure sounds like we should set LC_MESSAGES=C, but that doesn't work.

In our environment, our tool doesn't know that gcc is being invoked.
So the solution needs to be targetted.  That's why a solution like
GCC_COLOURS would be good.  In fact, it could probably be hacked into 
GCC_COLOURS.

Man pages in section 1 that explicitly reference LC_CTYPE:
enca
enconv
find
gcc
gnroff
grep
jove
koi8rxterm
less
locale
localedef
nroff
perl5004delta
perl5160delta
perl58delta
perlfunc
perllocale
perltoc
pico
pilot
sh
systemd
time
tree
uxterm
xterm
So I feel uncomfortable setting it.

Man pages in section 1 that explicitly reference LC_MESSAGES:
apropos
aspell
awk
bash
enca
enconv
find
gawk
gcc
grep
hunspell
install-tl
locale
localectl
localedef
lynx
man
nmcli
perllocale
perltoc
sh
systemd
systemd-firstboot
time
whatis
xdg-desktop-icon
xdg-desktop-menu
So setting this would hardly be safer.

Re: Update to GCC copyright assignment policy

[D. Hugh Redelmeier]

| From: Mark Wielaard 

| This seems a pretty bad policy to be honest.
| Why was there no public discussion on this?

Agreed.  I also agree with the rest of Mark's message.

(Note: I haven't contributed to GCC but I have contributed to other
copylefted code bases.)

It is important that the pool be trustable.  A tall order, but
solvable, I think.

Two pools (FSF for old stuff, something else, for new stuff if the
contributor prefers) should be quite managable.

This would allow, for example, moving to an updated copyleft if the
two pools agreed.  It is important that the governance of the pool be
trustable.

We've trusted the FSF and now some have qualms.  A second pool would
be a check on the power of the first pool.

Individual unassigned copyright pretty much guarantees that the
copyright terms can never be changed.  I don't think that that is
optimal.

Re: Mailing list reconfiguration: VERP Sender: header affected

[D. Hugh Redelmeier]

| From: Martin Liška 

| Which we recommend in the ection Filtering here:
| https://gcc.gnu.org/lists.html

Thanks for the useful information.

That document suggests:
 * ^List-Id: .*<.*.gcc.gnu.org>$

Surely this should be:
 * ^List-Id: .*<.*.gcc\.gnu\.org>$