My best guess, is the issue is related to the struct being 4 bytes.
A similar segfault occurs if you attempt to access in a similar manner
using c++.
A 4 byte struct will fit into a single register making pointers
unnecessary/slower and it's likely some part of the ABI has taken this
into consideration and the compiler is optimizing access to this.
However, I would imagine that such optimizations would not be allowed
with C++ and so by using a class it requires a pointer type and not the
optimized struct.
The following returns the value of 4 when I inspect the variable
refValue instead of the correct address and segfaults.
http://codepad.org/eepFTfbX
