Andrea,
My coworker Neil helped me in this afternoon to find out that when GCC 4.8
compiler assigns constant values to variables (value1 and value2) like this,
for(i = 0, value1 = 0.0, value2 = 1.0; i < (int)sizeof(long double); i++) {
value3 = value1;
value1 += value2;
value2 /= 256.0;
:
:
it introduces some garbage to the two padding bytes of value1 and value2. Then
the garbage confuses our algorithm, especially the value of "last_mbyte" gets
wrong. To fix it in a simple way, use an intermediate variable like this:
long double tmp_value, divisor;
tmp_value = 0.0;
value1 = tmp_value;
tmp_value = 1.0;
value2 = tmp_value;
tmp_value = 256.0;
divisor = tmp_value;
for(i = 0; i < (int)sizeof(long double); i++) {
value3 = value1;
value1 += value2;
value2 /= divisor;
:
:
How do you think about it?
Ray
On Sep 6, 2013, at 1:49 AM, Andrea Bedini wrote:
> Hi Rey,
>
> thanks for that, it really helped. I checked thoroughly and the memset of the
> temporary variables disappears randomly.
> It doesn't depends only on optimization though, on my machine putting a
> printf("%Lf\n", value2); just before the loop changes the result.
> I'm not sure who gets the blame here, poking into the padding bits of a long
> double might just be unspecified or undefined behaviour.
>
> Andrea
>
>
> On 6 September 2013 06:40, Raymond Lu <[email protected]> wrote:
> I isolated part of the DETECT_F into a C program as attached (detect.c). It
> only contains the algorithm for detecting the byte order of long double.
> When I compile it with gcc -g, -O0, or no flag, it reports little-endian.
> When I compile it with -O1, -O2, or -O3, it reports VAX order. I don't know
> where goes wrong yet. But I suspect GCC's optimization has bugs. Maybe you
> can help me.
>
> I haven't tried the algorithms for other parts in DETECT_F yet. The
> alignment problem you talked about is one of the other algorithms.
>
> Ray
>
>
>
>
>
>
>
> On Sep 4, 2013, at 8:40 PM, Andrea Bedini wrote:
>
>> Thanks George.
>>
>> For anyone interested in debugging this problem, debian has an extensive
>> collection of build logs over many architectures
>> https://buildd.debian.org/status/logs.php?pkg=hdf5 (going back 12 years!)
>>
>> As far as I know, the corruption is limited to the H5T_NATIVE_LDOUBLE type.
>> You can check your particular build with the following test
>>
>> #include <hdf5.h>
>> int main() {
>> return !(H5Tget_order(H5T_NATIVE_LDOUBLE) ==
>> H5Tget_order(H5T_NATIVE_DOUBLE));
>> }
>>
>> It exits with code 1 if the long double has different byte ordering than
>> double (which is technically possible, but highly suspicious).
>>
>> Otherwise the patch I sent earlier in this thread seems to do the trick,
>> although what exactly is going wrong is still beyond my understanding.
>>
>> Third option: you can define an equivalent of H5T_NATIVE_LDOUBLE yourself.
>> The following creates a data type representing a long double as implemented
>> by gcc on x86 architectures (see
>> http://en.wikipedia.org/wiki/Long_double#Implementations for details)
>>
>> hid_t ldouble_datatype = H5Tcopy(H5T_NATIVE_DOUBLE);
>> H5Tset_size(ldouble_datatype, sizeof(long double));
>> H5Tset_precision(ldouble_datatype, 80);
>> H5Tset_fields (ldouble_datatype, 79, 64, 15, 0, 64);
>> H5Tset_pad(ldouble_datatype, H5T_PAD_ZERO, H5T_PAD_ZERO);
>> H5Tset_inpad(ldouble_datatype, H5T_PAD_ZERO);
>> H5Tset_ebias(ldouble_datatype, 16383);
>> H5Tset_norm(ldouble_datatype, H5T_NORM_NONE);
>>
>> Best wishes,
>> Andrea
>>
>> On 4 September 2013 22:58, George N. White III <[email protected]> wrote:
>> Another historical reference to the obscurity of this code is:
>> <https://bugs.gentoo.org/show_bug.cgi?id=118777>.
>>
>> I've been building HDF5 libraries for use with NASA SeaDAS, and recently
>> have started using HDF5 with R and GDAL. The SeaDAS builds are static, and
>> I don't find the "unable to calculate alignment for long double" message in
>> my SeaDAS build logs on linux and OS X. For R and GDAL, however, I need
>> dynamic libraries and those build logs do have the "unable to calculate
>> alignment for long double" message on both linux and OS X.
>>
>>
>> On Tue, Sep 3, 2013 at 9:50 PM, Andrea Bedini <[email protected]>
>> wrote:
>> Hi,
>>
>> I found something else (I know, I should stop :)). I am not entirely sure
>> but it seems that when H5detect fails it writes "unable to calculate
>> alignment for long double" on stderr so this message should be observable on
>> build logs (although buried by other warnings). The packages on debian sid
>> and testing for both i386 and x86-64 seem to be affected:
>>
>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=amd64&ver=1.8.11-3%2Bb1&stamp=1377024563
>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=i386&ver=1.8.11-3%2Bb1&stamp=1377025110
>>
>> But here's the exciting part: look what I found
>>
>> http://www.unidata.ucar.edu/mailing_lists/archives/gembud/2010/msg00052.html
>>
>> It's a build log from 2010 for HDF5 v1.6.5 and gcc-4.4.3 that says "unable
>> to calculate alignment for long double".
>>
>> If my understanding is correct, nor 1.8.11 or gcc 4.8.0 would be the problem
>> and it would be that piece of code just doesn't work properly.
>>
>> Best wishes,
>> Andrea
>>
>>
>>
>> On 4 September 2013 08:00, Andrea Bedini <[email protected]> wrote:
>> Hi Ray,
>>
>> thanks for giving it a look. Antonio made me notice that something else
>> might be at work since the macro DETECT_F already zeroes the structure right
>> before anything else:
>>
>> memset(&INFO, 0, sizeof(INFO)); #L299
>>
>> so I don't understand how the perm fields need to be zeroed again around
>> line #L308. This still considering the "Byte Order" loop as a black box.
>>
>> As a side question: isn't there a more portable way of doing this? I am
>> pretty sure H5detect.c might invoke a bunch of undefined behaviours given
>> the amount of warning the compiler generates and of bit trickery.
>>
>> Best wishes,
>> Andrea
>>
>>
>>
>>
>> On 4 September 2013 05:43, Raymond Lu <[email protected]> wrote:
>> Andrea,
>>
>> We've verified that your solution is correct. We're putting your fix into
>> the library. Thanks for helping us.
>>
>> Ray
>>
>> On Sep 3, 2013, at 3:32 AM, Andrea Bedini wrote:
>>
>>> Hi there,
>>>
>>> I think I have found the problem. The issue is in H5detect.c. Macros
>>> DETECT_F and DETECT_I do not initialize properly the perm field in the
>>> detected_t struct. As a result the routine fix_order is passed some
>>> uninitialized memory which makes it fail. I have a small patch against
>>> H5detect.c which fixes the problem by simply initializing the perm field
>>> with zeros. Valgrind's tool memcheck would have exposed the problem.
>>>
>>> Best wishes,
>>> Andrea
>>>
>>>
>>>
>>> On 3 September 2013 15:30, Andrea Bedini <[email protected]> wrote:
>>> Hi,
>>>
>>> I am experiencing the following issue with hdf5 and gcc 4.8.0
>>>
>>> Consider this very simple test
>>>
>>> #include <hdf5.h>
>>>
>>> int main() {
>>> switch (H5Tget_order(H5T_NATIVE_LDOUBLE)) {
>>> case H5T_ORDER_LE:
>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE\n");
>>> break;
>>> case H5T_ORDER_BE:
>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_BE\n");
>>> break;
>>> case H5T_ORDER_VAX:
>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX\n");
>>> break;
>>> case H5T_ORDER_MIXED:
>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_MIXED\n");
>>> break;
>>> case H5T_ORDER_NONE:
>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_NONE\n");
>>> break;
>>> default:
>>> printf("here are dragons\n");
>>> }
>>> return 0;
>>> }
>>>
>>> on the same x86_64 GNU/Linux machine I get
>>>
>>> $ hdf5-1.8.11-gcc-4.7.0/my_test # compiled with gcc 4.7.0
>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE
>>>
>>> $ hdf5-1.8.11-gcc-4.8.0/my_test # compiled with gcc 4.8.0
>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX
>>>
>>> So H5T_NATIVE_LDOUBLE is mis-detected. I tried to dig deeper and basically
>>> the fault must be in src/H5detect.c which is used to generate the
>>> definitions in src/H5Tinit.c
>>> I could not figure out what H5detect.c does wrong (it is not very readable,
>>> given its extensive use of macros) but the compiler does emit a lot of
>>> warnings (see https://gist.github.com/andreabedini/6419975).
>>>
>>> I think this must be related to the failure of dt_arith long double test
>>> observed recently.
>>>
>>> Any suggestion on how to fix this ?
>>>
>>> Best wishes,
>>> Andrea
>>>
>>>
>>> --
>>> Andrea Bedini <[email protected]>
>>>
>>>
>>>
>>> --
>>> Andrea Bedini <[email protected]>
>>> <hdf5_uninitialized.patch>_______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>>
>>
>> --
>> Andrea Bedini <[email protected]>
>>
>>
>>
>> --
>> Andrea Bedini <[email protected]>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>>
>>
>> --
>> George N. White III <[email protected]>
>> Head of St. Margarets Bay, Nova Scotia
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>>
>>
>> --
>> Andrea Bedini <[email protected]>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
>
>
>
> --
> Andrea Bedini <[email protected]>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
