Hi Rey,
thanks for that, it really helped. I checked thoroughly and the memset of
the temporary variables disappears randomly.
It doesn't depends only on optimization though, on my machine putting a
printf("%Lf\n", value2); just before the loop changes the result.
I'm not sure who gets the blame here, poking into the padding bits of a
long double might just be unspecified or undefined behaviour.
Andrea
On 6 September 2013 06:40, Raymond Lu <[email protected]> wrote:
> I isolated part of the DETECT_F into a C program as attached (detect.c).
> It only contains the algorithm for detecting the byte order of long
> double. When I compile it with gcc -g, -O0, or no flag, it reports
> little-endian. When I compile it with -O1, -O2, or -O3, it reports VAX
> order. I don't know where goes wrong yet. But I suspect GCC's
> optimization has bugs. Maybe you can help me.
>
> I haven't tried the algorithms for other parts in DETECT_F yet. The
> alignment problem you talked about is one of the other algorithms.
>
> Ray
>
>
>
>
>
>
>
> On Sep 4, 2013, at 8:40 PM, Andrea Bedini wrote:
>
> Thanks George.
>
> For anyone interested in debugging this problem, debian has an extensive
> collection of build logs over many architectures
> https://buildd.debian.org/status/logs.php?pkg=hdf5 (going back 12 years!)
>
> As far as I know, the corruption is limited to the H5T_NATIVE_LDOUBLE
> type. You can check your particular build with the following test
>
> #include <hdf5.h>
> int main() {
> return !(H5Tget_order(H5T_NATIVE_LDOUBLE) == H5Tget_order(
> H5T_NATIVE_DOUBLE));
> }
>
> It exits with code 1 if the long double has different byte ordering than
> double (which is technically possible, but highly suspicious).
>
> Otherwise the patch I sent earlier in this thread seems to do the trick,
> although what exactly is going wrong is still beyond my understanding.
>
> Third option: you can define an equivalent of H5T_NATIVE_LDOUBLE yourself.
> The following creates a data type representing a long double as implemented
> by gcc on x86 architectures (see
> http://en.wikipedia.org/wiki/Long_double#Implementations for details)
>
> hid_t ldouble_datatype = H5Tcopy(H5T_NATIVE_DOUBLE);
> H5Tset_size(ldouble_datatype, sizeof(long double));
> H5Tset_precision(ldouble_datatype, 80);
> H5Tset_fields (ldouble_datatype, 79, 64, 15, 0, 64);
> H5Tset_pad(ldouble_datatype, H5T_PAD_ZERO, H5T_PAD_ZERO);
> H5Tset_inpad(ldouble_datatype, H5T_PAD_ZERO);
> H5Tset_ebias(ldouble_datatype, 16383);
> H5Tset_norm(ldouble_datatype, H5T_NORM_NONE);
>
> Best wishes,
> Andrea
>
> On 4 September 2013 22:58, George N. White III <[email protected]> wrote:
>
>> Another historical reference to the obscurity of this code is: <
>> https://bugs.gentoo.org/show_bug.cgi?id=118777>.
>>
>> I've been building HDF5 libraries for use with NASA SeaDAS, and recently
>> have started using HDF5 with R and GDAL. The SeaDAS builds are static, and
>> I don't find the "unable to calculate alignment for long double" message in
>> my SeaDAS build logs on linux and OS X. For R and GDAL, however, I need
>> dynamic libraries and those build logs do have the "unable to calculate
>> alignment for long double" message on both linux and OS X.
>>
>>
>> On Tue, Sep 3, 2013 at 9:50 PM, Andrea Bedini <[email protected]>wrote:
>>
>>> Hi,
>>>
>>> I found something else (I know, I should stop :)). I am not entirely
>>> sure but it seems that when H5detect fails it writes "unable to calculate
>>> alignment for long double" on stderr so this message should be observable
>>> on build logs (although buried by other warnings). The packages on debian
>>> sid and testing for both i386 and x86-64 seem to be affected:
>>>
>>>
>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=amd64&ver=1.8.11-3%2Bb1&stamp=1377024563
>>>
>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=i386&ver=1.8.11-3%2Bb1&stamp=1377025110
>>>
>>> But here's the exciting part: look what I found
>>>
>>>
>>> http://www.unidata.ucar.edu/mailing_lists/archives/gembud/2010/msg00052.html
>>>
>>> It's a build log from 2010 for HDF5 v1.6.5 and gcc-4.4.3 that says "unable
>>> to calculate alignment for long double".
>>>
>>> If my understanding is correct, nor 1.8.11 or gcc 4.8.0 would be the
>>> problem and it would be that piece of code just doesn't work properly.
>>>
>>> Best wishes,
>>> Andrea
>>>
>>>
>>>
>>> On 4 September 2013 08:00, Andrea Bedini <[email protected]>wrote:
>>>
>>>> Hi Ray,
>>>>
>>>> thanks for giving it a look. Antonio made me notice that something else
>>>> might be at work since the macro DETECT_F already zeroes the structure
>>>> right before anything else:
>>>>
>>>> memset(&INFO, 0, sizeof(INFO)); #L299
>>>>
>>>> so I don't understand how the perm fields need to be zeroed again
>>>> around line #L308. This still considering the "Byte Order" loop as a black
>>>> box.
>>>>
>>>> As a side question: isn't there a more portable way of doing this? I am
>>>> pretty sure H5detect.c might invoke a bunch of undefined behaviours given
>>>> the amount of warning the compiler generates and of bit trickery.
>>>>
>>>> Best wishes,
>>>> Andrea
>>>>
>>>>
>>>>
>>>>
>>>> On 4 September 2013 05:43, Raymond Lu <[email protected]> wrote:
>>>>
>>>>> Andrea,
>>>>>
>>>>> We've verified that your solution is correct. We're putting your fix
>>>>> into the library. Thanks for helping us.
>>>>>
>>>>> Ray
>>>>>
>>>>> On Sep 3, 2013, at 3:32 AM, Andrea Bedini wrote:
>>>>>
>>>>> Hi there,
>>>>>
>>>>> I think I have found the problem. The issue is in H5detect.c.
>>>>> Macros DETECT_F and DETECT_I do not initialize properly the perm field in
>>>>> the detected_t struct. As a result the routine fix_order is passed some
>>>>> uninitialized memory which makes it fail. I have a small patch against
>>>>> H5detect.c which fixes the problem by simply initializing the perm field
>>>>> with zeros. Valgrind's tool memcheck would have exposed the problem.
>>>>>
>>>>> Best wishes,
>>>>> Andrea
>>>>>
>>>>>
>>>>>
>>>>> On 3 September 2013 15:30, Andrea Bedini <[email protected]>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am experiencing the following issue with hdf5 and gcc 4.8.0
>>>>>>
>>>>>> Consider this very simple test
>>>>>>
>>>>>> #include <hdf5.h>
>>>>>>
>>>>>> int main() {
>>>>>> switch (H5Tget_order(H5T_NATIVE_LDOUBLE)) {
>>>>>> case H5T_ORDER_LE:
>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE\n");
>>>>>> break;
>>>>>> case H5T_ORDER_BE:
>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_BE\n");
>>>>>> break;
>>>>>> case H5T_ORDER_VAX:
>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX\n");
>>>>>> break;
>>>>>> case H5T_ORDER_MIXED:
>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_MIXED\n");
>>>>>> break;
>>>>>> case H5T_ORDER_NONE:
>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_NONE\n");
>>>>>> break;
>>>>>> default:
>>>>>> printf("here are dragons\n");
>>>>>> }
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>> on the same x86_64 GNU/Linux machine I get
>>>>>>
>>>>>> $ hdf5-1.8.11-gcc-4.7.0/my_test # compiled with gcc 4.7.0
>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE
>>>>>>
>>>>>> $ hdf5-1.8.11-gcc-4.8.0/my_test # compiled with gcc 4.8.0
>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX
>>>>>>
>>>>>> So H5T_NATIVE_LDOUBLE is mis-detected. I tried to dig deeper and
>>>>>> basically the fault must be in src/H5detect.c which is used to generate
>>>>>> the
>>>>>> definitions in src/H5Tinit.c
>>>>>> I could not figure out what H5detect.c does wrong (it is not very
>>>>>> readable, given its extensive use of macros) but the compiler does emit a
>>>>>> lot of warnings (see https://gist.github.com/andreabedini/6419975).
>>>>>>
>>>>>> I think this must be related to the failure of dt_arith long double
>>>>>> test observed recently.
>>>>>>
>>>>>> Any suggestion on how to fix this ?
>>>>>>
>>>>>> Best wishes,
>>>>>> Andrea
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andrea Bedini <[email protected]>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andrea Bedini <[email protected]>
>>>>> <hdf5_uninitialized.patch>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> [email protected]
>>>>>
>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> [email protected]
>>>>>
>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Andrea Bedini <[email protected]>
>>>>
>>>
>>>
>>>
>>> --
>>> Andrea Bedini <[email protected]>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> [email protected]
>>>
>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>>
>>>
>>
>>
>> --
>> George N. White III <[email protected]>
>> Head of St. Margarets Bay, Nova Scotia
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>>
>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>>
>
>
> --
> Andrea Bedini <[email protected]>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
>
> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
>
> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
>
--
Andrea Bedini <[email protected]>
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
