Hi Ray, yes, this is compatible with what I observed in my tests. Honestly I am not sure the problem has a simple solution at all. If the standard doesn't guarantee us the pad bit have a consistent value, there's no way we can expect this to work in a portable way: at some point in the future a compiler will be smart enough to revert all the obstacles we throw at its way.
If I may, I would suggest we replace the entire mechanism with something simpler. There are not that many floating point formats floating around (pardon the pun), even considering the many architectures. Can't we just hardcode them? The configuration system has surely enough information to determine the native floating point format without bit fiddling. What are the supported architectures? Andrea PS: it seems the disappearing memset problem can be solved by asking gcc to not replace memset with its builtin version, i.e. passing the option -fno-builtin-memset, but this of course won't work with other compilers. On 7 September 2013 08:24, Raymond Lu <[email protected]> wrote: > Andrea, > > My coworker Neil helped me in this afternoon to find out that when GCC 4.8 > compiler assigns constant values to variables (value1 and value2) like this, > > for(i = 0, value1 = 0.0, value2 = 1.0; i < (int)sizeof(long double); > i++) { > value3 = value1; > value1 += value2; > value2 /= 256.0; > : > : > > it introduces some garbage to the two padding bytes of value1 and value2. > Then the garbage confuses our algorithm, especially the value of > "last_mbyte" gets wrong. To fix it in a simple way, use an intermediate > variable like this: > > long double tmp_value, divisor; > > tmp_value = 0.0; > value1 = tmp_value; > tmp_value = 1.0; > value2 = tmp_value; > tmp_value = 256.0; > divisor = tmp_value; > > for(i = 0; i < (int)sizeof(long double); i++) { > value3 = value1; > value1 += value2; > value2 /= divisor; > : > : > > How do you think about it? > > Ray > > On Sep 6, 2013, at 1:49 AM, Andrea Bedini wrote: > > Hi Rey, > > thanks for that, it really helped. I checked thoroughly and the memset of > the temporary variables disappears randomly. > It doesn't depends only on optimization though, on my machine putting a > printf("%Lf\n", value2); just before the loop changes the result. > I'm not sure who gets the blame here, poking into the padding bits of a > long double might just be unspecified or undefined behaviour. > > Andrea > > > On 6 September 2013 06:40, Raymond Lu <[email protected]> wrote: > >> I isolated part of the DETECT_F into a C program as attached (detect.c). >> It only contains the algorithm for detecting the byte order of long >> double. When I compile it with gcc -g, -O0, or no flag, it reports >> little-endian. When I compile it with -O1, -O2, or -O3, it reports VAX >> order. I don't know where goes wrong yet. But I suspect GCC's >> optimization has bugs. Maybe you can help me. >> >> I haven't tried the algorithms for other parts in DETECT_F yet. The >> alignment problem you talked about is one of the other algorithms. >> >> Ray >> >> >> >> >> >> >> >> On Sep 4, 2013, at 8:40 PM, Andrea Bedini wrote: >> >> Thanks George. >> >> For anyone interested in debugging this problem, debian has an extensive >> collection of build logs over many architectures >> https://buildd.debian.org/status/logs.php?pkg=hdf5 (going back 12 years!) >> >> As far as I know, the corruption is limited to the H5T_NATIVE_LDOUBLE >> type. You can check your particular build with the following test >> >> #include <hdf5.h> >> int main() { >> return !(H5Tget_order(H5T_NATIVE_LDOUBLE) == H5Tget_order( >> H5T_NATIVE_DOUBLE)); >> } >> >> It exits with code 1 if the long double has different byte ordering than >> double (which is technically possible, but highly suspicious). >> >> Otherwise the patch I sent earlier in this thread seems to do the trick, >> although what exactly is going wrong is still beyond my understanding. >> >> Third option: you can define an equivalent of H5T_NATIVE_LDOUBLE >> yourself. The following creates a data type representing a long double as >> implemented by gcc on x86 architectures (see >> http://en.wikipedia.org/wiki/Long_double#Implementations for details) >> >> hid_t ldouble_datatype = H5Tcopy(H5T_NATIVE_DOUBLE); >> H5Tset_size(ldouble_datatype, sizeof(long double)); >> H5Tset_precision(ldouble_datatype, 80); >> H5Tset_fields (ldouble_datatype, 79, 64, 15, 0, 64); >> H5Tset_pad(ldouble_datatype, H5T_PAD_ZERO, H5T_PAD_ZERO); >> H5Tset_inpad(ldouble_datatype, H5T_PAD_ZERO); >> H5Tset_ebias(ldouble_datatype, 16383); >> H5Tset_norm(ldouble_datatype, H5T_NORM_NONE); >> >> Best wishes, >> Andrea >> >> On 4 September 2013 22:58, George N. White III <[email protected]> wrote: >> >>> Another historical reference to the obscurity of this code is: < >>> https://bugs.gentoo.org/show_bug.cgi?id=118777>. >>> >>> I've been building HDF5 libraries for use with NASA SeaDAS, and recently >>> have started using HDF5 with R and GDAL. The SeaDAS builds are static, and >>> I don't find the "unable to calculate alignment for long double" message in >>> my SeaDAS build logs on linux and OS X. For R and GDAL, however, I need >>> dynamic libraries and those build logs do have the "unable to calculate >>> alignment for long double" message on both linux and OS X. >>> >>> >>> On Tue, Sep 3, 2013 at 9:50 PM, Andrea Bedini >>> <[email protected]>wrote: >>> >>>> Hi, >>>> >>>> I found something else (I know, I should stop :)). I am not entirely >>>> sure but it seems that when H5detect fails it writes "unable to calculate >>>> alignment for long double" on stderr so this message should be observable >>>> on build logs (although buried by other warnings). The packages on debian >>>> sid and testing for both i386 and x86-64 seem to be affected: >>>> >>>> >>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=amd64&ver=1.8.11-3%2Bb1&stamp=1377024563 >>>> >>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=i386&ver=1.8.11-3%2Bb1&stamp=1377025110 >>>> >>>> But here's the exciting part: look what I found >>>> >>>> >>>> http://www.unidata.ucar.edu/mailing_lists/archives/gembud/2010/msg00052.html >>>> >>>> It's a build log from 2010 for HDF5 v1.6.5 and gcc-4.4.3 that says "unable >>>> to calculate alignment for long double". >>>> >>>> If my understanding is correct, nor 1.8.11 or gcc 4.8.0 would be the >>>> problem and it would be that piece of code just doesn't work properly. >>>> >>>> Best wishes, >>>> Andrea >>>> >>>> >>>> >>>> On 4 September 2013 08:00, Andrea Bedini <[email protected]>wrote: >>>> >>>>> Hi Ray, >>>>> >>>>> thanks for giving it a look. Antonio made me notice that something >>>>> else might be at work since the macro DETECT_F already zeroes the >>>>> structure >>>>> right before anything else: >>>>> >>>>> memset(&INFO, 0, sizeof(INFO)); #L299 >>>>> >>>>> so I don't understand how the perm fields need to be zeroed again >>>>> around line #L308. This still considering the "Byte Order" loop as a black >>>>> box. >>>>> >>>>> As a side question: isn't there a more portable way of doing this? I >>>>> am pretty sure H5detect.c might invoke a bunch of undefined behaviours >>>>> given the amount of warning the compiler generates and of bit trickery. >>>>> >>>>> Best wishes, >>>>> Andrea >>>>> >>>>> >>>>> >>>>> >>>>> On 4 September 2013 05:43, Raymond Lu <[email protected]> wrote: >>>>> >>>>>> Andrea, >>>>>> >>>>>> We've verified that your solution is correct. We're putting your fix >>>>>> into the library. Thanks for helping us. >>>>>> >>>>>> Ray >>>>>> >>>>>> On Sep 3, 2013, at 3:32 AM, Andrea Bedini wrote: >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I think I have found the problem. The issue is in H5detect.c. >>>>>> Macros DETECT_F and DETECT_I do not initialize properly the perm field in >>>>>> the detected_t struct. As a result the routine fix_order is passed some >>>>>> uninitialized memory which makes it fail. I have a small patch against >>>>>> H5detect.c which fixes the problem by simply initializing the perm field >>>>>> with zeros. Valgrind's tool memcheck would have exposed the problem. >>>>>> >>>>>> Best wishes, >>>>>> Andrea >>>>>> >>>>>> >>>>>> >>>>>> On 3 September 2013 15:30, Andrea Bedini <[email protected]>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am experiencing the following issue with hdf5 and gcc 4.8.0 >>>>>>> >>>>>>> Consider this very simple test >>>>>>> >>>>>>> #include <hdf5.h> >>>>>>> >>>>>>> int main() { >>>>>>> switch (H5Tget_order(H5T_NATIVE_LDOUBLE)) { >>>>>>> case H5T_ORDER_LE: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_BE: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_BE\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_VAX: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_MIXED: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_MIXED\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_NONE: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_NONE\n"); >>>>>>> break; >>>>>>> default: >>>>>>> printf("here are dragons\n"); >>>>>>> } >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> on the same x86_64 GNU/Linux machine I get >>>>>>> >>>>>>> $ hdf5-1.8.11-gcc-4.7.0/my_test # compiled with gcc 4.7.0 >>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE >>>>>>> >>>>>>> $ hdf5-1.8.11-gcc-4.8.0/my_test # compiled with gcc 4.8.0 >>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX >>>>>>> >>>>>>> So H5T_NATIVE_LDOUBLE is mis-detected. I tried to dig deeper and >>>>>>> basically the fault must be in src/H5detect.c which is used to generate >>>>>>> the >>>>>>> definitions in src/H5Tinit.c >>>>>>> I could not figure out what H5detect.c does wrong (it is not very >>>>>>> readable, given its extensive use of macros) but the compiler does emit >>>>>>> a >>>>>>> lot of warnings (see https://gist.github.com/andreabedini/6419975). >>>>>>> >>>>>>> I think this must be related to the failure of dt_arith long double >>>>>>> test observed recently. >>>>>>> >>>>>>> Any suggestion on how to fix this ? >>>>>>> >>>>>>> Best wishes, >>>>>>> Andrea >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Andrea Bedini <[email protected]> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Andrea Bedini <[email protected]> >>>>>> <hdf5_uninitialized.patch> >>>>>> _______________________________________________ >>>>>> Hdf-forum is for HDF software users discussion. >>>>>> [email protected] >>>>>> >>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Hdf-forum is for HDF software users discussion. >>>>>> [email protected] >>>>>> >>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Andrea Bedini <[email protected]> >>>>> >>>> >>>> >>>> >>>> -- >>>> Andrea Bedini <[email protected]> >>>> >>>> _______________________________________________ >>>> Hdf-forum is for HDF software users discussion. >>>> [email protected] >>>> >>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>> >>>> >>> >>> >>> -- >>> George N. White III <[email protected]> >>> Head of St. Margarets Bay, Nova Scotia >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> >>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> >>> >> >> >> -- >> Andrea Bedini <[email protected]> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> > > > -- > Andrea Bedini <[email protected]> > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > -- Andrea Bedini <[email protected]>
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
