Some links on memset (in the opposite order I have found them) https://news.ycombinator.com/item?id=4711346 http://www.eliteraspberries.com/blog/2012/10/zero-and-forget--caveats-of-zeroing-memory-in-c.html http://www.viva64.com/en/b/0178/ http://stackoverflow.com/questions/15538366/can-memset-function-call-be-removed-by-compiler
On 6 September 2013 16:49, Andrea Bedini <[email protected]> wrote: > Hi Rey, > > thanks for that, it really helped. I checked thoroughly and the memset of > the temporary variables disappears randomly. > It doesn't depends only on optimization though, on my machine putting a > printf("%Lf\n", value2); just before the loop changes the result. > I'm not sure who gets the blame here, poking into the padding bits of a > long double might just be unspecified or undefined behaviour. > > Andrea > > > On 6 September 2013 06:40, Raymond Lu <[email protected]> wrote: > >> I isolated part of the DETECT_F into a C program as attached (detect.c). >> It only contains the algorithm for detecting the byte order of long >> double. When I compile it with gcc -g, -O0, or no flag, it reports >> little-endian. When I compile it with -O1, -O2, or -O3, it reports VAX >> order. I don't know where goes wrong yet. But I suspect GCC's >> optimization has bugs. Maybe you can help me. >> >> I haven't tried the algorithms for other parts in DETECT_F yet. The >> alignment problem you talked about is one of the other algorithms. >> >> Ray >> >> >> >> >> >> >> >> On Sep 4, 2013, at 8:40 PM, Andrea Bedini wrote: >> >> Thanks George. >> >> For anyone interested in debugging this problem, debian has an extensive >> collection of build logs over many architectures >> https://buildd.debian.org/status/logs.php?pkg=hdf5 (going back 12 years!) >> >> As far as I know, the corruption is limited to the H5T_NATIVE_LDOUBLE >> type. You can check your particular build with the following test >> >> #include <hdf5.h> >> int main() { >> return !(H5Tget_order(H5T_NATIVE_LDOUBLE) == H5Tget_order( >> H5T_NATIVE_DOUBLE)); >> } >> >> It exits with code 1 if the long double has different byte ordering than >> double (which is technically possible, but highly suspicious). >> >> Otherwise the patch I sent earlier in this thread seems to do the trick, >> although what exactly is going wrong is still beyond my understanding. >> >> Third option: you can define an equivalent of H5T_NATIVE_LDOUBLE >> yourself. The following creates a data type representing a long double as >> implemented by gcc on x86 architectures (see >> http://en.wikipedia.org/wiki/Long_double#Implementations for details) >> >> hid_t ldouble_datatype = H5Tcopy(H5T_NATIVE_DOUBLE); >> H5Tset_size(ldouble_datatype, sizeof(long double)); >> H5Tset_precision(ldouble_datatype, 80); >> H5Tset_fields (ldouble_datatype, 79, 64, 15, 0, 64); >> H5Tset_pad(ldouble_datatype, H5T_PAD_ZERO, H5T_PAD_ZERO); >> H5Tset_inpad(ldouble_datatype, H5T_PAD_ZERO); >> H5Tset_ebias(ldouble_datatype, 16383); >> H5Tset_norm(ldouble_datatype, H5T_NORM_NONE); >> >> Best wishes, >> Andrea >> >> On 4 September 2013 22:58, George N. White III <[email protected]> wrote: >> >>> Another historical reference to the obscurity of this code is: < >>> https://bugs.gentoo.org/show_bug.cgi?id=118777>. >>> >>> I've been building HDF5 libraries for use with NASA SeaDAS, and recently >>> have started using HDF5 with R and GDAL. The SeaDAS builds are static, and >>> I don't find the "unable to calculate alignment for long double" message in >>> my SeaDAS build logs on linux and OS X. For R and GDAL, however, I need >>> dynamic libraries and those build logs do have the "unable to calculate >>> alignment for long double" message on both linux and OS X. >>> >>> >>> On Tue, Sep 3, 2013 at 9:50 PM, Andrea Bedini >>> <[email protected]>wrote: >>> >>>> Hi, >>>> >>>> I found something else (I know, I should stop :)). I am not entirely >>>> sure but it seems that when H5detect fails it writes "unable to calculate >>>> alignment for long double" on stderr so this message should be observable >>>> on build logs (although buried by other warnings). The packages on debian >>>> sid and testing for both i386 and x86-64 seem to be affected: >>>> >>>> >>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=amd64&ver=1.8.11-3%2Bb1&stamp=1377024563 >>>> >>>> https://buildd.debian.org/status/fetch.php?pkg=hdf5&arch=i386&ver=1.8.11-3%2Bb1&stamp=1377025110 >>>> >>>> But here's the exciting part: look what I found >>>> >>>> >>>> http://www.unidata.ucar.edu/mailing_lists/archives/gembud/2010/msg00052.html >>>> >>>> It's a build log from 2010 for HDF5 v1.6.5 and gcc-4.4.3 that says "unable >>>> to calculate alignment for long double". >>>> >>>> If my understanding is correct, nor 1.8.11 or gcc 4.8.0 would be the >>>> problem and it would be that piece of code just doesn't work properly. >>>> >>>> Best wishes, >>>> Andrea >>>> >>>> >>>> >>>> On 4 September 2013 08:00, Andrea Bedini <[email protected]>wrote: >>>> >>>>> Hi Ray, >>>>> >>>>> thanks for giving it a look. Antonio made me notice that something >>>>> else might be at work since the macro DETECT_F already zeroes the >>>>> structure >>>>> right before anything else: >>>>> >>>>> memset(&INFO, 0, sizeof(INFO)); #L299 >>>>> >>>>> so I don't understand how the perm fields need to be zeroed again >>>>> around line #L308. This still considering the "Byte Order" loop as a black >>>>> box. >>>>> >>>>> As a side question: isn't there a more portable way of doing this? I >>>>> am pretty sure H5detect.c might invoke a bunch of undefined behaviours >>>>> given the amount of warning the compiler generates and of bit trickery. >>>>> >>>>> Best wishes, >>>>> Andrea >>>>> >>>>> >>>>> >>>>> >>>>> On 4 September 2013 05:43, Raymond Lu <[email protected]> wrote: >>>>> >>>>>> Andrea, >>>>>> >>>>>> We've verified that your solution is correct. We're putting your fix >>>>>> into the library. Thanks for helping us. >>>>>> >>>>>> Ray >>>>>> >>>>>> On Sep 3, 2013, at 3:32 AM, Andrea Bedini wrote: >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I think I have found the problem. The issue is in H5detect.c. >>>>>> Macros DETECT_F and DETECT_I do not initialize properly the perm field in >>>>>> the detected_t struct. As a result the routine fix_order is passed some >>>>>> uninitialized memory which makes it fail. I have a small patch against >>>>>> H5detect.c which fixes the problem by simply initializing the perm field >>>>>> with zeros. Valgrind's tool memcheck would have exposed the problem. >>>>>> >>>>>> Best wishes, >>>>>> Andrea >>>>>> >>>>>> >>>>>> >>>>>> On 3 September 2013 15:30, Andrea Bedini <[email protected]>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am experiencing the following issue with hdf5 and gcc 4.8.0 >>>>>>> >>>>>>> Consider this very simple test >>>>>>> >>>>>>> #include <hdf5.h> >>>>>>> >>>>>>> int main() { >>>>>>> switch (H5Tget_order(H5T_NATIVE_LDOUBLE)) { >>>>>>> case H5T_ORDER_LE: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_BE: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_BE\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_VAX: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_MIXED: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_MIXED\n"); >>>>>>> break; >>>>>>> case H5T_ORDER_NONE: >>>>>>> printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_NONE\n"); >>>>>>> break; >>>>>>> default: >>>>>>> printf("here are dragons\n"); >>>>>>> } >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> on the same x86_64 GNU/Linux machine I get >>>>>>> >>>>>>> $ hdf5-1.8.11-gcc-4.7.0/my_test # compiled with gcc 4.7.0 >>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE >>>>>>> >>>>>>> $ hdf5-1.8.11-gcc-4.8.0/my_test # compiled with gcc 4.8.0 >>>>>>> H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX >>>>>>> >>>>>>> So H5T_NATIVE_LDOUBLE is mis-detected. I tried to dig deeper and >>>>>>> basically the fault must be in src/H5detect.c which is used to generate >>>>>>> the >>>>>>> definitions in src/H5Tinit.c >>>>>>> I could not figure out what H5detect.c does wrong (it is not very >>>>>>> readable, given its extensive use of macros) but the compiler does emit >>>>>>> a >>>>>>> lot of warnings (see https://gist.github.com/andreabedini/6419975). >>>>>>> >>>>>>> I think this must be related to the failure of dt_arith long double >>>>>>> test observed recently. >>>>>>> >>>>>>> Any suggestion on how to fix this ? >>>>>>> >>>>>>> Best wishes, >>>>>>> Andrea >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Andrea Bedini <[email protected]> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Andrea Bedini <[email protected]> >>>>>> <hdf5_uninitialized.patch> >>>>>> _______________________________________________ >>>>>> Hdf-forum is for HDF software users discussion. >>>>>> [email protected] >>>>>> >>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Hdf-forum is for HDF software users discussion. >>>>>> [email protected] >>>>>> >>>>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Andrea Bedini <[email protected]> >>>>> >>>> >>>> >>>> >>>> -- >>>> Andrea Bedini <[email protected]> >>>> >>>> _______________________________________________ >>>> Hdf-forum is for HDF software users discussion. >>>> [email protected] >>>> >>>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>> >>>> >>> >>> >>> -- >>> George N. White III <[email protected]> >>> Head of St. Margarets Bay, Nova Scotia >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> >>> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> >>> >> >> >> -- >> Andrea Bedini <[email protected]> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> > > > -- > Andrea Bedini <[email protected]> > -- Andrea Bedini <[email protected]>
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
