Re: [Hdf-forum] Issues with H5T_NATIVE_LDOUBLE

Thu, 05 Sep 2013 14:35:43 -0700

I isolated part of the DETECT_F into a C program as attached (detect.c).  It only contains the algorithm for detecting the byte order of long double.  When I compile it with gcc -g, -O0, or no flag, it reports little-endian.  When I compile it with -O1, -O2, or -O3, it reports VAX order.  I don't know where goes wrong yet.  But I suspect GCC's optimization has bugs.  Maybe you can help me.

I haven't tried the algorithms for other parts in DETECT_F yet.  The alignment problem you talked about is one of the other algorithms.

Ray



Attachment: detect.c
Description: Binary data

 


On Sep 4, 2013, at 8:40 PM, Andrea Bedini wrote:

Thanks George.

For anyone interested in debugging this problem, debian has an extensive collection of build logs over many architectures https://buildd.debian.org/status/logs.php?pkg=hdf5 (going back 12 years!)

As far as I know, the corruption is limited to the H5T_NATIVE_LDOUBLE type. You can check your particular build with the following test

#include <hdf5.h>
int main() {
  return !(H5Tget_order(H5T_NATIVE_LDOUBLE) == H5Tget_order(H5T_NATIVE_DOUBLE));
}

It exits with code 1 if the long double has different byte ordering than double (which is technically possible, but highly suspicious).

Otherwise the patch I sent earlier in this thread seems to do the trick, although what exactly is going wrong is still beyond my understanding.

Third option: you can define an equivalent of H5T_NATIVE_LDOUBLE yourself. The following creates a data type representing a long double as implemented by gcc on x86 architectures (see http://en.wikipedia.org/wiki/Long_double#Implementations for details)

hid_t ldouble_datatype = H5Tcopy(H5T_NATIVE_DOUBLE);
H5Tset_size(ldouble_datatype, sizeof(long double));
H5Tset_precision(ldouble_datatype, 80);
H5Tset_fields (ldouble_datatype, 79, 64, 15, 0, 64);
H5Tset_pad(ldouble_datatype, H5T_PAD_ZERO, H5T_PAD_ZERO);
H5Tset_inpad(ldouble_datatype, H5T_PAD_ZERO);
H5Tset_ebias(ldouble_datatype, 16383);
H5Tset_norm(ldouble_datatype, H5T_NORM_NONE);

Best wishes,
Andrea

On 4 September 2013 22:58, George N. White III <[email protected]> wrote:
Another historical reference to the obscurity of this code is: <https://bugs.gentoo.org/show_bug.cgi?id=118777>.

I've been building HDF5 libraries for use with NASA SeaDAS, and recently have started using HDF5 with R and GDAL.  The SeaDAS builds are static, and I don't find the "unable to calculate alignment for long double" message in my SeaDAS build logs on linux and OS X. For R and GDAL, however, I need dynamic libraries and those build logs do have the "unable to calculate alignment for long double" message on both linux and OS X.


On Tue, Sep 3, 2013 at 9:50 PM, Andrea Bedini <[email protected]> wrote:
Hi,

I found something else (I know, I should stop :)). I am not entirely sure but it seems that when H5detect fails it writes "unable to calculate alignment for long double" on stderr so this message should be observable on build logs (although buried by other warnings). The packages on debian sid and testing for both i386 and x86-64 seem to be affected:


But here's the exciting part: look what I found


It's a build log from 2010 for HDF5 v1.6.5 and gcc-4.4.3 that says "unable to calculate alignment for long double".

If my understanding is correct, nor 1.8.11 or gcc 4.8.0 would be the problem and it would be that piece of code just doesn't work properly. 

Best wishes,
Andrea



On 4 September 2013 08:00, Andrea Bedini <[email protected]> wrote:
Hi Ray,

thanks for giving it a look. Antonio made me notice that something else might be at work since the macro DETECT_F already zeroes the structure right before anything else:

memset(&INFO, 0, sizeof(INFO)); #L299

so I don't understand how the perm fields need to be zeroed again around line #L308. This still considering the "Byte Order" loop as a black box.

As a side question: isn't there a more portable way of doing this? I am pretty sure H5detect.c might invoke a bunch of undefined behaviours given the amount of warning the compiler generates and of bit trickery.

Best wishes,
Andrea




On 4 September 2013 05:43, Raymond Lu <[email protected]> wrote:
Andrea,

We've verified that your solution is correct.  We're putting your fix into the library.  Thanks for helping us.

Ray

On Sep 3, 2013, at 3:32 AM, Andrea Bedini wrote:

Hi there,

I think I have found the problem. The issue is in H5detect.c. Macros DETECT_F and DETECT_I do not initialize properly the perm field in the detected_t struct. As a result the routine fix_order is passed some uninitialized memory which makes it fail. I have a small patch against H5detect.c which fixes the problem by simply initializing the perm field with zeros. Valgrind's tool memcheck would have exposed the problem.

Best wishes,
Andrea



On 3 September 2013 15:30, Andrea Bedini <[email protected]> wrote:
Hi,

I am experiencing the following issue with hdf5 and gcc 4.8.0

Consider this very simple test

#include <hdf5.h>

int main() {
  switch (H5Tget_order(H5T_NATIVE_LDOUBLE)) {
  case H5T_ORDER_LE:
    printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE\n");
    break;
  case H5T_ORDER_BE:
    printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_BE\n");
    break;
  case H5T_ORDER_VAX:
    printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX\n");
    break;
  case H5T_ORDER_MIXED:
    printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_MIXED\n");
    break;
  case H5T_ORDER_NONE:
    printf("H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_NONE\n");
    break;
  default:
    printf("here are dragons\n");
  }
  return 0;
}

on the same x86_64 GNU/Linux machine I get 

$ hdf5-1.8.11-gcc-4.7.0/my_test # compiled with gcc 4.7.0
H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_LE

$ hdf5-1.8.11-gcc-4.8.0/my_test # compiled with gcc 4.8.0
H5Tget_order(H5T_NATIVE_LDOUBLE) = H5T_ORDER_VAX

So H5T_NATIVE_LDOUBLE is mis-detected. I tried to dig deeper and basically the fault must be in src/H5detect.c which is used to generate the definitions in src/H5Tinit.c
I could not figure out what H5detect.c does wrong (it is not very readable, given its extensive use of macros) but the compiler does emit a lot of warnings (see https://gist.github.com/andreabedini/6419975).

I think this must be related to the failure of dt_arith long double test observed recently.

Any suggestion on how to fix this ?

Best wishes,
Andrea


-- 
Andrea Bedini <[email protected]>



--
Andrea Bedini <[email protected]>
<hdf5_uninitialized.patch>_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org




--
Andrea Bedini <[email protected]>



--
Andrea Bedini <[email protected]>

_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org




--
George N. White III <[email protected]>
Head of St. Margarets Bay, Nova Scotia

_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org




--
Andrea Bedini <[email protected]>
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

Reply via email to