Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Hi,

On 12/27/2018 12:41 PM, Michael Crusoe wrote:

https://buildd.debian.org/status/fetch.php?pkg=htslib&arch=ppc64el&ver=1.9-7&stamp=1545236716&raw=0

Can I get some assistance here? Rebuilding using Qemu and the earlier source 
packages produces the same error, so maybe this is a regression in the compiler?


I'm not sure if it's a compiler regression, but I can only reproduce that
with gcc-8. Compiling with gcc-7 is fine.

Cheers,
Gustavo
 

--
Michael R. Crusoe
Co-founder & Lead, Common Workflow Language project 
Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
https://orcid.org/-0002-2961-9670 

m...@commonwl.org 
+1 480 627 9108 / +370 653 11125

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Hi Michael,

On 12/27/2018 04:35 PM, Gustavo Romero wrote:

On 12/27/2018 12:41 PM, Michael Crusoe wrote:

https://buildd.debian.org/status/fetch.php?pkg=htslib&arch=ppc64el&ver=1.9-7&stamp=1545236716&raw=0

Can I get some assistance here? Rebuilding using Qemu and the earlier source 
packages produces the same error, so maybe this is a regression in the compiler?


I'm not sure if it's a compiler regression, but I can only reproduce that
with gcc-8. Compiling with gcc-7 is fine.


I can see it fails with gcc-8 plus -O3, but not with -O2, because:

-O3
(gdb) call bam_auxB2f(p, i)
$5 = -3.141589879989624
(gdb) call (float) bam_auxB2f(p, i)
$1 = -3.14158988
(gdb) p bfvals[i]
$6 = -3.14159012

-O2
(gdb) call bam_auxB2f(p, i)
$1 = -3.1415901184082031
(gdb) call (float) bam_auxB2f(p, i)
$4 = -3.14159012
(gdb) p bfvals[i]
$2 = -3.14159012

Actually gcc will promote type float to double I think, but comparison will
fail anyway...

So, the code for this lib seems a little bit suspicious with all those cast /
missing casts, union (to convert uint32_t array to float), and different ways to
convert uint8_t* to uint32_t, like in le_to_u32(). So some implicit truncation
might be working by luck on gcc7 and gcc8 plus -O2 but optimization (O3) "broke"
it.

I can take a closer look tomorrow if nobody else bites the bait.

Cheers,
Gustavo


Cheers,
Gustavo


--
Michael R. Crusoe
Co-founder & Lead, Common Workflow Language project <http://www.commonwl.org/>
Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
https://orcid.org/-0002-2961-9670 
<https://impactstory.org/u/-0002-2961-9670>
m...@commonwl.org <mailto:m...@commonwl.org>
+1 480 627 9108 / +370 653 11125

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Hi Michael, Steffen,

On 12/30/2018 05:35 PM, Steffen Möller wrote:

I admit this is a bit over my head. It may be worthwhile to peek a boo at the 
latest gcc snapshots 
https://packages.debian.org/de/sid/ppc64/gcc-snapshot/download that may have a 
fix for that issue already.


I don't think it's a compiler issue... I can't explain it yet but it's probably
a float / double truncation issue exposed by some gcc-8 optimization. But trying
the snapshots can add more information to explain what's going on...
 

As a workaround to quickly proceed with what you are really after, and 
following the investigation by Gustavo, you may decide to have -O3 changed to 
-O2 if the arch is ppc64 and the compiler version is that faulty one.


Sorry, I meant -O0 instead of -O2, i.e. disabling any optimization, so it can't
be used as a workaround since it will destroy performance, specially for RISC.

That said, I dug a bit further and it looks like that sam_parse1() is actually
generating a different value for the floats in question, so when they are
loaded back for the comparison they are already screwed up, e.g.:

-- O3 --
0xc0490fcf
-3.14158988

-- O0 --
0xc0490fd0
-3.14159012 (expected)

because strtod() is used in float_to_le() and so also in u32_to_le(), which are
inlined and float_to_le() takes a float and not a double as the first argument
the use strtof() instead of strtod() in there looks the best, avoiding a
truncation from double to float, which might cause precision issues, specially
between different archs (like casts) plus optimization. So the following
change fixed the issue for me.

Michael, could you confirm that the following change fixes the issue at your
side  please?

diff --git a/sam.c b/sam.c
index aa94776..23233a0 100644
--- a/sam.c
+++ b/sam.c
@@ -1408,7 +1408,7 @@ int sam_parse1(kstring_t *s, bam_hdr_t *h, bam1_t *b)
 else if (type == 'S') while (q + 1 < p) { u16_to_le(strtoul(q + 1, 
&q, 0), (uint8_t *) str.s + str.l); str.l += 2; _skip_to_comma(q, p); }
 else if (type == 'i') while (q + 1 < p) { i32_to_le(strtol(q + 1, 
&q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
 else if (type == 'I') while (q + 1 < p) { u32_to_le(strtoul(q + 1, 
&q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
-else if (type == 'f') while (q + 1 < p) { float_to_le(strtod(q + 1, 
&q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
+else if (type == 'f') while (q + 1 < p) { float_to_le(strtof(q + 1, 
&q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
 else _parse_err_param(1, "unrecognized type B:%c", type);
 
 #undef _skip_to_comma



Cheers,
Gustavo


Cheers,

Steffen

On 27.12.18 15:41, Michael Crusoe wrote:

https://buildd.debian.org/status/fetch.php?pkg=htslib&arch=ppc64el&ver=1.9-7&stamp=1545236716&raw=0

Can I get some assistance here? Rebuilding using Qemu and the earlier source 
packages produces the same error, so maybe this is a regression in the compiler?

--
Michael R. Crusoe
Co-founder & Lead, Common Workflow Language project 
Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
https://orcid.org/-0002-2961-9670 

m...@commonwl.org 
+1 480 627 9108 / +370 653 11125

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Hello Michael,

On 12/31/2018 02:50 PM, Michael Crusoe wrote:

În lun., 31 dec. 2018 la 06:42, Gustavo Romero mailto:grom...@linux.vnet.ibm.com>> a scris:
diff --git a/sam.c b/sam.c
index aa94776..23233a0 100644
--- a/sam.c
+++ b/sam.c
@@ -1408,7 +1408,7 @@ int sam_parse1(kstring_t *s, bam_hdr_t *h, bam1_t *b)
               else if (type == 'S') while (q + 1 < p) { u16_to_le(strtoul(q + 
1, &q, 0), (uint8_t *) str.s + str.l); str.l += 2; _skip_to_comma(q, p); }
               else if (type == 'i') while (q + 1 < p) { i32_to_le(strtol(q + 
1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
               else if (type == 'I') while (q + 1 < p) { u32_to_le(strtoul(q + 
1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
-            else if (type == 'f') while (q + 1 < p) { float_to_le(strtod(q + 
1, &q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
+            else if (type == 'f') while (q + 1 < p) { float_to_le(strtof(q + 
1, &q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
               else _parse_err_param(1, "unrecognized type B:%c", type);

   #undef _skip_to_comma


Applying this patch and compiling under a qemu-based sid-ppc64el builder gets 
us to only a single test failure (yay!)

===
test_vcf_various:
         /build/htslib-1.9/htsfile -c /build/htslib-1.9/test/formatcols.vcf
         The outputs differ:
                 /build/htslib-1.9/test/formatcols.vcf
                 /build/htslib-1.9/test/formatcols.vcf.new
.. failed ...
===


Thanks for testing it!

hmm right... I'm not able to reproduce that failure on my setup.

Could you share the differences when it fails? I think a 'diff -u 
formatcols.vcf formatcols.vcf.new' suffices.



In the meantime, to unclog a chain of packages that have been held back from 
migrating to testing, I've uploaded a version of the package that uses -O0 for 
ppc64el only; which I agree is not ideal.

If you think this is a qemu-only test failure then I can upload a build to 
experimental so that real hardware is used (I don't have porter box access)


I don't know... but all debugging / testing at my side is on a ppc64el VM 
(qemu/kvm-only so), so probably not a VM vs baremetal issue.

Did you check if the same error happens on x86_64 when that patch is applied? 
Or it's still ppc64el-specific?


Cheers,
Gustavo


Cheers,
Gustavo

 > Cheers,
 >
 > Steffen
 >
 > On 27.12.18 15:41, Michael Crusoe wrote:
 >> 
https://buildd.debian.org/status/fetch.php?pkg=htslib&arch=ppc64el&ver=1.9-7&stamp=1545236716&raw=0
 >>
 >> Can I get some assistance here? Rebuilding using Qemu and the earlier 
source packages produces the same error, so maybe this is a regression in the 
compiler?
 >>
 >> --
 >> Michael R. Crusoe
 >> Co-founder & Lead, Common Workflow Language project 
<http://www.commonwl.org/>
 >> Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
 >> https://orcid.org/-0002-2961-9670 
<https://impactstory.org/u/-0002-2961-9670>
 >> m...@commonwl.org <mailto:m...@commonwl.org> <mailto:m...@commonwl.org 
<mailto:m...@commonwl.org>>
 >> +1 480 627 9108 / +370 653 11125
 >



--
Michael R. Crusoe
Co-founder & Lead, Common Workflow Language project <http://www.commonwl.org/>
Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
https://orcid.org/-0002-2961-9670 
<https://impactstory.org/u/-0002-2961-9670>
m...@commonwl.org <mailto:m...@commonwl.org>
+1 480 627 9108 / +370 653 11125

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

On 01/02/2019 11:27 AM, Gustavo Romero wrote:

Hello Michael,

On 12/31/2018 02:50 PM, Michael Crusoe wrote:

În lun., 31 dec. 2018 la 06:42, Gustavo Romero mailto:grom...@linux.vnet.ibm.com>> a scris:
    diff --git a/sam.c b/sam.c
    index aa94776..23233a0 100644
    --- a/sam.c
    +++ b/sam.c
    @@ -1408,7 +1408,7 @@ int sam_parse1(kstring_t *s, bam_hdr_t *h, bam1_t *b)
               else if (type == 'S') while (q + 1 < p) { u16_to_le(strtoul(q + 
1, &q, 0), (uint8_t *) str.s + str.l); str.l += 2; _skip_to_comma(q, p); }
               else if (type == 'i') while (q + 1 < p) { i32_to_le(strtol(q + 
1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
               else if (type == 'I') while (q + 1 < p) { u32_to_le(strtoul(q + 
1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
    -            else if (type == 'f') while (q + 1 < p) { float_to_le(strtod(q + 
1, &q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
    +            else if (type == 'f') while (q + 1 < p) { float_to_le(strtof(q + 
1, &q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, p); }
               else _parse_err_param(1, "unrecognized type B:%c", type);

   #undef _skip_to_comma


Applying this patch and compiling under a qemu-based sid-ppc64el builder gets 
us to only a single test failure (yay!)

===
test_vcf_various:
         /build/htslib-1.9/htsfile -c /build/htslib-1.9/test/formatcols.vcf
         The outputs differ:
                 /build/htslib-1.9/test/formatcols.vcf
                 /build/htslib-1.9/test/formatcols.vcf.new
.. failed ...
===


Thanks for testing it!

hmm right... I'm not able to reproduce that failure on my setup.

Could you share the differences when it fails? I think a 'diff -u 
formatcols.vcf formatcols.vcf.new' suffices.


I'm really not able to get any failure with the fix above applied.
I've tried with the latest sid/unstable updates, with both -O2 and -O3, on a 
VM, not baremetal.

As a reference, I have:

binutils 2.31.1-11
gcc-88.2.0-13
libc6-dev:ppc64el2.28-4
libstdc++-8-dev:ppc64el  8.2.0-13
libstdc++6:ppc64el   8.2.0-13
linux-libc-dev:ppc64el   4.19.13-1

Any chance there was stale file in the builder?
If you are able to grab the diff I can investigate it. I'm a bit worried about 
leaving -O0 for PPC64...

Thanks.

Cheers,
Gustavo
 



In the meantime, to unclog a chain of packages that have been held back from 
migrating to testing, I've uploaded a version of the package that uses -O0 for 
ppc64el only; which I agree is not ideal.

If you think this is a qemu-only test failure then I can upload a build to 
experimental so that real hardware is used (I don't have porter box access)


I don't know... but all debugging / testing at my side is on a ppc64el VM 
(qemu/kvm-only so), so probably not a VM vs baremetal issue.

Did you check if the same error happens on x86_64 when that patch is applied? 
Or it's still ppc64el-specific?


Cheers,
Gustavo


    Cheers,
    Gustavo

 > Cheers,
 >
 > Steffen
 >
 > On 27.12.18 15:41, Michael Crusoe wrote:
 >> 
https://buildd.debian.org/status/fetch.php?pkg=htslib&arch=ppc64el&ver=1.9-7&stamp=1545236716&raw=0
 >>
 >> Can I get some assistance here? Rebuilding using Qemu and the earlier 
source packages produces the same error, so maybe this is a regression in the 
compiler?
 >>
 >> --
 >> Michael R. Crusoe
 >> Co-founder & Lead, Common Workflow Language project 
<http://www.commonwl.org/>
 >> Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
 >> https://orcid.org/-0002-2961-9670 
<https://impactstory.org/u/-0002-2961-9670>
 >> m...@commonwl.org <mailto:m...@commonwl.org> <mailto:m...@commonwl.org 
<mailto:m...@commonwl.org>>
 >> +1 480 627 9108 / +370 653 11125
 >



--
Michael R. Crusoe
Co-founder & Lead, Common Workflow Language project <http://www.commonwl.org/>
Direktorius, VšĮ "Darbo eigos", Vilnius, Lithuania
https://orcid.org/-0002-2961-9670 
<https://impactstory.org/u/-0002-2961-9670>
m...@commonwl.org <mailto:m...@commonwl.org>
+1 480 627 9108 / +370 653 11125

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Hi Michael,

Thanks for the diff and the additional information on the failure.

Since I could not reproduce the issue locally, would you mind to point a buildd 
machine (I tried the buildd logs for htslib without success) where it fails, 
please?

Thank you.

Regards,
Gustavo

On 01/02/2019 02:41 PM, Michael Crusoe wrote:



În mie., 2 ian. 2019 la 15:27, Gustavo Romero mailto:grom...@linux.vnet.ibm.com>> a scris:

Hello Michael,

On 12/31/2018 02:50 PM, Michael Crusoe wrote:
 > În lun., 31 dec. 2018 la 06:42, Gustavo Romero mailto:grom...@linux.vnet.ibm.com> <mailto:grom...@linux.vnet.ibm.com 
<mailto:grom...@linux.vnet.ibm.com>>> a scris:
 >     diff --git a/sam.c b/sam.c
 >     index aa94776..23233a0 100644
 >     --- a/sam.c
 >     +++ b/sam.c
 >     @@ -1408,7 +1408,7 @@ int sam_parse1(kstring_t *s, bam_hdr_t *h, 
bam1_t *b)
 >                    else if (type == 'S') while (q + 1 < p) { 
u16_to_le(strtoul(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 2; 
_skip_to_comma(q, p); }
 >                    else if (type == 'i') while (q + 1 < p) { 
i32_to_le(strtol(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >                    else if (type == 'I') while (q + 1 < p) { 
u32_to_le(strtoul(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >     -            else if (type == 'f') while (q + 1 < p) { 
float_to_le(strtod(q + 1, &q), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >     +            else if (type == 'f') while (q + 1 < p) { 
float_to_le(strtof(q + 1, &q), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >                    else _parse_err_param(1, "unrecognized type B:%c", 
type);
 >
 >        #undef _skip_to_comma
 >
 >
 > Applying this patch and compiling under a qemu-based sid-ppc64el builder 
gets us to only a single test failure (yay!)
 >
 > ===
 > test_vcf_various:
 >          /build/htslib-1.9/htsfile -c 
/build/htslib-1.9/test/formatcols.vcf
 >          The outputs differ:
 >                  /build/htslib-1.9/test/formatcols.vcf
 >                  /build/htslib-1.9/test/formatcols.vcf.new
 > .. failed ...
 > ===

Thanks for testing it!

hmm right... I'm not able to reproduce that failure on my setup.


I've added the patch our git repo, so maybe try again by upcommenting it at 
https://salsa.debian.org/med-team/htslib/blob/master/debian/patches/series ?


Could you share the differences when it fails? I think a 'diff -u 
formatcols.vcf formatcols.vcf.new' suffices.

root@mrcdev:/build/htslib-1.9# diff -u 
/build/htslib-1.9/test/formatmissing-out.vcf 
/build/htslib-1.9/test/formatmissing-out.vcf.new
--- /build/htslib-1.9/test/formatmissing-out.vcf        2018-07-18 
08:33:10.0 +
+++ /build/htslib-1.9/test/formatmissing-out.vcf.new    2019-01-02 
16:14:27.0 +
@@ -3,4 +3,4 @@
  ##contig=
  ##FORMAT=
  #CHROM POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  S1     
 S2      S3
-1      100     a       A       T       .       .       .       .       .       
.       .
+1      100     a       A       T       3.40282e+38     .       .       .       
.       .       .

root@mrcdev:/build/htslib-1.9# diff -u /build/htslib-1.9/test/formatcols.vcf 
/build/htslib-1.9/test/formatcols.vcf.new
--- /build/htslib-1.9/test/formatcols.vcf       2018-07-18 08:33:10.0 
+
+++ /build/htslib-1.9/test/formatcols.vcf.new   2019-01-02 16:14:27.0 
+
@@ -3,4 +3,4 @@
  ##contig=
  ##FORMAT=
  #CHROM POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  S1     
 S2      S3
-1      100     a       A       T       .       .       .       S       a       
bbb c
+1      100     a       A       T       3.40282e+38     .       .       S       
a       bbb c

 > In the meantime, to unclog a chain of packages that have been held back 
from migrating to testing, I've uploaded a version of the package that uses -O0 
for ppc64el only; which I agree is not ideal.
 >
 > If you think this is a qemu-only test failure then I can upload a build 
to experimental so that real hardware is used (I don't have porter box access)

I don't know... but all debugging / testing at my side is on a ppc64el VM 
(qemu/kvm-only so), so probably not a VM vs baremetal issue.

Did you check if the same error happens on x86_64 when that patch is 
applied? Or it's still ppc64el-specific?


amd64 builds just fine with the patch

Cheers,
Gustavo
 >
 >     Cheers,
 >     Gustavo
 >
 >      > Cheers,
 >      &g

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Ping.

On 01/04/2019 08:01 PM, Gustavo Romero wrote:

Hi Michael,

Thanks for the diff and the additional information on the failure.

Since I could not reproduce the issue locally, would you mind to point a buildd 
machine (I tried the buildd logs for htslib without success) where it fails, 
please?

Thank you.

Regards,
Gustavo

On 01/02/2019 02:41 PM, Michael Crusoe wrote:



În mie., 2 ian. 2019 la 15:27, Gustavo Romero mailto:grom...@linux.vnet.ibm.com>> a scris:

    Hello Michael,

    On 12/31/2018 02:50 PM, Michael Crusoe wrote:
 > În lun., 31 dec. 2018 la 06:42, Gustavo Romero mailto:grom...@linux.vnet.ibm.com> <mailto:grom...@linux.vnet.ibm.com 
<mailto:grom...@linux.vnet.ibm.com>>> a scris:
 >     diff --git a/sam.c b/sam.c
 >     index aa94776..23233a0 100644
 >     --- a/sam.c
 >     +++ b/sam.c
 >     @@ -1408,7 +1408,7 @@ int sam_parse1(kstring_t *s, bam_hdr_t *h, 
bam1_t *b)
 >                    else if (type == 'S') while (q + 1 < p) { 
u16_to_le(strtoul(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 2; 
_skip_to_comma(q, p); }
 >                    else if (type == 'i') while (q + 1 < p) { 
i32_to_le(strtol(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >                    else if (type == 'I') while (q + 1 < p) { 
u32_to_le(strtoul(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >     -            else if (type == 'f') while (q + 1 < p) { 
float_to_le(strtod(q + 1, &q), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >     +            else if (type == 'f') while (q + 1 < p) { 
float_to_le(strtof(q + 1, &q), (uint8_t *) str.s + str.l); str.l += 4; 
_skip_to_comma(q, p); }
 >                    else _parse_err_param(1, "unrecognized type B:%c", 
type);
 >
 >        #undef _skip_to_comma
 >
 >
 > Applying this patch and compiling under a qemu-based sid-ppc64el builder 
gets us to only a single test failure (yay!)
 >
 > ===
 > test_vcf_various:
 >          /build/htslib-1.9/htsfile -c 
/build/htslib-1.9/test/formatcols.vcf
 >          The outputs differ:
 >                  /build/htslib-1.9/test/formatcols.vcf
 >                  /build/htslib-1.9/test/formatcols.vcf.new
 > .. failed ...
 > ===

    Thanks for testing it!

    hmm right... I'm not able to reproduce that failure on my setup.


I've added the patch our git repo, so maybe try again by upcommenting it at 
https://salsa.debian.org/med-team/htslib/blob/master/debian/patches/series ?


    Could you share the differences when it fails? I think a 'diff -u 
formatcols.vcf formatcols.vcf.new' suffices.

root@mrcdev:/build/htslib-1.9# diff -u 
/build/htslib-1.9/test/formatmissing-out.vcf 
/build/htslib-1.9/test/formatmissing-out.vcf.new
--- /build/htslib-1.9/test/formatmissing-out.vcf        2018-07-18 
08:33:10.0 +
+++ /build/htslib-1.9/test/formatmissing-out.vcf.new    2019-01-02 
16:14:27.0 +
@@ -3,4 +3,4 @@
  ##contig=
  ##FORMAT=
  #CHROM POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  S1     
 S2      S3
-1      100     a       A       T       .       .       .       .       .       
.       .
+1      100     a       A       T       3.40282e+38     .       .       .       
.       .       .

root@mrcdev:/build/htslib-1.9# diff -u /build/htslib-1.9/test/formatcols.vcf 
/build/htslib-1.9/test/formatcols.vcf.new
--- /build/htslib-1.9/test/formatcols.vcf       2018-07-18 08:33:10.0 
+
+++ /build/htslib-1.9/test/formatcols.vcf.new   2019-01-02 16:14:27.0 
+
@@ -3,4 +3,4 @@
  ##contig=
  ##FORMAT=
  #CHROM POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  S1     
 S2      S3
-1      100     a       A       T       .       .       .       S       a       
bbb c
+1      100     a       A       T       3.40282e+38     .       .       S       
a       bbb c

 > In the meantime, to unclog a chain of packages that have been held back 
from migrating to testing, I've uploaded a version of the package that uses -O0 
for ppc64el only; which I agree is not ideal.
 >
 > If you think this is a qemu-only test failure then I can upload a build 
to experimental so that real hardware is used (I don't have porter box access)

    I don't know... but all debugging / testing at my side is on a ppc64el VM 
(qemu/kvm-only so), so probably not a VM vs baremetal issue.

    Did you check if the same error happens on x86_64 when that patch is 
applied? Or it's still ppc64el-specific?


amd64 builds just fine with the patch

    Cheers,
    Gustavo
 >
 >     Cheers,
 >     Gu

Re: Help requested for htslib build failure on ppc64el

[Gustavo Romero]

Hi Michael!

On 01/09/2019 06:08 PM, Michael Crusoe wrote:

https://buildd.debian.org/status/package.php?p=htslib&suite=experimental

http://metadata.ftp-master.debian.org/changelogs/main/h/htslib/htslib_1.9-9~floatingpoint0_changelog
 
<http://metadata.ftp-master.debian.org/changelogs/main/h/htslib/htslib_1.9-9%7Efloatingpoint0_changelog>

Seems to work after all! Maybe a qemu bug?


Maybe, if host was amd64, etc... but hard to say with reproducing it and 
hunting.
I wonder if it's easy to trigger a build with -O3, just out of curiosity, to 
check it's gone.
btw, I didn't know you were using experimental instead of sid, but I 
suspected... :)

Thank you.

Best regards,
Gustavo


--
Michael R. Crusoe
Co-founder & Lead,
Common Workflow Language project
https://impactstory.org/u/-0002-2961-9670
m...@commonwl.org <mailto:m...@commonwl.org>

În mie., 9 ian. 2019, 19:38 Gustavo Romero mailto:grom...@linux.vnet.ibm.com> a scris:

    Ping.

On 01/04/2019 08:01 PM, Gustavo Romero wrote:
 > Hi Michael,
 >
 > Thanks for the diff and the additional information on the failure.
 >
 > Since I could not reproduce the issue locally, would you mind to point a 
buildd machine (I tried the buildd logs for htslib without success) where it 
fails, please?
 >
 > Thank you.
 >
 > Regards,
 > Gustavo
 >
 > On 01/02/2019 02:41 PM, Michael Crusoe wrote:
 >>
 >>
 >> În mie., 2 ian. 2019 la 15:27, Gustavo Romero mailto:grom...@linux.vnet.ibm.com> <mailto:grom...@linux.vnet.ibm.com 
<mailto:grom...@linux.vnet.ibm.com>>> a scris:
 >>
 >>     Hello Michael,
 >>
 >>     On 12/31/2018 02:50 PM, Michael Crusoe wrote:
 >>  > În lun., 31 dec. 2018 la 06:42, Gustavo Romero mailto:grom...@linux.vnet.ibm.com> <mailto:grom...@linux.vnet.ibm.com <mailto:grom...@linux.vnet.ibm.com>> 
<mailto:grom...@linux.vnet.ibm.com <mailto:grom...@linux.vnet.ibm.com> <mailto:grom...@linux.vnet.ibm.com 
<mailto:grom...@linux.vnet.ibm.com>>>> a scris:
 >>  >     diff --git a/sam.c b/sam.c
 >>  >     index aa94776..23233a0 100644
 >>  >     --- a/sam.c
 >>  >     +++ b/sam.c
 >>  >     @@ -1408,7 +1408,7 @@ int sam_parse1(kstring_t *s, bam_hdr_t 
*h, bam1_t *b)
 >>  >                    else if (type == 'S') while (q + 1 < p) { 
u16_to_le(strtoul(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 2; _skip_to_comma(q, 
p); }
 >>  >                    else if (type == 'i') while (q + 1 < p) { 
i32_to_le(strtol(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, 
p); }
 >>  >                    else if (type == 'I') while (q + 1 < p) { 
u32_to_le(strtoul(q + 1, &q, 0), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, 
p); }
 >>  >     -            else if (type == 'f') while (q + 1 < p) { 
float_to_le(strtod(q + 1, &q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, 
p); }
 >>  >     +            else if (type == 'f') while (q + 1 < p) { 
float_to_le(strtof(q + 1, &q), (uint8_t *) str.s + str.l); str.l += 4; _skip_to_comma(q, 
p); }
 >>  >                    else _parse_err_param(1, "unrecognized type 
B:%c", type);
 >>  >
 >>  >        #undef _skip_to_comma
 >>  >
 >>  >
 >>  > Applying this patch and compiling under a qemu-based sid-ppc64el 
builder gets us to only a single test failure (yay!)
 >>  >
 >>  > ===
 >>  > test_vcf_various:
 >>  >          /build/htslib-1.9/htsfile -c 
/build/htslib-1.9/test/formatcols.vcf
 >>  >          The outputs differ:
 >>  >                  /build/htslib-1.9/test/formatcols.vcf
 >>  >                  /build/htslib-1.9/test/formatcols.vcf.new
 >>  > .. failed ...
 >>  > ===
 >>
 >>     Thanks for testing it!
 >>
 >>     hmm right... I'm not able to reproduce that failure on my setup.
 >>
 >>
 >> I've added the patch our git repo, so maybe try again by upcommenting 
it at https://salsa.debian.org/med-team/htslib/blob/master/debian/patches/series ?
 >>
 >>
 >>     Could you share the differences when it fails? I think a 'diff -u 
formatcols.vcf formatcols.vcf.new' suffices.
 >>
 >> root@mrcdev:/build/htslib-1.9# diff -u 
/build/htslib-1.9/test/

Booting Debian 10 on powernv POWER9 machine

[Gustavo Romero]

Hi,

Has anybody tried to install, boot, and test successfully on a POWER9 powernv
machine?

If so, could I get and advice on which Debian installer (ISO, initrd, and
vmlinuz) I should use please?

Thanks,
Gustavo

Re: Booting Debian 10 on powernv POWER9 machine

[Gustavo Romero]

Hi Len,

On 10/17/2019 12:45 PM, Lennart Sorensen wrote:

On Wed, Oct 16, 2019 at 06:49:03PM -0300, Gustavo Romero wrote:

Has anybody tried to install, boot, and test successfully on a POWER9 powernv
machine?

If so, could I get and advice on which Debian installer (ISO, initrd, and
vmlinuz) I should use please?


I have not tried one, but I would expect the ppc64le to work on it since
it is meant for power8+.  I unfortunately have not had a chance to play
with anything newer than power7.


Sorry, I wrote my note super hastily and forgot to mention that I was wondering
about BE specifically. Currently I see kernel 4.16 is used by the installer and
I get a panic regarding a VMX Unavailable Exception on POWER9 powernv (both
baremetal / OPAL and qemu emulated), so I'm wondering if anybody else is
catching that panic too etc. I know BE support for PPC64 was dropped officially
by I understand buildd builders are still building it so there are fresh
packages out there.

Thanks for replying and for the comments!

Cheers,
Gustavo

Re: Booting Debian 10 on powernv POWER9 machine

[Gustavo Romero]

Hi Michael and Lennart,

On 10/17/2019 04:09 PM, Michael Cree wrote:

On Thu, Oct 17, 2019 at 02:58:37PM -0400, Lennart Sorensen wrote:

On Thu, Oct 17, 2019 at 02:57:12PM -0300, Gustavo Romero wrote:

Sorry, I wrote my note super hastily and forgot to mention that I was wondering
about BE specifically. Currently I see kernel 4.16 is used by the installer and
I get a panic regarding a VMX Unavailable Exception on POWER9 powernv (both
baremetal / OPAL and qemu emulated), so I'm wondering if anybody else is
catching that panic too etc. I know BE support for PPC64 was dropped officially
by I understand buildd builders are still building it so there are fresh
packages out there.

Thanks for replying and for the comments!


I don't believe 64 bit powerpc big endian was ever officially released
by Debian.  32 bit was but was dropped some time ago.


Got it, the drop was only for 32-bit. Thanks.



64 bit little endian on the other hand is supported.


Yep, I know.


 

32-bit BE powerpc and 64-bit BE ppc64 ports are unofficially maintained
at debian-ports:

https://www.ports.debian.org/

There are installers there.  Only unstable is available.


I could not find the installer's initrd.gz there, just the packages. I don't 
have
much experience on Debian, but I did some workarounds and was able to get Debian
Bullseye at least booting on POWER9 BE with upstream kernel 5.3:

root@gromero17:~# cat /etc/issue
Debian GNU/Linux bullseye/sid \n \l

root@gromero17:~# lscpu
Architecture:ppc64
CPU op-mode(s):  32-bit, 64-bit
Byte Order:  Big Endian
CPU(s):  1
On-line CPU(s) list: 0
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):   1
NUMA node(s):1
Model:   2.0 (pvr 004e 1200)
Model name:  POWER9, altivec supported
L1d cache:   32 KiB
L1i cache:   32 KiB
NUMA node0 CPU(s):   0
Vulnerability L1tf:  Not affected
Vulnerability Mds:   Not affected
Vulnerability Meltdown:  Mitigation; RFI Flush
Vulnerability Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio)
Vulnerability Spectre v1:Mitigation; __user pointer sanitization
Vulnerability Spectre v2:Vulnerable
root@gromero17:~# uname -a
Linux gromero17 5.3.0 #13 SMP Fri Oct 18 16:45:30 EDT 2019 ppc64 GNU/Linux
root@gromero17:~# ping -c 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=50 time=6.98 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=50 time=6.15 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=50 time=6.23 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=50 time=6.29 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 6.154/6.413/6.983/0.332 ms
root@gromero17:~#


Thanks a lot for all the help.

Best regards,
Gustavo