[Bug target/91450] __builtin_mul_overflow(A,B,R) wrong code if product < 0, *R is unsigned, and !(A&B)

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91450

--- Comment #4 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Yeah, the comment in expand_mul_overflow is correct:
     s1 * s2 -> ur
        t1 = (s1 & s2) < 0 ? (-(U) s1) : ((U) s1)
        t2 = (s1 & s2) < 0 ? (-(U) s2) : ((U) s2)
        res = t1 * t2
        ovf = (s1 ^ s2) < 0 ? (s1 && s2) : main_ovf (true)
but the implementation actually doesn't do s1 && s2 but s1 & s2, in both spots:
              tem = expand_binop (mode, and_optab, op0, op1, NULL_RTX, false,
                                  OPTAB_LIB_WIDEN);
              do_compare_rtx_and_jump (tem, const0_rtx, EQ, true, mode,
                                       NULL_RTX, NULL, done_label,
                                       profile_probability::very_likely ());
and
          /* One argument is negative here, the other positive.  This
             overflows always, unless one of the arguments is 0.  But
             if e.g. s2 is 0, (U) s1 * 0 doesn't overflow, whatever s1
             is, thus we can keep do_main code oring in overflow as is.  */
          do_compare_rtx_and_jump (tem, const0_rtx, EQ, true, mode, NULL_RTX,
                                   NULL, do_main_label,
profile_probability::very_likely ());
          expand_arith_set_overflow (lhs, target);
          emit_label (do_main_label);
          goto do_main;