So looking for thoughts from the community on this one....
Let's take this RTL:
(insn 10 9 11 2 (set (reg:SI 144)
(unspec_volatile [
(const_int 0 [0])
] UNSPECV_FRFLAGS)) "j.c":11:3 discrim 1 362 {riscv_frflags}
(nil))
(insn 11 10 55 2 (set (reg:DI 140 [ _12 ])
(sign_extend:DI (reg:SI 144))) "j.c":11:3 discrim 1 122
{*extendsidi2_internal}
(expr_list:REG_DEAD (reg:SI 144)
(nil)))
Assume we have a pass that can look at how (reg:DI 140) is used and
ultimately determine that bits 32..63 are never read. So that pass
turns the SIGN_EXTEND into a lowpart SUBREG:
(insn 10 9 11 2 (set (reg:SI 144)
(unspec_volatile [
(const_int 0 [0])
] UNSPECV_FRFLAGS)) "j.c":11:3 discrim 1 362 {riscv_frflags}
(nil))
(insn 11 10 55 2 (set (reg:DI 140 [ _12 ])
(subreg:DI (reg:SI 144) 0)) "j.c":11:3 discrim 1 206 {*movdi_64bit}
(expr_list:REG_DEAD (reg:SI 144)
(nil)))
Combine comes along and tries to substitute the UNSPEC_VOLATILE into the
use of (reg:SI 144). We end up calling simplify_subreg with the inner
mode being VOIDmode (from the UNSPEC_VOLATILE which has no mode). That
triggers an assertion failure.
So one obvious fix is to require an UNSPEC_VOLATILE to have a mode when
it is one arm of a SET. I see ~40 cases where this can happen with a
bit of grepping of the .md files across all the ports. So it's
probably a tractable problem. I could easily argue this is the right
fix. The docs for set indicate that if the destination is a reg, subreg
or mem (which all have a mode) that the source must be valid for the
destination's mode.
Another approach would be to remove the assert from simplify_subreg and
just return NULL_RTX. It's a trivial change, though I must admit I
don't like removing checks, even if it's the easiest way to fix this
problem.
A third approach would be to adjust combine to avoid the problem,
perhaps in can_combine_p or somewhere else. This feels a bit like a
hack to me and I would expect we can get into the same scenario from
other optimization passes.
Whatever we do for UNSPEC_VOLATILE we probably should be doing for
UNSPEC as well.
Again, looking for community input on this one. If left to my own
devices I'd probably be looking to add modes to the relevant modeless
unspecs.
Jeff