https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106763
--- Comment #10 from Richard Earnshaw <rearnsha at gcc dot gnu.org> --- If you don't have CONFIG_SMP enabled, it looks like the kernel will do lazy context switching of the FP registers (it can save time if a process doesn't do any FP). So another work around might be to enable that, even if you have only have a single core.