On Tue, Aug 20, 2024 at 9:38 AM LIU Hao <lh_mo...@126.com> wrote: > > 2024-08-20 16:13, Frederick Virchanza Gotham: > > > > I want to write a new calling convention into the GNU g++ compiler, > > specifically for the x86_64 instruction set. > > The x64 calling convention is much more complex than x86. Each of the first > four parameter is not > only assigned an integer register, but also an SSE register (XMM0 - XMM3) and > a shadow slot on the > stack.
My understanding of the SystemV calling convention used on x86_64 Linux, Apple, FreeBSD, is that the RDI register is used for the first integer parameter, and that the XMM0 register is used for the first floating point parameter. That is to say, if the function doesn't take any integer parameters, then RDI won't be used, and if the function doesn't take any floating point parameters, then XMM0 won't be used. If we change the first integer parameter register from RDI to RAX, I don't see why this has to have any effect on the floating point registers. Can you please point me to the source/header file in GCC where I would add a new calling convention? At first I just want to copy-paste the SystemV calling convention code and give it a new name like "gotham_call" to see if it works, and then later I'll make small changes to it such as changing RDI to RAX.
