Note on heap_prepare_freeze_tuple()'s fifth parameter, it's not valid to pass OldestXmin; you need a multixact limit there, not an Xid limit. I think the return value of GetOldestMultiXactId is a good choice. AFAICS this means that you'll need to add a new output argument to vacuum_set_xid_limits (You *could* call GetOldestMultiXactId again, but it seems a waste).
Maybe it's time for vacuum_set_xid_limits to have a caller's-stack- allocated struct for input and output values, rather than so many arguments and output pointers to fill.