Thanks for the hint, that was exactly it! I also asked the original author of the issue and he has confirmed the performance of pg 15+ now matches those of pg 13/14.
As mentioned "This issue is not necessarily a big deal to us, but can anyone please explain the why?" so it was partially out of curiosity, and partially because of the way the author of the issue is using Postgres which apparently is sensitive to these milliseconds. Not exactly sure what's their set up, but they did mention with the PostgresNIO SASL improvements which saves only some 10-20-30-40 milliseconds or so, their tests now run in 15s instead of 1m30s, which then it means they can now use pg versions higher than pg 13 and still have around the same performance. I think they run tests in parallel, and they possibly recreate the whole DB once for each test.