Hi Tomas, I was curious how Nix handles this, so I asked about it on the #nixos IRC. It looks like they run such tests in a VM. These are called NixOS tests, and they seem to be the equivalent of our system tests - they have to be run manually, separately from regular package tests.
So unless someone comes up with something innovative, it looks like that's the best answer to this question - figure out a minimal `operating-system` declaration that works for whatever you're testing, and use it to add a test in `(gnu tests ...)`.