The problem I hit usually: There are some packages that rely on X for some tests, that way, when I for example emerge tk, it's nearly impossible to keep working because I get tons of windows opened and closed.
Wouldn't be better to make that tests *always* use virtual X? Thanks for your thoughts