The problem with leaving failed dump files behind is that they can interfere with a following run of the testsuite (particularly if a pass is added/subtracted). I would vote strongly that the dump files for failing tests be removed.
Likewise. I've run into this problem more than once and gotten confused by it.