After checking a little more, it looks like it does happen every time the USB
device is reconnected.
Further, there is no need for two X servers to be running -- one is enough,
provided that it is made not active. That is, to reproduce:
0. Make sure your keyboard/mouse are connected via USB
1. Open a normal X session, log in
2. Go to a console tty (depending on your setup, you can probably do this by
pressing ctrl+alt+F2)
3. Diconnect and reconnect your keyboard/mouse
Expected Results: Nothing special happens
Actual Results: X session segfaults
HTH,
Shai.
