I would say look at the Xserver logs. I would bet you have something in there that X is searching for and not finding. If you have both a 'Configured Mouse' and 'Generic Mouse' section, disable the one that is NOT set as root pointer. This caused a few seconds of hangup on my box. Look for any warnings, like unused font paths and such, and correct the errors. If some fonts are not used by X anyway, comment them out of XF86Config-4.
Nathan
