[I am surprised you are able to link LyX. After all, swapping during.
linking was the reason I bought my laptop with 512 MB to _link_ LyX,
not to compile it. Are you sure you measure what you think you do?]
It is indeed the linking stage which is failing when I compile in
debugging symbols.
Part of the solution --- at least on linux --- may be this one,
suggested by Georg: http://permalink.gmane.org/gmane.editors.lyx.devel/79155
It helped me bring the linking time down from about 20 minutes to under
a minute...