Otherwise just about everything in contrib has dependency on non-free
software.That makes it simple--put the contrib packages on the server that has the non-free packages. A few months ago, I think someone mentioned that some packages were in contrib because their quality or utility was marginal, even though they had no dependence on non-free software. If that is true, those might perhaps want to be moved into main but somehow labeled as second-rate.
