If the loading time of the jQuery file is a real problem, you could look at gzipping it. This will make it more compact while it transfers from the server to the browser. It's something a lot of web developers over look and easily quadrupled the speed of the files I was looking at.
Also, the jQuery file only has to be transfered once to each person that used it. Then the browser just uses the cached file. Another trick to speed up loading time is the correct placement of the html tag calling jquery.js. It's recommended to put javascript last and let the rest of the page load. But if it is really important, called it first. It will then stated transferring as the other files are being called and transfered. There are often things you can do to make up for, what I guess is, the large file size of jQuery.
