I think something that may be even more important to consider than just the pure speed of your program, would be ease of design as well as the overall stability of your code.
My opinion would be that writing in Python would have many benefits over the speed gains of using C. For instance, you crawler will have to handle all types of input from all over the web. Who can say what types of malformed or poorly writen data it will come across. I think it would be easier to create a system to handle this type of data in Python than in C. I don't want to pigeon-hole your project, but if it is for any use other than a commercial product, I would say speed would be a concern lower on the list than accurracy or time to develop. As others have pointed out, if you hit many performance barriers chances are the problem is the algorithm and not Python itself. I wish you luck and hope you will experiment in Python first. If your crawler is still not up to par, at the very least you might come up with some ideas for how Python could be improved. -- http://mail.python.org/mailman/listinfo/python-list
