Steven D'Aprano wrote: > On Sun, 31 Dec 2006 02:03:34 +0100, Thomas Ploch wrote: > >> Hello fellow pythonists, >> >> I have a question concerning posting code on this list. >> >> I want to post source code of a module, which is a homework for >> university (yes yes, I know, please read on...). > > So long as you understand your university's policy on collaborations.
Well, collaborations are wanted by my prof, but I think he actually meant it in a way of getting students bonding with each other and establishing social contacts. He just said that he will reject copy & paste stuff and works that actually have nothing to do with the topic (when we were laughing, he said we couldn't imagine what sometimes is handed in). >> It is a web crawler (which I will *never* let out into the wide world) > > If you post it on Usenet, you will have let it out into the wide world. > People will see it. Some of those people will download it. Some of them > will run it. And some of them will run it, uncontrolled, on the WWW. > > Out of curiosity, if your web crawler isn't going to be used on the web, > what were you intending to use it on? It's a final homework, as I mentioned above, and it shouldn't be used anywhere but our university server to test it (unless timing of requests (i.e. only two fetches per second), handling of 'robots.txt' is implemented). But you are right with the Usenet thing, havn't thought about this actually, so I won't post the whole portion of the code. >> which uses regular expressions (and yes, I know, thats not good, too). > > Regexes are just a tool. Sometimes they are the right tool for the job. > Sometimes they aren't. Alright, my prof said '... to process documents written in structural markup languages using regular expressions is a no-no.' (Because of nested Elements? Can't remember) So I think he wants us to use regexes to learn them. He is pointing to HTMLParser though. >> I have finished it (as far as I can), but since I need a good mark to >> actually finish the course, I am wondering if I can post the code, and I >> am wondering if anyone of you can review it and give me possible hints >> on how to improve things. > It probably isn't a good idea to post a great big chunk of code and expect > people to read it all. If you have more specific questions than "how can > I make this better?", that would be good. Unless the code is fairly > short, it might be better to just post a few extracted functions and see > what people say about them, and then you can extend that to the rest of > your code. You are probably right. For me it boils down to these problems: - Implementing a stack for large queues of documents which is faster than list.pop(index) (Is there a lib for this?) - Getting Handlers for different MIME/ContentTypes and specify callbacks only for specific Content-Types / MIME-Types (a lot of work and complex checks) - Handle different encodings right. I will follow your suggestions and post my code concerning specifically these problems, and not the whole chunk. Thanks, Thomas -- http://mail.python.org/mailman/listinfo/python-list
