Hi Esmail. I've not looked at the site. however, i can give you some general pointers that might help you in solving your issue.
first off, try to accomplish your goal, using curl, or one of the other cmdline apps that fetch page data. this allows you to quickly nail down any issues that might occur, and to see if it's even possible to fetch the page. but even before you attempt curl, you might need to analyze the page/process of fetching the page. if you try a simple fetch using curl, and you don't get your page, then it might be a matter of using cookies and/or setting the user-agent, or any number of things. page/traffic analysis will allow you to see what traffic is happening between the server and your browser. if you're using firefox, you can get a plugin (livehttpheaders) which will allow you to see what's going on. this will tell you if coookies are being used/set, if you're using a straight get/post process, whether you're doing redirects, etc... once you have a better understanding of the process required, you can then setup curl to fetch the page. if you get the page with cur, you can then proceed to using python/urllib2/mech/cookies/etc.. to then fetch the page.. and once you get the page, you can then play with XPath, to manipulate/drive/parse the DOM to get the information you want... Have fun!!! -----Original Message----- From: python-list-bounces+bedouglas=earthlink....@python.org [mailto:python-list-bounces+bedouglas=earthlink....@python.org]on Behalf Of Esmail Sent: Thursday, April 09, 2009 2:47 PM To: python-list@python.org Subject: is this possible (getting info off web page) Hi, I am trying to keep track of two flight bookings on the kayak.com web site, and would like to automate my query using Python. If I enter the url below myself into the browser, a form gets filled out and the site searches for flights. I would love to be able to have a simple python script which I can supply with a url and have it simply display, in text if possible, the best 3 prices. I thought great opportunity to learn more about Python, and then presumably re as I parse the text.... However, when I use the code below (located via google), I end up with an alternate page, basically telling me that kayak doesn't like bots. So, it seems like Kayak has it set up to prevent automated access? Is there anyway to do what I would like to do? Thanks, Esmail -- import urllib2 url = 'http://www.kayak.com/s/search/air?l1=cmh&d1=4/23/2009&l2=yyz&d2=4/26/2009' res = urllib2.urlopen(url) page = res.read() -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
