Yes, the search engine API should be a better way to go than parsing HTML output. I will see whether I can use Google API (sandbox) for testing.
Thanks -----Original Message----- From: dan [mailto:d...@entropy.homelinux.org] Sent: 16 June 2009 00:01 To: Mimi Cafe Cc: beginners@perl.org Subject: Re: Price comparison-agent On Mon, 15 Jun 2009 14:10:14 +0100, Mimi Cafe <mimic...@googlemail.com> wrote: > I am trying to write a small online price comparison-agent, but I have no > experience doing something similar before. > My intention is to query search engines for a product and try to my own > algorithm to parse the returned values for the prices and compare them. > > Can any experienced developer tell me whether or not this is the right way > to proceed? if not, can you tell me how to go about this (also suggest Perl > modules which can be used)? I am unsure whether the libwww could be used > for this. There are 2 main ways you can go about this. The first is to get access to the search engines' API. With this, you'll be able to send SOAP requests and receive nicely structured responses. Depending on the search engine, they may charge you for access to their API. The other way, as you've guessed, is to simple send http requests for various products, and attempt to parse the HTML output. This is messy and time-consuming, and if the search engine output changes ( it happens ), you have to revise your code. > I have spent some time searching the web for an example of comparison agent > codes, but found none. It looks like such algorithms are kept secret by > their owners. There is no secret to the comparison algorithm. It's a simple sort, based on price. The secret bit is parsing the HTML output of each search engine. > Please give code examples if possible. Ha! Sorry. That's highly unlikely. It's quite a specialised task, and people who have done it will likely be guarding this code. I wrote some code a while back to parse a webpage. I used LWP::Simple to fetch the page contents, and HTML::TreeBuilder to parse it. Dan -- To unsubscribe, e-mail: beginners-unsubscr...@perl.org For additional commands, e-mail: beginners-h...@perl.org http://learn.perl.org/
