Hi Catonano, On Fri 30 Dec 2011 23:58, Catonano <caton...@gmail.com> writes:
> I´m a beginner, I never wrote a single line of LISP or Scheme in my life > and I´m here for asking for directions and suggestions. Welcome! :-) > I´m mumbling about a pet project. I would like to scrape the web site of > a comunitarian radio station and grab the flash streamed content they > publish. The license the material is published under is Creative Common > so what I´m planning is not illegal. Sounds like fun. > my boldness is such that I´d ask you to write for me an example > skeleton code. Hey, it's fair, I think; that is a new part of Guile, and there is not a lot of example code. Generally, we figure out how to solve problems at the REPL, so fire up your Guile: $ guile ... scheme@(guile-user)> (Here I'm assuming you have guile 2.0.3.) Use the web modules. Let's assume we're grabbing http://www.gnu.org/, for simplicity: > (use-modules (web client) (web uri)) > (http-get (string->uri "http://www.gnu.org/software/guile/";)) [here the text of the web page gets printed out] Actually there are two return values: the response object, corresponding to the headers, and the body. If you scroll your terminal up, you'll see that they get labels like $1 and $2. Now you need to parse the HTML. The best way to do this is with the pragmatic HTML parser, htmlprag. It's part of guile-lib. So download and install guile-lib (it's at http://www.non-gnu.org/guile-lib/), and then, assuming the html is in $2: > (use-modules (htmlprag)) > (define the-web-page (html->sxml $2)) That parses the web page to s-expressions. You can print the result nicely: > ,pretty-print the-web-page Now you need to get something out of the web page. The hackiest way to do it is just to match against the entire page. Maybe someone else can come up with an example, but I'm short on time, so I'll proceed to The Right Thing -- the problem is that whitespace is significant, and maybe all you want is the contents of "the <title> in the <head> in the <html>." So in XML you'd use XPATH. In SXML you'd use SXPATH. It's hard to use right now; we really need to steal http://www.neilvandyke.org/webscraperhelper/ from Neil van Dyke. But you can see from his docs that the thing would be > (use-modules (sxml xpath)) > (define matcher (sxpath '(// html head title))) > (matcher the-web-page) $3 = ((title "GNU Guile (About Guile)")) Et voila. I don't do much web scraping these days, but I know some others do. So if those others would chime in with better ways to do things, that is very welcome. Happy hacking, Andy -- http://wingolog.org/
