Bug#599670: [imdb-tools] New IMDb layout completely breaks parsing

[Thomas Breining]

Package: imdb-tools
Version: 0.9-1
Followup-For: Bug #599670

Hi,

I have tried applying the patch and it did allow to get informations.

However, as you can see in the attachments, the informations are not perfectly 
extracted and they include a lot of html nodes.

I guess it still is due to layout changes or maybe it never really was perfect.

Looking at the scripts I saw that everything was extracted from html code using 
sed.

It seems to me that the regular expressions necessary to achieve perfect 
extraction would be extremely complex, at least I don't think I could write 
them.

This is why I tried to take another approach and tested the use of XSLT to 
extract the relevant information from the html page.
(stylesheet and associated result are attached to this message).

Since we can count on IMDb to change its layout relatively often, stlesheets 
seem easier to maintain to me.

What do you think about using XSLT instead of sed ?

If you agree that it is a good idea, I am ready to help integrating it with the 
existing set of script.

Thanks anyway for your efforts so far.

-- System Information:
Debian Release: jessie/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: i386 (i686)

Kernel: Linux 3.9-1-686-pae (SMP w/2 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages imdb-tools depends on:
ii  curl      7.31.0-2
ii  realpath  1.18
ii  tidy      20091223cvs-1.2

imdb-tools recommends no packages.

imdb-tools suggests no packages.

-- no debconf information

stylesheet.imdb.xslt
Description: XML document

stylesheet.topgun.xml
Description: XML document

imdb-get -l tt0092099 -a
Getting details from IMDB
id: tt0092099
title: Top Gun
year: 1986
director: Director: Tony Scott Writers: Jim Cash, Jack Epps Jr., 1 more 
credit » Stars: Tom Cruise, Kelly McGillis, Val Kilmer | See full 
cast and crew Director: Tony Scott Director: Yimou Zhang Director: Kevin 
Reynolds Director: Robert Harmon Director: Gavin O'Connor Director: Michael Bay 
Director: Robert Zemeckis Director: Taylor Hackford Director: Jason Reitman 
Director: Brian Helgeland Director: Nick Cassavetes Director: David Frankel
writer: Writers: Jim Cash, Jack Epps Jr., 1 more credit » Stars: Tom 
Cruise, Kelly McGillis, Val Kilmer | See full cast and crew
tagline: 
plot: 
genre: Action | Drama | See all certifications »
cast: <img height="44" width="32" alt="Tom Cruise" title="Tom 
Cruise"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMTk1MjM3NTU5M15BMl5BanBnXkFtZTcwMTMyMjAyMg@@._V1_SY44_CR1,0,32,44_.jpg";
 /> as </td> Maverick, <img height="44" width="32" alt="Kelly McGillis" 
title="Kelly 
McGillis"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMTQ4NDIxNjU2MF5BMl5BanBnXkFtZTcwMjE4Njg0Mw@@._V1_SX32_CR0,0,32,44_.jpg";
 /> as </td> Charlie, <img height="44" width="32" alt="Val Kilmer" title="Val 
Kilmer"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMTk3ODIzMDA5Ml5BMl5BanBnXkFtZTcwNDY0NTU4Ng@@._V1_SX32_CR0,0,32,44_.jpg";
 
 /> as </td> Iceman, <img height="44" width="32" alt="Anthony Edwards" 
title="Anthony 
Edwards"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMTI1NTk5ODY3Nl5BMl5BanBnXkFtZTcwNTE5MjcyMg@@._V1_SX32_CR0,0,32,44_.jpg";
 /> as </td> Goose, <img height="44" width="32" alt="Tom Skerritt" title="Tom 
Skerritt"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BNzY3Mjk5MTMyMl5BMl5BanBnXkFtZTYwMjUxMTc1._V1_SX32_CR0,0,32,44_.jpg";
 /> as </td> Viper, <img height="44" width="32" alt="Michael Ironside" 
title="Michael 
Ironside"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMzI0MTI0MzE3N15BMl5BanBnXkFtZTcwNDQ1MzIzMQ@@._V1
 _SX32_CR0,0,32,44_.jpg" /> as </td> Jester, <img height="44" width="32" 
alt="John Stockwell" title="John 
Stockwell"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BNTAxNDUxNDk1N15BMl5BanBnXkFtZTYwNDAzOTg0._V1_SX32_CR0,0,32,44_.jpg";
 /> as </td> Cougar, <img height="44" width="32" alt="Barry Tubb" title="Barry 
Tubb"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMjEwMDY0OTk5OF5BMl5BanBnXkFtZTcwODQ2NzQ1OQ@@._V1_SY44_CR23,0,32,44_.jpg";
 /> as </td> Wolfman, <img height="44" width="32" alt="Rick Rossovich" 
title="Rick 
Rossovich"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BODQ2NDMwMzg1N15BMl5BanBnXkFtZT
 cwODY4Nzg5Nw@@._V1_SX32_CR0,0,32,44_.jpg" /> as </td> Slider, <img height="44" 
width="32" alt="Tim Robbins" title="Tim 
Robbins"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMTI1OTYxNzAxOF5BMl5BanBnXkFtZTYwNTE5ODI4._V1_SY44_CR1,0,32,44_.jpg";
 /> as </td> Merlin, <img height="44" width="32" alt="Clarence Gilyard Jr." 
title="Clarence Gilyard 
Jr."src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMjQwNjMzNzMwM15BMl5BanBnXkFtZTcwNTIzNjQwOA@@._V1_SX32_CR0,0,32,44_.jpg";
 /> as </td> Sundown, <img height="44" width="32" alt="Whip Hubley" title="Whip 
Hubley"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " loadlate="http://ia.media-imdb.com/images/M/MV5BMzE4M
 Tk4ODM1OF5BMl5BanBnXkFtZTcwODM1ODkyMQ@@._V1_SX32_CR0,0,32,44_.jpg" /> as </td> 
Hollywood, <img height="44" width="32" alt="James Tolkan" title="James 
Tolkan"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BMTc0NDcyNzQ4Nl5BMl5BanBnXkFtZTcwMjExNTgyMQ@@._V1_SY44_CR15,0,32,44_.jpg";
 /> as </td> Stinger, <img height="44" width="32" alt="Meg Ryan" title="Meg 
Ryan"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " 
loadlate="http://ia.media-imdb.com/images/M/MV5BNTEyMzEzMDc1NF5BMl5BanBnXkFtZTcwNTQzMDM0Mw@@._V1_SY44_CR0,0,32,44_.jpg";
 /> as </td> Carole, <img height="44" width="32" alt="Adrian Pasdar" 
title="Adrian 
Pasdar"src="http://ia.media-imdb.com/images/G/01/imdb/images/nopicture/32x44/name-2138558783._V397576332_.png"class="loadlate
 hidden " loadlate="http://ia.media-imdb.com/ima
 
ges/M/MV5BMTMzOTg0NzI3NF5BMl5BanBnXkFtZTcwOTMyMDM5MQ@@._V1_SY44_CR1,0,32,44_.jpg"
 /> as </td> Chipper,
coverart: 
rating: