Oh, and thanks for the suggestion to use Luke. Doug
-----Original Message----- From: Erik Hatcher [mailto:[EMAIL PROTECTED] Sent: Sunday, May 29, 2005 7:39 PM To: java-user@lucene.apache.org Subject: Re: Indexing multiple keywords in one field? On May 29, 2005, at 8:29 AM, Doug Hughes wrote: > Hi, > > I'm working on a pretty typical web page search system based on > lucene. > Pretty much everything works great. However, I'm having one problem. > I want to have a feature in this system where I can find all pages > which link to another page. So, for instance, I might search for all > the pages linked to http://www.foobar.com/index.html. The search term > does not need to be fuzzy in any way. http://www.foobar.com would not > match http://www.foobar.com/. The thing is that any for any given > document I could have any number of associated links. > > I think that each page's links could be treated as an array of > keywords. > However, I don't know the best practice for indexing this data or how > to find matches for specific links. One possibility is to extract the links (XPath could do this with a "//a" pattern) during a parsing phase, not during an analyzer. Build a list of links and index each one as a separate Field.Keyword() field for a single document. > I tried creating a LinebreakAnalyzer which (I think) tokenized phrases > based on CRs and LFs. I converted the array of links to a list of > links delimited by LFs. When indexing I used the > PerFieldAnalyzerWrapper and set the links field to use the > LinebreakAnalyzer. My understanding is that the lucene index should > now have each of the links indexed as separate terms or keywords > (sorry if my vocabulary is wrong!) Links are broken per line? For general HTML parsing you certainly cannot assume that, but maybe in your documents you can? I'd be surprised at that though. > Now, all that seems to work fine. However, when I search I build I > query using this code: > > QueryParser.parse(link, "links", new LinebreakAnalyzer()) > > The link is the link I'm searching for, "links" is the field I'm > searching. > I'm using the same analyzer I used to index the links. The problem is > I don't get any matches at all when I execute the search. > > Does anyone know of any better techniques for this? Or does anyone > see anything I'm doing wrong The first thing to do is ensure what you think was indexed really was. I highly recommend you get Luke - http://www.getopt.org/luke/ - and explore the index you've built and see what terms were indexed in your links field. Then experiment using the TermQuery API for the _exact_ terms indexed. Only then move up to QueryParser if you need that kind of thing, using Query.toString() to dump the generated query instance and see what it is made of. QueryParser introduces a level of complexity that can be confusing because there is query expression operators, parsing, and analysis all mixed together - and some characters in a URL within a QueryParser expression will need to escaped to be interpreted properly (like the ":" in "http://";). Erik --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
