On Monday 02 May 2005 06:01, Yonik Seeley wrote: > I can't say whats actually ready, but I am very interested in sparse > filter representations. I'm working on a project that needs to > dynamic categorization of search results, and this requires caching > thousands of filters.
The need for many filters is what started me off on the boolean scorer changes in the first place. Caching many filters implies filters must be sparse (ie. take little memory). Sparse filters implies that they must be used with iterator access (no more random access as on BitSet). Iterator access implies all scoring should be done in order of document numbers, which is not provided by the 1.4.3 BooleanScorer. > http://issues.apache.org/bugzilla/show_bug.cgi?id=32965 > http://issues.apache.org/bugzilla/show_bug.cgi?id=32921 > It does currently require Java 1.4, so I guess it needs to wait for Lucene 2.0? It is the need to apply skipTo on the BitSet that introduces the need for java 1.4 . In case one would only use SortedVIntList, (the sparse filter), the java 1.4 dependency would not be there. Btw. this also shows why both filters only work in RAM: they have no underlying support for skipTo in their data structure. Java has bolted this on in 1.4 over the BitSet (probably by a linear search over the underlying array), and for the SkipFilter skipTo is implemented by using the iterator, ie. a linear search. > And I'm not sure if I'm reading it correctly, but it looks like > BooleanScorer1 could be a replacement for both BooleanScorer and > BooleanScorer2. > http://issues.apache.org/bugzilla/show_bug.cgi?id=33019 It's not stated there, but BooleanScorer2 currently depends on DisjunctionScorer in svn. Regards, Paul Elschot --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
