On May 7, 2009, at 9:11 AM, Adrian Dimulescu wrote:


Thank you for these precisions. As I had to do something fast, I coded the thing as illustrated by the following pseudocode:


IndexReader index;

TermPositions iterator = this.index.termPositions(t); // for each doc where this term appears

while (iterator.next()) {
          int docNr = iterator.doc();
          int freq = iterator.freq();

int[] apparitionPositions = new int[freq]; // these are the positions in the crt doc of the crt term
          for (int i = 0; i < freq; i++) {
              apparitionPositions[i] = iterator.nextPosition();
          }
...
TermPositionVector tpv = (TermPositionVector) this.index.getTermFreqVector(docNr, "text");
...
// for all possible terms, see if it is close to one of the elements in apparitionPositions
         for (int i = 0; i < terms.length; i++) {
              int[] pos = tpv.getTermPositions(i);
... // for each element in pos, check close distance to the crt term
         }
}


My understanding is that this is a less object-oriented way of doing the same thing as your proposition but please correct me if I'm wrong.


This use case is in fact why I added the TermVectorMapper stuff into Lucene. In my case, I used SpanQuery to get me the position of the term within the doc, then I materialized the vector via a TermVectorMapper such that the TVM only stored the window around the span. It's not much different CPU wise, but it does save on memory, I think.

I finally managed to retrieve what I wanted with this code. The problem is that it is not really parallelizable. If several threads call getTermFreqVector at the same time, they have to wait after each other. My multithreaded scenario involved a unique IndexReader on which all threads ask for term vectors. I wonder if it is possible to avoid this problem (perhaps by having a pool of IndexReaders, is this a good practice, wouldn't there be memory problems?). I welcome any ideas on this subject.

Are you saying there are synchronizations happening? Even with multiple Readers, don't you end up with the disk access being a problem? Or, are you all in memory?



Thank you,
Adrian.

Grant Ingersoll wrote:
There isn't a very clean way to do this just yet, but it is doable. Index with positions (you might find offsets useful too) and then use the TermVectorMapper and TermVector API call on the IndexReader (not the termPositions). Then, you will need to implement a TermVectorMapper that takes in your position and then reads in the term vector and gets just those positions around the interested position. Once you are outside of your window, you can then short circuit out of the TermVM (I think).

HTH,
Grant

On May 3, 2009, at 2:39 PM, Adrian Dimulescu wrote:

Hello,

I am post-processing a positional index -- with a field like the following:

doc.add(new Field(Constants.FIELD_TEXT, txt, Store.NO, Index.ANALYZED, TermVector.WITH_POSITIONS));

At post-processing, I want to retrieve the neighbours of a given term within a given range. That is, if document x contains the sequence :

"Alabama experienced significant /recovery as the economy of the state/ transitioned from agriculture to diversified interests in heavy manufacturing"

for range = 3 and term = "economy", I want to retrieve "recovery as the *economy* of the state".

I see there is an API call :

IndexReader.termPositions(term)

which retrieves the actual positions of the given term. Is there a quick way to retrieve its neighbours too, instead of browsing all terms for all document and see if their position is close to the position of the central term ?



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org


--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to