Hey, On Mon, Feb 21, 2011 at 8:56 PM, Ajay Anandan <ajayda...@gmail.com> wrote: > Hi > I am trying to implement an Expectation Maximization algorithm for document > clustering. I am planning to use Lucene Term Vectors for finding similarity > between 2 documents. There are 2 kinds of EM algos using naive Bayes: the > multivariate model and the multinomial model. In simple terms, the > multinomial model uses the frequencies of different words in the documents > which the multivariate model just uses the info of whether a word is present > or not in the document(a boolean vector). > > I know that the term vectors in Lucene store the terms present in the > current document along with their frequencies. This is exactly what is > needed for the multinomial model. > > But the multivariate model requires the following: > A vector which stores the presence or absence of a particular term. Thus > all the terms in all the documents must be handled by this vector. > > As an example: > > doc1 : field CONTENT has the following terms : this is the world of > pleasure. > > doc2 : field CONTENT has the following terms : this amazing world is full of > sarcastic people. > > now the vector that I need should be > > < this is the world of pleasure amazing full sarcastic people > ( it > contains all the words in all the documents ) > > for doc1 the value of this vector is <1 1 1 1 1 1 0 0 0 0> > > for doc2 the vakue of this vector is <1 1 0 1 0 0 1 1 1 1> > > Is there any way to generate such a boolean vector in Lucene?
Lucene stores as you found correctly a sparse representation of TermVectors along with their frequencies, positions and / or offsets if needed. The terms which occur in other document but are not in a particular one are omitted (they might even not be present when the vector is stored). Yet, what lucene offers you is one big vector with all unique terms for each field, its term dictionary. you can simply build your dense vector as needed at runtime. You can simply traverse the TermEnum for that field and the TermVector at the same time. for each term in the TermEnum not present in you vector you insert a 0. Maybe there is a better way to do it or already existing infrastructure for this I don't know about just a possibility. simon > > -- > > Ajay Anandan. > MSc,computing Science, University of Alberta. > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
