Hi Nicola, I think this limit denotes the number of bytes you can write in a single DV value. So this actually means much less number of facets you index. Do you know how many categories are indexed for that one document?
Also, do you expect to index large number of facets for most documents, or is this one extreme example? Basically I think you can achieve that by enabling partitions. Partitions let you split the categories space into smaller sets, so that each DV value contains less values, and also the RAM consumption during search is lower since FacetArrays is allocated the size of the partition and not the taxonomy. But you also incur search performance loss because counting a certain dimension requires traversing multiple DV fields. To enable partitions you need to override FacetIndexingParams partition size. You can try to play with it. In am intetested though to understand the general scenario. Perhaps this can be solved some other way... Shai On Apr 26, 2013 5:44 PM, "Nicola Buso" <nb...@ebi.ac.uk> wrote: > Hi all, > > I'm encountering a problem to index a document with a large number of > values for one facet. > > Caused by: java.lang.IllegalArgumentException: DocValuesField "$facets" > is too large, must be <= 32766 > at > > org.apache.lucene.index.BinaryDocValuesWriter.addValue(BinaryDocValuesWriter.java:57) > at > > org.apache.lucene.index.DocValuesProcessor.addBinaryField(DocValuesProcessor.java:111) > at > > org.apache.lucene.index.DocValuesProcessor.addField(DocValuesProcessor.java:57) > at > > org.apache.lucene.index.TwoStoredFieldsConsumers.addField(TwoStoredFieldsConsumers.java:36) > at > > org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:242) > at > > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:256) > at > > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376) > at > org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473) > > > It's obviously hard to visualize such a big number of facets to the user > and is also hard to evaluate which of these values to skip to permit to > store this document into the index. > > Do you have any suggestion on how to overcome this number? is it > possible? > > > > Nicola > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
