Re: Unable to remove from Lucene index

[Manish]

I guess you are trying to remove or requesting to remove null referenced
object.
Manish B. Joshi
(Adserving Team)



On Tue, May 5, 2009 at 1:58 PM, Enrico Goosen
wrote:

>  Hi,
>
>
>
> I’m new to Lucene, and I’m getting an exception while trying to do a manual
> indexing operation on one of my entities.
>
> It works fine for the Product entity, but fails for the ProductInfo entity
> (see attached).
>
>
>
> Versions:
>
> hibernate-search 3.0.1.GA <http://3.0.1.ga/>
>
> Lucene 2.3
>
>
>
> 10:26:57,167 ERROR [IndexerBean] Exception during indexing operation.
>
> *org.hibernate.search.SearchException*: Unable to remove from Lucene
> index: class za.co.mymed.model.Product#null
>
>   at org.hibernate.search.backend.impl.lucene.LuceneWorker.remove(*
> LuceneWorker.java:109*)
>
>   at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(
> *LuceneWorker.java:80*)
>
>   at org.hibernate.search.backend.impl.lucene.LuceneWorker.performWork(
> *LuceneWorker.java:46*)
>
>   at
> org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.run(*
> LuceneBackendQueueProcessor.java:98*)
>
>   at
> org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(*
> BatchedQueueingProcessor.java:149*)
>
>   at org.hibernate.search.backend.impl.TransactionalWorker.performWork(
> *TransactionalWorker.java:45*)
>
>   at org.hibernate.search.impl.FullTextSessionImpl.index(*
> FullTextSessionImpl.java:140*)
>
>   at za.co.mymed.action.IndexerBean.indexAllClasses(*
> IndexerBean.java:93*)
>
>   at za.co.mymed.action.IndexerBean.index(*IndexerBean.java:63*)
>
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(*Native Method*)
>
>   at sun.reflect.NativeMethodAccessorImpl.invoke(*
> NativeMethodAccessorImpl.java:39*)
>
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(*
> DelegatingMethodAccessorImpl.java:25*)
>
>   at java.lang.reflect.Method.invoke(*Method.java:585*)
>
>   at org.jboss.seam.util.Reflections.invoke(*Reflections.java:22*)
>
>   at org.jboss.seam.intercept.RootInvocationContext.proceed(*
> RootInvocationContext.java:31*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:56*)
>
>   at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(*
> RollbackInterceptor.java:28*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:68*)
>
>   at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(*
> BijectionInterceptor.java:77*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:68*)
>
>   at org.jboss.seam.bpm.BusinessProcessInterceptor.aroundInvoke(*
> BusinessProcessInterceptor.java:51*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:68*)
>
>   at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(*
> MethodContextInterceptor.java:44*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:68*)
>
>   at org.jboss.seam.async.AsynchronousInterceptor.aroundInvoke(*
> AsynchronousInterceptor.java:52*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:68*)
>
>   at org.jboss.seam.intercept.RootInterceptor.invoke(*
> RootInterceptor.java:107*)
>
>   at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(
> *JavaBeanInterceptor.java:166*)
>
>   at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(*
> JavaBeanInterceptor.java:102*)
>
>   at
> za.co.mymed.action.IndexerBean_$$_javassist_5.index(IndexerBean_$$_javassist_5.java)
>
>   at za.co.mymed.action.IndexerBean.startProcess(*IndexerBean.java:48*
> )
>
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(*Native Method*)
>
>   at sun.reflect.NativeMethodAccessorImpl.invoke(*
> NativeMethodAccessorImpl.java:39*)
>
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(*
> DelegatingMethodAccessorImpl.java:25*)
>
>   at java.lang.reflect.Method.invoke(*Method.java:585*)
>
>   at org.jboss.seam.util.Reflections.invoke(*Reflections.java:22*)
>
>   at org.jboss.seam.intercept.RootInvocationContext.proceed(*
> RootInvocationContext.java:31*)
>
>   at org.jboss.seam.intercept.SeamInvocationContext.proceed(*
> SeamInvocationContext.java:56*)
>
>   at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(*
> RollbackInterceptor.java:28*)
>
>   at or

Re: HighFreqTerms for results set

[Manish Bafna]

Use Facet by that field. It will bring up top words.

On Mon, Jul 18, 2011 at 6:03 PM, Mihai Caraman wrote:

> So I looked around and found no viable solution for this problem:
> How to extract the most frequent terms in the search result set after
> submitting the query.
>
> HighFreqTerms
> and
> docFreq
> <
> http://lucene.apache.org/java/3_2_0/api/core/org/apache/lucene/index/FilterIndexReader.html#docFreq%28org.apache.lucene.index.Term%29
> >don't
> do the job for specific documents.
>
> - is it plausible to make a vector of resulted docID's and intersect it
> with
> each term's posting list in the index? bigger intersection meaning higher
> frequency.
>  *because search results could be really custom, this method can't be
> optimize to intersect only the highest frequency terms for the entire
> index.
>

Re: search match documents and pagination in lucene3.x

[Manish Bafna]

You can use SOLR.

On Wed, Sep 21, 2011 at 1:37 PM, Ian Lea  wrote:
>>   i  want to implements pagination like google search result page in my 
>> project.We use lucene3.3.Here are 2 issue:
>>   1.How can i get the number of matched documents
>
> TopDocs.totalHits
>
>>   2.What is the best practice for lucene search results pagination
>
> http://wiki.apache.org/lucene-java/LuceneFAQ#How_do_I_implement_paging.2C_i.e._showing_result_from_1-10.2C_11-20_etc.3F
>
>
> --
> Ian.
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>

-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Query on searchAfter API usage in IndexSearcher

[manish gupta]

 Hi Team,

I am new to Lucene and I am trying to use Lucene for text search in my
project to achieve better results in terms of query performance.

Initially I was facing lot of GC issues while using lucene as I was using
search API and passing all the documents count. As my data size is around 4
billion the number of documents created by Lucene were huge. Internally
search API uses TopScoreDocCollector which internally creates a
PriorityQueue of given documents count thus causing lot of GC.

*To avoid this problem I am trying to query using a pagination way wherein
I am query only 10 documents at a time and after that I am using
seacrhAfter API to query further passing the lastScoreDoc from previous
result. This has resolved the GC problem but the query time has increased
by a huge margin from 3 sec to 600 sec.*

*When I debugged I found that even though I use the searchAfter API, it is
not avoiding the IO and every time it is reading the data from disk again.
It is only skipping the results filled in previous search. Is my
understanding correct?. If yes please let me know if there is a better way
to query the results in incremental order so as to avoid GC and with
minimal impact on query performance.*

Regards
Manish Gupta

Re: Index Concurrent access

[Manish Joshi]

Concurrently adding updating deleting may not be good idea.It may corrupt
the index
-Manish B. Joshi

On Tue, Jun 16, 2009 at 2:33 PM, João Silva  wrote:

> Hi,
> how can i access the index in a concurrently way,
> so i can perform add/update/delete documents concurrently?
>
> Cheers,
> João
>
> --
> Cumprimentos,
> João Carlos Galaio da Silva
>



-- 
Manish Joshi
Adserving Team
Zedo India

Re: Insuffient resources during optimize merge

[Manish Joshi]

I assume it must be memory related issue.Not sure just a guess.
-Manish

On Fri, Jul 3, 2009 at 3:42 PM, Carl Austin  wrote:

> Hi,
>
> I have an issue when optimizing a large index (40GB). Basically the
> optimize falls over with an IOException stating there are not enough
> system resources, within a RandomAccessFile.readBytes call followed back
> to the CompoundFileWriter close method. I assume this is when creating a
> new CFS file from the separate other files.
> Unfortunately there is no hint at which system resource, and I have no
> access to the machine this fails on, otherwise I would run some testing
> to see what resource usage is going on.
> The process running this, has 1.3GB of memory allowance.
>
> If anyone has any insights they would be much appreciated. Stack trace
> included now:
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
> Insufficient system resources exist to complete the requested service
>at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(Co
> ncurrentMergeScheduler.java:309)
>at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:286)
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
>at java.io.RandomAccessFile.readBytes(Native Method)
>at java.io.RandomAccessFile.read(Unknown Source)
>at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
>at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
>at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
> ava:228)
>at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
> :184)
>at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
> ava:204)
>at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
>at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
> Scheduler.java:205)
>at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:260)
> Exception in thread "Thread-12957887" java.lang.RuntimeException:
> java.io.IOException: background merge hit exception: _cx:c3072736
> _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791 _dn:c4045 _do:c3643
> _dp:c3197 into _dq [optimize]
>at a.b.c.d.e.f.ImportThread$ImportRunner.run(Unknown Source)
>at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: background merge hit exception:
> _cx:c3072736 _d8:c55434 _d9:c9791 _da:c10097 _dl:c54262 _dm:c4791
> _dn:c4045 _do:c3643 _dp:c3197 into _dq [optimize]
>at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
>at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2203)
>at
> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2183)
>... 2 more
> Caused by: java.io.IOException: Insufficient system resources exist to
> complete the requested service
>at java.io.RandomAccessFile.readBytes(Native Method)
>at java.io.RandomAccessFile.read(Unknown Source)
>at
> org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirector
> y.java:596)
>at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.
> java:136)
>at
> org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.j
> ava:228)
>at
> org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java
> :184)
>at
> org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.j
> ava:204)
>at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4260)
>at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
>at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMerge
> Scheduler.java:205)
>at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Concurr
> entMergeScheduler.java:260)
>
> Thanks
>
> Carl Austin
>
>
>
>
>
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory.  The contents of this email may relate to
> dealings with other companies within the Detica Group Limited group of
> companies.
>
> Detica Group Limited is registered in England under No: 03328242.
>
> Registered offices: Warwick House, PO Box 87, Farnborough Aerospace Centre,
> Farnborough, Hampshire, GU14 6YU, England.
>
>


-- 
Manish Joshi
Adserving Team
Zedo India