Re: Can lucene index tokenized files?

2014-09-15 Thread Sachin Kulkarni
Hi Erick, Thank you. Yes the data is in text form with the space delimited tokens. The queries are categories that the documents belong to. They are regular text files and will need the transformation at my end. Regards, Sachin On Mon, Sep 15, 2014 at 12:31 PM, Erick Erickson wrote: > How are

Re: Can lucene index tokenized files?

2014-09-15 Thread Erick Erickson
How are they delimited? If they're just a text stream, it seems all you need is a whitespace tokenizer. Won' How are you going to search them though? Is your query submission process going to _also_ do the transformations or will you have to construct a query-time analysis chain that mimics the pr

How to search keywords using apache lucene on apache tike indexed data.

2014-09-15 Thread MOHAMED MUSTHAFA
Hi Team, I indexed some html,doc and pdf files using apache tika. Now I want to search some keywords on this indexed data by apache lucene. Anyone to help me on this. -- = |* Thanks & Regards * | | * MOHAMED MUSTHAFA * | =

Exception while using a custom analyzer in a parallel indexing!

2014-09-15 Thread andi rexha
Hi, I have an index writer that is used to from a pool of threads to index. The index writer is using a "PerFieldAnalyzerWrapper": this.analyzer = new PerFieldAnalyzerWrapper(DEFAULT_ANALYZER, fields); If I add the documents single threaded I dont get any exception. In the case that I add th

RE: Insufficient system resources exist to complete the requested service

2014-09-15 Thread Vincent Sevel
Hi, Most of the physical memory was being used. But the virtual memory's peak was far away from the limit. But thanks for the idea. I will keep an eye on it. Regards, vince Vincent Sevel Banque Lombard Odier & Cie SA 11, rue de la Corraterie - 1204 Genève - Suisse T +41 22 709 3376 - F +41 22 70

Re: Insufficient system resources exist to complete the requested service

2014-09-15 Thread Robert Muir
SimpleFSDirectory doesn't use memory mapping. I'd check you dont have leaks of indexreaders or similar. This error happens in windows when it runs out of open file handles. On Mon, Sep 15, 2014 at 3:52 AM, Michael McCandless wrote: > Maybe your OS is running out of total virtual memory? Try loo

Re: 回复: Speed up searching in multiple-thread?

2014-09-15 Thread Toke Eskildsen
On Mon, 2014-09-15 at 11:41 +0200, Harry Yu wrote: > 17ms / searches is the whole process of search service, include > accessing complete data form db, calling REST service etc. Try looking at QTime in solr.log and compare it with your measured response times, to see if it is Solr or your other s

回复: Speed up searching in multiple-thread?

2014-09-15 Thread Harry Yu
Hi Toke,‍ Thanks for your reply. I expect the outcome of my application would be 150~200 searches / per. Now I'm looking for methods to optimize my application, and want to improve the performance of the throughout and response time. 17ms / searches is the whole process of search service, inc

Re: Speed up searching in multiple-thread?

2014-09-15 Thread Toke Eskildsen
On Mon, 2014-09-15 at 09:10 +0200, Harry Yu wrote: > I'm developing poi search application using lucene 4.8 . Recently, I > met a trouble that the performance of IndexSearcher.search is bad in > multiple-thread environment. According the test results, I found that > if thread number is 1, the resp

回复: Speed up searching in multiple-thread?

2014-09-15 Thread Harry Yu
Hi Michael, Thanks for your reply. Yes, core i5 is 2 real (4 with hyperthreading) cores. And is this the core reason? Are my test methods wrong? In my test environment, I used JMeter to simulate 30 threads, and access my search ‍ service. I recorded per-search time and analyzed them. Best

Re: Insufficient system resources exist to complete the requested service

2014-09-15 Thread Michael McCandless
Maybe your OS is running out of total virtual memory? Try looking in task manager? Mike McCandless http://blog.mikemccandless.com On Mon, Sep 15, 2014 at 3:19 AM, Vincent Sevel wrote: > Hi, > > I have had this issue : > > > java.io.FileNotFoundException: > F:\logserver\index\INFRA-LOGSERVER2

Re: Speed up searching in multiple-thread?

2014-09-15 Thread Michael McCandless
If you run 30 search threads on a core i5 it's expected there will be big slowdowns in the per-query latency since core i5 only has 2 real (4 with hyperthreading) cores? Mike McCandless http://blog.mikemccandless.com On Mon, Sep 15, 2014 at 3:10 AM, Harry Yu <502437...@qq.com> wrote: > Dear mem

Speed up searching in multiple-thread?

2014-09-15 Thread Harry Yu
Dear members at Lucene project, I'm developing poi search application using lucene 4.8 . Recently, I met a trouble that the performance of IndexSearcher.search is bad in multiple-thread environment. According the test results, I found that if thread number is 1, the response time of searching

Insufficient system resources exist to complete the requested service

2014-09-15 Thread Vincent Sevel
Hi, I have had this issue : java.io.FileNotFoundException: F:\logserver\index\INFRA-LOGSERVER2_UNIV_UNIV_DBIZ\cpp\cpp_D_2014-09-13\_xr_Lucene45_0.dvd (Insufficient system resources exist to complete the requested service) at java.io.RandomAccessFile.open(Native Method)