Share Index on NFS

Hi Everyone, We are planning to distribute searches on the index and have a single indexing node. We want to mount the index on NFS so that it can be shared by the indexer and searcher nodes. To optimize several of our search workflows, we are caching the IndexSearcher and refreshing it every h

Query Analyzer Issue

Hi Everyone, I am facing some strange behaviour with Analyzers. I am using SimpleAnalyzer for some fields in my Compass entity, but I also wrote a custom Analyzer that is slightly different from the SimpleAnalyzer as I wanted to allow even letters and digits in company name column. So custom analy

Searching with too many clauses + Out of Memory

Hi Everyone, I am using Compass 1.1 M2 which supports Lucene 2.2 to store & search huge amount of company, executive and employment data. There are some usecases where I need to search for executives/employments on the result set of company search. But when I try to create a compass query to sear

Re: Lucene 2.2 + Not Merging Segments

McCandless <[EMAIL PROTECTED]> wrote: This is spooky. Could you provide more details about how you are using IndexWriter? Are you changing "max buffered docs" from the default? Also, can you call writer.setInfoStream(System.out), capture that log, and post it? Mike "Har

Lucene 2.2 + Not Merging Segments

Hi All, I just upgraded my application to use lucene 2.2. Prior to this, I was using lucene 1.9.1 and the addDocument method in the IndexWriter used to keep merging smaller segments while adding new documents depending on the mergeFactor. I was using the default mergeFactor settings. But after th

Re: unused tmp fdt files in index

etable, it is should be safe to delete these files. Please backup your data before testing. Andy -Original Message- From: Harini Raghavan [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 10, 2007 2:29 PM To: java-user@lucene.apache.org Subject: unused tmp fdt files in index Hi All, I have a

unused tmp fdt files in index

Hi All, I have a large lucene index of size 60G. We have had Out Of Memory issues a few times in the past due to which the indexing had got interrupted. This has resulted in a lot of .fnm, .fdt,.tmp files which don't get removed even through optimizing the index. We have data for last 90 days in

QueryParser stripping special char

Hi All, I am trying to create a lucene query to search for companies based on areacode. The phone no. is stored in the lucene index in the form of '415-567-2323'. I need to create a query like +areaCode:"415-". But the QueryParser is stripping off the hyphen(-). Here is the code to create the qu

Re: ERROR opening the Index - contact sysadmin!

Hi All, I am trying to create a lucene query to search for companies based on areacode. The phone no. is stored in the lucene index in the form of '415-567-2323'. I need to create a query like +areaCode:"415-". But the QueryParser is stripping off the hyphen(-). Here is the code to create the qu

Scalability Issues with Indexing

Hi Everyone, We have been using Lucene integrated with our application for over a year now. The indexing and searching has been pretty fast until recently. But now we are having some scalability issues. We have a job that indexes around 2 documents in to index every day. There are 2 processes

Re: Delete document with keyword field

what you really have in your index is, as you suspect, 839930494, it all depends upon the analyzer you used. Erick On 3/11/07, Harini Raghavan <[EMAIL PROTECTED]> wrote: > > Hi All, > > I have a lucene index with many fields, one of which is a Keyword field > IS. > The v

Delete document with keyword field

Hi All, I have a lucene index with many fields, one of which is a Keyword field IS. The values stored in this field are the document ids like _839930494, _839930492. But I am unable to delete the documents using this id. Is this something to do with the underscore? Can someone suggest how I shou

RE: Counting term frequency without using Explanation

Hi Erick, I have a similar requirement to know the frequency of occurrence of a keyword in a given content to find out the relevancy of the article to a set of keywords. If the keyword is mentioned more than once in the article, then I want to treat it as more relevant. Can you please point me t

Modelling Relational Lucene Index

s a text-search engine, not a relational database. This kind of relation may be perfectly valid to implement in Lucene, but you want to be careful if you find yourself trying to do any more RDBMS-like things. Best Erick On 12/26/06, Harini Raghavan <[EMAIL PROTECTED]> wrote: Hi, I have

Re: Merge Index Filling up Disk Space

the index for searching. - Mark Harini Raghavan wrote: Hi Mike, Thank you for the response. I don't have readers open on the index, but while the optimize/merge was running I was searching on the index. Would that make any difference? Also after the optimizing the index I had some .tmp

Re: Merge Index Filling up Disk Space

Yes I think I got hit IOException. I assumed that the.tmp files are not required and deleted them manually from the indes directory as they were more than 10G. Is that ok? Michael McCandless wrote: Harini Raghavan wrote: Thank you for the response. I don't have readers open on the

Re: Merge Index Filling up Disk Space

uld that also be related to having searchers open while running optimize? -Harini Michael McCandless wrote: Harini Raghavan wrote: I am using lucene 1.9.1 for search functionality in my j2ee application using JBoss as app server. The lucene index directory size is almost 20G right now. Th

Merge Index Filling up Disk Space

size to be allotted in such scenario. Any suggestions would be appreciated. Thanks, Harini -- Harini Raghavan Software Engineer Office : +91-40-23556255 [EMAIL PROTECTED] we think, you sell www.InsideView.com InsideView - To

Re: Highlighting "really" found terms

ce). Is there a way to highlight only the terms really found ? Thanks a lot ! Pierre - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL P

Re: indexing in lucene 1.9.1

Richmond wrote: Hello Harini, When you are finished indexing the documents are you running the optimize() method on the IndexWriter before closing it? This should reduce the number of segments and make searching faster. Just a thought. --Mike On 5/22/06, Harini Raghavan <[EMAIL PROTEC

indexing in lucene 1.9.1

Hi All, We have recently upgraded from lucene 1.4.3 to lucene 1.9.1 version. After the upgrade, we are facing some issues: 1. Indexing seems to be behaving differently. There were more than 300 segment files(.cfs) in the index and the IndexSearcher is taking forever to refresh the index. Have t

adding new fields to index

Hi All, I have an existing lucene index whose size is around 10G. I need to add/delete some fields for all the documents in the index. Is there any way to do this without reindexing all the documents again? Thanks, Harini - To

Re: highlighting phrases

, Harini Erik Hatcher wrote: On Jan 9, 2006, at 1:16 PM, Harini Raghavan wrote: I am using the highlighter package to highlight my search results. The query I am passing to the Highlighter is: +(Content:"Apple Computer" Content:"Apple Comp") +(Title:"Apple Computer&quo

Lock obtain timed out + IndexSearcher

Hi All, All of a sudden I have started getting LockTimeOut exception while searching the index. There is no write.lock file in the index directory, so why should this issue come while searching? I tried to delete the index directory and restarted the server, but still no luck. What could be w

highlighting phrases

Hi All, I am using the highlighter package to highlight my search results. The query I am passing to the Highlighter is: +(Content:"Apple Computer" Content:"Apple Comp") +(Title:"Apple Computer" Title:"Apple Comp") But the Highlighter is highlighting even occurances of terms 'Computer'/'Comp'.

Re: Deleting a Document

Hi Koji, Thanks for the suggestion. It worked when I closed the reader before refreshing the IndexSearcher instance. Harini Koji Sekiguchi wrote: Hi Harini, I meant you close the reader first, then get a new searcher. regards, Koji -Original Message- From: Harini Raghavan

Re: Deleting a Document

Hi Koji, I am closing the reader after deleting. You can see the close being called in the finally block in the code. -Harini Koji Sekiguchi wrote: Hi Harini, Close the reader after delete docs to take effect. Hope this helps, Koji -Original Message- From: Harini Raghavan

Deleting a Document

Hi, I want to update a document in the lucene index. As mentioned in the documentation, I tried to delete the document using IndexReader.delete method. But even after I delete the document, I am able to see the document when I perform a search. I thought this could be because, I am caching the

Re: Query Scoring

Thank you Chris. That seems like a good suggestion. I will try to pass a different Query object to the Highlighter api that the one used for searching. I plan to break down the HTML document and store the title/sub title/content in different fields of the index. So if I create a new query com

Re: Query Scoring

e returned, but again this is unrelated to highlighting. Could you elaborate on what you're after? Erik On Dec 30, 2005, at 12:02 PM, Harini Raghavan wrote: Hi, I have a requirement to highlight search keywords in the results and display the matching fragment of the text with the re

Query Scoring

Hi, I have a requirement to highlight search keywords in the results and display the matching fragment of the text with the results. I am using the Hits highlighting mentioned in Lucene in Action. Here is the search query(BooleanQuery) I am passing to the IndexSearcher and QueryScorer: +Document

Re: how to control terms to be highlighted?

the keywords appear(similar to google) and display below the search result. But, the above text extracted is not really the best fragment, it seems to be the first fragment which has the keywords. Has someone implemented this kind of functionality? -Harini Harini Raghavan wrote: Hi Chris

Re: how to control terms to be highlighted?

: >>Content:restructuring Content:restructure) ...just give the highlighter... (Content:"cost saving" Content:"cost savings" Content:outsource Content:outsources Content:downsize Content:downsizes Content:restructuring Content:restructure) : Date: Th

Re: how to control terms to be highlighted?

o have the effect of filtering out any query terms that weren't for this field. Would this change be a problem for anyone? Cheers, Mark --- Harini Raghavan <[EMAIL PROTECTED]> wrote: Hi, I have a requirement to highlight search keywords in the results and display the matching frag

how to control terms to be highlighted?

Hi, I have a requirement to highlight search keywords in the results and display the matching fragment of the text with the results. I am using the Hits highlighting mentioned in Lucene in Action. Here is the search query(BooleanQuery) I am passing to the IndexSearcher and QueryScorer: +Doc

Re: Lucene search is very slow

caching data in each index searcher. For large index, it's definitely a waste to re-create index searcher every time. Chris -- Full-Text Search on Any Databases http://www.dbsight.net On 10/20/05, Harini Raghavan <[EMAIL PROTECTED]> wrote: Hi

Re: Lucene search is very slow

http://www.dbsight.net On 10/10/05, Koji Sekiguchi <[EMAIL PROTECTED]> wrote: Is it really the part of Lucene slow? Please take thread dumps every 15 secs, 3 to 4 times. What can you look at them? Koji -Original Message- From: Harini Raghavan [mailto:[EMAIL

Lucene search is very slow

Hi, I am using lucene for search functionality in my j2ee application using JBoss as app server. The lucene index directory size is almsot 10G. The performance has been quite good until now. But after the last deploy, when the server was restarted , the lucene search has become very slow. It t

Stale NFS file handle Exception

Hi All, I have 2 servers in the production environment, one running some Quartz jobs and the other one running the application. There is a common NFS mount which has the lucene index directory. The jobs fetch the latest data and update the lucene index. And the user can search on the index to

Re: IOException : Lock obtain timed out

The default value of IndexWriter.WRITE_LOCK_TIMEOUT property is 1000ms. Can this value be increased to some optimum value? - Original Message - From: "Harini Raghavan" <[EMAIL PROTECTED]> To: Sent: Saturday, July 30, 2005 11:23 PM Subject: IOException : Lock obtain

IOException : Lock obtain timed out

lise a new IndexWriter. So I am wondering why this problem is occuring. Can someone please help? Thanks, Harini : Date: Mon, 18 Jul 2005 10:12:39 +0530 : From: Harini Raghavan <[EMAIL PROTECTED]> : Reply-To: java-user@lucene.apache.org : To: [EMAIL PROTECTED] : Cc: java-user@lucene.apache.o

Re: OutOfMemory errors while indexing large documents

ire document and not just the first 10,000 or so terms - I set maxFieldLength to Integer.MAX_VALUE. Erik On Jul 25, 2005, at 7:30 AM, Harini Raghavan wrote: Hi All, I am using lucene to index large documents(HTML pages). The application is running on JBoss and MySQL on UNIX. The index

OutOfMemory errors while indexing large documents

Hi All, I am using lucene to index large documents(HTML pages). The application is running on JBoss and MySQL on UNIX. The indexing is throwing OutOfMemory errors beyond a certain point. I am not sure why this is happening. I am using the default IndexWriter properties, but the lucene documenta

Implementing paging functionality in lucene

Hi All, I am trying to add paging functionality while using lucene search. I have created a PageFilter what takes in the current page num and the number of records as input and invoking the IndexSearcher passing the a Boolean Query object and the PageFilter. The search returns around 1000 records

Re: Index locked exception while updating index

Hi All, I am trying to add paging functionality while using lucene search. I have created a PageFilter what takes in the current page num and the number of records as input and invoking the IndexSearcher passing the a Boolean Query object and the PageFilter. The search returns around 1000 reco

Re: Index locked exception while updating index

ses the current writer to persist the additions, and opens a new one it it's palce for future updates. ...if you're using JDK1.5, take a look at java.util.concurrent.atomic.AtomicReference, it should make managing a singleton IndexWriter really easy. : Date: Mon, 18 Jul 2005 10:12

Re: Index locked exception while updating index

You are catching IOException in the finally block, but you are not even printing out the exception stack trace. Perhaps you are not able to close your IndexWriter for some reason. Otis --- Harini Raghavan <[EMAIL PROTECTED]> wrote: Hi All, I am quite new to Lucene and I have problem

Index locked exception while updating index

Hi All, I am quite new to Lucene and I have problem with locking. I have a MessageDrivenBean that sends messages to my Lucene indexer whenever there is a new database update. The indexer updates the index incrementally . Below is the code fragment in the indexer method that gets invoked by the