Re: Speed up searching on index created using JdbcDirectory
HI All, Please help me out to resolve this issue. Your help is really appriciated. Thanks Mahesh On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar < mahesh.charegaon...@gmail.com> wrote: > Hi Lucene masters, > > I was using lucene couple of years back. We have developed application > which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have > writing and reading data from database. > > Over the time data has increased tremendously and that why we are facing > performance issue with searching. I am using Lucene core jar 2.4 and JDK 5. > Could you please suggests good ways to increase the performance. > > I have limitation to use JDK5 , so suggest the changes which will be > supported by JDK5 with any new API in lucene. > > Thanks > Mahesh >
RE: Speed up searching on index created using JdbcDirectory
Don't use JDBCDirectory. It does not scale and has very poor performance. This is why it was removed in Lucene 3. Uwe - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com] > Sent: Saturday, August 23, 2014 11:12 PM > To: java-user@lucene.apache.org > Subject: Re: Speed up searching on index created using JdbcDirectory > > HI All, > > Please help me out to resolve this issue. Your help is really appriciated. > > Thanks > Mahesh > > > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar < > mahesh.charegaon...@gmail.com> wrote: > > > Hi Lucene masters, > > > > I was using lucene couple of years back. We have developed application > > which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have > > writing and reading data from database. > > > > Over the time data has increased tremendously and that why we are > > facing performance issue with searching. I am using Lucene core jar 2.4 and > JDK 5. > > Could you please suggests good ways to increase the performance. > > > > I have limitation to use JDK5 , so suggest the changes which will be > > supported by JDK5 with any new API in lucene. > > > > Thanks > > Mahesh > > - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Speed up searching on index created using JdbcDirectory
Thanks Uwe for your response. Could you please tell me if i have to still need to keep index in database what will be good option for searching. Thanks Mahesh On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler wrote: > Don't use JDBCDirectory. It does not scale and has very poor performance. > This is why it was removed in Lucene 3. > > Uwe > > - > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -Original Message- > > From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com] > > Sent: Saturday, August 23, 2014 11:12 PM > > To: java-user@lucene.apache.org > > Subject: Re: Speed up searching on index created using JdbcDirectory > > > > HI All, > > > > Please help me out to resolve this issue. Your help is really > appriciated. > > > > Thanks > > Mahesh > > > > > > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar < > > mahesh.charegaon...@gmail.com> wrote: > > > > > Hi Lucene masters, > > > > > > I was using lucene couple of years back. We have developed application > > > which uses lucene's JdbcDirecory feature. Using JdbcDirecory we have > > > writing and reading data from database. > > > > > > Over the time data has increased tremendously and that why we are > > > facing performance issue with searching. I am using Lucene core jar > 2.4 and > > JDK 5. > > > Could you please suggests good ways to increase the performance. > > > > > > I have limitation to use JDK5 , so suggest the changes which will be > > > supported by JDK5 with any new API in lucene. > > > > > > Thanks > > > Mahesh > > > > > > - > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
RE: Speed up searching on index created using JdbcDirectory
Hi, there is no need to have an index in a relational database. Lucene indexes are commonly stored as files on local disks. Use FSDirectory subclasses to do this! For more details about performance problem, you should maybe give us more details. Uwe - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com] > Sent: Saturday, August 23, 2014 11:42 PM > To: java-user@lucene.apache.org > Subject: Re: Speed up searching on index created using JdbcDirectory > > Thanks Uwe for your response. > > Could you please tell me if i have to still need to keep index in database > what > will be good option for searching. > > Thanks > Mahesh > > > On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler wrote: > > > Don't use JDBCDirectory. It does not scale and has very poor performance. > > This is why it was removed in Lucene 3. > > > > Uwe > > > > - > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > > > > -Original Message- > > > From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com] > > > Sent: Saturday, August 23, 2014 11:12 PM > > > To: java-user@lucene.apache.org > > > Subject: Re: Speed up searching on index created using JdbcDirectory > > > > > > HI All, > > > > > > Please help me out to resolve this issue. Your help is really > > appriciated. > > > > > > Thanks > > > Mahesh > > > > > > > > > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar < > > > mahesh.charegaon...@gmail.com> wrote: > > > > > > > Hi Lucene masters, > > > > > > > > I was using lucene couple of years back. We have developed > > > > application which uses lucene's JdbcDirecory feature. Using > > > > JdbcDirecory we have writing and reading data from database. > > > > > > > > Over the time data has increased tremendously and that why we are > > > > facing performance issue with searching. I am using Lucene core > > > > jar > > 2.4 and > > > JDK 5. > > > > Could you please suggests good ways to increase the performance. > > > > > > > > I have limitation to use JDK5 , so suggest the changes which will > > > > be supported by JDK5 with any new API in lucene. > > > > > > > > Thanks > > > > Mahesh > > > > > > > > > > - > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Speed up searching on index created using JdbcDirectory
Hi,
My company policy not allowed me to place files on websphere server JVM, so
thats why we are placing index files in database and searching using
JDBCDirectory.
I am using 2.4 Lucene API jars API with below ways to search the fields
from index,
Directory dir; = new JdbcDirectory (ds,dialect, indexFileName,true);
Searcher searcher = new IndexSearcher(dir);
Hits hits = searcher.search(searchQuery);
for (int i = 0; i < hits.length(); i++) {
Document doc = hits.doc(i);
if(doc.get("Name") != null && !"".equals(doc.get("Name"))){
other biz logic
}
}
Thanks
Mahesh
On Sat, Aug 23, 2014 at 2:48 PM, Uwe Schindler wrote:
> Hi,
>
> there is no need to have an index in a relational database. Lucene indexes
> are commonly stored as files on local disks. Use FSDirectory subclasses to
> do this!
>
> For more details about performance problem, you should maybe give us more
> details.
>
> Uwe
>
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
> > -Original Message-
> > From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com]
> > Sent: Saturday, August 23, 2014 11:42 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Speed up searching on index created using JdbcDirectory
> >
> > Thanks Uwe for your response.
> >
> > Could you please tell me if i have to still need to keep index in
> database what
> > will be good option for searching.
> >
> > Thanks
> > Mahesh
> >
> >
> > On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler wrote:
> >
> > > Don't use JDBCDirectory. It does not scale and has very poor
> performance.
> > > This is why it was removed in Lucene 3.
> > >
> > > Uwe
> > >
> > > -
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: u...@thetaphi.de
> > >
> > >
> > > > -Original Message-
> > > > From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com]
> > > > Sent: Saturday, August 23, 2014 11:12 PM
> > > > To: java-user@lucene.apache.org
> > > > Subject: Re: Speed up searching on index created using JdbcDirectory
> > > >
> > > > HI All,
> > > >
> > > > Please help me out to resolve this issue. Your help is really
> > > appriciated.
> > > >
> > > > Thanks
> > > > Mahesh
> > > >
> > > >
> > > > On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
> > > > mahesh.charegaon...@gmail.com> wrote:
> > > >
> > > > > Hi Lucene masters,
> > > > >
> > > > > I was using lucene couple of years back. We have developed
> > > > > application which uses lucene's JdbcDirecory feature. Using
> > > > > JdbcDirecory we have writing and reading data from database.
> > > > >
> > > > > Over the time data has increased tremendously and that why we are
> > > > > facing performance issue with searching. I am using Lucene core
> > > > > jar
> > > 2.4 and
> > > > JDK 5.
> > > > > Could you please suggests good ways to increase the performance.
> > > > >
> > > > > I have limitation to use JDK5 , so suggest the changes which will
> > > > > be supported by JDK5 with any new API in lucene.
> > > > >
> > > > > Thanks
> > > > > Mahesh
> > > > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > > For additional commands, e-mail: java-user-h...@lucene.apache.org
> > >
> > >
>
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
SegmentReader heap usage with stored field compression on
Is it reasonable to assume that using stored field compression with a lot of stored fields per document in a very large index (100+ GB) could potentially lead to a significant heap utilization? If I am reading the code in CompressingStoredFieldsIndexReader correctly, there's a non-trivial accounting overhead, per segment, to maintain fields index reader state, which appears to be a function of both compression chunk size and overall segment size. Not sure if my hunch is correct here, but we have run into situations when loading stored fields for a relatively small number of search results (<100K) after a single query for an index of the above size would result in OOME with 5+ GB heap sizes, with dominating objects in heap dump being SegmentReader... hence the question. Thank you.
Re: Speed up searching on index created using JdbcDirectory
trace the DB operation Lucene is performing for your search operation and pass
on the explain plan to the list.
There is little you can do to tune this from a Lucene point of view but you can
probably tune the database to perform this operation faster.
This solution would just get you going for now.
Using Lucene file system indexing is the best option. You can encrypt the
indexes if you don't want them to be in clear, but that would add its own
overhead.
- Pradeep
> On Aug 23, 2014, at 5:01 PM, Mahesh Charegaonkar
> wrote:
>
> Hi,
>
> My company policy not allowed me to place files on websphere server JVM, so
> thats why we are placing index files in database and searching using
> JDBCDirectory.
>
> I am using 2.4 Lucene API jars API with below ways to search the fields
> from index,
>
>
> Directory dir; = new JdbcDirectory (ds,dialect, indexFileName,true);
> Searcher searcher = new IndexSearcher(dir);
> Hits hits = searcher.search(searchQuery);
> for (int i = 0; i < hits.length(); i++) {
> Document doc = hits.doc(i);
> if(doc.get("Name") != null && !"".equals(doc.get("Name"))){
> other biz logic
> }
> }
>
> Thanks
> Mahesh
>
>
>> On Sat, Aug 23, 2014 at 2:48 PM, Uwe Schindler wrote:
>>
>> Hi,
>>
>> there is no need to have an index in a relational database. Lucene indexes
>> are commonly stored as files on local disks. Use FSDirectory subclasses to
>> do this!
>>
>> For more details about performance problem, you should maybe give us more
>> details.
>>
>> Uwe
>>
>> -
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: u...@thetaphi.de
>>
>>
>>> -Original Message-
>>> From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com]
>>> Sent: Saturday, August 23, 2014 11:42 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Re: Speed up searching on index created using JdbcDirectory
>>>
>>> Thanks Uwe for your response.
>>>
>>> Could you please tell me if i have to still need to keep index in
>> database what
>>> will be good option for searching.
>>>
>>> Thanks
>>> Mahesh
>>>
>>>
On Sat, Aug 23, 2014 at 2:15 PM, Uwe Schindler wrote:
Don't use JDBCDirectory. It does not scale and has very poor
>> performance.
This is why it was removed in Lucene 3.
Uwe
-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
> -Original Message-
> From: Mahesh Charegaonkar [mailto:mahesh.charegaon...@gmail.com]
> Sent: Saturday, August 23, 2014 11:12 PM
> To: java-user@lucene.apache.org
> Subject: Re: Speed up searching on index created using JdbcDirectory
>
> HI All,
>
> Please help me out to resolve this issue. Your help is really
appriciated.
>
> Thanks
> Mahesh
>
>
> On Wed, Aug 20, 2014 at 1:57 PM, Mahesh Charegaonkar <
> mahesh.charegaon...@gmail.com> wrote:
>
>> Hi Lucene masters,
>>
>> I was using lucene couple of years back. We have developed
>> application which uses lucene's JdbcDirecory feature. Using
>> JdbcDirecory we have writing and reading data from database.
>>
>> Over the time data has increased tremendously and that why we are
>> facing performance issue with searching. I am using Lucene core
>> jar
2.4 and
> JDK 5.
>> Could you please suggests good ways to increase the performance.
>>
>> I have limitation to use JDK5 , so suggest the changes which will
>> be supported by JDK5 with any new API in lucene.
>>
>> Thanks
>> Mahesh
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
>> -
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
