setRAMBufferSizeMB and setRAMPerThreadHardLimitMB

2016-07-28 Thread Gimantha Bandara
each thread reaches 1945 MB? -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-25 Thread Gimantha Bandara
t; Torsten > > > > > > > > > > > > - > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Lucene 5.0.0 - StringField and Sorting

2016-04-21 Thread Gimantha Bandara
-- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: GROUP BY in Lucene

2016-03-19 Thread Gimantha Bandara
ene has a grouping module that has several approaches for grouping > > search hits, though it's only by a single field I believe. > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > > > On Sun, Aug 9, 2015 at 2:55 PM, Gimantha B

Re: GROUP BY in Lucene

2016-03-18 Thread Gimantha Bandara
aryDocValues(this.indexFieldName); > > //Here find the oridinals of the group-by-fields and the > arrgegate fields. > //Create a multi ordinal key MultiIntKey from the > group-by-ordinals and use that to add the current value of the fiels to do > the agggregation to the facet-aggregates > > .. > > > Hope this helps :) > -Rob > > -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

How to get the top facets values of a field/dimension which has the highest number of immediate children

2016-03-15 Thread Gimantha Bandara
Hi, Lets say I have thousands of Lucene documents which have a FacetField which has the format of the following. doc.add(new FacetField("Category", "level0", "level1", "level2", "level3", ...)); "Category" is the dimension name. "level0" represents the first hierarchical level, "level1" represen

Re: How does sorting work in Lucene?

2016-03-12 Thread Gimantha Bandara
compared to another document's score of a different index). If we assume that the score is properly returned, can I use the scores of the docs of different indices and sort all the doc ids from all the indexes( probably using mergesort)? On Tue, Mar 1, 2016 at 11:04 AM, Gimantha Bandara

Getting an Exception while searching when (numHits = Large Number) in TopScoreDocCollector

2016-03-01 Thread Gimantha Bandara
I know that I am getting this exception because the priorityQueue allocate memory more than my PC can allocate from the RAM. ERROR {org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsDataIndexer} - Error in index search: null java.lang.NegativeArraySizeException at org.apache.lucene

Re: How does sorting work in Lucene?

2016-02-29 Thread Gimantha Bandara
Any thoughts? On Monday, February 29, 2016, Gimantha Bandara wrote: > Hi all, > > We are using lucene to index our data and are maintaining millions of > documents in sharded indices. Currently what we do is, reading each shard > separately and collecting the TopDocs using TopDo

How does sorting work in Lucene?

2016-02-28 Thread Gimantha Bandara
Lucene support pagination? Help is much appreciated. -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Equivalent API in Lucene 5.x.x for ResultMode.setResultMode and ResultMode.setDepth

2015-10-09 Thread Gimantha Bandara
any help on this please? On 10/6/15, Gimantha Bandara wrote: > Hi, > > I want to get the top categories (all the categories) recursively in one > call. I know that in Lucene 4.x.x we can simply set the ResultMode to > PER_NODE_IN_TREE [1] and set the depth, so we can get

Equivalent API in Lucene 5.x.x for ResultMode.setResultMode and ResultMode.setDepth

2015-10-06 Thread Gimantha Bandara
Hi, I want to get the top categories (all the categories) recursively in one call. I know that in Lucene 4.x.x we can simply set the ResultMode to PER_NODE_IN_TREE [1] and set the depth, so we can get the categories recursively to the level we want in the tree. How do I achieve the same in Lucene

GROUP BY in Lucene

2015-08-09 Thread Gimantha Bandara
guess Faceting would help, But is it possible get all the categoryPaths along with the matching records? ) Is there any other way other than using Facets? -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Document updates work as delete/add under the hood

2015-07-10 Thread Gimantha Bandara
documents and re-index new document. I need to update only a single field. > Re-indexing a new document with updated field + other fields seems to be > big overhead. My question is, why lucene does that and currently is there a > way we can avoid this ? > > Thanks, > Chalitha > >

Re: Document updates work as delete/add under the hood

2015-07-10 Thread Gimantha Bandara
use these > delete/add for updates as it feels like a real bottleneck. > > Is there any way to do single fields updates without using delete/add ? > > Thanks, > Chalitha > > -- > J.M Chalitha Udara Perera > > *Department of Computer Science and Engineering,* > *U

Re: Using lucene queries to search StringFields

2015-06-21 Thread Gimantha Bandara
i, Jun 19, 2015 at 10:45 AM, Ahmet Arslan > wrote: > > > Hi, > > > > Why don't you create your query with API? > > > > Term term = new Term("B", "1 2"); > > Query query = new TermQuery(term); > > > > Ahmet > > >

Re: Using lucene queries to search StringFields

2015-06-18 Thread Gimantha Bandara
"B:1 2\"*"); Can someone point out the correct way to query for StringFields? Thanks, On Thu, Jun 18, 2015 at 2:12 PM, Gimantha Bandara wrote: > Hi all, > > I have created lucene documents like below. > > Document doc = new Document(); > doc.add(new TextField(&quo

Using lucene queries to search StringFields

2015-06-18 Thread Gimantha Bandara
e.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:277) at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) Is my searching query wrong? (Note: I am using whitespace analyzer everywhere) -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Exception While searching through indices.

2015-06-16 Thread Gimantha Bandara
n/java/org/wso2/carbon/analytics/dataservice/indexing/AnalyticsDataIndexer.java On Sun, Jun 14, 2015 at 7:13 PM, Đạt Cao Mạnh wrote: > Can you post you scenario in detail along with your modification please? > > On 14:09, Sun, 14 Jun 2015 Gimantha Bandara wrote: > >> Hi Dat

Re: Exception While searching through indices.

2015-06-14 Thread Gimantha Bandara
single lucene index to index billions > documents is not a proper ways. You should consider using Solr Cloud or > Elasticsearch to index your documents. > > On 19:43, Fri, 12 Jun 2015 Gimantha Bandara wrote: > > > Hi all, > > > > We are using Lucene 4.10.

Exception While searching through indices.

2015-06-12 Thread Gimantha Bandara
) -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Migrating from Lucene 4.10.3 to Lucene 5.10

2015-05-22 Thread Gimantha Bandara
dea. The only > available method is, as said before, Directory#makeLock that should return > a Lock instance. > > Uwe > > - > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > -Original Me

Migrating from Lucene 4.10.3 to Lucene 5.10

2015-05-21 Thread Gimantha Bandara
5.1.0. How does Lucene 5.1.0 handle releasing the lock? -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Exception while updating a lucene document

2015-04-26 Thread Gimantha Bandara
ems like you forgot to do facetsConfig.setMultiValued(`field`, true) too > . > > On Sat, Apr 25, 2015 at 7:37 AM, Gimantha Bandara > wrote: > > > Hi, > > > > I was able to fix the problem.. the issue was with my wrong usage of > > FacetConfig class. I was cre

Re: Exception while updating a lucene document

2015-04-25 Thread Gimantha Bandara
document, Call the facetConfig.build at last. Then call the updateDocument on the created document. On Fri, Apr 24, 2015 at 10:13 PM, Gimantha Bandara wrote: > Hi all, > > I have documents which have some facetfields. If I tried to update a > document where the facet values are same i

Exception while updating a lucene document

2015-04-24 Thread Gimantha Bandara
Hi all, I have documents which have some facetfields. If I tried to update a document where the facet values are same in both facet fields i am getting the following error. Note I am using "updateDocument" in indexWriter to create the document. Exception in thread "pool-23-thread-2" java.lang.Ill

Joining two Indices in Lucene

2015-04-24 Thread Gimantha Bandara
Hi, I am now looking into BlockJoinQuery where I can join two indices and execute search queries. My concern is this... Is it possible to perform "AND" and "OR" operations between two seperate indices? As an example, I have a common field "_id" in both indices. I will relate the two indices( using

Re: How to read multiple indices in parallel.

2015-04-07 Thread Gimantha Bandara
ver B would use leaves E, F. > Searching over a MultiReader on (A, B) would use leaves B, C, E, F. > > This runs serially by default but can run in parallel if you provide an > ExecutorService to the IndexSearcher and use a compatible search() method > on it. > > --Terry

Re: How to read multiple indices in parallel.

2015-04-07 Thread Gimantha Bandara
pply your own Collector). > > The not-yet-released Lucene 5.1 includes some changes (LUCENE-6294 > <https://issues.apache.org/jira/browse/LUCENE-6294>) that enable better > parallel query support. > > --Terry > > > On Tue, Apr 7, 2015 at 8:09 AM, Gimantha Bandara >

How to read multiple indices in parallel.

2015-04-07 Thread Gimantha Bandara
Hi all, As I can see the Multireader is reading the multiple indices sequentially (correct me if I am wrong). So using a IndexSearcher on a multireader will also perform sequential searches right? Is there a lucene-built-in class to search several indices parallely? -- Gimantha Bandara Software

Re: How to merge several Taxonomy indexes

2015-04-02 Thread Gimantha Bandara
gt; On Thu, Apr 2, 2015 at 1:51 PM, Gimantha Bandara > wrote: > > > Btw I was using a RAMDirectory for just testing purposes.. > > > > On Thu, Apr 2, 2015 at 5:16 PM, Gimantha Bandara > > wrote: > > > > > Hi Christoph and Shai, > > > > &

Re: How to merge several Taxonomy indexes

2015-04-02 Thread Gimantha Bandara
Btw I was using a RAMDirectory for just testing purposes.. On Thu, Apr 2, 2015 at 5:16 PM, Gimantha Bandara wrote: > Hi Christoph and Shai, > > Thanks for the quick response!. > Indices are stored in a relational database ( using a custom Directory > implementation ). The Prob

Re: How to merge several Taxonomy indexes

2015-04-02 Thread Gimantha Bandara
> > why do you use a RAMDirectory? If your merged index fits into RAM > > completely, a MMapDirectory should offer almost the same performance. And > > if not, it is definitely the better choice. > > > > Regards > > Christoph > > > > > > Am 02.0

Re: How to merge several Taxonomy indexes

2015-04-02 Thread Gimantha Bandara
handle this issue? On Tue, Mar 24, 2015 at 12:18 PM, Gimantha Bandara wrote: > Hi Christoph, > > My mistake. :) It does the exactly what i need. figured it out later.. > Thanks a lot! > > On Tue, Mar 24, 2015 at 3:14 AM, Gimantha Bandara > wrote: > >&g

Re: for check similarity of two sentences

2015-04-02 Thread Gimantha Bandara
cation to suggest > those questions.can i do that using Apache Lucene? > Thank You! > regards, > Heshan jayasinghe > -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Would Like to contribute to Lucene

2015-03-27 Thread Gimantha Bandara
book. However, there is some interesting content > > > that you can find about Lucene and Solr internals scattered in blog > > > posts and conference presentations. I would recommend having a look at > > > Mike's blog http://blog.mikemccandless.com/ and videos of Luce

Getting the doc values grouped by Facets

2015-03-26 Thread Gimantha Bandara
grouped by facets. Is it possible? -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: How to merge several Taxonomy indexes

2015-03-23 Thread Gimantha Bandara
Hi Christoph, My mistake. :) It does the exactly what i need. figured it out later.. Thanks a lot! On Tue, Mar 24, 2015 at 3:14 AM, Gimantha Bandara wrote: > Hi Christoph, > > I think TaxonomyMergeUtils is to merge a taxonomy directory and an index > together (Correct me if I am wr

Re: How to merge several Taxonomy indexes

2015-03-23 Thread Gimantha Bandara
tha, > > have a look at the class org.apache.lucene.facet.taxonomy.TaxonomyMergeUtils, > which does exactly what you need. > > Best regards, > Christoph > > Am 23.03.2015 um 15:44 schrieb Gimantha Bandara: > >> Hi all, >> >> Can anyone point me how to merge s

How to merge several Taxonomy indexes

2015-03-23 Thread Gimantha Bandara
is easy. To Merge document indexes, all I have to do is create a MultiReader and pass it to IndexSearcher. But I am stuck at merging the taxonomy indexes. Is there a way to merge taxonomy indexes? -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Would Like to contribute to Lucene

2015-03-19 Thread Gimantha Bandara
Any clue on where to start from? On Fri, Mar 13, 2015 at 11:24 AM, Gimantha Bandara wrote: > Hi all, > > I am willing to contribute to Lucene project. I have already been > referring to "Lucene in Action" 2nd edition recently. But I think it is > outdated. It is base

Would Like to contribute to Lucene

2015-03-12 Thread Gimantha Bandara
he lack of up-to-date resources. Can someone recommend a recently released book on lucene internals or has someone planned to write one? What would be the starting point if I need to learn the internals of Lucene? Thanks, -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Sampled Hit counts using Lucene Facets.

2015-03-11 Thread Gimantha Bandara
> be an additional constructor on LongRangeFacetCounts (and Double), or a > separate utility class which given min/max values and numBuckets, creates > the proper Range[]? > > Shai > > On Tue, Mar 10, 2015 at 4:07 PM, Gimantha Bandara > wrote: > > > Hi Shai, > > >

Difference between StoredField vs Other Fields with Field.Store.YES

2015-03-11 Thread Gimantha Bandara
Hi all, Is there a difference between using StoredField and using other types of fields with Field.Store.YES? Another question, Is it a good practise to use NumericDocValuesField instead of using usual Fields (IntField, LongField, StringField ...etc) with Field.Store.NO ? -- Gimantha Bandara

Re: Sampled Hit counts using Lucene Facets.

2015-03-10 Thread Gimantha Bandara
; "sampled" too. This is at least how we implemented facet sampling. > > Shai > > On Tue, Mar 10, 2015 at 10:21 AM, Gimantha Bandara > wrote: > > > What I am planning to do is, split the given time range into smaller time > > ranges by myself and pass them to a Lo

Re: Lucene index

2015-03-10 Thread Gimantha Bandara
ym in lucene PLEASE > > > Also, is lucene index graph database or not? > > > > Thank U > -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Sampled Hit counts using Lucene Facets.

2015-03-10 Thread Gimantha Bandara
What I am planning to do is, split the given time range into smaller time ranges by myself and pass them to a LongRangeFacetsCount object and get the counts for each sub range. Is this the correct way? On Tue, Mar 10, 2015 at 12:01 AM, Gimantha Bandara wrote: > Any updates on this please? D

Re: Sampled Hit counts using Lucene Facets.

2015-03-09 Thread Gimantha Bandara
Any updates on this please? Do I have to write my own code to sample and get the hitcount? On Sat, Mar 7, 2015 at 2:14 PM, Gimantha Bandara wrote: > Any help on this please? > > On Fri, Mar 6, 2015 at 3:13 PM, Gimantha Bandara > wrote: > >> Hi, >> >> I am

Re: Sampled Hit counts using Lucene Facets.

2015-03-07 Thread Gimantha Bandara
Any help on this please? On Fri, Mar 6, 2015 at 3:13 PM, Gimantha Bandara wrote: > Hi, > > I am trying to create some APIs using lucene facets APIs. First I will > explain my requirement with an example. Lets say I am keeping track of the > count of people who enter through

Sampled Hit counts using Lucene Facets.

2015-03-06 Thread Gimantha Bandara
guide for lucene 4.10.3 or lucene 5.0.0 ? Thanks, -- Gimantha Bandara Software Engineer WSO2. Inc : http://wso2.com Mobile : +94714961919

Re: Difference between CategoryPath and Plain FacetFields with hierarchy

2015-03-05 Thread Gimantha Bandara
ndless > > http://blog.mikemccandless.com > > > On Thu, Mar 5, 2015 at 2:13 AM, Gimantha Bandara > wrote: > > Hi, > > > > Any help on this? Or Can someone point me to Faceted User guide of > 4.10.3. > > I cannot find it. Is it only available for Older ver

Re: Difference between CategoryPath and Plain FacetFields with hierarchy

2015-03-04 Thread Gimantha Bandara
Hi, Any help on this? Or Can someone point me to Faceted User guide of 4.10.3. I cannot find it. Is it only available for Older version? On Wed, Mar 4, 2015 at 2:38 PM, Gimantha Bandara wrote: > Hi, > > I am new to Lucene faceting and taxonomy. I saw few examples in some blogs > a

Difference between CategoryPath and Plain FacetFields with hierarchy

2015-03-04 Thread Gimantha Bandara
both these approaches? What is the recommended approach to to create a faceted search application? For so called two different approaches please refer to [1] and [2] [1] http://www.hascode.com/2012/08/lucene-snippets-faceting-search/ [2] http://www.norconex.com/facets-with-lucene/ -- Gimantha