Re: Handling Indian regional languages

Hi Robert Muir, we will check on this. Thanks a lot for the pointers. -- *K*umaran *R* On Mon, Jan 16, 2023 at 11:16 PM Robert Muir wrote: > On Tue, Jan 10, 2023 at 2:04 AM Kumaran Ramasubramanian > wrote: > > > > For handling Indian regional languages, what is the

Handling Indian regional languages

For handling Indian regional languages, what is the advisable approach? 1. Indexing each language data(Tamil, Hindi etc) in specific fields like content_tamil, content_hindi with specific per field Analyzer like Tamil for content_tamil, HindiAnalyzer for content_hindi? 2. Indexing all language da

Re: currency based search using query time calculated field match with expression

If you want useful > suggestions, you need to be much more explicit about your use case, > what you've tried, why it didn't work, etc. > > On Fri, Sep 3, 2021 at 6:08 AM Kumaran Ramasubramanian > wrote: > > > > Hi Michael, Thanks for the response. > &

Re: currency based search using query time calculated field match with expression

simple > expression language. It might prove useful to you if the exchange rate > needs to be tracked very dynamically. > > On Thu, Sep 2, 2021 at 2:15 PM Kumaran Ramasubramanian > wrote: > > > > I am having one use case regarding currency based search. I want to get

currency based search using query time calculated field match with expression

I am having one use case regarding currency based search. I want to get any suggestions or pointers.. For example, Assume, 1USD = 75 INR 1USD = 42190 IRR similarly, we have support for 100 currencies as of now. Record1 created with PRICE 150 INR & EXCHANGE_RATE 75 for USD Record2 created with PRI

Re: Autocompletion based on one field in index

Hi All, Any input would be appreciated.. thanks in advance. Kumaran R On Tue, Mar 10, 2020, 11:44 PM Kumaran Ramasubramanian wrote: > > > Hi Mikhail > > Thanks for the input. But i would like to suggest title of the available > documents (for the query typed in search b

Re: Autocompletion based on one field in index

its help widget ) [image: Screenshot from 2020-03-10 23-40-14.png] -- *K*umaran *R* On Wed, Mar 4, 2020 at 2:44 AM Mikhail Khludnev wrote: > Hi, > > org.apache.lucene.search.spell.DirectSpellChecker > > > On Tue, Mar 3, 2020 at 8:14 AM Kumaran Ramasubramanian > > wr

Autocompletion based on one field in index

Hi All, I would like to compute autocompletion based on one field's data. For example, title field of a list of webpages. Is there anyway to achieve this? Regards Kumaran R

Index fields configuration - suggestions

Hi all, Regarding configurations about every fields( stored? analyzed? sort needed? numeric ? ), elastic search designed cluster state to hold these configurations index wise.. solr have those configurations in xml format. If we have data center in multiple locations, is there any better way of ma

Re: Encryption at lucene index

t; ichattopadhy...@gmail.com> wrote: > > > >> Harry Ochiai (Hitachi) has some index encryption solution, > >> https://www.slideshare.net/maggon/securing-solr-search- > data-in-the-cloud > >> I think it is proprietary, but I'm not sure. Maybe m

Re: Encryption at lucene index

is proprietary, but I'm not sure. Maybe more googling might help > find the exact page where his solution is described. > > On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian < > kums@gmail.com> > wrote: > > > Hi Erick, i want to encrypt some fields

Re: Encryption at lucene index

large to conduct a seminar through the user's > list. > > Best, > Erick > > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian > wrote: > > Hi Erick, > > > > Thanks for the information. Any pointers about encryption options in > > so

Re: Encryption at lucene index

eople ask for encryption they soon discover > it's much more difficult than they imagine and settle for just putting > the indexes on an encrypting file system. When they move beyond that > it gets complex and you'd be well advised to consult with Solr > security experts. &

Encryption at lucene index

Hi All, After looking at all below discussions, i have one doubt which may be silly or novice but i want to throw this to lucene user list. if we have encryption layer included in our analyzer's flow of filters like EncryptionFilter to control field-level encryption. what are the consequences ?

Re: Filters Vs queries - for terms more than 1024

ery$FilterStrategy.filteredBulkScorer(FilteredQuery.java:504) > at > org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:150) If i use BooleanQuery and wrap it with ConstantScoreQuery, shall i use 1 lakh boolean clauses in booleanquery ? - ​- Kumaran R ​ On Wed, Jul 19,

Re: Filters Vs queries - for terms more than 1024

f you have many clauses. But like TermsQuery, BooleanFilter always consume all matching documents from all its clauses. So if you intersect it with a selective query, it is wasteful. Le mar. 18 juil. 2017 à 11:42, Kumaran Ramasubramanian a écrit : > ​Hi Adrien, > > Thanks for your in

Re: Filters Vs queries - for terms more than 1024

ries. > > Otherwise you would probably need to rethink how you design your documents > in order to be able to run simpler queries. > > Le lun. 17 juil. 2017 à 16:28, Kumaran Ramasubramanian > > a écrit : > > > Hi All, > > > > i am using lucene 4.10.4 >

Filters Vs queries - for terms more than 1024

Hi All, i am using lucene 4.10.4 In lucene search, i know we have 1024 limitation in number of boolean query clauses. i know we can increase this limit.. but i want to understand queries vs filter in lucene 4.10.4... i want to make queries larger than 1024.. Relevance is not needed for me. What

Re: email field - analyzed and not analyzed in single field using custom analyzer

ertAnalyzesTo(a, "will.sm...@yahoo.com", > new String[] { "will.sm...@yahoo.com", "will", "smith", "yahoo", > "com" }, > null, null, null, > new int[] { 1, 0, 1, 1, 1 }, > null, false); > a.close(

email field - analyzed and not analyzed in single field using custom analyzer

Hi All, i want to index email fields as both analyzed and not analyzed using custom analyzer. for example, sm...@yahoo.com will.sm...@yahoo.com that is, indexing sm...@yahoo.com as single token as well as analyzed tokens in same email field... My existing custom analyzer, public class Custom

Re: Recommended number of fields in one lucene index

y documents and provide search for them, which cannot scale, or to > the fact that the index contained many unrelated documents that should have > been put into different indices. This limit has been very useful to catch > such design problems early instead of waiting for the production server t

Recommended number of fields in one lucene index

Hi All, Elasticsearch allows 1000 fields by default. In lucene, What are the indexing and searching performance impacts of having 10 fields vs 3000 fields in a lucene index? In my case, while indexing, i index and store all fields and so i can provide update on one field where we use to take out

Re: Replacement for Filter-as-abstract-class in Lucene 5.4?

I always use filter when i need to add more than 1024 ( for no scoring cases ). If filter is removed in lucene 6, what will happen to maxbooleanclauses limit? Am i missing anything? - Kumaran R On Jan 12, 2017 5:01 AM, "Trejkaz" wrote: On Thu, Jan 21, 2016 at 4:25 AM, Adrien Grand wrote: >

Re: Heavy usage of final in Lucene classes

Hi I want to know the purpose of having final in analyzers. For eg: classicanalyzer. It will be easy to add asciifolding filter over classicanalyzer. - Kumaran R On Jan 12, 2017 5:41 AM, "Michael McCandless" wrote: I don't think it's about efficiency but rather about not exposing possibly tr

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

and sorting (numeric). > > > > A field with docValues="true" just supports faceting, grouping and > > sorting without having to "uninvert" the field in the Java heap, the > > data is out in OS cache. See Uwe's excellent blog here: > > http://

Re: Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

Thank you Adrien. "NumericDocValuesField is the one that supports sorting." Does this mean LongField/IntField just supports lexicographic order in sorting? - Kumaran R On Dec 22, 2016 11:28 PM, "Adrien Grand" wrote: Le jeu. 22 déc. 2016 à 18:50, Kumaran Ramasubrama

Sorting, Range Query, faceting - NumericDocValuesField Vs LongField

Hi All, I want to provide sorting, range search and faceting in numeric fields. AFAIK, Purpose of different numeric field types are, NumericDocValuesField supports sorting and faceting LongField/IntField supports range query and sorting 1. Should i duplicate one field in above mentioned type

Re: how do lucene read large index files?

rhaps NIOFSDirectory won't be much slower. > > Why not use MMapDirectory for both indexing and searching? > Mike McCandless > > http://blog.mikemccandless.com > > > On Mon, Nov 28, 2016 at 7:20 AM, Kumaran Ramasubramanian > wrote: > > Thanks a lot Uwe!!! Do we g

Re: how do lucene read large index files?

u, Nov 24, 2016 at 9:41 AM, Uwe Schindler wrote: > > Hi Kumaran, hi Erick, > > > >> Not really, as I don't know that code well, Uwe and company > >> are the masters of that realm ;) > >> > >> Sorry I can't be more help there.... > >

Re: how do lucene read large index files?

/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Short form: files are read into the OS's memory as needed. the whole > file isn't read at once. > > Best, > Erick > > On Wed, Nov 23, 2016 at 12:04 AM, Kumaran Ramasubramanian > wrote: > > Hi All, > &

how do lucene read large index files?

Hi All, how do lucene read large index files? for example, if one file (for eg: .dat file) is 4GB. lucene read only part of file to RAM? or is it different approach for different lucene file formats? Related Link: How do applications (and OS) handle very big files? http://superuser.com/a/361201

Re: indexing analyzed and not_analyzed values in same field

:07 AM, Kumaran Ramasubramanian < kums@gmail.com> wrote: > > > Hi All, > > i have indexed 4 documents in an index where BANKNAME field is analyzed > in two documents and it is not_analyzed in another two documents. i have > mentioned search cases below where i a

Faceting : what are the limitations of Taxonomy (Separate index and hierarchical facets) and SortedSetDocValuesFacetField ( flat facets and no sidecar index) ?

Hi All, We all know that Lucene supports faceting by providing Taxonomy(Separate index and hierarchical facets) and SortedSetDocValuesFacetField ( flat facets and no sidecar index). Then why did solr and elastic search go for its own implementation ? ( that is, solr uses block j

Re: Indexing values of different datatype under same field

Hi Rajnish It is not advisable to index values with two data types in a field. Features like phrase query, sorting may break in those indexes. related previous discussion : http://www.gossamer-threads.com/lists/lucene/java-user/289159?do=post_view_flat#289159 - Kumaran R On Fri, Nov 4,

indexing analyzed and not_analyzed values in same field

Hi All, i have indexed 4 documents in an index where BANKNAME field is analyzed in two documents and it is not_analyzed in another two documents. i have mentioned search cases below where i am able to search using both analyzed ( using classic analyzer ) and not_analyzed ( using keyword analyzer )

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

n you could use AnalyzerWrapper. ​ > > > > > > > > > > > > > ​- > > Kumaran R​ > > > > > > > > > > > > > > > > > > > > On Tue, Oct 11, 2016 at 9:59 PM, Kumaran Ramasubramanian < > > kums@gmail.com > > >

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

Hi Adrien How to do this? Any Pointers? ​ > If it is fine to add the ascii folding filter at the end of the analysis chain, then you could use AnalyzerWrapper. ​ > ​- Kumaran R​ On Tue, Oct 11, 2016 at 9:59 PM, Kumaran Ramasubramanian wrote: > > > @Ahmet, Uwe: T

Re: How to add ASCIIFoldingFilter in ClassicAnalyzer

; > Hi Kumaran, > > > > If it is fine to add the ascii folding filter at the end of the analysis > > chain, then you could use AnalyzerWrapper. Otherwise, you need to create > a > > new analyzer that has the same analysis chain as ClassicAnalyzer, plus an > > A

How to add ASCIIFoldingFilter in ClassicAnalyzer

Hi All, Is there any way to add ASCIIFoldingFilter over ClassicAnalyzer without writing a new custom analyzer ? should i extend StopwordAnalyzerBase again? I know that ClassicAnalyzer is final. any special purpose for making it as final? Because, StandardAnalyzer was not final before ? public

Re: Clarification on LUCENE 4795 discussions ( Add FacetsCollector based on SortedSetDocValues )

b76f97891c8593ef5fbc;hb=HEAD > > Hope this helps! > > Shai > > On Tue, Sep 27, 2016 at 7:20 AM Kumaran Ramasubramanian < kums@gmail.com> > wrote: > > > Hi mike, > > > > Thanks for the clarification. Any example about difference in using flat vs >

Re: Clarification on LUCENE 4795 discussions ( Add FacetsCollector based on SortedSetDocValues )

t facets) > though this is possible to fix, I think (patches welcome!). > > Mike McCandless > > http://blog.mikemccandless.com > > On Mon, Sep 26, 2016 at 5:24 PM, Kumaran Ramasubramanian > wrote: > > > > > > Hi All, > > > > i want to kn

Clarification on LUCENE 4795 discussions ( Add FacetsCollector based on SortedSetDocValues )

​ ​Hi All,​ ​i want to know the list of features ​which can be used by applications using facet module of lucene. https://issues.apache.org/jira/browse/LUCENE-4795?focusedCommentId=13599687 I ask because it seems that the only thing that we get from this SortedSet > approach is not having to ma

Re: parent-child relationship in lucene - to avoid reindexing if parent information changes

ield in your lucene document. > > Then, when you want to search for all messages assigned to groups the user > is member of, first search for the groups where the user is member to get > the id, and next search all messages with that ids. > > So there should no need to reind

parent-child relationship in lucene - to avoid reindexing if parent information changes

Hi All, Am building a sample application, where a group of members can interact as a chat room. i am trying to enable search for message level search... If i denormalize group_name & group_members in every lucene document, then below cases will reindex more number of lucene documents... 1. edi

In lucene faceting, advantages of taxonomy index over ​SortedSetDocValuesFacetField

Hi All, As per documentation of SortedSetDocValuesFacetCounts, Compute facets counts from previously indexed > ​​ > SortedSetDocValuesFacetField, without require a separate taxonomy index. > Faceting is a bit slower (~25%), and there is added cost on every > IndexReader open to create a new Sor

Re: How to store and retrieve latest utf8mb4 emoji / smiley characters in lucene index

find😃f😄g😅h​ Any further pointers on the same ?? -- Kumaran R On Mon, Aug 1, 2016 at 12:07 AM, Kumaran Ramasubramanian wrote: > > Hi All, > > Is there any pointers on storing smileys in lucene index?? Any help is much appreciated. > > Thank you. > > -- >

Re: How to store and retrieve latest utf8mb4 emoji / smiley characters in lucene index

Hi All, Is there any pointers on storing smileys in lucene index?? Any help is much appreciated. Thank you. -- Kumaran R On Jul 30, 2016 12:24 PM, "Kumaran Ramasubramanian" wrote: > > Hi All, > > Am using lucene 4.10.4. Using lucene index, Is there any way to store and

How to store and retrieve latest utf8mb4 emoji / smiley characters in lucene index

Hi All, Am using lucene 4.10.4. Using lucene index, Is there any way to store and retrieve latest utf8 and utf8mb4 emoji / smiley characters?? In any latest lucene version?? Thanks in advance. -- Kumaran R

Re: Indexing and storing Long fields

candless.com > > On Thu, Jul 28, 2016 at 9:31 AM, Kumaran Ramasubramanian < > kums@gmail.com > > wrote: > > > Hi Mike, > > > > For your information, am using lucene 4.10.4.. am i missing anything? > > > > > > > > ​-- > >

Re: Indexing and storing Long fields

Hi Mike, For your information, am using lucene 4.10.4.. am i missing anything? ​-- Kumaran R​ On Wed, Jul 27, 2016 at 1:52 AM, Kumaran Ramasubramanian wrote: > > Hi Mike, > > 1.if we index one field as analyzed and not analyzed using same name, > phrase queries are not

Re: Indexing and storing Long fields

t; Expect many more anomalies. At best you need to > index both forms as text rather than mixing numeric > and text data. Thanks in advance... -- Kumaran R On Sun, Jul 24, 2016 at 1:54 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > On Sat, Jul 23, 2016 at 4:48

Re: Indexing and storing Long fields

Hi Mike, *Two different fields can be the same name* Is it so? You mean we can index one field as docvaluefield and also stored field, Using same name? And AFAIK, We cannot index one field as analyzed and not analyzed using the same name. Am i right? Kumaran R On Jul 21, 2016 11:50 PM, "Michae

query norm understanding when we use term boosting during search time

Hi All, i have read some discussions on impact of querynorm values in lucene score when we specify more boosting. Impact is, if boosting increases, querynorm decreases a lot, and so more boosting does not results in final lucene score Consider an example like, i need to order results based on gi

Re: Lucene term modifiers during search time - importance based on listed order

Kumaran R On Tue, Jun 7, 2016 at 10:16 PM, Kumaran Ramasubramanian wrote: > > > Hi All, > > i am trying to retrieve most related results using user given search > queries. If there is no results for exact queries, i want to use term > modifiers during search tim

Lucene term modifiers during search time - importance based on listed order

Hi All, i am trying to retrieve most related results using user given search queries. If there is no results for exact queries, i want to use term modifiers during search time. And so, i want to give more importance to term modifiers in the defined listed order given below 1. exact quer

Re: Difference in Retrieving values of a docvalue field using atomicreader over duplicate StoredField

some doc values + some stored fields you're looking at > multiple seeks per docID you retrieve. > > Really you should only do this for a few hits, e.g. one page worth. > > Mike McCandless > > http://blog.mikemccandless.com > > On Wed, Jun 1, 2016 at 3:15 PM,

Difference in Retrieving values of a docvalue field using atomicreader over duplicate StoredField

Hi All, In javadoc of every docvalue field , there is line like ​ > If you also need to store the value, you should add a separate StoredField >

IntField is not indexing properly - field properties changed to "just stored"

Hi All, i am using lucene 4.10.4. i am using below code to index IntField. All properties of IntField is vanished except stored. Any help is much appreciated. IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_4_10_4, new > ClassicAnalyzer(Version.LUCENE_30, new StringReader(""))); > i

Re: Need change one field type from IntField to String including indexOptions to store positions & Norms

a new index and then use the new index. That's > if all of the indexed field values are stored. > > -- Jack Krupansky > > On Thu, Dec 17, 2015 at 2:10 PM, Kumaran Ramasubramanian < > kums@gmail.com > > wrote: > > > Hi Jack Krupansky > > > > Th

Re: Need change one field type from IntField to String including indexOptions to store positions & Norms

standard answer is that you need to reindex all of your data. > > > > > > -- Jack Krupansky > > > > > > On Thu, Dec 17, 2015 at 6:10 AM, Kumaran Ramasubramanian < > > kums@gmail.com > > >> wrote: > > &g

Re: Need change one field type from IntField to String including indexOptions to store positions & Norms

Dear All i am using lucene 4.10.4. Is there any more information i missed to provide? Please let me know. -- Kumaran R*​* On Wed, Dec 16, 2015 at 10:35 PM, Kumaran Ramasubramanian < kums@gmail.com> wrote: > > Hi All, > > Previous Post - > http://www.gossamer-thr

Need change one field type from IntField to String including indexOptions to store positions & Norms

Hi All, Previous Post - http://www.gossamer-threads.com/lists/lucene/java-user/289159 i have indexed one field "STATUS" as both IntField & String field in same index. Now i want to take IntField containing documents and change the value of field "STATUS" to string with norms & positions ( t

Re: Using ​phrase query in Termfilters

Query > (using Occur.FILTER). > > Uwe > > - > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Kumaran Ramasubramanian [mailto:kums@gmail.

Using ​phrase query in Termfilters

Hi All, Am using lucene 4.10.4. Is it right to add analyzed multi valued fields & phrase query for the same field in boolean filter. i believe we could not apply analyzers to values in filters. So am not getting results for those filters' match. String phraseTerm = "hello world" > Term term =

Re: Two different types of values in same field name in single index

data. Since you're not sorting you should be > OK with the caveat that searching for > "02" won't match an indexed value of "2" unless you > remove all leading zeros at both index and query > time. > > Best, > Erick > > On Tue, Oct 27, 2015 at

Re: Two different types of values in same field name in single index

n't you creating an unworkable scenario for sorting? > > -will > > On 10/27/15 5:49 AM, Kumaran Ramasubramanian wrote: > >> Hi All, >> >> i have indexed module wise data in same index. In this case, we index >> two >> types of field in same name in two diffe

Two different types of values in same field name in single index

Hi All, i have indexed module wise data in same index. In this case, we index two types of field in same name in two different document like this. *document1:* > module:1 > status:4 ( as LongField ) > *code:* > long longValue=Long.parseLong(value); > LongField field=new LongField(f > ield > Na

Re: How does Lucene decides which fields have termvectors stored and which not?

{ > > String fieldName = strItr.next(); > > System.out.println("next field " + fieldName); > > } > > Document DocList= reader.document(indexedDocID); > > List field_list = DocList.getFields(); > > for(int j=0; j < field_list.size(); j++) > >

Re: How does Lucene decides which fields have termvectors stored and which not?

Hi Sachin Kulkarni, If possible, Please share your code. - Kumaran R On Tue, Aug 19, 2014 at 9:07 AM, Sachin Kulkarni wrote: > Hi, > > I am using Lucene 4.6.0. > > I have been storing 5 fields for my documents in the index, namely body, > title, docname, docdate and docid. > > But whe

Re: Is housekeeping of Lucene indexes block index update but allow search ?

Hi Gaurav Thanks for the clarification. If possible, please share your NRT manager API related code example. i believe, it will help me to understand little better. - Kumaran R On Tue, Aug 5, 2014 at 12:39 PM, Gaurav gupta wrote: > Thanks Kumaran and Erik for resolving my queries. > > K

Is it right to index multi-valued fields with hash value?

Hi All During search, i find it difficult to handle every multi-valued field with different analyzers. So i believe indexing multi-valued field with hash value may solve the problem of searching with different analyzers. Any of you tried hash value in lucene index? if so, please share limit