Re: Lucene 5.0.0 - StringField and Sorting

[Torsten Krah]

Hi,

unfortunately i did not and no one did answered to this yet - although
javadoc states it should work it does not.
In the meantime i've used my own custom field but still i am interested
on a solution to this.

Lucene 6 still does have the same javadoc:

https://lucene.apache.org/core/6_0_0/core/org/apache/lucene/document/StringField.html

Cheers

Torsten


Am Donnerstag, den 21.04.2016, 13:13 +0530 schrieb Gimantha Bandara:
> Hi Torsten,
> 
> 
> Did you find a solution for this? I am having the same issue.. I am
> planning to create a custom Field with DocValueType.SORTED. Is there
> any other way to do that without creating a custom Field?
> 
> 
> On Fri, Mar 6, 2015 at 3:34 PM, Torsten Krah 
> wrote:
> Hi,
> 
> looking at the JavaDoc of StringField it says:
> 
> /** A field that is indexed but not tokenized: the entire
>  *  String value is indexed as a single token.  For example
>  *  this might be used for a 'country' field or an 'id'
>  *  field, or any field that you intend to use for sorting
>  *  or access through the field cache. */
> 
> So i intend to use some StringFields for sorting.
> However trying to sort on them fails with:
> 
> java.lang.IllegalStateException: unexpected docvalues type
> NONE for
> field 'NAME_KEYWORD' (expected=SORTED).
> 
> Was indexed as StringField and Store.YES.
> 
> So is the JavaDoc wrong here or is it correct and StringField
> should
> set:
> 
> TYPE.setDocValuesType(DocValuesType.SORTED);
> 
> so it would work?
> 
> kind regards
> 
> Torsten
> 
> 
> 
> 
> 
> -
> To unsubscribe, e-mail:
> java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail:
> java-user-h...@lucene.apache.org
> 
> 
> 
> 
> -- 
> Gimantha Bandara
> Software Engineer
> WSO2. Inc : http://wso2.com
> Mobile : +94714961919



-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Evaluate if a document satisfies a query

[Andres de la Peña]

Hi all,

Is it possible to evaluate if a document satisfies a query? Of course it
can be done indexing the document in a RAMIndex and querying it, but I
wonder if it is possible to do it in a more efficient way.

Thanks,

-- 
Andrés de la Peña

Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
*

Re: Evaluate if a document satisfies a query

[Ahmet Arslan]

Hi,

MemoryIndex is used for that purpose.

Please see :

https://github.com/flaxsearch/luwak

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html

http://lucene.apache.org/core/6_0_0/memory/index.html?org/apache/lucene/index/memory/MemoryIndex.html
Ahmet




On Monday, April 25, 2016 5:04 PM, Andres de la Peña  
wrote:
Hi all,

Is it possible to evaluate if a document satisfies a query? Of course it
can be done indexing the document in a RAMIndex and querying it, but I
wonder if it is possible to do it in a more efficient way.

Thanks,

-- 
Andrés de la Peña

Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
*

-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Re: Lucene 5.0.0 - StringField and Sorting

[Erick Erickson]

This works fine for me in a current Solr (5x). What version are you using?

Note that _any_ field you use for sorting _must_ be indexed="true" or, in
recent Solrs, docValues="true", just like any other field. You can't
sort on a field that
only has stored='"true" which is the only bit of the field definition
posted so far..

Best,
Erick

On Mon, Apr 25, 2016 at 6:13 AM, Torsten Krah  wrote:
> Hi,
>
> unfortunately i did not and no one did answered to this yet - although
> javadoc states it should work it does not.
> In the meantime i've used my own custom field but still i am interested
> on a solution to this.
>
> Lucene 6 still does have the same javadoc:
>
> https://lucene.apache.org/core/6_0_0/core/org/apache/lucene/document/StringField.html
>
> Cheers
>
> Torsten
>
>
> Am Donnerstag, den 21.04.2016, 13:13 +0530 schrieb Gimantha Bandara:
>> Hi Torsten,
>>
>>
>> Did you find a solution for this? I am having the same issue.. I am
>> planning to create a custom Field with DocValueType.SORTED. Is there
>> any other way to do that without creating a custom Field?
>>
>>
>> On Fri, Mar 6, 2015 at 3:34 PM, Torsten Krah 
>> wrote:
>> Hi,
>>
>> looking at the JavaDoc of StringField it says:
>>
>> /** A field that is indexed but not tokenized: the entire
>>  *  String value is indexed as a single token.  For example
>>  *  this might be used for a 'country' field or an 'id'
>>  *  field, or any field that you intend to use for sorting
>>  *  or access through the field cache. */
>>
>> So i intend to use some StringFields for sorting.
>> However trying to sort on them fails with:
>>
>> java.lang.IllegalStateException: unexpected docvalues type
>> NONE for
>> field 'NAME_KEYWORD' (expected=SORTED).
>>
>> Was indexed as StringField and Store.YES.
>>
>> So is the JavaDoc wrong here or is it correct and StringField
>> should
>> set:
>>
>> TYPE.setDocValuesType(DocValuesType.SORTED);
>>
>> so it would work?
>>
>> kind regards
>>
>> Torsten
>>
>>
>>
>>
>>
>> -
>> To unsubscribe, e-mail:
>> java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail:
>> java-user-h...@lucene.apache.org
>>
>>
>>
>>
>> --
>> Gimantha Bandara
>> Software Engineer
>> WSO2. Inc : http://wso2.com
>> Mobile : +94714961919
>
>
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Re: Lucene 5.0.0 - StringField and Sorting

[Michael McCandless]

The Lucene level javadocs are definitely stale ... I'll fix.

You should separately add a SortedDocValuesField if you also need to sort
on this field.

Mike McCandless

http://blog.mikemccandless.com

On Fri, Mar 6, 2015 at 5:04 AM, Torsten Krah  wrote:

> Hi,
>
> looking at the JavaDoc of StringField it says:
>
> /** A field that is indexed but not tokenized: the entire
>  *  String value is indexed as a single token.  For example
>  *  this might be used for a 'country' field or an 'id'
>  *  field, or any field that you intend to use for sorting
>  *  or access through the field cache. */
>
> So i intend to use some StringFields for sorting.
> However trying to sort on them fails with:
>
> java.lang.IllegalStateException: unexpected docvalues type NONE for
> field 'NAME_KEYWORD' (expected=SORTED).
>
> Was indexed as StringField and Store.YES.
>
> So is the JavaDoc wrong here or is it correct and StringField should
> set:
>
> TYPE.setDocValuesType(DocValuesType.SORTED);
>
> so it would work?
>
> kind regards
>
> Torsten
>
>
>
>
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>

Re: Evaluate if a document satisfies a query

[Andres de la Peña]

Hi Ahmet,

MemoryIndex is exactly what I was looking for.

Thanks a lot for the quick response :)

2016-04-25 15:40 GMT+01:00 Ahmet Arslan :

> Hi,
>
> MemoryIndex is used for that purpose.
>
> Please see :
>
> https://github.com/flaxsearch/luwak
>
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-percolate.html
>
>
> http://lucene.apache.org/core/6_0_0/memory/index.html?org/apache/lucene/index/memory/MemoryIndex.html
> Ahmet
>
>
>
>
> On Monday, April 25, 2016 5:04 PM, Andres de la Peña <
> adelap...@stratio.com> wrote:
> Hi all,
>
> Is it possible to evaluate if a document satisfies a query? Of course it
> can be done indexing the document in a RAMIndex and querying it, but I
> wonder if it is possible to do it in a more efficient way.
>
> Thanks,
>
> --
> Andrés de la Peña
>
> Vía de las dos Castillas, 33, Ática 4, 3ª Planta
> 28224 Pozuelo de Alarcón, Madrid
> Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
> *
>
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>


-- 
Andrés de la Peña

Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
*

Re: Lucene 5.0.0 - StringField and Sorting

[Gimantha Bandara]

Yep.. adding a SortedDocValuesField did work for me! thanks..

On Mon, Apr 25, 2016 at 8:39 PM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> The Lucene level javadocs are definitely stale ... I'll fix.
>
> You should separately add a SortedDocValuesField if you also need to sort
> on this field.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Fri, Mar 6, 2015 at 5:04 AM, Torsten Krah  wrote:
>
> > Hi,
> >
> > looking at the JavaDoc of StringField it says:
> >
> > /** A field that is indexed but not tokenized: the entire
> >  *  String value is indexed as a single token.  For example
> >  *  this might be used for a 'country' field or an 'id'
> >  *  field, or any field that you intend to use for sorting
> >  *  or access through the field cache. */
> >
> > So i intend to use some StringFields for sorting.
> > However trying to sort on them fails with:
> >
> > java.lang.IllegalStateException: unexpected docvalues type NONE for
> > field 'NAME_KEYWORD' (expected=SORTED).
> >
> > Was indexed as StringField and Store.YES.
> >
> > So is the JavaDoc wrong here or is it correct and StringField should
> > set:
> >
> > TYPE.setDocValuesType(DocValuesType.SORTED);
> >
> > so it would work?
> >
> > kind regards
> >
> > Torsten
> >
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: java-user-h...@lucene.apache.org
> >
> >
>



-- 
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919

Re: Lucene 5.0.0 - StringField and Sorting

[Torsten Krah]

Hi Erick,

i guess you've muddled the lists - this is lucenes one, not solr. I know
how to define it in solr but that wasn't the question as i am using pure
lucene and it did not work as expected from the javadocs there.

Cheers

Torsten 


-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Re: Lucene 5.0.0 - StringField and Sorting

[Torsten Krah]

Thanks for going to fix the stale docs.

Torsten

Am Montag, den 25.04.2016, 11:09 -0400 schrieb Michael McCandless:
> The Lucene level javadocs are definitely stale ... I'll fix.
> 
> 
> You should separately add a SortedDocValuesField if you also need to
> sort on this field.
> 
> Mike McCandless



-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Need guidance to start contributing to Lucene.

[Prateek Singhal]

Hi Folks,

I am very new to Lucene. Used it in a couple of projects but haven't really
got into the insights of how it works and
*never Deep Dived into its code-base*
Just for a head-start I want some guidance from *all the smart folks* on
this group regarding *which knowledge source will be a good start-point for
me.*

I understand Java and OOPs concept well so language is not a hinderance.
Just want to get familiar with Lucene's inner functionality and also *looking
forward to contribute some useful stuff*

Thanks in advance.

Regards,
Prateek Singhal

Re: Need guidance to start contributing to Lucene.

[Taher Galal]

I would say Start with reading lucene in action
On 25 Apr 2016 19:39, "Prateek Singhal"  wrote:

> Hi Folks,
>
> I am very new to Lucene. Used it in a couple of projects but haven't really
> got into the insights of how it works and
> *never Deep Dived into its code-base*
> Just for a head-start I want some guidance from *all the smart folks* on
> this group regarding *which knowledge source will be a good start-point for
> me.*
>
> I understand Java and OOPs concept well so language is not a hinderance.
> Just want to get familiar with Lucene's inner functionality and also
> *looking
> forward to contribute some useful stuff*
>
> Thanks in advance.
>
> Regards,
> Prateek Singhal
>

Re: Need guidance to start contributing to Lucene.

[Prateek Singhal]

Okay. Thanks a lot for your suggestion.

When will you think is the right time to start reading the *lucene's
official apache docs ?*

Thanks.

On Mon, Apr 25, 2016 at 11:11 PM, Taher Galal 
wrote:

> I would say Start with reading lucene in action
> On 25 Apr 2016 19:39, "Prateek Singhal"  wrote:
>
> > Hi Folks,
> >
> > I am very new to Lucene. Used it in a couple of projects but haven't
> really
> > got into the insights of how it works and
> > *never Deep Dived into its code-base*
> > Just for a head-start I want some guidance from *all the smart folks* on
> > this group regarding *which knowledge source will be a good start-point
> for
> > me.*
> >
> > I understand Java and OOPs concept well so language is not a hinderance.
> > Just want to get familiar with Lucene's inner functionality and also
> > *looking
> > forward to contribute some useful stuff*
> >
> > Thanks in advance.
> >
> > Regards,
> > Prateek Singhal
> >
>



-- 
Regards,
Prateek Singhal
Software Development Engineer @ Amazon.com

"Believe in yourself and you can do unbelievable things."

Re: Need guidance to start contributing to Lucene.

[Taher Galal]

Depending on what you are looking for everything you need is in the docs.
The book just has a good structure a problem you will face is the book is
not up to date with the latest API changes but the main concepts are there
so if you want to read the Docs you can always do that of you want some
insight how things work in a book form then you have lucene in action but
take the functions there with a pinch of salt as the book is a bit old that
is all.
On 25 Apr 2016 19:47, "Prateek Singhal"  wrote:

> Okay. Thanks a lot for your suggestion.
>
> When will you think is the right time to start reading the *lucene's
> official apache docs ?*
>
> Thanks.
>
> On Mon, Apr 25, 2016 at 11:11 PM, Taher Galal 
> wrote:
>
> > I would say Start with reading lucene in action
> > On 25 Apr 2016 19:39, "Prateek Singhal"  wrote:
> >
> > > Hi Folks,
> > >
> > > I am very new to Lucene. Used it in a couple of projects but haven't
> > really
> > > got into the insights of how it works and
> > > *never Deep Dived into its code-base*
> > > Just for a head-start I want some guidance from *all the smart folks*
> on
> > > this group regarding *which knowledge source will be a good start-point
> > for
> > > me.*
> > >
> > > I understand Java and OOPs concept well so language is not a
> hinderance.
> > > Just want to get familiar with Lucene's inner functionality and also
> > > *looking
> > > forward to contribute some useful stuff*
> > >
> > > Thanks in advance.
> > >
> > > Regards,
> > > Prateek Singhal
> > >
> >
>
>
>
> --
> Regards,
> Prateek Singhal
> Software Development Engineer @ Amazon.com
>
> "Believe in yourself and you can do unbelievable things."
>

RE: Lucene 5.0.0 - StringField and Sorting

[Uwe Schindler]

In Lucene you have to use the new DocValues fields (available since 4.0) to 
sort on.

For backwards compatibility, you can still use the older "uninversion" hack 
(also known as "Field Cache"). In Lucene 5 and 6 this part was removed from 
Lucene Core. To sort, you have 2 options:

- Reindex your stuff with DocValues fields enabled. This new field type is 
optimized for random access as used by sorting. This is the recommended 
approach.

- Alternatively you can use UninvertingReader from the misc module and wrap 
your index reader. This "emulates" the DocValues APIs on indexes without 
DocValues. Internally it uses the old FieldCache code to do this. This has the 
same problems as earlier Lucene versions: Heavy heap usage and slow uninversion 
of field values. To use it you must wrap every segment of your index separately 
(e.g. by implementing FilterDirectoryReader) using 
https://lucene.apache.org/core/5_0_0/misc/org/apache/lucene/uninverting/UninvertingReader.html

Erick: Solr has UninvertingReader used by default in SolrIndexSearcher.

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de

> -Original Message-
> From: Torsten Krah [mailto:krah...@gmail.com]
> Sent: Monday, April 25, 2016 6:25 PM
> To: Erick Erickson 
> Cc: java-user 
> Subject: Re: Lucene 5.0.0 - StringField and Sorting
> 
> Hi Erick,
> 
> i guess you've muddled the lists - this is lucenes one, not solr. I know
> how to define it in solr but that wasn't the question as i am using pure
> lucene and it did not work as expected from the javadocs there.
> 
> Cheers
> 
> Torsten
> 
> 
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org


-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

RE: Lucene 5.0.0 - StringField and Sorting

[Uwe Schindler]

Hi,

> - Alternatively you can use UninvertingReader from the misc module and
> wrap your index reader. This "emulates" the DocValues APIs on indexes
> without DocValues. Internally it uses the old FieldCache code to do this. This
> has the same problems as earlier Lucene versions: Heavy heap usage and
> slow uninversion of field values. To use it you must wrap every segment of
> your index separately (e.g. by implementing FilterDirectoryReader) using
> https://lucene.apache.org/core/5_0_0/misc/org/apache/lucene/uninverting
> /UninvertingReader.html

Of course there is also the static factory 
UninvertingReader.wrap(DirectoryReader) to wrap any DirectoryReader for 
convenience.

Uwe

> > -Original Message-
> > From: Torsten Krah [mailto:krah...@gmail.com]
> > Sent: Monday, April 25, 2016 6:25 PM
> > To: Erick Erickson 
> > Cc: java-user 
> > Subject: Re: Lucene 5.0.0 - StringField and Sorting
> >
> > Hi Erick,
> >
> > i guess you've muddled the lists - this is lucenes one, not solr. I know
> > how to define it in solr but that wasn't the question as i am using pure
> > lucene and it did not work as expected from the javadocs there.
> >
> > Cheers
> >
> > Torsten
> >
> >
> > -
> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: java-user-h...@lucene.apache.org
> 
> 
> -
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org


-
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

upgrading lucene 4 to 6

[Jamie]

Hi Uwe

Looking at the magnitude of the API changes from Lucene 4 to 6, I don't 
think we'll ever be able to upgrade. It seems the API has been modified 
to a large extent.


Is there any chance that the bug fixes will back ported to the 4.0 branch?

Jamie