It has never worked, though I do think the metadata has changed over time,
so the degree to which it didn't work has changed?

Mike McCandless

http://blog.mikemccandless.com

On Mon, Aug 22, 2016 at 4:41 PM, Stuart Goldberg <sgoldb...@fixflyer.com>
wrote:

> Understood, but did it used to work?
>
>
>
> Stuart M Goldberg
>
> Senior Vice President of Software Develpment
> *FIX Flyer LLC*
> http://www.FIXFlyer.com/
>
> NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
> WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS
> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>
>
> *From:* Michael McCandless [mailto:luc...@mikemccandless.com]
> *Sent:* Monday, August 22, 2016 4:38 PM
> *To:* Stuart Goldberg <sgoldb...@fixflyer.com>
> *Cc:* Lucene Users <java-user@lucene.apache.org>
>
> *Subject:* Re: Problems Refactoring a Lucene Index
>
>
>
> The design is indeed trappy, and many users have hit the situation you
> have, and we have tried to fix this before (to change IndexReader.document
> to return a different class than Document), but it didn't "take":
> https://issues.apache.org/jira/browse/LUCENE-6971
>
>
>
> Have a look at FieldInfo.java to see the metadata it records.
>
>
>
> The challenge here is Lucene's schema-less-ness.  For example, on a
> document by document basis, you can change how term vectors are indexed,
> whether a field is stored, or omits norms, or indexes only docs and not
> freqs, etc., for the same field across documents, across segments.
>
>
>
> Lucene only stores in FieldInfo what is necessary for it to read the index
> files, and does not store metadata beyond that.
>
>
>
> Patches welcome :)  We should fix this trap; it's just that doing so is
> apparently not so easy.
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
>
> On Mon, Aug 22, 2016 at 11:04 AM, Stuart Goldberg <sgoldb...@fixflyer.com>
> wrote:
>
> Thanks for the quick response.
>
>
>
> I kind of figured on my own that I had to recreate the document from
> scratch
>
>
>
> But there is something in your response that I don’t understand. You say 
> “Lucene
> only preserves the metadata it needs for each field”. What does that mean?
> In my posting I gave examples of metadata returned that is clearly the
> exact opposite of the metadata that was there when originally indexed.
>
>
>
> According to what you are saying there is metadata that is preserved
> correctly. What metadata is that?
>
>
>
> Not sure if you are just a Lucene guru (I have your Lucene in Action
> books!) or an actual author/contributor to the code, so my observation
> might not be appropriately directed at you. But it seems a questionable API
> design to return a “Document” from the index that has properties described
> by the Javadoc that give back bogus data.
>
>
>
> And what about the FieldInfo class that purports to give back field
> information. Why have such an API if the data it provides is bogus?
>
>
>
> Stuart M Goldberg
>
> Senior Vice President of Software Develpment
> *FIX Flyer LLC*
> http://www.FIXFlyer.com/
>
> NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
> WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS
> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>
>
> *From:* Michael McCandless [mailto:luc...@mikemccandless.com]
> *Sent:* Monday, August 22, 2016 10:48 AM
> *To:* Lucene Users <java-user@lucene.apache.org>; sgoldb...@fixflyer.com
> *Subject:* Re: Problems Refactoring a Lucene Index
>
>
>
> This is unfortunately "by design": Lucene makes no guarantees that the
> Document you retrieve from an IndexReader is precisely the same Document
> you had indexed.
>
>
>
> Lucene only preserves the metadata it needs for each field.
>
>
>
> Your only recourse is to create a new Document using your application
> level information about which fields are tokenized, indexed, etc.
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
>
> On Fri, Jul 8, 2016 at 12:12 PM, Stuart Goldberg <sgoldb...@fixflyer.com>
> wrote:
>
> As our software goes through its lifecycle, we sometimes have to alter
> existing Lucene indexes. The way I have done that in the past is to open
> the
> existing index for reading, read each Document, modify it and write that
> Document to a new index. At the end of the process, I delete the old index
> and rename the new index to the old name.
>
> I do not do any tokenizing and use no analyzers.
>
> I recently upgraded from Lucene 3.x to 4.10.4. Now I have the following
> problem: Suppose the existing document has 10 fields in it and there's one
> I
> have to modify. I remove that field and re-add it with the new settings.
> Then I add the Document in its entirety to the new index. I run into the
> following problems:
>
> *       I get Exceptions thrown for the fields I don't even touch. That's
> because their FieldType has 'tokenized' set to true and it fails because I
> am using no analyzers. 'tokenized' is set to true even though when I
> originally added the field to the original index I had 'tokenized' set to
> false!
>
> *       I have LongFields that come back with 'indexed' set to false even
> though in the original index they were indexed! This makes the new index
> not
> searchable on these fields and hence unusable.
>
> *       I can't even alter 'indexed' for these LongFields because for some
> reason the FieldType instance comes back frozen from the IndexReader. Once
> frozen,  you can't alter it. Even if I create a new FieldType, there is no
> way to change the FieldType of a Field
>
> It seems the returned FieldType contents are kind of random!
>
> I did see in the Javadoc of IndexReader.document() that field metadata is
> not returned and that, in fact, that they should have new kind of object
> returned like 'StoredField' so there is no pretense of there being any
> metadata.
>
> I thought perhaps I could use FieldInfos. But that class returns the same
> bogus metadata.  What then is the purpose of FieldInfos if the info is
> bogus?
>
> Am I not understanding something here? This is not very usable. What can I
> do to work around this? Is this a Lucene bug? Oversight?
>
>
>
>
>

Reply via email to