RE: not indexing analyzed field

2010-11-28 Thread Uwe Schindler
Hi Bernd, > Nevertheless I'm in the need of the functionality to change the stored value > during analysis or tokenization or filtering (what ever works). > > Thats how it can be done in FAST FDS/ESP (full processing) compared to > Lucene/Solr (sparse processing). > > Do you have any idea, how

Re: not indexing analyzed field

2010-11-28 Thread Bernd Fehling
t any result! > > I'd suggest to read a book about Lucene/Solr first :-) > > Uwe > > - > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > >> -Original Message- >> From: Bernd

RE: not indexing analyzed field

2010-11-27 Thread Uwe Schindler
@uni-bielefeld.de] > Sent: Friday, November 26, 2010 8:23 AM > To: java-user@lucene.apache.org > Subject: Re: not indexing analyzed field > > Hi Uwe, > > my fieldType and fields are as follows: > > >class="de.ubbielefeld.solr.analysis.TextMessageDigestAna

Re: not indexing analyzed field

2010-11-26 Thread Bernd Fehling
Hi Erik, the "problem" can be described as follows: - we have a database for users - users can search and mark/store records for watching - the record marker is the unique path to the source and also the unique record id of the database - therefore we decided to sha256 the id as backreference fro

Re: not indexing analyzed field

2010-11-26 Thread Erick Erickson
Can you "define the problem away"? That is, why do you want to store it at all? If there's no value to the users in seeing the encoded value, just don't store it. You can still search on the encoded value if in that case Which is a way of saying that I don't know, off the top of my head, how y

Re: not indexing analyzed field

2010-11-26 Thread Bernd Fehling
Hi Erik, I see my problem, caused by a misunderstanding of the indexing by lucene. I guess its due to the fact that FAST Data Search has real processing pipelines. Youre right I use Solr but, as a matter of fact, in this special case I really want to change the indexed _and_ stored data. For secu

Re: not indexing analyzed field

2010-11-26 Thread Erick Erickson
So, you're using Solr, right? And have a custom analyzer? If that's the case, Uwe pointed you in the right direction and I think everything may be working fine, or at least as I'd expect. Specifying stored="true" puts a verbatim, unanalyzed copy of the data in the index. When you display a field i

Re: not indexing analyzed field

2010-11-25 Thread Bernd Fehling
--- >> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] >> Sent: Thursday, November 25, 2010 2:08 PM >> To: java-user@lucene.apache.org >> Subject: not indexing analyzed field >> >> I used KeywordAnalyzer and KeywordTokenizer as templates for a new >> ana

Re: not indexing analyzed field

2010-11-25 Thread Bernd Fehling
Hi Erik, my evidence is that I load a single document into an empty index with a field "id" and a second field "dcdocid". The field "dcdocid" has the word "foo". This goes through my analyzer and changes to MD5 string which is then "acbd18db4cc2f85cedef654fccc4a4d8". After indexing and commit a se

RE: not indexing analyzed field

2010-11-25 Thread Uwe Schindler
.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > Sent: Thursday, November 25, 2010 2:08 PM > To: java-user@lucene.apache.org > Subject: not indexing analyzed fi

Re: not indexing analyzed field

2010-11-25 Thread Erick Erickson
What is your evidence that "the result never reaches the index?" Are you sure: 1> you commit afterwards 2> you reopen the underlying reader to see 3> if you don't store the value for the field, how are you sure? 4> If you search and don't find it, did you index it? First, I'd be sure the value in

not indexing analyzed field

2010-11-25 Thread Bernd Fehling
I used KeywordAnalyzer and KeywordTokenizer as templates for a new analyzer. The analyzer works fine but the result never reaches the index. My analyzer is called in "DocInverterPerField.processFields" with "stream.incrementToken()". ... try { boolean hasMoreTokens = stream.incrementToken();