Re: Fields with the same name?? - Was Re: Payloads and tokenizers

2008-08-20 Thread Doron Cohen
On Tue, Aug 19, 2008 at 2:15 AM, Antony Bowesman <[EMAIL PROTECTED]> wrote: > > Thanks for you time and I appreciate your valuable insight Doron. > Antony > I'm glad I could help! Doron

Re: Fields with the same name?? - Was Re: Payloads and tokenizers

2008-08-18 Thread Antony Bowesman
Doron Cohen wrote: The API definitely doesn't promise this. AFAIK implementation wise it happens to be like this but I can be wrong and plus it might change in the future. It would make me nervous to rely on this. I made some tests and it 'seems' to work, but I agree, it also makes me nervous

Re: Fields with the same name?? - Was Re: Payloads and tokenizers

2008-08-18 Thread Doron Cohen
> > payload and the other part for storing, i.e. something like this: >> >>Token token = new Token(...); >>token.setPayload(...); >>SingleTokenTokenStream ts = new SingleTokenTokenStream(token); >> >>Field f1 = new Field("f","some-stored-content",Store.YES,Index.NO); >>Field f2

Fields with the same name?? - Was Re: Payloads and tokenizers

2008-08-17 Thread Antony Bowesman
I assume you already know this but just to make sure what I meant was clear - on tokenization but still indexing just means that the entire field's text becomes a single unchanged token. I believe this is exactly what SingleTokenTokenStream can buy you - a single token, for which you can pre set a

Re: Payloads and tokenizers

2008-08-17 Thread Doron Cohen
> > Implementing payloads via Tokens explicitly prevents the use of payloads > for untokenized fields, as they only support field.stringValue(). There > seems no way to override this. I assume you already know this but just to make sure what I meant was clear - on tokenization but still indexing

Re: Payloads and tokenizers

2008-08-14 Thread Antony Bowesman
Thanks for your comments Doron. I found the earlier discussions on the dev list (21/12/06), where this issue is discussed - my use case is similar to Nadav Har'El. Implementing payloads via Tokens explicitly prevents the use of payloads for untokenized fields, as they only support field.string

Re: Payloads and tokenizers

2008-08-14 Thread Doron Cohen
IIRC first versions of patches that added payloads support had this notion of payload by field rather than by token, but later it was modified to be by token only. I have seen two code patterns to add payloads to tokens. The first one created the field text with a reserved separator/delimiter whi

Payloads and tokenizers

2008-08-13 Thread Antony Bowesman
I started playing with payloads and have been trying to work out how to get the data into the payload I have a field where I want to add the following untokenized fields A1 A2 A3 With these fields, I would like to add the payloads B1 B2 B3 Firstly, it looks like you cannot add payloads to un