Re: Lucene/Solr and BERT

2021-05-26 Thread Julie Tibshirani
These JIRA issues contain results against two ann-benchmarks datasets. It'd be great to get your thoughts/ feedback if you have any: * Searching: https://issues.apache.org/jira/browse/LUCENE-9937 * Indexing: https://issues.apache.org/jira/browse/LUCENE-9941 The benchmarks are based on the setup he

Re: Index backwards compatibility

2021-05-26 Thread Michael Wechner
using lucene-backward-codecs-9.0.0-SNAPSHOT.jar makes it work :-) Thank you very much! But IIUC it is recommended to reindex when upgrading, right? I guess similar to what Solr is recommending https://solr.apache.org/guide/8_0/reindexing.html Am 26.05.21 um 21:26 schrieb Michael Sokolov: I

Re: Lucene/Solr and BERT

2021-05-26 Thread Alex K
Thanks Michael. IIRC, the thing that was taking so long was merging into a single segment. Is there already benchmarking code for HNSW available somewhere? I feel like I remember someone posting benchmarking results on one of the Jira tickets. Thanks, Alex On Wed, May 26, 2021 at 3:41 PM Michael

Re: Lucene/Solr and BERT

2021-05-26 Thread Michael Sokolov
This java implementation will be slower than the C implementation. I believe the algorithm is essentially the same, however this is new and there may be bugs! I (and I think Julie had similar results IIRC) measured something like 8x slower than hnswlib (using ann-benchmarks). It is also surprising

Re: Index backwards compatibility

2021-05-26 Thread Michael Sokolov
I think you need backward-codecs-9.0.0-SNAPSHOT there. It enables 9.0 to read 8.x indexes. On Wed, May 26, 2021 at 9:27 AM Michael Wechner wrote: > > Hi > > I am using Lucene 8.8.2 in production and I am currently doing some > tests using 9.0.0-SNAPSHOT, whereas I have included > lucene-backward-

Re: Lucene/Solr and BERT

2021-05-26 Thread Michael Wechner
Hi Alex Thank you very much for your feedback and the various insights! Am 26.05.21 um 04:41 schrieb Alex K: Hi Michael and others, Sorry just now getting back to you. For your three original questions: - Yes, I was referring to the Lucene90Hnsw* classes. Michael S. had a thorough response. -

Index backwards compatibility

2021-05-26 Thread Michael Wechner
Hi I am using Lucene 8.8.2 in production and I am currently doing some tests using 9.0.0-SNAPSHOT, whereas I have included lucene-backward-codecs, because in the log files it was asking me whether I have forgotten to include lucene-backward-codecs.jar         org.apache.lucene