Re: how do i improve Indexing and Searching performance of 2 billion documents over SolrCloud

2017-02-14 Thread Duke DAI
SSD or in-memory index Best regards, Duke If not now, when? If not me, who? On Wed, Feb 15, 2017 at 12:32 AM, Adrien Grand wrote: > This list is for users of the Lucene Java API, maybe try solr-user instead? > > Le lun. 13 févr. 2017 à 21:24, yeshwanth kumar a > écrit : > &g

Re: Hardcoded checksum mechanism in BlockTreeTermsReader

2016-12-25 Thread Duke DAI
benefit is obvious, I've added checksum in my customized IndexOutput. Thanks for your help and happy jolly holidays! Best regards, Duke If not now, when? If not me, who? On Tue, Dec 6, 2016 at 9:39 PM, Uwe Schindler wrote: > Hi, > > The checksum is also written for a second re

Re: Hardcoded checksum mechanism in BlockTreeTermsReader

2016-12-06 Thread Duke DAI
esign/API is pluggable with default way, it'll be good enough for various scenario. Best regards, Duke If not now, when? If not me, who? On Tue, Dec 6, 2016 at 6:36 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > We have learned over time not to trust the underlying s

Hardcoded checksum mechanism in BlockTreeTermsReader

2016-12-06 Thread Duke DAI
uffix, TERMS_INDEX_EXTENSION); indexIn = state.directory.openInput(indexName, state.context); CodecUtil.checkIndexHeader(indexIn, TERMS_INDEX_CODEC_NAME, version, version, state.segmentInfo.getId(), state.segmentSuffix); CodecUtil.checksumEntireFile(indexIn); Best regards, Duke If not now, whe

Re: Having some trouble running tests with custom codec

2015-10-11 Thread Duke DAI
How about add line feed for the single line? It seems I have the impression that line feed is required. Best regards, Duke If not now, when? If not me, who? On Fri, Oct 9, 2015 at 10:07 PM, Sigbjørn Lund Olsen < sigbjorn.lund.ol...@gmail.com> wrote: > As part of my master's thesis

Re: Standard highlighter returns whole document as a fragment

2015-08-11 Thread Duke DAI
ize * currentNumFrags) <-true && (textSize - offsetAtt.endOffset()) >= (fragmentSize >>> 1); <--FALSE I am pursuing input from the community instead of changing/maintaining code by myself. Best regards, Duke If not now, when? If not me, who? O

Re: bug of highlighter/SimpleSpanFragmenter, returned longer fragment than expected?

2015-08-11 Thread Duke DAI
Greetings! Any body has input on this? Best regards, Duke If not now, when? If not me, who? On Fri, Aug 7, 2015 at 10:58 AM, Duke DAI wrote: > Hi experts, > > I'm trying to reproduce a bug from Lucene side, and found something. > > In latest codeline, 5.2.1, I

Re: Re: memory cost in forceMerge(1)

2015-08-11 Thread Duke DAI
10 minute(5 minute???). The server is so common on hardware, 4G heap assigned. Best regards, Duke If not now, when? If not me, who? On Tue, Aug 11, 2015 at 7:00 PM, Phaneendra N wrote: > There could be other applications running on the machine with 24 GB memory? > Which would result

bug of highlighter/SimpleSpanFragmenter, returned longer fragment than expected?

2015-08-06 Thread Duke DAI
gment: boolean isNewFrag = offsetAtt.endOffset() >= (fragmentSize * currentNumFrags) && (textSize - offsetAtt.endOffset()) >= (fragmentSize >>> 1); At the end of text, fragmenter can't stop well and following logic also does not do the trim work. Is it possible to handle this corner case in standard highlighter code? Best regards, Duke If not now, when? If not me, who?

Inconsistency of LogMergePolicy and IWC.useCompoundFile

2014-06-19 Thread Duke DAI
small enough for 10 documents. 3. for merging, small segment will be created as compound file if selected segments size is small. Both for small segment, but behavior is inconsistent. Do you think the API contract is good enough? Best regards, Duke If not now, when? If not me, who?

Re: Retrieving values for a NumericDocValuesField [SEC=UNOFFICIAL]

2013-10-23 Thread Duke DAI
Hi Stephen, I have the same scenario with you. I verified with simple pure Lucene test, same way as Mike mentioned, performance with NumericDocValue is 10x faster than retrieving stored field. Hope you can get similar performance measurement. Best regards, Duke If not now, when? If not me, who

Re: problem found with DiskDocValuesFormat

2013-10-22 Thread Duke DAI
problem was resolved. Thank you guys for building this feature into lucene-core.jar, it dispels my worry about compatibility by using lucene-codecs.jar Best regards, Duke If not now, when? If not me, who? On Tue, Oct 22, 2013 at 12:48 AM, Michael McCandless < luc...@mikemccandless.com>

Re: problem found with DiskDocValuesFormat

2013-10-21 Thread Duke DAI
ith proper RejectedExecutionHandler(now one sub rejected, cancel and fail all subs). For simple, what's the private status of per-thread NumericDocValues instance? The private status can be re-used for different queries? Best regards, Duke If not now, when? If not me, who? On Mon, Oct 21,

Re: problem found with DiskDocValuesFormat

2013-10-21 Thread Duke DAI
cases. Do you have any idea about this? Information is enough? Thanks, Duke Best regards, Duke If not now, when? If not me, who? On Tue, Aug 13, 2013 at 4:54 PM, Duke DAI wrote: > Hi experts, > > I'm upgrading Lucene 4.4 and trying to use DocValues instead of store > fiel

Re: SPI class of type org.apache.lucene.codecs.Codec error

2013-08-21 Thread Duke DAI
e. If no, then the workaround is to have way to add all META-INF/services/xxx into your fat jar. If yes, the problem may be with classloader, classloader can't locate the SPI file within its context, I guess it's not this case. Hope this help. Best regards, Duke If not now, when? If no

Re: Question on wildcard queries, filters, scoring and TooManyClauses exception

2013-08-21 Thread Duke DAI
ery.setMaxBooleanClause at start to avoid the exception, but be careful about memory usage if you set too big one. Best regards, Duke If not now, when? If not me, who? On Fri, Aug 16, 2013 at 10:24 PM, Bill Chesky wrote: > Thanks for the reply Ian. > > > I can't explain all

Re: SPI class of type org.apache.lucene.codecs.Codec error

2013-08-20 Thread Duke DAI
The link http://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer will help. Best regards, Duke If not now, when? If not me, who? On Mon, Aug 19, 2013 at 8:48 PM, Amal Kammoun wrote: > Dear All, > > Please do you have a

Re: problem found with DiskDocValuesFormat

2013-08-13 Thread Duke DAI
the one which has problem is not a multi-thread scenario for index. Best regards, Duke If not now, when? If not me, who? On Tue, Aug 13, 2013 at 7:34 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > DiskDVFormat does not have index back compatibility between minor > rel

problem found with DiskDocValuesFormat

2013-08-13 Thread Duke DAI
ession? Or I didn't use DiskDocValuesFormat correctly? Seems no other parameters for it. Sorry that I have no pure Lucene test case yet. Hope someone shed some light on this. Best regards, Duke If not now, when? If not me, who?

Re: DocValues memory usage

2013-03-26 Thread Duke
I made the same experiment and got same result. Then I used per-field codec with DiskDocValuesFormat, it works like DirectSource in 4.0.0, but I'm not feeling confident with this usage. Anyone can say more about removing DirectSource API? On 2013-3-26, at 22:59, Peter Keegan wrote: > Inspir

Re: PayloadFunctions don't work the same since 4.1

2013-03-22 Thread Duke DAI
anted. Best regards, Duke If not now, when? If not me, who? On Fri, Mar 22, 2013 at 6:37 PM, jimtronic wrote: > Thanks for the response. I wrote some new custom payload functions to > verify > that I'm getting the value correctly and I think I am, but I did unearth > this clue.

Re: PayloadFunctions don't work the same since 4.1

2013-03-21 Thread Duke DAI
s you want). Hope this help. Best regards, Duke If not now, when? If not me, who? On Fri, Mar 22, 2013 at 12:47 PM, jimtronic wrote: > I have some custom Payload Functions that worked great up until version 4.1 > and now I can't get any of them to work correctly. In fact

Re: ArrayIndexOutOfBoundsException: -65536

2012-01-19 Thread Duke DAI
ifferent. What kind of hardware issue can cause AIOOBE? Is there any possible? Best regards, Duke If not now, when? If not me, who? On Wed, Jan 18, 2012 at 9:47 PM, Michael McCandless < luc...@mikemccandless.com> wrote: > Hmm, are you certain your RAM buffer is 3 MB? > > Is i

Re: ArrayIndexOutOfBoundsException: -65536

2012-01-18 Thread Duke DAI
ument(IndexWriter.java:1972) ... (following are business related method stack and also had been obfuscated) Best regards, Duke If not now, when? If not me, who? M 13818420095 On Mon, Jan 16, 2012 at 9:09 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > Do y

ArrayIndexOutOfBoundsException: -65536

2012-01-15 Thread Duke DAI
ce is appreciated. Best regards, Duke If not now, when? If not me, who?