original-brownbear commented on PR #13472:
URL: https://github.com/apache/lucene/pull/13472#issuecomment-2176357069
> So for these results, I would expect to see ~4X QPS gain (ish) simply
because wall-clock elapsed time for the query got ~4X faster
That's why I added the perf numbers :) We already lose about 10% for
instructions/cycle already + need to deal with context switches that take a
rather unpredictable amount of time.
> These are not actual red-line (capacity) QPS numbers (CPU is not normally
saturated during these runs), but rather "effective QPS".
Jup that's my point in all of this to some extent :) we're not saturated but
perf and wall-clock time numbers taken together show we often burn more CPU for
no speedup whatsoever. The number of executed CPU instructions is the same
across runs pretty much, just takes much longer to get through them because we
have the stalls from context switching.
It's also not that we're just paying for context switching here. We're also
waking up threads in some cases which is mostly what shows as costly in
profiling.
> Or, if context switching really explains it all, and collect is much
faster, it would be sampled less
Yes, the CAS operations in collect get much much faster as far as I can
tell. Note that the it's not even blind CAS here. So we do read -> cas and then
later another read on the same number in many cases. That's a huge cost.
Check index says this:
```
0.00% total deletions; 33332620 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=10.0.0
id=adt5pnf860s3oddr3g83e6jp3 userData={userData=multi}
1 of 15: name=_h2 maxDoc=56399
version=10.0.0
id=adt5pnf860s3oddr3g83e6jnh
codec=Lucene99
compound=false
numFiles=17
size (MB)=34.555
diagnostics = {source=flush, lucene.version=10.0.0,
os.version=5.15.0-112-generic, os.arch=amd64, java.vendor=Ubuntu, os=Linux,
java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1, timestamp=1718716766205}
no deletions
test: open reader.........OK [took 0.045 sec]
test: check integrity.....OK [took 0.027 sec]
2 of 15: name=_h1 maxDoc=60066
version=10.0.0
id=adt5pnf860s3oddr3g83e6jlw
codec=Lucene99
compound=false
numFiles=17
size (MB)=37.469
diagnostics = {source=flush, lucene.version=10.0.0,
os.version=5.15.0-112-generic, os.arch=amd64, java.vendor=Ubuntu, os=Linux,
java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1, timestamp=1718716761312}
no deletions
test: open reader.........OK [took 0.043 sec]
test: check integrity.....OK [took 0.033 sec]
3 of 15: name=_h0 maxDoc=60065
version=10.0.0
id=adt5pnf860s3oddr3g83e6jkb
codec=Lucene99
compound=false
numFiles=17
size (MB)=38.403
diagnostics = {source=flush, lucene.version=10.0.0,
os.version=5.15.0-112-generic, os.arch=amd64, java.vendor=Ubuntu, os=Linux,
java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1, timestamp=1718716755990}
no deletions
test: open reader.........OK [took 0.043 sec]
test: check integrity.....OK [took 0.033 sec]
4 of 15: name=_gz maxDoc=60065
version=10.0.0
id=adt5pnf860s3oddr3g83e6jit
codec=Lucene99
compound=false
numFiles=17
size (MB)=44.224
diagnostics = {source=flush, lucene.version=10.0.0,
os.version=5.15.0-112-generic, os.arch=amd64, java.vendor=Ubuntu, os=Linux,
java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1, timestamp=1718716750529}
no deletions
test: open reader.........OK [took 0.046 sec]
test: check integrity.....OK [took 0.040 sec]
5 of 15: name=_gy maxDoc=60066
version=10.0.0
id=adt5pnf860s3oddr3g83e6jhb
codec=Lucene99
compound=false
numFiles=17
size (MB)=50.305
diagnostics = {source=flush, lucene.version=10.0.0,
os.version=5.15.0-112-generic, os.arch=amd64, java.vendor=Ubuntu, os=Linux,
java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1, timestamp=1718716744376}
no deletions
test: open reader.........OK [took 0.044 sec]
test: check integrity.....OK [took 0.041 sec]
6 of 15: name=_gm maxDoc=600653
version=10.0.0
id=adt5pnf860s3oddr3g83e6j15
codec=Lucene99
compound=false
numFiles=17
size (MB)=349.349
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718716660478, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.052 sec]
test: check integrity.....OK [took 0.174 sec]
7 of 15: name=_gb maxDoc=600654
version=10.0.0
id=adt5pnf860s3oddr3g83e6ikk
codec=Lucene99
compound=false
numFiles=17
size (MB)=356.852
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718716598071, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.044 sec]
test: check integrity.....OK [took 0.180 sec]
8 of 15: name=_g0 maxDoc=600654
version=10.0.0
id=adt5pnf860s3oddr3g83e6i3z
codec=Lucene99
compound=false
numFiles=17
size (MB)=375.264
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718716534641, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.044 sec]
test: check integrity.....OK [took 0.164 sec]
9 of 15: name=_fp maxDoc=600654
version=10.0.0
id=adt5pnf860s3oddr3g83e6hmz
codec=Lucene99
compound=false
numFiles=17
size (MB)=387.276
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718716464813, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.067 sec]
test: check integrity.....OK [took 0.182 sec]
10 of 15: name=_gx maxDoc=600654
version=10.0.0
id=adt5pnf860s3oddr3g83e6jh8
codec=Lucene99
compound=false
numFiles=17
size (MB)=388.355
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718716728124, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.044 sec]
test: check integrity.....OK [took 0.154 sec]
11 of 15: name=_cb maxDoc=6006538
version=10.0.0
id=adt5pnf860s3oddr3g83e6bpn
codec=Lucene99
compound=false
numFiles=17
size (MB)=3,224.495
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718715629725, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.056 sec]
test: check integrity.....OK [took 0.940 sec]
12 of 15: name=_fe maxDoc=6006538
version=10.0.0
id=adt5pnf860s3oddr3g83e6h4h
codec=Lucene99
compound=false
numFiles=17
size (MB)=3,245.9
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718716334512, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.044 sec]
test: check integrity.....OK [took 0.790 sec]
13 of 15: name=_98 maxDoc=6006538
version=10.0.0
id=adt5pnf860s3oddr3g83e663z
codec=Lucene99
compound=false
numFiles=17
size (MB)=3,248.36
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718714905816, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.046 sec]
test: check integrity.....OK [took 0.805 sec]
14 of 15: name=_65 maxDoc=6006538
version=10.0.0
id=adt5pnf860s3oddr3g83e60a5
codec=Lucene99
compound=false
numFiles=17
size (MB)=3,312.925
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718714177660, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.046 sec]
test: check integrity.....OK [took 1.095 sec]
15 of 15: name=_32 maxDoc=6006538
version=10.0.0
id=adt5pnf860s3oddr3g83e5u7k
codec=Lucene99
compound=false
numFiles=17
size (MB)=3,385.773
diagnostics = {java.runtime.version=21.0.3+9-Ubuntu-1ubuntu122.04.1,
mergeFactor=10, java.vendor=Ubuntu, os=Linux, os.version=5.15.0-112-generic,
timestamp=1718713437198, mergeMaxNumSegments=-1, lucene.version=10.0.0,
source=merge, os.arch=amd64}
no deletions
test: open reader.........OK [took 0.044 sec]
test: check integrity.....OK [took 0.955 sec]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
