[
https://issues.apache.org/jira/browse/LUCENE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599687#comment-13599687
]
Shai Erera commented on LUCENE-4795:
------------------------------------
bq. I also suspect there's a non-trivial performance hit to it (haven't yet
tested)
Mike, we did experiment with that during the overall refactoring/optimization
work. I was able to dig up the results -- we tried to simulate "column-stride
ordinals" by indexing different dimensions into different DV fields, using
different taxoWriters. I created a SingleValuedFacetField (over the old DV,
using VAR_INTS) and indexed the ordinal of the CP there. So I think kind of
simulates SortedSet when there's only one value?
The results were not good:
{noformat}
Per Dim Trunk Diff (%)
AndHighHigh 64.35 89.41 38.94%
AndHighLow 358.7 1033.32 188.07%
AndHighMed 181.22 318.36 75.68%
Fuzzy1 78.49 116.99 49.05%
Fuzzy2 72.79 104.12 43.04%
HighPhrase 36.83 43.67 18.57%
HighSloppyPhrase 3.12 3.25 4.17%
HighSpanNear 10.76 11.74 9.11%
HighTerm 40.23 65.07 61.74%
IntNRQ 109.46 114.52 4.62%
LowPhrase 60.63 72.59 19.73%
LowSloppyPhrase 66.01 82.9 25.59%
LowSpanNear 38.5 43.81 13.79%
LowTerm 187.58 378.86 101.97%
MedPhrase 103.57 153.26 47.98%
MedSloppyPhrase 70.86 89.78 26.70%
MedSpanNear 69.29 89.2 28.73%
MedTerm 120.72 237.39 96.65%
OrHighHigh 26.41 38.55 45.97%
OrHighLow 36.6 55.38 51.31%
OrHighMed 37.62 56.81 51.01%
PKLookup 250.59 301.5 20.32%
Prefix3 170.76 176.48 3.35%
Respell 101.4 136.58 34.69%
Wildcard 225.08 252.69 12.27%
{noformat}
Trunk was trunk at the time, and per-dim is the new FacetField. We suspected
that that's because when you index different dimensions into different DV
fields, you need to go over the result docs X times (where X is the number of
dimensions you asked to count; we tested 9).
bq. I think its healthier to keep as much of the facet modules existing approach
Do you have an idea how can one use the new field to index weighted facets?
I.e. {{category/Computers=0.9}} and {{category/Sceience=0.84}}? If the only way
is to use BinaryDV, is there a way we can use the ordinals from SortedSet to
put in a {{BDV 1=0.9, 2=0.84}}? Hmm, but since those ords are per-segment, we'd
need to rewrite the BDV upon merge right?
I ask because it seems that the only thing that we get from this SortedSet
approach is not having to maintain a sidecar index (which for some reason
freaks everybody), and we even lose performance. Plus, I don't see how we can
support other facet features with it. So perhaps we should focus on how to use
the search index to build a taxonomy? Maybe it's all in-memory, that's fine. If
we manage to support on-disk lookups too, even better. But if we do that, then
we should have no problems supporting all current facet features, because all
that the taxonomy index gives us is a global-ordinal (plus hierarchy
management, but I think we can do that w/ SortedSet too). We can of course
explore that in a different issue.
> Add FacetsCollector based on SortedSetDocValues
> -----------------------------------------------
>
> Key: LUCENE-4795
> URL: https://issues.apache.org/jira/browse/LUCENE-4795
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/facet
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Attachments: LUCENE-4795.patch, LUCENE-4795.patch, LUCENE-4795.patch,
> LUCENE-4795.patch, pleaseBenchmarkMe.patch
>
>
> Recently (LUCENE-4765) we added multi-valued DocValues field
> (SortedSetDocValuesField), and this can be used for faceting in Solr
> (SOLR-4490). I think we should also add support in the facet module?
> It'd be an option with different tradeoffs. Eg, it wouldn't require
> the taxonomy index, since the main index handles label/ord resolving.
> There are at least two possible approaches:
> * On every reopen, build the seg -> global ord map, and then on
> every collect, get the seg ord, map it to the global ord space,
> and increment counts. This adds cost during reopen in proportion
> to number of unique terms ...
> * On every collect, increment counts based on the seg ords, and then
> do a "merge" in the end just like distributed faceting does.
> The first approach is much easier so I built a quick prototype using
> that. The prototype does the counting, but it does NOT do the top K
> facets gathering in the end, and it doesn't "know" parent/child ord
> relationships, so there's tons more to do before this is real. I also
> was unsure how to properly integrate it since the existing classes
> seem to expect that you use a taxonomy index to resolve ords.
> I ran a quick performance test. base = trunk except I disabled the
> "compute top-K" in FacetsAccumulator to make the comparison fair; comp
> = using the prototype collector in the patch:
> {noformat}
> Task QPS base StdDev QPS comp StdDev
> Pct diff
> OrHighLow 18.79 (2.5%) 14.36 (3.3%)
> -23.6% ( -28% - -18%)
> HighTerm 21.58 (2.4%) 16.53 (3.7%)
> -23.4% ( -28% - -17%)
> OrHighMed 18.20 (2.5%) 13.99 (3.3%)
> -23.2% ( -28% - -17%)
> Prefix3 14.37 (1.5%) 11.62 (3.5%)
> -19.1% ( -23% - -14%)
> LowTerm 130.80 (1.6%) 106.95 (2.4%)
> -18.2% ( -21% - -14%)
> OrHighHigh 9.60 (2.6%) 7.88 (3.5%)
> -17.9% ( -23% - -12%)
> AndHighHigh 24.61 (0.7%) 20.74 (1.9%)
> -15.7% ( -18% - -13%)
> Fuzzy1 49.40 (2.5%) 43.48 (1.9%)
> -12.0% ( -15% - -7%)
> MedSloppyPhrase 27.06 (1.6%) 23.95 (2.3%)
> -11.5% ( -15% - -7%)
> MedTerm 51.43 (2.0%) 46.21 (2.7%)
> -10.2% ( -14% - -5%)
> IntNRQ 4.02 (1.6%) 3.63 (4.0%)
> -9.7% ( -15% - -4%)
> Wildcard 29.14 (1.5%) 26.46 (2.5%)
> -9.2% ( -13% - -5%)
> HighSloppyPhrase 0.92 (4.5%) 0.87 (5.8%)
> -5.4% ( -15% - 5%)
> MedSpanNear 29.51 (2.5%) 27.94 (2.2%)
> -5.3% ( -9% - 0%)
> HighSpanNear 3.55 (2.4%) 3.38 (2.0%)
> -4.9% ( -9% - 0%)
> AndHighMed 108.34 (0.9%) 104.55 (1.1%)
> -3.5% ( -5% - -1%)
> LowSloppyPhrase 20.50 (2.0%) 20.09 (4.2%)
> -2.0% ( -8% - 4%)
> LowPhrase 21.60 (6.0%) 21.26 (5.1%)
> -1.6% ( -11% - 10%)
> Fuzzy2 53.16 (3.9%) 52.40 (2.7%)
> -1.4% ( -7% - 5%)
> LowSpanNear 8.42 (3.2%) 8.45 (3.0%)
> 0.3% ( -5% - 6%)
> Respell 45.17 (4.3%) 45.38 (4.4%)
> 0.5% ( -7% - 9%)
> MedPhrase 113.93 (5.8%) 115.02 (4.9%)
> 1.0% ( -9% - 12%)
> AndHighLow 596.42 (2.5%) 617.12 (2.8%)
> 3.5% ( -1% - 8%)
> HighPhrase 17.30 (10.5%) 18.36 (9.1%)
> 6.2% ( -12% - 28%)
> {noformat}
> I'm impressed that this approach is only ~24% slower in the worst
> case! I think this means it's a good option to make available? Yes
> it has downsides (NRT reopen more costly, small added RAM usage,
> slightly slower faceting), but it's also simpler (no taxo index to
> manage).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
