ilya-biryukov added a comment.
Here are the stats for the latest version. We now get a significant bump for
initialisms and a much lower hit in non-initialism cases.
==================================================================================================
OVERALL (excl. CROSS_NAMESPACE and
INITIALISMS)
==================================================================================================
Total measurements: 108483 (+0)
Average latency (ms): 223.570343018 (-5)
All measurements:
MRR: 69.72 (-0.31) Top-1: 60.32% (-0.40%) Top-5: 81.56% (-0.13%)
Top-100: 96.12% (-0.03%)
Full identifiers:
MRR: 97.72 (-0.48) Top-1: 96.69% (-0.88%) Top-5: 98.97% (-0.04%)
Top-100: 99.15% (+0.00%)
Filter length 0-5:
MRR: 29.15 (-0.00) 62.03 (+0.42) 71.61 (-0.50)
74.13 (-0.43) 76.52 (-0.60) 80.52 (-0.74)
Top-1: 17.52% (+0.00%) 49.21% (+0.50%) 60.51%
(-0.46%) 63.35% (-0.48%) 66.59% (-0.74%) 71.83% (-0.93%)
Top-5: 42.53% (-0.01%) 78.96% (+0.40%) 86.18%
(-0.29%) 87.76% (-0.31%) 88.92% (-0.38%) 91.36% (-0.39%)
Top-100: 84.57% (-0.01%) 96.74% (+0.15%) 98.15%
(-0.03%) 98.31% (-0.13%) 98.43% (-0.16%) 98.60% (-0.08%)
==================================================================================================
INITIALISMS
==================================================================================================
Total measurements: 16489 (+0)
Average latency (ms): 207.854690552 (-5)
All measurements:
MRR: 82.31 (+3.81) Top-1: 74.67% (+4.40%) Top-5: 91.76% (+2.93%)
Top-100: 98.41% (+0.28%)
Initialism length 2-4:
MRR: 80.43 (+2.49) 85.02 (+3.48) 86.64 (+13.05)
Top-1: 71.77% (+3.06%) 78.81% (+3.58%) 81.47%
(+15.15%)
Top-5: 91.22% (+1.60%) 92.54% (+3.27%) 92.94%
(+10.37%)
Top-100: 98.34% (+0.19%) 98.57% (+0.26%) 98.40%
(+0.86%)
==================================================================================================
DEFAULT
==================================================================================================
Total measurements: 51805 (+0)
Average latency (ms): 256.839630127 (-3)
All measurements:
MRR: 61.93 (-0.44) Top-1: 51.91% (-0.52%) Top-5: 74.55% (-0.21%)
Top-100: 92.72% (-0.07%)
Full identifiers:
MRR: 96.38 (-0.15) Top-1: 95.18% (-0.20%) Top-5: 97.96% (-0.05%)
Top-100: 98.30% (+0.00%)
Filter length 0-5:
MRR: 16.76 (-0.01) 48.89 (+0.32) 63.24 (-0.85)
67.15 (-0.58) 71.30 (-0.78) 73.47 (-1.25)
Top-1: 9.05% (+0.00%) 34.65% (+0.26%) 50.29%
(-0.82%) 54.37% (-0.67%) 59.95% (-0.89%) 63.15% (-1.56%)
Top-5: 23.55% (-0.01%) 68.04% (+0.51%) 80.76%
(-0.33%) 83.50% (-0.46%) 85.67% (-0.61%) 86.59% (-0.67%)
Top-100: 70.69% (-0.01%) 93.65% (+0.31%) 96.64%
(-0.06%) 96.96% (-0.27%) 97.20% (-0.34%) 97.35% (-0.17%)
==================================================================================================
EXPLICIT_MEMBER_ACCESS
==================================================================================================
Total measurements: 30162 (+0)
Average latency (ms): 113.366485596 (-9)
All measurements:
MRR: 68.79 (-0.18) Top-1: 59.23% (-0.28%) Top-5: 80.68% (-0.05%)
Top-100: 98.64% (+0.00%)
Full identifiers:
MRR: 98.20 (-1.43) Top-1: 96.72% (-2.71%) Top-5: 99.82% (-0.07%)
Top-100: 99.89% (+0.00%)
Filter length 0-5:
MRR: 27.83 (-0.00) 62.41 (+0.79) 69.19 (-0.27)
71.31 (-0.13) 72.93 (-0.07) 82.67 (-0.11)
Top-1: 16.48% (+0.00%) 49.23% (+1.06%) 57.90%
(-0.09%) 60.58% (+0.00%) 62.56% (-0.02%) 74.12% (-0.13%)
Top-5: 39.63% (+0.00%) 80.07% (+0.55%) 83.96%
(-0.47%) 85.22% (-0.19%) 85.93% (-0.12%) 93.20% (-0.08%)
Top-100: 94.21% (+0.00%) 99.21% (+0.02%) 99.24%
(+0.00%) 99.26% (+0.00%) 99.28% (+0.00%) 99.71% (+0.00%)
==================================================================================================
WANT_LOCAL
==================================================================================================
Total measurements: 26516 (+0)
Average latency (ms): 283.928375244 (-2)
All measurements:
MRR: 86.00 (-0.22) Top-1: 77.98% (-0.32%) Top-5: 96.24% (-0.07%)
Top-100: 99.88% (+0.00%)
Full identifiers:
MRR: 99.66 (-0.08) Top-1: 99.42% (-0.16%) Top-5: 99.93% (+0.00%)
Top-100: 99.93% (+0.00%)
Filter length 0-5:
MRR: 53.11 (+0.00) 86.95 (+0.19) 90.98 (-0.09)
91.46 (-0.46) 91.57 (-0.87) 92.96 (-0.43)
Top-1: 34.05% (+0.00%) 77.27% (+0.35%) 83.75%
(-0.15%) 84.66% (-0.68%) 85.12% (-1.31%) 87.60% (-0.57%)
Top-5: 80.17% (+0.00%) 98.79% (+0.00%) 99.47%
(+0.00%) 99.34% (-0.14%) 99.21% (-0.24%) 99.30% (-0.17%)
Top-100: 99.67% (+0.00%) 99.93% (+0.00%) 99.92%
(+0.00%) 99.92% (+0.00%) 99.91% (+0.00%) 99.90% (+0.00%)
==================================================================================================
CROSS_NAMESPACE
==================================================================================================
Total measurements: 17928 (+0)
Average latency (ms): 272.283966064 (0)
All measurements:
MRR: 33.81 (-1.01) Top-1: 26.33% (-0.90%) Top-5: 41.87% (-1.17%)
Top-100: 74.37% (-1.88%)
Full identifiers:
MRR: 80.65 (-1.27) Top-1: 73.96% (-1.22%) Top-5: 88.87% (-1.37%)
Top-100: 98.70% (+0.00%)
Filter length 0-5:
MRR: 1.21 (-0.00) 11.45 (+0.28) 25.79 (-0.45)
30.97 (-1.92) 41.27 (-2.02) 48.48 (-2.01)
Top-1: 0.59% (+0.00%) 5.79% (+0.04%) 16.00%
(-0.71%) 21.13% (-1.49%) 31.06% (-1.78%) 37.93% (-1.38%)
Top-5: 1.45% (+0.00%) 16.51% (+0.45%) 35.07%
(-0.49%) 42.18% (-2.31%) 52.22% (-2.33%) 61.26% (-2.54%)
Top-100: 9.46% (-0.04%) 63.65% (+0.89%) 87.10%
(+0.00%) 85.64% (-5.05%) 89.84% (-4.87%) 91.67% (-5.16%)
==================================================================================================
WITH EXPECTED_TYPE
==================================================================================================
Total measurements: 51958 (+0)
Average latency (ms): 224.306686401 (-6)
All measurements:
MRR: 71.24 (-0.02) Top-1: 62.54% (-0.08%) Top-5: 82.20% (+0.10%)
Top-100: 94.85% (-0.10%)
Full identifiers:
MRR: 94.81 (-1.30) Top-1: 92.51% (-2.15%) Top-5: 97.59% (-0.42%)
Top-100: 98.93% (+0.00%)
Filter length 0-5:
MRR: 35.97 (-0.00) 62.80 (+0.82) 75.03 (+0.17)
74.65 (+0.00) 76.16 (+0.83) 78.62 (-0.86)
Top-1: 24.23% (+0.00%) 52.13% (+1.04%) 65.95%
(+0.22%) 65.33% (-0.04%) 67.13% (+1.05%) 70.13% (-0.90%)
Top-5: 50.87% (+0.00%) 76.24% (+0.71%) 86.68%
(+0.22%) 86.46% (+0.31%) 87.31% (+0.50%) 89.11% (-0.85%)
Top-100: 79.22% (-0.01%) 94.35% (+0.29%) 97.77%
(+0.08%) 97.11% (-0.39%) 97.57% (-0.44%) 98.16% (-0.32%)
Repository:
rG LLVM Github Monorepo
CHANGES SINCE LAST ACTION
https://reviews.llvm.org/D59300/new/
https://reviews.llvm.org/D59300
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
