Hello Fred, riak-admin aae-status: ================================== Exchanges ================================== Index Last (ago) All (ago) ------------------------------------------------------------------------------- 0 12.9 min 14.2 min 57089907708238395242331438777979805455309864960 9.1 min 30.6 min 114179815416476790484662877555959610910619729920 7.6 min 29.1 min 171269723124715185726994316333939416365929594880 6.1 min 27.6 min 228359630832953580969325755111919221821239459840 4.6 min 26.1 min 285449538541191976211657193889899027276549324800 3.1 min 24.6 min 342539446249430371453988632667878832731859189760 1.6 min 23.1 min 399629353957668766696320071445858638187169054720 6.4 s 21.6 min 456719261665907161938651510223838443642478919680 56.5 s 36.4 min 513809169374145557180982949001818249097788784640 18.6 min 38.9 min 570899077082383952423314387779798054553098649600 17.1 min 37.4 min 627988984790622347665645826557777860008408514560 15.6 min 35.9 min 685078892498860742907977265335757665463718379520 14.1 min 34.4 min 742168800207099138150308704113737470919028244480 12.6 min 32.9 min 799258707915337533392640142891717276374338109440 11.1 min 31.4 min 856348615623575928634971581669697081829647974400 9.6 min 29.9 min 913438523331814323877303020447676887284957839360 8.1 min 28.4 min 970528431040052719119634459225656692740267704320 6.6 min 26.9 min 1027618338748291114361965898003636498195577569280 5.1 min 25.4 min 1084708246456529509604297336781616303650887434240 3.6 min 23.9 min 1141798154164767904846628775559596109106197299200 2.1 min 22.4 min 1198888061873006300088960214337575914561507164160 33.2 s 20.9 min 1255977969581244695331291653115555720016817029120 15.1 min 36.6 min 1313067877289483090573623091893535525472126894080 13.6 min 35.1 min 1370157784997721485815954530671515330927436759040 12.1 min 33.6 min 1427247692705959881058285969449495136382746624000 10.6 min 32.1 min
================================ Entropy Trees ================================ Index Built (ago) ------------------------------------------------------------------------------- 0 6.8 d 57089907708238395242331438777979805455309864960 14.5 hr 114179815416476790484662877555959610910619729920 2.5 hr 171269723124715185726994316333939416365929594880 6.5 hr 228359630832953580969325755111919221821239459840 5.5 hr 285449538541191976211657193889899027276549324800 7.0 d 342539446249430371453988632667878832731859189760 1.4 d 399629353957668766696320071445858638187169054720 6.6 d 456719261665907161938651510223838443642478919680 1.5 hr 513809169374145557180982949001818249097788784640 4.5 hr 570899077082383952423314387779798054553098649600 1.4 d 627988984790622347665645826557777860008408514560 1.4 d 685078892498860742907977265335757665463718379520 6.7 d 742168800207099138150308704113737470919028244480 6.8 d 799258707915337533392640142891717276374338109440 6.9 d 856348615623575928634971581669697081829647974400 1.3 d 913438523331814323877303020447676887284957839360 3.5 hr 970528431040052719119634459225656692740267704320 6.9 d 1027618338748291114361965898003636498195577569280 7.5 hr 1084708246456529509604297336781616303650887434240 6.9 d 1141798154164767904846628775559596109106197299200 6.7 d 1198888061873006300088960214337575914561507164160 1.4 d 1255977969581244695331291653115555720016817029120 13.5 hr 1313067877289483090573623091893535525472126894080 30.0 min 1370157784997721485815954530671515330927436759040 1.3 d 1427247692705959881058285969449495136382746624000 1.2 d ================================ Keys Repaired ================================ Index Last Mean Max ------------------------------------------------------------------------------- 0 0 0 0 57089907708238395242331438777979805455309864960 0 0 0 114179815416476790484662877555959610910619729920 0 0 0 171269723124715185726994316333939416365929594880 0 0 0 228359630832953580969325755111919221821239459840 0 0 0 285449538541191976211657193889899027276549324800 0 0 0 342539446249430371453988632667878832731859189760 0 0 1 399629353957668766696320071445858638187169054720 0 0 0 456719261665907161938651510223838443642478919680 0 0 1 513809169374145557180982949001818249097788784640 0 0 0 570899077082383952423314387779798054553098649600 0 0 0 627988984790622347665645826557777860008408514560 0 0 0 685078892498860742907977265335757665463718379520 0 0 0 742168800207099138150308704113737470919028244480 0 0 0 799258707915337533392640142891717276374338109440 0 0 0 856348615623575928634971581669697081829647974400 0 0 0 913438523331814323877303020447676887284957839360 0 0 0 970528431040052719119634459225656692740267704320 0 0 0 1027618338748291114361965898003636498195577569280 0 0 0 1084708246456529509604297336781616303650887434240 0 0 0 1141798154164767904846628775559596109106197299200 0 0 0 1198888061873006300088960214337575914561507164160 0 0 0 1255977969581244695331291653115555720016817029120 0 0 0 1313067877289483090573623091893535525472126894080 0 0 0 1370157784997721485815954530671515330927436759040 0 0 0 1427247692705959881058285969449495136382746624000 0 0 0 riak-admin search aae-status ================================== Exchanges ================================== Index Last (ago) All (ago) ------------------------------------------------------------------------------- 0 5.1 min 5.6 min 57089907708238395242331438777979805455309864960 4.4 min 4.9 min 114179815416476790484662877555959610910619729920 3.6 min 4.1 min 171269723124715185726994316333939416365929594880 2.9 min 3.4 min 228359630832953580969325755111919221821239459840 2.1 min 2.6 min 285449538541191976211657193889899027276549324800 1.4 min 1.9 min 342539446249430371453988632667878832731859189760 36.5 s 1.1 min 399629353957668766696320071445858638187169054720 6.5 s 19.4 min 456719261665907161938651510223838443642478919680 18.6 min 19.1 min 513809169374145557180982949001818249097788784640 17.9 min 18.4 min 570899077082383952423314387779798054553098649600 17.1 min 17.6 min 627988984790622347665645826557777860008408514560 16.4 min 16.9 min 685078892498860742907977265335757665463718379520 15.6 min 16.1 min 742168800207099138150308704113737470919028244480 14.9 min 15.4 min 799258707915337533392640142891717276374338109440 14.1 min 14.6 min 856348615623575928634971581669697081829647974400 13.4 min 13.9 min 913438523331814323877303020447676887284957839360 12.6 min 13.1 min 970528431040052719119634459225656692740267704320 11.9 min 12.4 min 1027618338748291114361965898003636498195577569280 11.1 min 11.6 min 1084708246456529509604297336781616303650887434240 10.4 min 10.9 min 1141798154164767904846628775559596109106197299200 9.6 min 10.1 min 1198888061873006300088960214337575914561507164160 8.9 min 9.4 min 1255977969581244695331291653115555720016817029120 8.1 min 8.6 min 1313067877289483090573623091893535525472126894080 7.4 min 7.9 min 1370157784997721485815954530671515330927436759040 6.6 min 7.1 min 1427247692705959881058285969449495136382746624000 5.9 min 6.4 min ================================ Entropy Trees ================================ Index Built (ago) ------------------------------------------------------------------------------- 0 1.5 d 57089907708238395242331438777979805455309864960 1.2 d 114179815416476790484662877555959610910619729920 20.5 hr 171269723124715185726994316333939416365929594880 1.4 d 228359630832953580969325755111919221821239459840 22.5 hr 285449538541191976211657193889899027276549324800 19.5 hr 342539446249430371453988632667878832731859189760 1.6 d 399629353957668766696320071445858638187169054720 1.1 d 456719261665907161938651510223838443642478919680 1.2 d 513809169374145557180982949001818249097788784640 16.5 hr 570899077082383952423314387779798054553098649600 15.5 hr 627988984790622347665645826557777860008408514560 1.5 d 685078892498860742907977265335757665463718379520 1.4 d 742168800207099138150308704113737470919028244480 21.5 hr 799258707915337533392640142891717276374338109440 18.5 hr 856348615623575928634971581669697081829647974400 1.6 d 913438523331814323877303020447676887284957839360 1.3 d 970528431040052719119634459225656692740267704320 1.1 d 1027618338748291114361965898003636498195577569280 14.5 hr 1084708246456529509604297336781616303650887434240 1.3 d 1141798154164767904846628775559596109106197299200 1.4 d 1198888061873006300088960214337575914561507164160 13.5 hr 1255977969581244695331291653115555720016817029120 1.0 d 1313067877289483090573623091893535525472126894080 23.5 hr 1370157784997721485815954530671515330927436759040 1.1 d 1427247692705959881058285969449495136382746624000 17.5 hr ================================ Keys Repaired ================================ Index Last Mean Max ------------------------------------------------------------------------------- 0 0 0 0 57089907708238395242331438777979805455309864960 0 0 0 114179815416476790484662877555959610910619729920 0 0 0 171269723124715185726994316333939416365929594880 0 0 0 228359630832953580969325755111919221821239459840 0 0 0 285449538541191976211657193889899027276549324800 0 0 0 342539446249430371453988632667878832731859189760 0 0 0 399629353957668766696320071445858638187169054720 0 0 1 456719261665907161938651510223838443642478919680 0 0 0 513809169374145557180982949001818249097788784640 0 0 0 570899077082383952423314387779798054553098649600 0 0 0 627988984790622347665645826557777860008408514560 0 0 0 685078892498860742907977265335757665463718379520 0 0 0 742168800207099138150308704113737470919028244480 0 0 0 799258707915337533392640142891717276374338109440 0 0 0 856348615623575928634971581669697081829647974400 0 0 0 913438523331814323877303020447676887284957839360 0 0 0 970528431040052719119634459225656692740267704320 0 0 0 1027618338748291114361965898003636498195577569280 0 0 0 1084708246456529509604297336781616303650887434240 0 0 1 1141798154164767904846628775559596109106197299200 1 0 1 1198888061873006300088960214337575914561507164160 0 0 0 1255977969581244695331291653115555720016817029120 0 0 0 1313067877289483090573623091893535525472126894080 0 0 0 1370157784997721485815954530671515330927436759040 0 0 0 1427247692705959881058285969449495136382746624000 0 0 0 I have tried to read the data from RiakKV with r=3, it successfully retrieves the data of this entry(read repair maybe fail). But to use Riak Search Query again, it still returns results in discrepancy. I am curious that why just map to all nodes then once the data in the range got, then return and reduce. I found that this entry in solr via solr-webapp, it is located in 2 nodes of 10. While some other entry located in 3 nodes of 10. Best regards, Witeman > On 24 Feb 2017, at 11:43 AM, Fred Dushin <fdus...@basho.com> wrote: > > Hello Witeman, > > What you are seeing with your two queries is the result of two different > coverage plans, querying different parts of the cluster. Riak Search > translates coverage plans to Solr sharded queries, and will periodically > change the coverage plan, so as to more evenly distribute queries across the > cluster. So what you are seeing is effectively two different sharded > queries, hitting different solr instances in your cluster. You are seeing > inconsistent search results, which suggests there is a discrepancy between > what is stored in Solr and what is stored in Riak. Generally speaking, AAE > should detect these repairs and correct them over time. > > Can you please send the output of the following two commands: > > riak-admin aae-status > riak-admin search aae-status > > That will tell you something about the behavior of the underlying KV and > Yokozuna AAE subsystems. > > Out of curiosity, can you you read the "2772439" key in the "data_201702" > bucket in Riak with quorum equal to all? If you do that read, does that > affect the behavior of your query you posted? (I am wondering if you trigger > read repair that will repair the entry in Solr) > > Thanks, > -Fred > >> On Feb 23, 2017, at 10:11 PM, Witeman Zheng <witema...@gmail.com >> <mailto:witema...@gmail.com>> wrote: >> >> Hi, >> >> >> I am having a 10 nodes of RiakKV 2.2.0, and turn on Riak Search(Yokozuna). >> Having about 3million records in one bucket with index, every record has >> about 1k size. >> >> Then when it is triggered a Yokozuna query for one specific id, sometimes >> return the record, sometimes return NOT FOUND, it is very weird. >> >> FOUND case: >> wt=json&q=*:*&rows=1000&start=0&sort=collect_id_l%20asc&fq=agent_uid_i:1191&fq=id_l:2772439&indent=true" >> { >> "responseHeader":{ >> "status":0, >> "QTime":57, >> "params":{ >> "10.100.205.80:8093":"_yz_pn:230 OR _yz_pn:200 OR _yz_pn:170 OR >> _yz_pn:140 OR _yz_pn:110 OR _yz_pn:80 OR _yz_pn:50 OR _yz_pn:20", >> "10.100.205.79:8093":"_yz_pn:239 OR _yz_pn:209 OR _yz_pn:179 OR >> _yz_pn:149 OR _yz_pn:119 OR _yz_pn:89 OR _yz_pn:59 OR _yz_pn:29", >> "indent":"true", >> "10.100.205.73:8093":"_yz_pn:233 OR _yz_pn:203 OR _yz_pn:173 OR >> _yz_pn:143 OR _yz_pn:113 OR _yz_pn:83 OR _yz_pn:53 OR _yz_pn:23", >> "start":"0", >> "sort":"collect_id_l asc", >> "fq":["agent_uid_i:1191", >> "id_l:2772439"], >> "rows":"1000", >> "10.100.205.76:8093":"_yz_pn:236 OR _yz_pn:206 OR _yz_pn:176 OR >> _yz_pn:146 OR _yz_pn:116 OR _yz_pn:86 OR _yz_pn:56 OR _yz_pn:26", >> "q":"*:*", >> >> "shards":"10.100.205.71:8093/internal_solr/game_data_records_index,10.100.205.72:8093/internal_solr/game_data_records_index,10.100.205.73:8093/internal_solr/game_data_records_index,10.100.205.74:8093/internal_solr/game_data_records_index,10.100.205.75:8093/internal_solr/game_data_records_index,10.100.205.76:8093/internal_solr/game_data_records_index,10.100.205.77:8093/internal_solr/game_data_records_index,10.100.205.78:8093/internal_solr/game_data_records_index,10.100.205.79:8093/internal_solr/game_data_records_index,10.100.205.80:8093/internal_solr/game_data_records_index", >> "10.100.205.71:8093":"_yz_pn:251 OR _yz_pn:221 OR _yz_pn:191 OR >> _yz_pn:161 OR _yz_pn:131 OR _yz_pn:101 OR _yz_pn:71 OR _yz_pn:41 OR >> _yz_pn:11", >> "10.100.205.74:8093":"_yz_pn:224 OR _yz_pn:194 OR _yz_pn:164 OR >> _yz_pn:134 OR _yz_pn:104 OR _yz_pn:74 OR _yz_pn:44 OR _yz_pn:14", >> "10.100.205.77:8093":"_yz_pn:227 OR _yz_pn:197 OR _yz_pn:167 OR >> _yz_pn:137 OR _yz_pn:107 OR _yz_pn:77 OR _yz_pn:47 OR _yz_pn:17", >> "10.100.205.78:8093":"_yz_pn:248 OR _yz_pn:218 OR _yz_pn:188 OR >> _yz_pn:158 OR _yz_pn:128 OR _yz_pn:98 OR _yz_pn:68 OR _yz_pn:38 OR _yz_pn:8", >> "10.100.205.75:8093":"_yz_pn:255 OR _yz_pn:245 OR _yz_pn:215 OR >> _yz_pn:185 OR _yz_pn:155 OR _yz_pn:125 OR _yz_pn:95 OR _yz_pn:65 OR >> _yz_pn:35 OR _yz_pn:5", >> "wt":"json", >> "10.100.205.72:8093":"(_yz_pn:252 AND (_yz_fpn:252)) OR _yz_pn:242 OR >> _yz_pn:212 OR _yz_pn:182 OR _yz_pn:152 OR _yz_pn:122 OR _yz_pn:92 OR >> _yz_pn:62 OR _yz_pn:32 OR _yz_pn:2"}}, >> "response":{"numFound":1,"start":0,"docs":[ >> { >> “some_field_1_l":0, >> "some_field_2_l":0, >> "some_field_3_s":"[]", >> "some_field_4_s":"[]", >> "some_field_5_l":0, >> "some_field_6_l":0, >> "some_field_7_s":"[]", >> >> "some_field_8_s":"[[1,0],[2,0],[3,0],[4,0],[5,0],[6,0],[7,0],[8,0],[9,0],[10,0],[11,0],[12,0],[13,0],[14,0],[15,0],[16,0],[17,0],[18,0],[19,0],[20,0]]", >> "collect_id_l":2765608, >> "some_field_9_i":1191, >> "some_field_10_i":1191, >> "some_field_11_i":1, >> "some_field_12_l":0, >> "some_field_13_l":2000, >> "some_field_14_l":764846, >> "some_field_15_l":766846, >> "some_field_16_i":57, >> "some_field_17_i":1, >> "some_field_18_s":"UTC_-4", >> "some_field_19_i":1487822270, >> "some_field_20_s":"61.221.181.7", >> "some_field_21_l":2869104, >> "some_field_22_s":"[1,4,10,4,5,3,5,6,8,4,3,2,2,1,10]", >> "some_field_23_i":20, >> "some_field_24_i":1, >> "some_field_25_i":1, >> "agent_uid_i":1191, >> "some_field_26_l":2772439, >> "some_field_27_i":100, >> "_yz_id":"1*default*data_201702*2772439*203", >> "_yz_rk":"2772439", >> "_yz_rt":"default", >> "_yz_rb":"data_201702"}] >> >> }} >> >> >> >> >> NOTFOUND case: >> >> wt=json&q=*:*&rows=1000&start=0&sort=collect_id_l%20asc&fq=agent_uid_i:1191&fq=id_l:2772439&indent=true" >> { >> "responseHeader":{ >> "status":0, >> "QTime":62, >> "params":{ >> "10.100.205.80:8093":"_yz_pn:240 OR _yz_pn:210 OR _yz_pn:180 OR >> _yz_pn:150 OR _yz_pn:120 OR _yz_pn:90 OR _yz_pn:60 OR _yz_pn:30", >> "10.100.205.79:8093":"_yz_pn:249 OR _yz_pn:219 OR _yz_pn:189 OR >> _yz_pn:159 OR _yz_pn:129 OR _yz_pn:99 OR _yz_pn:69 OR _yz_pn:39 OR _yz_pn:9", >> "indent":"true", >> "10.100.205.73:8093":"(_yz_pn:253 AND (_yz_fpn:253)) OR _yz_pn:243 OR >> _yz_pn:213 OR _yz_pn:183 OR _yz_pn:153 OR _yz_pn:123 OR _yz_pn:93 OR >> _yz_pn:63 OR _yz_pn:33 OR _yz_pn:3", >> "start":"0", >> "sort":"collect_id_l asc", >> "fq":["agent_uid_i:1191", >> "id_l:2772439"], >> "rows":"1000", >> "10.100.205.76:8093":"_yz_pn:256 OR _yz_pn:246 OR _yz_pn:216 OR >> _yz_pn:186 OR _yz_pn:156 OR _yz_pn:126 OR _yz_pn:96 OR _yz_pn:66 OR >> _yz_pn:36 OR _yz_pn:6", >> "q":"*:*", >> >> "shards":"10.100.205.71:8093/internal_solr/game_data_records_index,10.100.205.72:8093/internal_solr/game_data_records_index,10.100.205.73:8093/internal_solr/game_data_records_index,10.100.205.74:8093/internal_solr/game_data_records_index,10.100.205.75:8093/internal_solr/game_data_records_index,10.100.205.76:8093/internal_solr/game_data_records_index,10.100.205.77:8093/internal_solr/game_data_records_index,10.100.205.78:8093/internal_solr/game_data_records_index,10.100.205.79:8093/internal_solr/game_data_records_index,10.100.205.80:8093/internal_solr/game_data_records_index", >> "10.100.205.71:8093":"_yz_pn:231 OR _yz_pn:201 OR _yz_pn:171 OR >> _yz_pn:141 OR _yz_pn:111 OR _yz_pn:81 OR _yz_pn:51 OR _yz_pn:21", >> "10.100.205.74:8093":"_yz_pn:234 OR _yz_pn:204 OR _yz_pn:174 OR >> _yz_pn:144 OR _yz_pn:114 OR _yz_pn:84 OR _yz_pn:54 OR _yz_pn:24", >> "10.100.205.77:8093":"_yz_pn:237 OR _yz_pn:207 OR _yz_pn:177 OR >> _yz_pn:147 OR _yz_pn:117 OR _yz_pn:87 OR _yz_pn:57 OR _yz_pn:27", >> "10.100.205.78:8093":"_yz_pn:228 OR _yz_pn:198 OR _yz_pn:168 OR >> _yz_pn:138 OR _yz_pn:108 OR _yz_pn:78 OR _yz_pn:48 OR _yz_pn:18", >> "10.100.205.75:8093":"_yz_pn:225 OR _yz_pn:195 OR _yz_pn:165 OR >> _yz_pn:135 OR _yz_pn:105 OR _yz_pn:75 OR _yz_pn:45 OR _yz_pn:15", >> "wt":"json", >> "10.100.205.72:8093":"_yz_pn:252 OR _yz_pn:222 OR _yz_pn:192 OR >> _yz_pn:162 OR _yz_pn:132 OR _yz_pn:102 OR _yz_pn:72 OR _yz_pn:42 OR >> _yz_pn:12"}}, >> "response":{"numFound":0,"start":0,"docs":[] >> }} >> >> >> I can select this record from solr webapp from some of these 10 nodes. So >> this record should be indexed by solr. So I guessed the problem cause this >> is related with Yokozuna’s distributed shards, since one query would shard >> to every solr instance by Yokozuna, then Yokozuna would collect all the >> returns and reduce to a fix result, something like map-reduce mechanism. >> >> So the problem here may underlay in map phase or reduce phase. >> >> The default configuration of riak search applied in these 10 nodes. >> >> Would anyone have some insight on how to fix this? By modified the >> configuration of search? >> >> >> Best regards, >> Witeman >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com <mailto:riak-users@lists.basho.com> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
