Hi Ryan, I had an incident when removing a replica back in November. But then I managed to fix the state of the cluster. Then in December I passed the script to re-save all docs so it worked (i.e. documents matched), and now in January we were still noticing discrepancies.
I noticed the problem only twice, perhaps it did happen more often. Avg load is way under control, no spikes, no faulty disks. The cluster hasn't changed at all since November. Francisco 2012/1/10 Ryan Zezeski <rzeze...@basho.com>: > Typically, something like this would be caused by replicas being lost as > Search currently has no anti-entropy. I imagine you would have mentioned > replica loss if that was the case, though. You've seen this problem at > least twice, correct? Were there other occurrences? If so how many and > about how often do you notice them? Did any other system events happen that > might be correlated such as spike in load, faulty disks, node > join/leave/remove, node down/up, etc. > > On Tue, Jan 10, 2012 at 11:31 AM, francisco treacy > <francisco.tre...@gmail.com> wrote: >> >> 2012/1/10 Ryan Zezeski <rzeze...@basho.com>: >> > When you say "doesn't show up on the search query results *anymore*" >> > does >> > that imply that at one time they did? >> >> Absolutely. As a matter of fact that's what some of my users were >> asking: "why can't I see X anymore?". >> >> > I'm trying to understand if the index >> > entries appear to have been lost or if it was never successfully written >> > in >> > the first place. >> >> They are definitely written to the index in the first place. I recall >> noticing a similar situation early December, where I compared item >> counts from a full-bucket map/reduce vs. a search map/reduce and they >> didn't match. >> >> Went about it by re-saving all items in the bucket as I said before. >> Did a check just after, and numbers matched. >> >> And this morning I was back to the inequality after a user complaint, >> so passed the re-save script once again. Obviously this is not >> sustainable. >> >> The curious thing is that those documents are not even regularly >> updated or deleted. >> >> > Any errors you see in the logs may be relevant. Feel free to include >> > snippets of anything you find. >> >> I went through all sasl-error.log from all 3 nodes and couldn't find >> anything related to search. Where else could I look? Do you have any >> idea what could be causing this behaviour? >> >> Francisco >> >> > >> > On Tue, Jan 10, 2012 at 4:16 AM, francisco treacy >> > <francisco.tre...@gmail.com> wrote: >> >> >> >> Hi all, >> >> >> >> I am running (still) 0.14.2 in production, and using Riak Search to >> >> index certain buckets. Those buckets have the Riak Search pre-commit >> >> hook enabled. >> >> >> >> Once in a while I get complaints of missing documents from my clients, >> >> and it just happened. >> >> >> >> I checked and the document: >> >> - is correctly stored in Riak >> >> - doesn't show up on the search query results *anymore* >> >> >> >> My workaround before was to go through every key in the bucket and >> >> save it again, as to trigger the search hook. Of course, this is less >> >> than ideal. >> >> >> >> Should I be looking for something specific in the logs? What could be >> >> going on here? >> >> >> >> Thanks, >> >> >> >> Francisco >> >> >> >> _______________________________________________ >> >> riak-users mailing list >> >> riak-users@lists.basho.com >> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > >> > > > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
