Kresten, Thanks for the help! I successfully joined my additional node to the cluster and *hopefully* the rest go smoothly now.
I took your advice and stopped the node, moved the bad merge index out of the 'merge_index' directory and then restarted the node. After watching the log, and viewing the ring_status and member_status values (awesome additions by the way the riak-admin utility!), it was clear this was what was blocking the transfer of the partitions. I had to do this to a few more that were also stuck: 2 on node1 1 on node2 2 on node3 After doing the last one on node3 the ownership handoff was complete. I did run into one more issue that seems sorta related but looked a little different... after the ownership handoff was complete, there were 7 pending partitions 'riak-admin transfers' that needed to complete. I found it interesting that it was actually from the joining node back to the other nodes (so from my new node5, back to the original nodes). After watching the console log again I could see that there was some more crashing and right before the crash the message: I saw http://riak-users.197444.n3.nabble.com/Failed-to-compact-in-RiakSearch-tp2816484p2821431.html Rusty's post a while back that shed a bit of light, but in the end I went with the same solution as before. Not sure if it was the best, or correct but like before: stop riak and move that directory out of the merge_index folder and then start riak back up. I had to do this four more times when it would get stuck, but finally the 'riak-admin transfer' showed no more pending/active transfers. The only thing I'm seeing now in the logs printed over and over on all the nodes is: But maybe that is normal? I would still be interested in your question about knowing if there is a better way to recover from this? Also since I'm sure my search indices are now in bad shape, if there is a way to rebuild all the indices besides using the python driver to read/write each key again? Possibly the 'search-cmd' or maybe something from the erlang console? Also if anyone would like the logs to help in future development let me know. -- View this message in context: http://riak-users.197444.n3.nabble.com/v1-0-3-search-merge-index-preventing-partition-handoff-tp3688803p3690487.html Sent from the Riak Users mailing list archive at Nabble.com. _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
