Awesome. Thanks. Zeeshan Lakhani programmer | software engineer at @basho | org. member/founder of @papers_we_love | twitter => @zeeshanlakhani
> On Apr 24, 2015, at 8:56 PM, Jason Campbell <xia...@xiaclo.net> wrote: > > This may be a case of force-replace vs replace vs reip. I'm happy to see if > I can get new cluster from backup to keep the Solr indexes. > > The disk backup was all of /var/lib/riak, so definitely included the YZ > indexes before the force-replace, and they were kept on the first node that > was changed with reip. I stopped each node before the snapshot to ensure > consistency. So I would expect the final restored cluster to be somewhere > between the first and last node snapshot in terms of data, and AAE to repair > things to a consistent state for that few minute gap. > > I'll experiment with different methods of rebuilding the cluster on Monday > and see if I can get it to keep the Solr indexes. Maybe moving the YZ > indexes out of the way during the force-replace, then stopping the node and > putting them back could help as well. I'll let you know the results of the > experiments either way. > > Thanks, > Jason > >> On 25 Apr 2015, at 09:25, Zeeshan Lakhani <zlakh...@basho.com> wrote: >> >> Hey Jason, >> >> Yeah, nodes can normally be joined without a cluster dropping its Solr Index >> and AAE normally rebuilds the missing KV bits. >> >> In the case of restoring from a backup and having missing data, we can only >> recommend a reindex (the indexes that have the issue) with aggressive AAE >> settings to speed things up. It can be pretty fast. Recreating indexes are >> cheap in Yokozuna, but are the `data/yz` directories missing from the nodes >> that were force-replaced? Unless someone else wants to chime in, I’ll gather >> more info on what occurred from the reip vs the force-replace. >> >> Zeeshan Lakhani >> programmer | >> software engineer at @basho | >> org. member/founder of @papers_we_love | paperswelove.org >> twitter => @zeeshanlakhani >> >>> On Apr 24, 2015, at 7:02 PM, Jason Campbell <xia...@xiaclo.net> wrote: >>> >>> Is there a way to do a restore without rebuilding these indexes though? >>> Obviously this could take a long time depending on the amount of indexed >>> data in the cluster. It's a fairly big gotcha to say that Yokozuna fixes a >>> lot of the data access issues that Riak has, but if you restore from a >>> backup, it could be useless for days or weeks. >>> >>> As far as disk consistency, the nodes were stopped during the snapshot, so >>> I'm assuming on-disk it would be consistent within a single node. And >>> cluster wide, I would expect the overall data to fall somewhere between the >>> first and last node snapshot. AAE should still repair the bits left over, >>> but it shouldn't have to rebuild the entire Solr index. >>> >>> So the heart of the question can I join a node to a cluster without >>> dropping it's Solr index? force-replace obviously doesn't work, what is >>> the harm in running reip on every node instead of just the first? >>> >>> Thanks for the help, >>> Jason >>> >>>> On 25 Apr 2015, at 00:36, Zeeshan Lakhani <zlakh...@basho.com> wrote: >>>> >>>> Hey Jason, >>>> >>>> Here’s a little more discussion on Yokozuna backup strategies: >>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2014-January/014514.html. >>>> >>>> Nonetheless, I wouldn’t say the behavior’s expected, but we’re going to be >>>> adding more to the docs on how to rebuild indexes. >>>> >>>> To do so, you could just remove the yz_anti_entropy directory, and make >>>> AAE more aggressive, via >>>> >>>> ``` >>>> rpc:multicall([node() | nodes()], application, set_env, [yokozuna, >>>> anti_entropy_build_limit, {100, 1000}]). >>>> rpc:multicall([node() | nodes()], application, set_env, [yokozuna, >>>> anti_entropy_concurrency, 4]) >>>> ``` >>>> >>>> and the indexes will rebuild. You can try to initialize the building of >>>> trees with `yz_entropy_mgr:init([])` via `riak attach`, but a restart >>>> would also kick AAE into gear. There’s a bit more related info on this >>>> thread: >>>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016929.html. >>>> >>>> Thanks. >>>> >>>> Zeeshan Lakhani >>>> programmer | >>>> software engineer at @basho | >>>> org. member/founder of @papers_we_love | paperswelove.org >>>> twitter => @zeeshanlakhani >>>> >>>>> On Apr 24, 2015, at 1:34 AM, Jason Campbell <xia...@xiaclo.net> wrote: >>>>> >>>>> I think I figured it out. >>>>> >>>>> I followed this guide: >>>>> http://docs.basho.com/riak/latest/ops/running/nodes/renaming/#Clusters-from-Backups >>>>> >>>>> The first Riak node (changed with riak-admin reip) kept it's Solr index. >>>>> However, the other nodes when joined via riak-admin cluster >>>>> force-replace, dropped their Solr indexes. >>>>> >>>>> Is this expected? If so, it should really be in the docs, and there >>>>> should be another way to restore a cluster keeping Solr intact. >>>>> >>>>> Also, is there a way to rebuild a Solr index? >>>>> >>>>> Thanks, >>>>> Jason >>>>> >>>>>> On 24 Apr 2015, at 15:16, Jason Campbell <xia...@xiaclo.net> wrote: >>>>>> >>>>>> I've just done a backup and restore of our production Riak cluster, and >>>>>> Yokozuna has dropped from around 125 million records to 25million. >>>>>> Obviously the IPs have changed, and although the Riak cluster is stable, >>>>>> I'm not sure Solr handled the transition as nicely. >>>>>> >>>>>> Is there a way to force Solr to rebuild the indexes, or at least get >>>>>> back to the state it was in before the backup? >>>>>> >>>>>> Also, is this expected behaviour? >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> _______________________________________________ >>>>>> riak-users mailing list >>>>>> riak-users@lists.basho.com >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> riak-users@lists.basho.com >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
