Riak2.0 with Solr Search: index on one node contains not all entries
Hi all,
I have on a test environment two riak nodes and each of them has solr
activated which index 3 buckets using a custom schema.
After testing a bit back and forth, I have the case that on one solr node,
an entry is not in the index (I know in which node though).
Fetching the specific key in the bucket works however, both nodes have the
respective entry.
1) How can this happen? I don't see any error/warning in the logs (neither
solr nor riak logs).
2) Is there a possibility to fix this without having to do e.g. a PUT on
the specific key with the same content in order to update it?
I tried to run a repair on the failing node using the guide:
http://docs.basho.com/riak/1.4.7/ops/running/recovery/repairing-indexes/#Repairing-Search-Indexes
When running the repair command on the partitions I received, it gave me
however a [{, down}, {...}] response, which gives me an
uncomfortable feeling, but I didn't really figure out yet what this means
exactly.
Thanks a lot for your help :)
Best,
Michael
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak2.0 with Solr Search: index on one node contains not all entries
Hi Zeeshan,
Thanks for your answer.
> Just to be sure, does your custom schema include the required fields, as
> mentioned in the docs:
> http://docs.basho.com/riak/latest/dev/advanced/search-schema/#Custom-Schemas?
>
Yes, I double checked that now to make sure, and the schema includes the
required fields.
> Are these Riak nodes joined? What’s your ring size, n_val value?
>
They run in a cluster, yes. Output of "riak-admin status":
ring_creation_size : 64
ring_members : ['riak@IPADDRESS','riak@IPADDRESS']
ring_num_partitions : 64
ring_ownership : <<"[{'riak@IPADDRESS',32},{'riak@IPADDRESS',32}]">>
rings_reconciled : 0
rings_reconciled_total : 33
n_val is 3, the initial one. According to the docs we should probably
either add another node or reduce it to "2" though..correct? (Sorry, newbie
in riak here :D)
> How are you querying the Solr nodes to know which node has the data and
> which one doesn't? Coverage is R=1, so you would be getting a different
> number on some search queries (using the standard
> /search/query/?...) if its only on one of the Solr cores.
>
Yes exactly. That's how I figured out that there's a difference, later on I
queried the separate Solr instances using the solr admin interface itself.
I also then fetched the missing key on both the riak nodes (without going
to solr, just fetching it directly using the riak HTTP API) and they exist
on both nodes.
> Can you also post me a screenshot of your search AAE exchanges, e.g.
> `riak-admin search aae-status`? You could look at this thread,
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html,
> for answers on how to perform read-repair/repair the AAE tree.
>
aae-status is in the .log file attached.
So you'd propose to perform a read-repair on the AAE tree?
Best,
Michael
>
> Thanks.
>
> Zeeshan Lakhani
> programmer |
> software engineer at @basho |
> org. member/founder of @papers_we_love | paperswelove.org
> twitter => @zeeshanlakhani
>
> On Mar 25, 2015, at 6:42 AM, Michael Weibel
> wrote:
>
> Hi all,
>
> I have on a test environment two riak nodes and each of them has solr
> activated which index 3 buckets using a custom schema.
> After testing a bit back and forth, I have the case that on one solr node,
> an entry is not in the index (I know in which node though).
> Fetching the specific key in the bucket works however, both nodes have the
> respective entry.
>
> 1) How can this happen? I don't see any error/warning in the logs (neither
> solr nor riak logs).
> 2) Is there a possibility to fix this without having to do e.g. a PUT on
> the specific key with the same content in order to update it?
>
> I tried to run a repair on the failing node using the guide:
> http://docs.basho.com/riak/1.4.7/ops/running/recovery/repairing-indexes/#Repairing-Search-Indexes
> When running the repair command on the partitions I received, it gave me
> however a [{, down}, {...}] response, which gives me an
> uncomfortable feeling, but I didn't really figure out yet what this means
> exactly.
>
> Thanks a lot for your help :)
>
> Best,
> Michael
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
riak-admin-aae-status.log
Description: Binary data
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak2.0 with Solr Search: index on one node contains not all entries
Hi Zeeshan,
Ok, will do that and report back as soon as I have it.. Might take a while
though because I first also have to figure out whether I have still the
same issue or not.. ;)
Thanks!
Michael
2015-03-25 16:56 GMT+01:00 Zeeshan Lakhani :
> Hey Michael,
>
> Ideally, for this “testing" setup, n_val=2 would be the effective choice.
> I’d create a new bucket_type/bucket and re-PUT your data in and test search
> again to be sure.
>
> Let me know. Thanks.
>
> Zeeshan Lakhani
> programmer |
> software engineer at @basho |
> org. member/founder of @papers_we_love | paperswelove.org
> twitter => @zeeshanlakhani
>
> On Mar 25, 2015, at 11:02 AM, Michael Weibel
> wrote:
>
> Hi Zeeshan,
>
> Thanks for your answer.
>
>> Just to be sure, does your custom schema include the required fields, as
>> mentioned in the docs:
>> http://docs.basho.com/riak/latest/dev/advanced/search-schema/#Custom-Schemas?
>>
>
> Yes, I double checked that now to make sure, and the schema includes the
> required fields.
>
>
>> Are these Riak nodes joined? What’s your ring size, n_val value?
>>
>
> They run in a cluster, yes. Output of "riak-admin status":
>
> ring_creation_size : 64
> ring_members : ['riak@IPADDRESS','riak@IPADDRESS']
> ring_num_partitions : 64
> ring_ownership : <<"[{'riak@IPADDRESS',32},{'riak@IPADDRESS',32}]">>
> rings_reconciled : 0
> rings_reconciled_total : 33
>
> n_val is 3, the initial one. According to the docs we should probably
> either add another node or reduce it to "2" though..correct? (Sorry, newbie
> in riak here :D)
>
>
>> How are you querying the Solr nodes to know which node has the data and
>> which one doesn't? Coverage is R=1, so you would be getting a different
>> number on some search queries (using the standard
>> /search/query/?...) if its only on one of the Solr cores.
>>
>
> Yes exactly. That's how I figured out that there's a difference, later on
> I queried the separate Solr instances using the solr admin interface
> itself. I also then fetched the missing key on both the riak nodes (without
> going to solr, just fetching it directly using the riak HTTP API) and they
> exist on both nodes.
>
>
>> Can you also post me a screenshot of your search AAE exchanges, e.g.
>> `riak-admin search aae-status`? You could look at this thread,
>> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016926.html,
>> for answers on how to perform read-repair/repair the AAE tree.
>>
>
> aae-status is in the .log file attached.
>
> So you'd propose to perform a read-repair on the AAE tree?
>
> Best,
> Michael
>
>
>
>>
>> Thanks.
>>
>> Zeeshan Lakhani
>> programmer |
>> software engineer at @basho |
>> org. member/founder of @papers_we_love | paperswelove.org
>> twitter => @zeeshanlakhani
>>
>> On Mar 25, 2015, at 6:42 AM, Michael Weibel
>> wrote:
>>
>> Hi all,
>>
>> I have on a test environment two riak nodes and each of them has solr
>> activated which index 3 buckets using a custom schema.
>> After testing a bit back and forth, I have the case that on one solr
>> node, an entry is not in the index (I know in which node though).
>> Fetching the specific key in the bucket works however, both nodes have
>> the respective entry.
>>
>> 1) How can this happen? I don't see any error/warning in the logs
>> (neither solr nor riak logs).
>> 2) Is there a possibility to fix this without having to do e.g. a PUT on
>> the specific key with the same content in order to update it?
>>
>> I tried to run a repair on the failing node using the guide:
>> http://docs.basho.com/riak/1.4.7/ops/running/recovery/repairing-indexes/#Repairing-Search-Indexes
>> When running the repair command on the partitions I received, it gave me
>> however a [{, down}, {...}] response, which gives me an
>> uncomfortable feeling, but I didn't really figure out yet what this means
>> exactly.
>>
>> Thanks a lot for your help :)
>>
>> Best,
>> Michael
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Query on Riak Search in a cluster of 3 nodes behind ELB is giving different result everytime
Hey Vitaly,
experienced the same issue recently (see
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2015-March/016972.html).
When you have a look at your now working solr instances admin pages, do you
have everywhere the index counts the same? (i.e. the numDocs, maxDocs,
deletedDocs)
Thanks,
Michael
2015-04-01 9:50 GMT+02:00 Vitaliy E <13vitam...@gmail.com>:
> Hello everyone,
>
> I've just joined the list, and am a bit late to the party. Sorry about
> that. Thought I would contribute an answer anyway.
>
> Santi, what is you n_val?
>
> I observed the behavior you are describing on Riak 2.0.0 with n_val=3 in
> two cases:
>
> 1. One of the nodes was not part of the cluster although the cluster was
> thinking it was. Don't ask me how that happened. Obviously, when a request
> hit that node part of the entries could not be found there.
>
> 2. Look for indexing errors in Solr console and Riak logs. Each Riak node
> has its own Solr "repository", so if an entry fails to be indexed on any of
> them, search results will be inconsistent depending on which set of nodes
> returns it. Let's say you have replicas on nodes A, B, and C. Entry X
> failed to be indexed on A, entry Y failed to be indexed on A and B, and
> entry Z was indexed OK on all nodes. Then you may get {X,Y,Z}, {X,Z}, or
> {Z} as your search results.
>
> In our case the indexing failures were caused by disk/filesystem errors.
>
> Regards,
> Vitaly
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
