On 07/21/2013 07:24 AM, Russell Brown wrote:> Hi,
>
> On 21 Jul 2013, at 02:09, Siraaj Khandkar <sir...@khandkar.net> wrote:
>
>> I (sequentially) made 146204 inserts of unique objects to a single
>> bucket. Several secondary indices (most with unique values) were set
>> for each object, one of which was "bucket" = BucketName (to use 2i
>> for listing all keys).
>
> There is a special $bucket index for this already, please see the docs
> here http://docs.basho.com/riak/latest/dev/using/2i/
>
Yeah... I stumbled on that piece of info in another doc about two days
ago - made me feel both stupid and validated :)
However, it doesn't seem to work for me - I always get: {ok,{keys,[]}}
>>
>> 6 of the objects appear to have been lost - they're consistently not
>> found by GETs (by key) and are not found by 2i queries to the indices
>> with unique values.
>
> Are you sure they were inserted?
I am sure that the client returned 'ok' on _every_ insertion in the
batch.
> Was there an error during your batch insert?
>
None. It was a sequential loop matching on 'ok' for every put and
expecting to crash on any error - there were no crashes.
>>
>> However, the "bucket" index search returns them _sometimes_.
>
> Oh. Erm. Have you deleted some keys? 2i is essentially an r=1 query.
>
Sort-of. This was a second instance of this batch insertion (a slightly
extended set of keys), the first one was deleted ~6 hours prior to
executing the second one.
At the end of the deletion there _were_ some tombstones left. Frankly I
do not remember with certainty if there are overlaps between tombstones
from previous delete and the keys in question. In retrospect - it was
big failure on my part not to take note of those.
After the second instance of the set insertion - there were _no_
more deletions.
So, in summary:
1) Inserted the set
2) Deleted the set
3) 6 hours passed
4) Inserted the set
5) Observed the problem
>>
>> Now, I understand there may be a replication lag, but this state has
>> remained for over 3 days now.
>>
>> "What is fucked, and why?" :)
>
> Good question.
>
I was hoping this list would appreciate the reference :)
> Could you provide some more details to help me figure it out: How many
> nodes are you running?
5
> Can you provide an example of the 2i queries you're running?
This is how I am testing it:
Compare = fun(PID, Bucket) ->
B = Bucket,
L1 = riakc_pb_socket:get_index(PID, B, {binary_index,
"bucket"}, B),
L2 = riakc_pb_socket:get_index(PID, B, {binary_index,
"bucket"}, B),
io:format("L1: ~b, L2: ~b~n",[length(L1), length(L2)]),
Diff_L1_L2 = L1 -- L2,
Diff_L2_L1 = L2 -- L1,
io:format("=== L1 -- L2 ===~n~p~n~n", [Diff_L1_L2]),
io:format("=== L2 -- L1 ===~n~p~n~n", [Diff_L2_L1]),
Fetch = fun(Key) ->
case riakc_pb_socket:get(PID, B, Key) of
{ok, _} -> io:format("FOUND: ~p~n", [Key]);
{error, _} -> io:format("NOT FOUND: ~p~n", [Key])
end
end,
io:format("=== L1 -- L2 ===~n"),
lists:foreach(Fetch, Diff_L1_L2),
io:format("=== L2 -- L1 ===~n"),
lists:foreach(Fetch, Diff_L2_L1)
end.
Which results in differences _sometimes_, but _always_ fails on get.
> If this is just a dev cluster, can you verify the keys are present /
> absent using either a range 2i $keys query, or a key list, please?
>
Unfortunately this is prod, so brute-force key list is out of the
question.
Running:
curl "http://127.0.0.1:8098/buckets/$bucket/index/\$keys_bin/0/z";
Returns:
{"keys":[]}
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
