Re: riak2 erlang mapreduce counters
Hi Bryce,
Sorry about this, and thanks for the detailed info. I do need to add MapReduce
friendly functions.
On 23 Jan 2014, at 23:51, Bryce Verdier wrote:
> Thank you both Eric & Russeli for the answer, sadly it leads to more
> questions. Regardless of the type (though I can say in this case the counters
> were pushed from the python 2.0.2 client, so I assume its riak_dt_pncounter)
It is a riak_kv_pncounter, I don’t think any of the clients support the new API
end points (bucket types) so you’ll be using the 1.4 counter.
>
> I get this error:
> {"phase":0,"error":"badarg","input":"{ok,{r_object,<<\"ogir-fp\">>,<<\"682l2fp6\">>,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<\"content-type\">>,97,112,112,108,105,99,97,116,105,111,110,47,114,105,97,107,95,99,111,117,110,116,101,114],[<<\"X-Riak-VTag\">>,52,103,66,88,71,122,56,55,111,105,66,112,103,65,75,99,54,72,55,69,110,79]],[[<<\"index\">>]],[],[[<<\"X-Riak-Last-Modified\">>|{1390,508268,984013}]],[],[]}}},<<69,1,71,1,0,0,0,29,70,1,131,108,0,0,0,1,104,2,109,...>>}],...},...}","type":"error","stack":"[{erlang,apply,[<<\"riak_kv_pncounter\">>,new,[]],[]},{riak_kv_crdt,crdt_value,2,[{file,\"src/riak_kv_crdt.erl\"},{line,94}]},{riak_kv_crdt,value,2,[{file,\"src/riak_kv_crdt.erl\"},{line,86}]},{mr_kv_counters,value,3,[{file,\"mr_kv_counters.erl\"},{line,38}]},{riak_kv_mrc_map,map,3,[{file,\"src/riak_kv_mrc_map.erl\"},{line,165}]},{riak_kv_mrc_map,process,3,[{file,\"src/riak_kv_mrc_map.erl\"},{line,141}]},{riak_pipe_vnode_worker,process_input,3,[{file,\"src/riak_pipe_vnode_worker.erl\"},{line,445}]},{riak_pipe_vnode_worker,...}]”}
That binary (<<69,1,71…>>) says CRDT(69), Version 1(1), riak_kv_pncounter (71).
The error says that erlang is trying to apply the Module, Function, Arguments
of <<“riak_kv_pncounter”>>, new, []. But modules need to be atoms.
> Just to help, this is my erlang MR code:
> value(RiakObject, _KeyData, _Arg) ->
>Key = riak_object:key(RiakObject),
>Count = riak_kv_crdt:value(RiakObject, <<"riak_kv_pncounter">>),
Type needs to be an atom, as it is the module name (so riak_kv_pncounter).
Also, the return value is now a tuple of
{{Context :: binary(), Value :: term()}, Stats :: proplist()}. You probably
only want the Value bit. So:
{{_Ctx, Count}, _Stats} = riak_kv_crdt:value(RiakObject, riak_kv_pncounter),
Should be what you need, let me know if that works, please?
Cheers
Russell
>[ {Key, Count} ].
>
> What am I doing wrong? I can't seem to figure it out... I'm sure its
> something simple thing I'm just not seeing.
>
> Thanks again,
> Bryce
>
> On 01/23/2014 01:07 PM, Russell Brown wrote:
>> On 23 Jan 2014, at 20:51, Eric Redmond wrote:
>>
>>> For version 1.4 counters, riak_kv_pncounter. For 2.0 CRDT counters,
>>> riak_dt_pncounter.
>> As in, if the data was written in 1.4, or in 2.0 using the legacy, backwards
>> compatible 1.4 API endpoints, the the type is risk_kv_pncounter. If the
>> counter is 2.0, bucket types counter, then risk_dt_pncounter.
>>
>> Really, we need to re-introduce the riak_kv_counter module for backwards
>> compatibility, and add some friendly `value’ functions to risk_kv_crdt. I’m
>> opening an issue for just this now.
>>
>> The other option is to include the riak_kv_types.hrl and use the macros
>> ?MAP_TYPE, ?SET_TYPE, ?V1_COUNTER_TYPE, ?COUNTER_TYPE for now, and assume
>> that we’ll have some helper functions for MapReduce in before 2.0.
>>
>> Cheers
>>
>> Russell
>>
>>> Eric
>>>
>>> On Jan 23, 2014, at 3:44 PM, Bryce Verdier wrote:
>>>
In 1.4 there was just the simple function riak_kv_counters:value. In 2.0 I
found the riak_kv_crdt module, which has a value function in it. But I'm
not sure what "type" to use for second value argument for a counter.
Can someone share that with me?
Thanks in advance,
Bryce
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Java client counter object
Hi, Is there any small doc that could explain its usage a little bit: From the Java perspective it would be nice if it point out its counter part methods with Atomic Integer like: * How to create it? Does incrementing a counter will just create it with zero as initial value and then increment it? What if you need to know its value and then increment it before it is created? (getAndIncrement) * get() * getAndIncrement() * incrementAndGet() Regards, Guido. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Search is not working in RIAK
Ajit,
Riak Search doesn't exactly support nested fields. The default extractor
will flatten the fields in the input by joining the field and subfield names
with underscore. (example below)
I'm not totally clear on what you are trying to accomplish, so I'll run
through an example and hope it clears up some confusion.
The data as pasted in the original email confuses the extractor since the
same fields are duplicated with the object:
```
root@sitea2:~# search-cmd install testbucket
:: Installing Riak Search <--> KV hook on bucket 'testbucket'.
root@sitea2:~# curl http://sitea2:8098/buckets/testbucket/keys/emp1 -XPUT -H
"content-type: application/json" -d '
{
"employee":{
"employee_name":"Raghav Maitra",
"employee_id":"1001",
"designation":"Technical Analyst",
"emp_contract_no":"1987-XXVI",
"department":
{"dept_name":"ADW",
"sub_dept_name":"A1",
"dept_id":"A1-2001",
"dept_description":"Analysis and Devlopement Wing",
"dept_site_name":"Diego Gracia"
}
"employee_name":"Ajit",
"employee_id":"1002",
"designation":"Technical Analyst",
"emp_contract_no":"2001-XXVI",
"department":
{"dept_name":"ADW",
"sub_dept_name":"A1",
"dept_id":"A1-2001",
"dept_description":"Analysis and Devlopement Wing",
"dept_site_name":"Diego Gracia"
}
}'
500 Internal Server
ErrorInternal Server ErrorThe server
encountered an error while processing this request:{error,
{error,badarg,
[{erlang,iolist_to_binary,
[{hook_crashed,
{riak_search_kv_hook,precommit,error,function_clause}}],
[]},
{wrq,append_to_response_body,2,[{file,"src/wrq.erl"},{line,215}]},
{riak_kv_wm_object,handle_common_error,3,
[{file,"src/riak_kv_wm_object.erl"},{line,1046}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,accept_helper,1,
[{file,"src/webmachine_decision_core.erl"},{line,612}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,580}]}]}}mochiweb+webmachine web server
```
Adding them separately allows each to be indexed:
```
root@sitea2:~# curl http://sitea2:8098/buckets/testbucket/keys/emp1 -XPUT -H
"content-type: application/json" -d '{
"employee":{
"employee_name":"Raghav Maitra",
"employee_id":"1001",
"designation":"Technical Analyst",
"emp_contract_no":"1987-XXVI",
"department":
{"dept_name":"ADW",
"sub_dept_name":"A1",
"dept_id":"A1-2001",
"dept_description":"Analysis and Devlopement Wing",
"dept_site_name":"Diego Gracia"
},
}
}'
root@sitea2:~# curl http://sitea2:8098/buckets/testbucket/keys/emp2 -XPUT -H
"content-type: application/json" -d '{
"employee":{
"employee_name":"Ajit",
"employee_id":"1002",
"designation":"Technical Analyst",
"emp_contract_no":"1987-XXVI",
"department":
{"dept_name":"ADW",
"sub_dept_name":"A1",
"dept_id":"A1-2001",
"dept_description":"Analysis and Devlopement Wing",
"dept_site_name":"Diego Gracia"
},
}
}'
```
After these puts, the following terms exist in the search index:
```
{{<<"testbucket">>,{<<"employee_department_dept_description">>,<<"Analysis"
>>}}
{{<<"testbucket">>,{<<"employee_department_dept_description">>,<<"and">>}}
{{<<"testbucket">>,{<<"employee_department_dept_description">>,<<"Devlopeme
nt">>}}
{{<<"testbucket">>,{<<"employee_department_dept_description">>,<<"Wing">>}}
{{<<"testbucket">>,{<<"employee_department_dept_id">>,<<"A1-2001">>}}
{{<<"testbucket">>,{<<"employee_department_dept_name">>,<<"ADW">>}}
{{<<"testbucket">>,{<<"employee_department_dept_site_name">>,<<"Diego">>}}
{{<<"testbucket">>,{<<"employee_department_dept_site_name">>,<<"Gracia">>}}
{{<<"testbucket">>,{<<"employee_department_sub_dept_name">>,<<"A1">>}}
{{<<"testbucket">>,{<<"employee_designation">>,<<"Analyst">>}}
{{<<"testbucket">>,{<<"employee_designation">>,<<"Technical">>}}
{{<<"testbucket">>,{<<"employee_emp_contract_no">>,<<"1987-XXVI">>}}
{{<<"te
Thank You, Riak Community
Hi All, For anyone keeping score at home, "m...@basho.com" has managed to spam this list with exactly 710 messages [1] since March 31, 2010 [2]. This, the 711th, is coming from my personal address (msphilllena...@gmail.com) as today is my last day at Basho. This message is to say thank you. I've been at Basho for the better part of six years, with the last four being dedicated to the creation and proliferation of the Riak community. This roughly lines up with when Riak was open-sourced (which, for any historians out there, was 1,631 days ago [3] as of this writing). I officially started in February of 2010 as Community Manager. About two years after that, thanks in large part to our community and customers validating the power of Riak through production usage, I was given the opportunity to build a team. That team now stands at six, with Tom, Hector, Seth, Michelle, John, and Joel all being far more capable than yours truly. The Community Team, alongside all of Basho, have big plans for 2014. Better docs, better meetups, better blog posts, better swag, better RICONs, better code, and better products. The future for Riak is brighter than ever. 2.0 is on the cusp of being official (patience, patience), and the features and fixes in that release will put Riak into another category of database all together. With Yokozuna, CRDTs, Strong Consistency, Security, and Cuttlefish, (each of which could be their own major-release leading feature), I truly believe there's no better database out there for companies building scalable, real-time systems that can't go down. If you have one use case for Riak 1.4.x, you'll have another three or four thanks to the additions in 2.0. (And if you throw CS into the mix, almost all your problems are solved...) The Riak Community was largely my introduction to open source software. It was a never-ending adventure, and I was thankful and honored every day to be a part of it. And even though I'm moving on from Basho to try something new, I'll still be yelling about Riak and Riak CS every chance I get. You won't be getting rid of me that easily. :) (Also, for anyone interested, today's Riak Community Hangout will feature John Daily, Hector Castro, Andy Gross, and myself taking a look at the last six years of Riak and Basho. We'll be live at around 11 AM Pacific - about 14 minutes from now - on Basho's YouTube page [4].) Thanks for everything. See you all soon. Mark twitter.com/pharkmillups 413.335.2332 [1] http://riak.markmail.org/search/?q= [2] http://riak.markmail.org/search/?q=#query:from%3A%22Mark%20Phillips%22%20date%3A201003%20+page:1+mid:2au3o6gk7qodaurz+state:results [3] https://news.ycombinator.com/item?id=748079 [4] http://www.youtube.com/bashotechnologies ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak2 erlang mapreduce counters
Hi Russell,
Thank you for the code, it worked like a charm.
Warm regards,
Bryce
On 01/23/2014 11:59 PM, Russell Brown wrote:
Hi Bryce,
Sorry about this, and thanks for the detailed info. I do need to add MapReduce
friendly functions.
On 23 Jan 2014, at 23:51, Bryce Verdier wrote:
Thank you both Eric & Russeli for the answer, sadly it leads to more questions.
Regardless of the type (though I can say in this case the counters were pushed from
the python 2.0.2 client, so I assume its riak_dt_pncounter)
It is a riak_kv_pncounter, I don’t think any of the clients support the new API
end points (bucket types) so you’ll be using the 1.4 counter.
I get this error:
{"phase":0,"error":"badarg","input":"{ok,{r_object,<<\"ogir-fp\">>,<<\"682l2fp6\">>,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<\"content-type\">>,97,112,112,108,105,99,97,116,105,111,110,47,114,105,97,107,95,99,111,117,110,116,101,114],[<<\"X-Riak-VTag\">>,52,103,66,88,71,122,56,55,111,105,66,112,103,65,75,99,54,72,55,69,110,79]],[[<<\"index\">>]],[],[[<<\"X-Riak-Last-Modified\">>|{1390,508268,984013}]],[],[]}}},<<69,1,71,1,0,0,0,29,70,1,131,108,0,0,0,1,104,2,109,...>>}],...},...}","type":"error","stack":"[{erlang,apply,[<<\"riak_kv_pncounter\">>,new,[]],[]},{riak_kv_crdt,crdt_value,2,[{file,\"src/riak_kv_crdt.erl\"},{line,94}]},{riak_kv_crdt,value,2,[{file,\"src/riak_kv_crdt.erl\"},{line,86}]},{mr_kv_counters,value,3,[{file,\"mr_kv_counters.erl\"},{line,38}]},{riak_kv_mrc_map,map,3,[{file,\"src/riak_kv_mrc_map.erl\"},{line,165}]},{riak_kv_mrc_map,process,3,[{file,\"src/riak_kv_mrc_map.erl\"},{line,141}]},{riak_pipe_vnode_worker,process_input,3,[{file,\"src/riak_pipe_vnode_worker.erl\"},{line,445}]},{riak_pipe_vnode_worker,...}]”}
That binary (<<69,1,71…>>) says CRDT(69), Version 1(1), riak_kv_pncounter (71). The error
says that erlang is trying to apply the Module, Function, Arguments of
<<“riak_kv_pncounter”>>, new, []. But modules need to be atoms.
Just to help, this is my erlang MR code:
value(RiakObject, _KeyData, _Arg) ->
Key = riak_object:key(RiakObject),
Count = riak_kv_crdt:value(RiakObject, <<"riak_kv_pncounter">>),
Type needs to be an atom, as it is the module name (so riak_kv_pncounter).
Also, the return value is now a tuple of
{{Context :: binary(), Value :: term()}, Stats :: proplist()}. You
probably only want the Value bit. So:
{{_Ctx, Count}, _Stats} = riak_kv_crdt:value(RiakObject,
riak_kv_pncounter),
Should be what you need, let me know if that works, please?
Cheers
Russell
[ {Key, Count} ].
What am I doing wrong? I can't seem to figure it out... I'm sure its something
simple thing I'm just not seeing.
Thanks again,
Bryce
On 01/23/2014 01:07 PM, Russell Brown wrote:
On 23 Jan 2014, at 20:51, Eric Redmond wrote:
For version 1.4 counters, riak_kv_pncounter. For 2.0 CRDT counters,
riak_dt_pncounter.
As in, if the data was written in 1.4, or in 2.0 using the legacy, backwards
compatible 1.4 API endpoints, the the type is risk_kv_pncounter. If the counter
is 2.0, bucket types counter, then risk_dt_pncounter.
Really, we need to re-introduce the riak_kv_counter module for backwards
compatibility, and add some friendly `value’ functions to risk_kv_crdt. I’m
opening an issue for just this now.
The other option is to include the riak_kv_types.hrl and use the macros
?MAP_TYPE, ?SET_TYPE, ?V1_COUNTER_TYPE, ?COUNTER_TYPE for now, and assume that
we’ll have some helper functions for MapReduce in before 2.0.
Cheers
Russell
Eric
On Jan 23, 2014, at 3:44 PM, Bryce Verdier wrote:
In 1.4 there was just the simple function riak_kv_counters:value. In 2.0 I found the
riak_kv_crdt module, which has a value function in it. But I'm not sure what
"type" to use for second value argument for a counter.
Can someone share that with me?
Thanks in advance,
Bryce
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Impending Community Hangout
If you’ve been a part of the Riak community for a while, you won’t want to miss today’s Community Hangout. Hector Castro and I will be joined by two people who have been among the more prominent faces for Basho for a very long time: Andy Gross and Mark Phillips. We’ll take a look back at where we’ve been as a company, product, but most importantly, as a community. And if you saw Mark’s email a few minutes ago, you’ll know why today is a good moment to reflect. Hope to see you there. https://plus.google.com/events/cpql4g74af2p5r8ccfnh3i5g9gk -John ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Thank You, Riak Community
One might say "sorry to see you go" but I think we all know you'll never really leave. And that's the only way I'll have it. Thanks for making this a great community. Mazal Tov! -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Jan 24, 2014, at 13:46, Mark S Phillips wrote: > > Hi All, > > For anyone keeping score at home, "m...@basho.com" has managed to spam > this list with exactly 710 messages [1] since March 31, 2010 [2]. > This, the 711th, is coming from my personal address > (msphilllena...@gmail.com) as today is my last day at Basho. This > message is to say thank you. > > I've been at Basho for the better part of six years, with the last > four being dedicated to the creation and proliferation of the Riak > community. This roughly lines up with when Riak was open-sourced > (which, for any historians out there, was 1,631 days ago [3] as of > this writing). I officially started in February of 2010 as Community > Manager. About two years after that, thanks in large part to our > community and customers validating the power of Riak through > production usage, I was given the opportunity to build a team. That > team now stands at six, with Tom, Hector, Seth, Michelle, John, and > Joel all being far more capable than yours truly. > > The Community Team, alongside all of Basho, have big plans for 2014. > Better docs, better meetups, better blog posts, better swag, better > RICONs, better code, and better products. > > The future for Riak is brighter than ever. 2.0 is on the cusp of being > official (patience, patience), and the features and fixes in that > release will put Riak into another category of database all together. > With Yokozuna, CRDTs, Strong Consistency, Security, and Cuttlefish, > (each of which could be their own major-release leading feature), I > truly believe there's no better database out there for companies > building scalable, real-time systems that can't go down. If you have > one use case for Riak 1.4.x, you'll have another three or four thanks > to the additions in 2.0. (And if you throw CS into the mix, almost all > your problems are solved...) > > The Riak Community was largely my introduction to open source > software. It was a never-ending adventure, and I was thankful and > honored every day to be a part of it. And even though I'm moving on > from Basho to try something new, I'll still be yelling about Riak and > Riak CS every chance I get. You won't be getting rid of me that > easily. :) > > (Also, for anyone interested, today's Riak Community Hangout will > feature John Daily, Hector Castro, Andy Gross, and myself taking a > look at the last six years of Riak and Basho. We'll be live at around > 11 AM Pacific - about 14 minutes from now - on Basho's YouTube page > [4].) > > Thanks for everything. See you all soon. > > Mark > twitter.com/pharkmillups > 413.335.2332 > > [1] http://riak.markmail.org/search/?q= > [2] > http://riak.markmail.org/search/?q=#query:from%3A%22Mark%20Phillips%22%20date%3A201003%20+page:1+mid:2au3o6gk7qodaurz+state:results > [3] https://news.ycombinator.com/item?id=748079 > [4] http://www.youtube.com/bashotechnologies > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Java client counter object
Guido,
I’m sorry, there are not currently any docs available. Rest assured they are a
work in progress. In the meantime, please feel free to check-out the current
pre-release [1]. The pre-release covers the low level API we have built on top
of the PB interface. There is a start to a friendlier API in the develop branch
of riak-java-client [2].
If you would like to get started with the low level API to begin getting a feel
for Riak datatypes, I have included a few snippets that will hopefully help.
The counters (and the data structures in general) are implemented as two
separate operations: Fetch and Update. Updating a counter that does not exist
will create it and initialize it to the given value. Fetching a counter that
has not had a value stored will return an empty value.
While the low level API does not have convenience methods for getAndIncrement
and the like, they can all be easily accomplished given fetch and update. It
would clearly be easy for us to provide methods for these typical access
patterns and in fact, it is something we are planning on doing.
RiakCluster cluster = …
BinaryValue bucket = …
BinaryValue key = …
BinaryValue type = ...
fetch():
DtFetchOperation get =
new DtFetchOperation.Builder(bucket, key)
.withBucketType(type)
.build();
DtFetchOperation.Response response = cluster.execute(get).get();
long value;
CrdtElement element = response.getCrdtElement();
if (crdtElement.isCounter()) {
value = element.getCounter().getValue();
} else {
...
}
update():
DtUpdateOperation update =
new DtUpdateOperation.Builder(bucket, key)
.withBucketType(type)
.withReturnBody(true)
.withOp(new CounterOp(1)) // increment by 1
.build();
DtUpdateOperation.Response response = cluster.execute(update).get();
long value;
CrdtElement element = response.getCrdtElement();
if (crdtElement.isCounter()) {
value = element.getCounter().getValue();
} else {
...
}
getAndIncrement();
long value = fetch();
update(1);
incrementAndGet():
long value = update(1); // make sure withReturnBody == true
[1] https://github.com/basho/riak-java-client
[2]https://github.com/basho/riak-java-client/tree/develop/src/main/java/com/basho/riak/client/operations
--
Dave Rusek
Basho Technologies
@davidjrusek
On January 24, 2014 at 8:41:07 AM, Guido Medina (guido.med...@temetra.com)
wrote:
Hi,
Is there any small doc that could explain its usage a little bit:
From the Java perspective it would be nice if it point out its counter part
methods with Atomic Integer like:
How to create it? Does incrementing a counter will just create it with zero as
initial value and then increment it? What if you need to know its value and
then increment it before it is created? (getAndIncrement)
get()
getAndIncrement()
incrementAndGet()
Regards,
Guido.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
