Re: Riak pre-commit hook crashing unexpectedly
Your first suggestion solves the problem. signed_pb:decode_sign is
auto-generated by erlang_protobuffs, and I thought it wanted binary
input. However, omitting term_to_binary fixed the error.
Thanks,
~ David.
On 12/7/15 10:44 AM, Jon Meredith wrote:
Does your signed_pb:decode_sign(Msg) function expect the binary to be
in external term format? I would guess you probably want the direct
value there without the term to binary - it should already be a binary.
Is your hook expected to work with allow_mult=true, if so the
get_metadata and get_value calls could get badmatch errors if called
with siblings.
Jon
On Mon, Dec 7, 2015 at 1:52 AM David Rogers
mailto:predictivestatm...@gmail.com>>
wrote:
Hello,
I'm using riak {release,"riak","2.1.0","5.10.3",
[{kernel,"2.16.3",
"$HOME/src/riak-2.1.1/rel/riak/lib/kernel-2.16.3"} ...}
on a cluster with 1 Linux 3.16.0-38 and 4 OSX machines.
I have installed the following pre-commit hook (see end of email) to
validate protobuf-formatted values whose key should be the
sha1-hash of
part of their payload.
props =
{"props":{"allow_mult":false,"basic_quorum":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"dvv_enabled":false,"dw":"quorum","last_write_wins":false,"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"n_val":3,"name":"sil/code","notfound_ok":true,"old_vclock":86400,"postcommit":[],"pr":0,"precommit":[{"mod":"validate_hash","fun":"validate"}],"pw":0,"r":"quorum","rw":"quorum","small_vclock":50,"w":"quorum","write_once":false,"young_vclock":20}})
I attached the failed data element. I'm using the riak-c-client to
interface.
Although almost everything I've added to the bucket this way has
worked so far, a few (like the sample I sent) fail, with riak_put
returning only the unhelpful ERIAK_SERVER_ERROR ("An error was
returned
from the server"). This happens even though manually running the
validation works fine. It is consistently reproducible on this input.
In my attempt to track down the error, I added a test at the end of
riak_sync_request (below) that will log all server errors. The only
problem is that cfg->log_fn is always NULL when riak_log_error is
called, rather than what I set it to initially with
riak_config_set_logging! Gdb can't seem to catch any writes there, so
maybe cfg gets incompletely copied somewhere?
Anyway, I can get out the error message from the debugger.
riak_sync_request (rop_target=rop_target@entry=0x7fffe0b0,
response=response@entry=0x7fffe130)
at src/riak.c:81
81riak_log_error(cxn, "%.*s\n", (int)msg->len, msg->data);
(gdb) print msg->data
$13 = (riak_uint8_t *) 0x670a40
"{precommit_fail,{hook_crashed,{validate_hash,validate,error,badarg}}}"
so the server error shows only that the precommit hook
(validate_hash:validate/1) crashes. Again, manually running doesn't
crash it...
Questions:
First, I don't understand why riak-c-client doesn't call the
function
I supplied to riak_config_set_logging. Second, I don't know where to
look for more details on why the pre-commit hook crashes only when I
actually try to add this particular key/value. Can anyone spot a
problem
in its error handling or suggest a way to debug?
Note: I get slightly more info. when posting from curl,
$ curl -XPOST
http://127.0.0.1:8098/types/default/buckets/sil%2Fcode/keys/$hash -H
'Content-Type: application/octet-stream' --data-binary @"$hash"
500 Internal Server
ErrorInternal Server ErrorThe server
encountered an error while processing this request:{error,
{error,badarg,
[{erlang,iolist_to_binary,
[{hook_crashed,{validate_hash,validate,error,badarg}}],
[]},
{wrq,append_to_response_body,2,[{file,"src/wrq.erl"},{line,215}]},
{riak_kv_wm_object,handle_common_error,3,
[{file,"src/riak_kv_wm_object.erl"},{line,1178}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,490}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]}]}}mochiweb+webmachine
web server
Sincerely,
~ David M. Rogers
signed.proto:
```
message sign {
required bytes signer = 1;
required uint32 dig_alg = 2;
required bytes sign= 3;
required uint64 ctime = 4;
optional uint32 flags = 5;
riak nodes on vmware
We thinking of deploying multiple riak nodes on a few servers using vmware, 2 vm with riak nodes per physical server. Is this a bad idea? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
A put is not put in Riak
Good-Evening everyone, I'm reading this maling for minths now wioth great interest, and now we are using Riak, I have a question to not say an issue that I can not figure out. We have migrated leveldb files from Riak 1.3.1 to Riak 2.1.1. The clusters has 6 Nodes and 6 computers. We are running now on CentOS Linux release 7.1.1503 (Core) (we were under Fedora when we are under Riak 1.3.1). We have kept the default Riak paramaters. We are not using 'bucket_type'. The only paramaters we have tuned are: erlang.distribution_buffer_size = 128MB erlang.async_threads = 128 My issue is on a Bucket without sibling (default settings) I'm making a put and when I do the get the value has not been changed, and old value stayed under the key. So, I have made a test: I've made a put of an objet A, I can see in the precommit hook that it's well the object A, but in the postcommit the object (argument passed) is an old one. I thought that I'm not sending the right object, but when I see it in precommit hook, I'm sure now that I send what I expect to send. It's not for all the objects, but it appears time to time under load. Do you already seen this issue ? Do you have an idea or something I can enhance/tune ? Help me mailing list, you are my only hope :-) (dixit Princess Leia). -- Best Reagrds, Patrice Bruno Kuantic --- ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: A put is not put in Riak
This smells like a clock problem. Are you confident all of your servers’ clocks are set to the same time? Updates can be treacherous if clocks are not synchronized and multiple writes to the same key occur in a period of time short enough to fit within the clock skew. I wrote about this after a similar question arose here: http://basho.com/posts/technical/clocks-are-bad-or-welcome-to-distributed-systems/ -John > On Dec 8, 2015, at 9:39 AM, Patrice Bruno wrote: > > Good-Evening everyone, > > I'm reading this maling for minths now wioth great interest, and now we are > using Riak, I have a question to not say an issue that I can not figure out. > > We have migrated leveldb files from Riak 1.3.1 to Riak 2.1.1. > The clusters has 6 Nodes and 6 computers. > We are running now on CentOS Linux release 7.1.1503 (Core) (we were under > Fedora when we are under Riak 1.3.1). > > We have kept the default Riak paramaters. We are not using 'bucket_type'. The > only paramaters we have tuned are: > erlang.distribution_buffer_size = 128MB > erlang.async_threads = 128 > > My issue is on a Bucket without sibling (default settings) I'm making a put > and when I do the get the value has not been changed, and old value stayed > under the key. So, I have made a test: I've made a put of an objet A, I can > see in the precommit hook that it's well the object A, but in the postcommit > the object (argument passed) is an old one. > I thought that I'm not sending the right object, but when I see it in > precommit hook, I'm sure now that I send what I expect to send. > > It's not for all the objects, but it appears time to time under load. > > Do you already seen this issue ? > Do you have an idea or something I can enhance/tune ? > > > Help me mailing list, you are my only hope :-) (dixit Princess Leia). > > -- > > Best Reagrds, > >Patrice Bruno >Kuantic > --- > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak nodes on vmware
Hi Travis, Most likely you will have reduced performance when compared to running a single Riak node on physical hardware. Benchmarking is the only way to be certain. -- Luke Bakken Engineer lbak...@basho.com On Mon, Dec 7, 2015 at 10:57 AM, Travis Kirstine wrote: > We thinking of deploying multiple riak nodes on a few servers using vmware, > 2 vm with riak nodes per physical server. Is this a bad idea? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Forming inputs to MR job
Hello, I'm trying to write a simple MR job using Javascript and hit a wall right at start. I can't figure out how to specify "inputs". Here's the code: curl -XPOST "my_riak_server:8098/mapred" -H "Content-Type: application/json" -d @-
Re: Forming inputs to MR job
Hi Timur,
For your inputs line, try:
"inputs":["mybucket_type", "mybucket"]
Thanks,
Alex
On Wed, Dec 30, 2015 at 3:01 PM Timur Fayruzov
wrote:
> Hello,
>
> I'm trying to write a simple MR job using Javascript and hit a wall right
> at start. I can't figure out how to specify "inputs". Here's the code:
> curl -XPOST "my_riak_server:8098/mapred" -H "Content-Type:
> application/json" -d @- < {
> "input": "my_bucket"
> "query":[{
> "map":{
> "language":"javascript",
> "source":"function(riakObject, keydata, arg) {
> var m = riakObject.values[0].data;
> return [m];
> }"
> }
> }]
> }
> EOF
>
> this returns empty array.
>
> Aside: I know that listing all keys is slow but for now I can live with
> this.
>
> Note, that I'm using non-default bucket type, so the actual location of my
> keys is my_riak_server/types/my_bucket_type/buckets/my_bucket/my_key, but I
> can't figure out how to communicate this location properly in the "input"
> field. I have found this "documentation":
> https://github.com/basho/riak_kv/blob/2.1/src/riak_kv_mapred_json.erl#L101,
> but it does not explain how to specify bucket type and I'm not proficient
> enough in Erlang to follow the code easily. I did not find any other
> documentation on this field.
>
> Following returns all keys successfully, so data is there:
> curl '
> http://my_riak_cluster:8098/types/my_bucket_type/buckets/my_bucket/keys?keys=true
> '
>
> Any pointers are highly appreciated.
>
> Thanks,
> Timur
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Forming inputs to MR job
Take a look at this:https://gist.github.com/binarytemple/6bf84b041db0fabbdc74Specifically : Sample mapreduce job sectionFrom: Timur FayruzovSent: Wednay, December 30, 2015 8:01 PMTo: riak-usersSubject: Forming inputs to MR jobHello,I'm trying to write a simple MR job using _javascript_ and hit a wall right at start. I can't figure out how to specify "inputs". Here's the code:curl -XPOST "my_riak_server:8098/mapred" -H "Content-Type: application/json" -d @- <{ "input": "my_bucket" "query":[{ "map":{ "language":"_javascript_", "source":"function(riakObject, keydata, arg) { var m = riakObject.values[0].data; return [m]; }" } }]}EOFthis returns empty array.Aside: I know that listing all keys is slow but for now I can live with this.Note, that I'm using non-default bucket type, so the actual location of my keys is my_riak_server/types/my_bucket_type/buckets/my_bucket/my_key, but I can't figure out how to communicate this location properly in the "input" field. I have found this "documentation":https://github.com/basho/riak_kv/blob/2.1/src/riak_kv_mapred_json.erl#L101, but it does not explain how to specify bucket type and I'm not proficient enough in Erlang to follow the code easily. I did not find any other documentation on this field. Following returns all keys successfully, so data is there:curl 'http://my_riak_cluster:8098/types/my_bucket_type/buckets/my_bucket/keys?keys=true'Any pointers are highly appreciated.Thanks,Timur
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak nodes on vmware
Hi Travis, Beyond performance reasons, this architecture is a bad idea from an availability perspective. If you lose one physical machine you'll lose two segments of your Riak cluster. And that's generally "not a good thing." -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Dec 7, 2015, at 12:57, Travis Kirstine > wrote: > > We thinking of deploying multiple riak nodes on a few servers using vmware, 2 > vm with riak nodes per physical server. Is this a bad idea? > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Connection multiplexing in the Erlang client
Hi, Does the Erlang Riak client support multiplexing multiple concurrent calls in a single TCP connection? Specifically when using the PB interface (riakc_pb_socket:start_link). Thanks. Regards, Paulo ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: A put is not put in Riak
Patrice, since Riak is distributed it uses a quorum (default 2 when n_val is 3) to read your data. Since you're doing a post commit hook to read your data immediately after you write there may be delay in getting the new value from two of three copies written due simply to timing. So the rule of thumb is to do a w+r>n transaction where you ensure that you write/read with some total value more than your number of replicas. Read these posts (part four talks about RYOW): http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/ http://basho.com/posts/technical/riaks-config-behaviors-part-2/ http://basho.com/posts/technical/riaks-config-behaviors-part-3/ http://basho.com/posts/technical/riaks-config-behaviors-part-4/ -Alexander @siculars http://siculars.posthaven.com Sent from my iRotaryPhone > On Dec 30, 2015, at 11:14, John Daily wrote: > > This smells like a clock problem. Are you confident all of your servers’ > clocks are set to the same time? > > Updates can be treacherous if clocks are not synchronized and multiple writes > to the same key occur in a period of time short enough to fit within the > clock skew. > > I wrote about this after a similar question arose here: > http://basho.com/posts/technical/clocks-are-bad-or-welcome-to-distributed-systems/ > > -John > >> On Dec 8, 2015, at 9:39 AM, Patrice Bruno wrote: >> >> Good-Evening everyone, >> >> I'm reading this maling for minths now wioth great interest, and now we are >> using Riak, I have a question to not say an issue that I can not figure out. >> >> We have migrated leveldb files from Riak 1.3.1 to Riak 2.1.1. >> The clusters has 6 Nodes and 6 computers. >> We are running now on CentOS Linux release 7.1.1503 (Core) (we were under >> Fedora when we are under Riak 1.3.1). >> >> We have kept the default Riak paramaters. We are not using 'bucket_type'. >> The only paramaters we have tuned are: >> erlang.distribution_buffer_size = 128MB >> erlang.async_threads = 128 >> >> My issue is on a Bucket without sibling (default settings) I'm making a put >> and when I do the get the value has not been changed, and old value stayed >> under the key. So, I have made a test: I've made a put of an objet A, I can >> see in the precommit hook that it's well the object A, but in the postcommit >> the object (argument passed) is an old one. >> I thought that I'm not sending the right object, but when I see it in >> precommit hook, I'm sure now that I send what I expect to send. >> >> It's not for all the objects, but it appears time to time under load. >> >> Do you already seen this issue ? >> Do you have an idea or something I can enhance/tune ? >> >> >> Help me mailing list, you are my only hope :-) (dixit Princess Leia). >> >> -- >> >> Best Reagrds, >> >> Patrice Bruno >> Kuantic >> --- >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak PHP Client v3 and official Protocol Buffers support v1
This week we released a significant update to the PHP client to support API
bridge classes other than the Http class that ships with the client
library. This work was released as version 3 due to a few minor interface
changes that break backwards compatibility. Those breaking changes are:
- Basho\Riak\Command\Object\Response::getVclock() has been moved to
Basho\Riak\Object
- Basho\Riak\Command\Object\Response::getStatusCode() has been
renamed Basho\Riak\Command\Object\Response::getCode()
- Basho\Riak\Command\Object\Response::getBody() has been removed
We also released official support for Protocol Buffers using PHP. This
support is delivered via an independent package that has the core library
above included as a dependency. This allows us to keep the same programming
interface while keeping the dependencies needed for PB separate from those
needed by HTTP.
Both releases can be retrieved from packagist.com @
https://packagist.org/packages/basho/
Enjoy!
Chris
--
Sincerely,
Christopher Mancini
-
employee = {
purpose: solve problems with code,
phone:7164625591,
email: cmanc...@basho.com,
github:http://www.github.com/christophermancini
}
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Connection multiplexing in the Erlang client
Paulo,You can find a list of community maintained PBC pooling libraries in the Erlang sub-section of this page: http://docs.basho.com/riak/latest/dev/using/libraries/#Community-LibrariesII was under the impression that Riak Erlang client ships with poolboy so I'm uncertain of the distinction between the different libraries listed.Perhaps someone could comment to clarify?Bryan From: Paulo AlmeidaSent: Wednesday, December 30, 2015 10:40 PMTo: riak-users@lists.basho.comSubject: Connection multiplexing in the Erlang clientHi,Does the Erlang Riak client support multiplexing multiple concurrent calls in a single TCP connection? Specifically when using the PB interface (riakc_pb_socket:start_link).Thanks.Regards,Paulo ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
