loading erlang terms
Erlang has a function, file:consult, that allows one to load erlang terms from a file. Is there a content type I can use when I'm loading data from disk using curl that ensures that what I'm loading are interpreted as erlang terms, not strings? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: loading erlang terms
On Mon, Oct 25, 2010 at 07:34:13PM -0700, Dan Reverri wrote:
> I don't understand your use case; can you expand on what you are doing?
Perhaps this in the category of "too much information," but since you asked, I
had written an OAI-PMH provider in erlang. My backing store is a dets file,
which I load into memory and read into an ets table when the server starts up.
The memory-resident database consists of keys associated with values which are
erlang terms. I use these terms for filtering queries conforming to the OAI-PMH
protocol specification. The way I create the dets file is to begin with a file
of plain text. Suppose I had one term in it (I'll typically have hundreds or
thousands), it might look like this:
{oai_dc,"AEP-WYS97",
{{2004,10,28},{17,21,25}},
["aep"],
"/storage/aep2003/metadata/oai_dc/oai_dc-AEP-WYS97.xml"}.
In erlang itself, if I've called the file AEP-WYS97, I can do something like
this:
1> {_, Term} = file:consult("AEP-WYS97").
{ok,[{oai_dc,"AEP-WYS97",
{{2004,10,28},{17,21,25}},
["aep"],
"/storage/aep2003/metadata/oai_dc/oai_dc-AEP-WYS97.xml"}]}
2> Term.
[{oai_dc,"AEP-WYS97",
{{2004,10,28},{17,21,25}},
["aep"],
"/storage/aep2003/metadata/oai_dc/oai_dc-AEP-WYS97.xml"}]
3> element(2, hd(Term)).
"AEP-WYS97"
Now, if I replace dets/ets with riak, I can do two things (I think). The first
would be to write an erlang function that does something like the above. But
the second (which is where my question comes into play) is, if I can load
arbitrary data types into riak, then is there a way I can specify that what I'm
loading are erlang terms. If I don't do that, and load strings, I run into this
sort of nastiness:
(r...@127.0.0.1)295> {ok, Tk} = C:get(<<"oai_dc">>, <<"dsalhensley-m025">>).
(r...@127.0.0.1)296> V = riak_object:get_value(Tk).
<<"{oai_dc,dsalhensley-m025,{{2004,11,9},{15,38,27}},
[dsal,dsal:hensley],/storage/dsal/hensley/"...>>
(r...@127.0.0.1)300> A = binary_to_list(V).
"{oai_dc,dsalhensley-m025,{{2004,11,9},{15,38,27}},
[dsal,dsal:hensley],
/storage/dsal/hensley/metadata/oai_dc/oai_dc-dsalhensley-m025.xml}"
(r...@127.0.0.1)301> {ok, Tokens, _} = erl_scan:string(A).
{ok,[{'{',1},
{atom,1,oai_dc},
{',',1},
{atom,1,dsalhensley},
{'-',1},
{atom,1,m025},
{',',1},
{'{',1},
{'{',1},
{integer,1,2004},
{',',1},
{integer,1,11},
{',',1},
{integer,1,9},
{'}',1},
{',',1},
{'{',1},
{integer,1,15},
{',',1},
{integer,1,38},
{',',1},
{integer,1,27},
{'}',1},
{'}',1},
{',',1},
{'[',...},
{...}|...],
1}
So far, so good. However:
(r...@127.0.0.1)302> erl_parse:parse_term(Tokens).
{error,{1,erl_parse,["syntax error before: ","'/'"]}}
This happens because I have unescaped quotation marks in the input. If I had
intended my input to be strings, then that's something I should have taken care
of beforehand, but if I intend the input to be terms, then I shouldn't have to.
I'm prepared to use option 1, above, or else convert my terms into strings, and
parse strings, but string handling in erlang is not particularly efficient, so
before I go either of those routes, I'm curious whether I can tell riak, this
is an erlang term, have it be stored as such, so when I go to use it, my code
doesn't have to do any further transformation that isn't related to the job at
hand (filtering).
I'm relatively new to riak (less than a week), so perhaps there is a better
approach entirely.
Thanks.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: loading erlang terms
On Tue, Oct 26, 2010 at 09:13:17AM -0700, Seth Falcon wrote: > Perhaps I'm missing something... How about using Erlang's native > binary term serialization. > > B = term_to_binary(Terms), > % now store B in Riak > > % ... > > % now get B out and do: > binary_to_term(B) Thanks, Seth. That's a fine solution if I'm loading terms into riak from an erlang client. My question, though, is what if I want to use the http interface to do the loading. Since I can load arbitrary data types into riak, it strikes me that, in principle, there might be a way to do this. In practice, though, perhaps there isn't. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: loading erlang terms
Btw, the documentation has this: encode_term(Object, Term) -> riakc_obj:update_value(Object, term_to_binary(Term, [compressed]), <<"application/x-erlang-term">>). This gave me the idea that perhaps by encoding the content type as application/x-erlang-term I could get things to work as I envisioned they might, but I couldn't. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
erlang client: dropping connections
I have successfully connected an erlang client to a running riak server and
gotten some work done, but if the server returns an error response, the
connection is dropped. A typical error might result in a message such as
** exception error: no function clause matching [whatever]
Grovelling through the source code, I've tried establishing a connection by
using this option:
{ok,Pid} = riakc_pb_socket:start_link({123,456,78,90}, 8087, [{auto_reconnect,
true}]).
But that doesn't seem to help. Reconnecting is easy enough, but the problem is
annoying. Has anyone else run into this, and is there a fix?
Thanks.
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: loading erlang terms
Thanks, all.
I think my issues at this time boil down to a difference between how I
can interact with a running riak system using the shell in riak-admin
attach, and using another shell to contact the running system. So, for
example, following the README file, this works fine (shell prompts
omitted):
RiakNode = 'r...@123.456.78.90'.
{ok, C} = riak:client_connect(RiakNode).
{ok, Terms} = file:consult("/path/to/file").
Term = hd(Terms).
Object = riak_object:new(<<"oai_test">>, <<"AEP-WYS97">>, Term).
C:put(Object,1).
{ok, O} = C:get(<<"oai_test">>, <<"AEP-WYS97">>, 1).
V = riak_object:get_value(O).
V.
{oai_dc,"AEP-WYS97",
{{2004,10,28},{17,21,25}},
["aep"],
"/storage/aep2003/metadata/oai_dc/oai_dc-AEP-WYS97.xml"}
element(2, V).
"AEP-WYS97"
The above proves that I can put and get an erlang term.
Now let's try this:
erl -pa /usr/local/riak-0.13.0/rel/riak/lib/riak_core-0.13.0/ebin
/usr/local/riak-0.13.0/rel/riak/lib/riak_kv-0.13.0/ebin
/usr/local/riak-0.13.0/rel/riak/lib/riakc-1.0.1/ebin
/usr/local/riak-0.13.0/rel/riak/lib/protobuffs-0.5.0/ebin
1> {ok,Pid} = riakc_pb_socket:start_link("128.135.53.253", 8087).
{ok,<0.33.0>}
2> {ok, C} = riakc_pb_socket:get(Pid, <<"people">>, <<"chas">>).
{ok,{riakc_obj,<<"people">>,<<"chas">>,
<<107,206,97,96,96,96,204,96,202,5,82,44,236,70,107,239,
100,48,37,50,230,177,50,...>>,
[{{dict,3,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[],[],[],[],[],[],[],[],...}}},
<<"Charles Blair">>}],
undefined,undefined}}
3> {ok, C} = riakc_pb_socket:get(Pid, <<"oai_test">>,
<<"AEP-WYS97">>).
=ERROR REPORT 27-Oct-2010::16:34:48 ===
** Generic server <0.33.0> terminating
** Last message in was {tcp_closed,#Port<0.913>}
** When Server state ==
{state,"128.135.53.253",8087,false,false,undefined,
undefined,
{[],[]},
1,[],100}
** Reason for termination ==
** disconnected
** exception exit: disconnected
4>
In step 2 I can retrieve a key whose value is a string. In step 3, I
fail to retrieve the value. But this is the value that I successfully
put and fetched up above. I have failed to put this value from a script.
The record types returned by the riak-attached shell, and a remote
shell are not the same.
C:get(<<"people">>, <<"chas">>, 1).
{ok,{r_object,<<"people">>,<<"chas">>,
[{r_content,{dict,5,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[[<<"Links">>]],[],[],[],[],[],[],[],...}}},
<<"Charles Blair">>}],
[{<<7,50,173,220>>,{1,63455400780}}],
{dict,1,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[],[],[],[],[],[],[],[],[],...}}},
undefined}}
riakc_pb_socket:get(Pid, <<"people">>, <<"chas">>).
{ok,{riakc_obj,<<"people">>,<<"chas">>,
<<107,206,97,96,96,96,204,96,202,5,82,44,236,70,107,239,
100,48,37,50,230,177,50,...>>,
[{{dict,3,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[],[],[],[],[],[],[],[],...}}},
<<"Charles Blair">>}],
undefined,undefined}}
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: loading erlang terms
Thanks for spotting the "brainos". But even without those, here is a
repeat of the experiment.
erl -pa /usr/local/riak-0.13.0/rel/riak/lib/riak_core-0.13.0/ebin
/usr/local/riak-0.13.0/rel/riak/lib/riak_kv-0.13.0/ebin
/usr/local/riak-0.13.0/rel/riak/lib/riakc-1.0.1/ebin
/usr/local/riak-0.13.0/rel/riak/lib/protobuffs-0.5.0/ebin
Erlang R14B (erts-5.8.1) [source] [smp:16:16] [rq:16] [async-threads:0] [hipe]
[kernel-poll:false]
Eshell V5.8.1 (abort with ^G)
1> {ok,Pid} = riakc_pb_socket:start_link("123.456.78.90", 8087).
{ok,<0.33.0>}
2> riakc_pb_socket:get(Pid, <<"people">>, <<"chas">>).
{ok,{riakc_obj,<<"people">>,<<"chas">>,
<<107,206,97,96,96,224,206,96,202,5,82,44,236,70,107,239,
100,48,37,50,230,177,50,...>>,
[{{dict,4,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],
[[<<"Links">>,
{{<<"images">>,<<"abc">>},<<"avatar">>},
{{<<"posi"...>>,<<...>>},<<"posi"...>>}]],
[],[],[],[],[],[],[],...}}},
<<"Charles Blair">>}],
undefined,undefined}}
3> riakc_pb_socket:get(Pid, <<"oai_test">>,<<"AEP-WYS97">>).
{error,disconnected}
4>
=ERROR REPORT 28-Oct-2010::14:30:25 ===
** Generic server <0.33.0> terminating
** Last message in was {tcp_closed,#Port<0.913>}
** When Server state == {state,"128.135.53.253",8087,false,false,undefined,
undefined,
{[],[]},
1,[],100}
** Reason for termination ==
** disconnected
** exception error: disconnected
4>
The directly attached shell also provides an error message:
=ERROR REPORT 28-Oct-2010::14:30:25 ===
** Generic server <0.3660.0> terminating
** Last message in was {tcp,#Port<0.2753>,
[9|
<<10,8,111,97,105,95,116,101,115,116,18,9,65,69,
80,45,87,89,83,57,55>>]}
** When Server state == {state,#Port<0.2753>,
{riak_client,'r...@128.135.53.253',
<<5,45,154,107>>},
undefined,undefined}
** Reason for termination ==
** {function_clause,
[{riakclient_pb,encode,
[oai_dc,
{oai_dc,"AEP-WYS97",
{{2004,10,28},{17,21,25}},
["aep"],
"/storage/aep2003/metadata/oai_dc/oai_dc-AEP-WYS97.xml"}]},
{riakclient_pb,pack,5},
{riakclient_pb,iolist,2},
{riakclient_pb,encode,2},
{riakclient_pb,pack,5},
{riakclient_pb,pack,5},
{riakclient_pb,iolist,2},
{riakc_pb,encode,1}]}
---
I've repeated this several times with the same result. Trigger the error as
many times as I want, but in the directly attached shell I can do this
afterwards:
C:get(<<"oai_test">>, <<"AEP-WYS97">>, 1).
{ok,{r_object,<<"oai_test">>,<<"AEP-WYS97">>,
[{r_content,{dict,2,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],...},
{{[],[],[],[],[],[],[],[],[],[],...}}},
{oai_dc,"AEP-WYS97",
{{2004,10,28},{17,21,25}},
["aep"],
"/storage/aep2003/metadata/oai_dc/oai_dc-AEP-WYS97.xml"}}],
I can't find the source for riakclient_pb in my distribution to see what might
be going on:
% find riak-0.13.0 -name riakclient_pb"*"
riak-0.13.0/deps/riakc/ebin/riakclient_pb.beam
riak-0.13.0/deps/riakc/include/riakclient_pb.hrl
riak-0.13.0/rel/riak/lib/riakc-1.0.1/ebin/riakclient_pb.beam
riak-0.13.0/rel/riak/lib/riakc-1.0.1/include/riakclient_pb.hrl
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
