Re: Pros/Cons to not storing JSON

[Runar Jordahl]

I am also experimenting with storing objects serialized as binary
data. I think this is a great way to use Riak, and you covered the
pros and cons.

The upcoming support for secondary indices will use metadata, so you
will be able to index your binary data too:
http://blog.epigent.com/2011/05/soon-in-riak-secondary-indices.html

Basically you will pass tuples of index-name and index-data as
metadata to the put operation.

Runar

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Announcing Riak Pipe (BETA)

[Jonathan Langevin]

So to be clear, will Riak Pipe negate the need for someone to use Hadoop in
front of Riak KV?*


Jonathan Langevin
Systems Administrator
Loom Inc.
Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
www.loomlearning.com - Skype: intel352
*


On Mon, Jun 13, 2011 at 9:44 PM, Sean Cribbs  wrote:

>
> On Jun 13, 2011, at 9:40 PM, Antonio Rohman Fernandez wrote:
> >
> > Can a same job be performed by several servers distributed as a
> > cluster? or only 1 Pipe server will do the job and Riak KV will be the
> > distributed cluster?... the good thing of Hadoop is that you can have a
> > Hadoop cluster and a Riak cluster and both clusters would work
> > distributed to perform jobs ( Hadoop ) and to deliver data ( Riak KV
> > )... i wonder how Riak Pipe will work, i hope that it would be like
> > Hadoop and the computation of a job can be distributed.
>
>
> Riak Pipe builds on the abstractions already available in riak_core for
> distribution, namely vnodes and consistent hashing. The existing MapReduce
> in Riak KV also uses these abstractions, but Pipe is a lot smarter about how
> it coordinates and monitors data processing.
>
> Sean Cribbs 
> Developer Advocate
> Basho Technologies, Inc.
> http://basho.com/
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Announcing Riak Pipe (BETA)

[Sean Cribbs]

It will not negate that need necessarily (Hadoop is a large ecosystem), but it 
could be an alternative for run-of-the-mill batch mapred jobs.  There's still a 
long way to go for that to be possible, but we encourage you to push its limits 
and find the pain points.

Sean Cribbs 
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On Jun 14, 2011, at 11:41 AM, Jonathan Langevin wrote:

> So to be clear, will Riak Pipe negate the need for someone to use Hadoop in 
> front of Riak KV?
> 
> 
> Jonathan Langevin
> Systems Administrator
> Loom Inc.
> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - 
> www.loomlearning.com - Skype: intel352
> 
> 
> 
> On Mon, Jun 13, 2011 at 9:44 PM, Sean Cribbs  wrote:
> 
> On Jun 13, 2011, at 9:40 PM, Antonio Rohman Fernandez wrote:
> >
> > Can a same job be performed by several servers distributed as a
> > cluster? or only 1 Pipe server will do the job and Riak KV will be the
> > distributed cluster?... the good thing of Hadoop is that you can have a
> > Hadoop cluster and a Riak cluster and both clusters would work
> > distributed to perform jobs ( Hadoop ) and to deliver data ( Riak KV
> > )... i wonder how Riak Pipe will work, i hope that it would be like
> > Hadoop and the computation of a job can be distributed.
> 
> 
> Riak Pipe builds on the abstractions already available in riak_core for 
> distribution, namely vnodes and consistent hashing. The existing MapReduce in 
> Riak KV also uses these abstractions, but Pipe is a lot smarter about how it 
> coordinates and monitors data processing.
> 
> Sean Cribbs 
> Developer Advocate
> Basho Technologies, Inc.
> http://basho.com/
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Pros/Cons to not storing JSON

[Jonathan Langevin]

Would the Riak standard map/reduce be able to deal with BSON properly, or
would map/reduce functionality be less efficient than if it were inspecting
JSON objects?


*Jonathan Langevin
Systems Administrator
Loom Inc.
Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
www.loomlearning.com - Skype: intel352*



On Thu, Jun 9, 2011 at 5:26 PM, Will Moss  wrote:

> Hey Andrew,
>
> We're using BSON (bsonspec.org), because it stores binary (and other) data
> types better than JSON and is also faster and more wire efficient (sounds
> like about the same reasons you're considering leaving JSON). There are also
> libraries to parse BSON it in just about every language.
>
> I haven't tried using it in a Erlang map-reduce yet (we don't do
> map-reduces for any of our production work), but there is a library out
> there so it shouldn't be too hard.
>
> Will
>
>
> On Thu, Jun 9, 2011 at 2:24 PM, Sean Cribbs  wrote:
>
>> Andrew,
>>
>> I think you're on the right track here, but I might add that you'll want
>> to have upgrade paths available if you're using records -- that is, version
>> them -- so that you can evolve their structure over time.  That could be a
>> little hairy unless done carefully.
>>
>> That said, you could use BERT as the serialization format, making
>> implementing JavaScript M/R functions a little easier, and interop with
>> other languages.
>>
>> Sean Cribbs 
>> Developer Advocate
>> Basho Technologies, Inc.
>> http://basho.com/
>>
>> On Jun 9, 2011, at 5:14 PM, Andrew Berman wrote:
>>
>> > I am using Riak using the Erlang Client API (PB) and I was storing my
>> documents as JSON and then converting them to records when I pull them out
>> of Riak, but I got to thinking that maybe this isn't the greatest approach.
>>  I'm thinking that maybe it's better to store documents just as the record
>> itself (Erlang binary) and then just converting the binary back to the
>> record when I pull them from Riak.  I was wondering what the pros/cons are
>> to this approach.  Here's my list so far:
>> >
>> > Pros:
>> >
>> > Native Erlang is stored, so less time to convert to the record
>> > Better support for nested records
>> > Smaller storage requirements and hence faster on the wire (?)
>> >
>> > Cons:
>> >
>> > Not readable through Rekon (or other utils) without modification
>> > Can't use standard M/R functions which analyze the document (have to
>> write all custom functions using Erlang)
>> > Not portable across languages
>> >
>> > Thanks,
>> >
>> > Andrew
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Pros/Cons to not storing JSON

[Will Moss]

I still haven't had a chance to try out using BSON for a map-reduce, but in
general, parsing BSON is faster than parsing JSON and I'd imagine that's
true in Erlang as well. So, I'd assume using BSON is faster, but you know
what they say about assumptions...

Will


On Tue, Jun 14, 2011 at 9:40 AM, Jonathan Langevin <
jlange...@loomlearning.com> wrote:

> Would the Riak standard map/reduce be able to deal with BSON properly, or
> would map/reduce functionality be less efficient than if it were inspecting
> JSON objects?
>
>  
> * Jonathan Langevin
> Systems Administrator
> Loom Inc.
> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
> www.loomlearning.com - Skype: intel352 *
>
>
>
> On Thu, Jun 9, 2011 at 5:26 PM, Will Moss  wrote:
>
>> Hey Andrew,
>>
>> We're using BSON (bsonspec.org), because it stores binary (and other)
>> data types better than JSON and is also faster and more wire efficient
>> (sounds like about the same reasons you're considering leaving JSON). There
>> are also libraries to parse BSON it in just about every language.
>>
>> I haven't tried using it in a Erlang map-reduce yet (we don't do
>> map-reduces for any of our production work), but there is a library out
>> there so it shouldn't be too hard.
>>
>>  Will
>>
>>
>> On Thu, Jun 9, 2011 at 2:24 PM, Sean Cribbs  wrote:
>>
>>> Andrew,
>>>
>>> I think you're on the right track here, but I might add that you'll want
>>> to have upgrade paths available if you're using records -- that is, version
>>> them -- so that you can evolve their structure over time.  That could be a
>>> little hairy unless done carefully.
>>>
>>> That said, you could use BERT as the serialization format, making
>>> implementing JavaScript M/R functions a little easier, and interop with
>>> other languages.
>>>
>>> Sean Cribbs 
>>> Developer Advocate
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>>
>>> On Jun 9, 2011, at 5:14 PM, Andrew Berman wrote:
>>>
>>> > I am using Riak using the Erlang Client API (PB) and I was storing my
>>> documents as JSON and then converting them to records when I pull them out
>>> of Riak, but I got to thinking that maybe this isn't the greatest approach.
>>>  I'm thinking that maybe it's better to store documents just as the record
>>> itself (Erlang binary) and then just converting the binary back to the
>>> record when I pull them from Riak.  I was wondering what the pros/cons are
>>> to this approach.  Here's my list so far:
>>> >
>>> > Pros:
>>> >
>>> > Native Erlang is stored, so less time to convert to the record
>>> > Better support for nested records
>>> > Smaller storage requirements and hence faster on the wire (?)
>>> >
>>> > Cons:
>>> >
>>> > Not readable through Rekon (or other utils) without modification
>>> > Can't use standard M/R functions which analyze the document (have to
>>> write all custom functions using Erlang)
>>> > Not portable across languages
>>> >
>>> > Thanks,
>>> >
>>> > Andrew
>>> > ___
>>> > riak-users mailing list
>>> > riak-users@lists.basho.com
>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Pros/Cons to not storing JSON

[Jonathan Langevin]

How would we tell the m/r that the data type is BSON? Does the js engine
already have BSON support included?

- Jon Langevin -- sent from my Android phone
On Jun 14, 2011 12:47 PM, "Will Moss"  wrote:
> I still haven't had a chance to try out using BSON for a map-reduce, but
in
> general, parsing BSON is faster than parsing JSON and I'd imagine that's
> true in Erlang as well. So, I'd assume using BSON is faster, but you know
> what they say about assumptions...
>
> Will
>
>
> On Tue, Jun 14, 2011 at 9:40 AM, Jonathan Langevin <
> jlange...@loomlearning.com> wrote:
>
>> Would the Riak standard map/reduce be able to deal with BSON properly, or
>> would map/reduce functionality be less efficient than if it were
inspecting
>> JSON objects?
>>
>> 
>> * Jonathan Langevin
>> Systems Administrator
>> Loom Inc.
>> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
>> www.loomlearning.com - Skype: intel352 *
>>
>>
>>
>> On Thu, Jun 9, 2011 at 5:26 PM, Will Moss  wrote:
>>
>>> Hey Andrew,
>>>
>>> We're using BSON (bsonspec.org), because it stores binary (and other)
>>> data types better than JSON and is also faster and more wire efficient
>>> (sounds like about the same reasons you're considering leaving JSON).
There
>>> are also libraries to parse BSON it in just about every language.
>>>
>>> I haven't tried using it in a Erlang map-reduce yet (we don't do
>>> map-reduces for any of our production work), but there is a library out
>>> there so it shouldn't be too hard.
>>>
>>> Will
>>>
>>>
>>> On Thu, Jun 9, 2011 at 2:24 PM, Sean Cribbs  wrote:
>>>
 Andrew,

 I think you're on the right track here, but I might add that you'll
want
 to have upgrade paths available if you're using records -- that is,
version
 them -- so that you can evolve their structure over time. That could be
a
 little hairy unless done carefully.

 That said, you could use BERT as the serialization format, making
 implementing JavaScript M/R functions a little easier, and interop with
 other languages.

 Sean Cribbs 
 Developer Advocate
 Basho Technologies, Inc.
 http://basho.com/

 On Jun 9, 2011, at 5:14 PM, Andrew Berman wrote:

 > I am using Riak using the Erlang Client API (PB) and I was storing my
 documents as JSON and then converting them to records when I pull them
out
 of Riak, but I got to thinking that maybe this isn't the greatest
approach.
 I'm thinking that maybe it's better to store documents just as the
record
 itself (Erlang binary) and then just converting the binary back to the
 record when I pull them from Riak. I was wondering what the pros/cons
are
 to this approach. Here's my list so far:
 >
 > Pros:
 >
 > Native Erlang is stored, so less time to convert to the record
 > Better support for nested records
 > Smaller storage requirements and hence faster on the wire (?)
 >
 > Cons:
 >
 > Not readable through Rekon (or other utils) without modification
 > Can't use standard M/R functions which analyze the document (have to
 write all custom functions using Erlang)
 > Not portable across languages
 >
 > Thanks,
 >
 > Andrew
 > ___
 > riak-users mailing list
 > riak-users@lists.basho.com
 > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


 ___
 riak-users mailing list
 riak-users@lists.basho.com
 http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

>>>
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Pros/Cons to not storing JSON

[Will Moss]

You wouldn't be able to do it in Javascript, you'd have to use Erlang. Then,
you'd just import the BSON library to parse the values.

Will


On Tue, Jun 14, 2011 at 9:50 AM, Jonathan Langevin <
jlange...@loomlearning.com> wrote:

> How would we tell the m/r that the data type is BSON? Does the js engine
> already have BSON support included?
>
> - Jon Langevin -- sent from my Android phone
> On Jun 14, 2011 12:47 PM, "Will Moss"  wrote:
> > I still haven't had a chance to try out using BSON for a map-reduce, but
> in
> > general, parsing BSON is faster than parsing JSON and I'd imagine that's
> > true in Erlang as well. So, I'd assume using BSON is faster, but you know
> > what they say about assumptions...
> >
> > Will
> >
> >
> > On Tue, Jun 14, 2011 at 9:40 AM, Jonathan Langevin <
> > jlange...@loomlearning.com> wrote:
> >
> >> Would the Riak standard map/reduce be able to deal with BSON properly,
> or
> >> would map/reduce functionality be less efficient than if it were
> inspecting
> >> JSON objects?
> >>
> >> 
> >> * Jonathan Langevin
>
> >> Systems Administrator
> >> Loom Inc.
> >> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
> >> www.loomlearning.com - Skype: intel352 *
> >>
> >>
> >>
> >> On Thu, Jun 9, 2011 at 5:26 PM, Will Moss  wrote:
> >>
> >>> Hey Andrew,
> >>>
> >>> We're using BSON (bsonspec.org), because it stores binary (and other)
> >>> data types better than JSON and is also faster and more wire efficient
> >>> (sounds like about the same reasons you're considering leaving JSON).
> There
> >>> are also libraries to parse BSON it in just about every language.
> >>>
> >>> I haven't tried using it in a Erlang map-reduce yet (we don't do
> >>> map-reduces for any of our production work), but there is a library out
> >>> there so it shouldn't be too hard.
> >>>
> >>> Will
> >>>
> >>>
> >>> On Thu, Jun 9, 2011 at 2:24 PM, Sean Cribbs  wrote:
> >>>
>  Andrew,
> 
>  I think you're on the right track here, but I might add that you'll
> want
>  to have upgrade paths available if you're using records -- that is,
> version
>  them -- so that you can evolve their structure over time. That could
> be a
>  little hairy unless done carefully.
> 
>  That said, you could use BERT as the serialization format, making
>  implementing JavaScript M/R functions a little easier, and interop
> with
>  other languages.
> 
>  Sean Cribbs 
>  Developer Advocate
>  Basho Technologies, Inc.
>  http://basho.com/
> 
>  On Jun 9, 2011, at 5:14 PM, Andrew Berman wrote:
> 
>  > I am using Riak using the Erlang Client API (PB) and I was storing
> my
>  documents as JSON and then converting them to records when I pull them
> out
>  of Riak, but I got to thinking that maybe this isn't the greatest
> approach.
>  I'm thinking that maybe it's better to store documents just as the
> record
>  itself (Erlang binary) and then just converting the binary back to the
>  record when I pull them from Riak. I was wondering what the pros/cons
> are
>  to this approach. Here's my list so far:
>  >
>  > Pros:
>  >
>  > Native Erlang is stored, so less time to convert to the record
>  > Better support for nested records
>  > Smaller storage requirements and hence faster on the wire (?)
>  >
>  > Cons:
>  >
>  > Not readable through Rekon (or other utils) without modification
>  > Can't use standard M/R functions which analyze the document (have to
>  write all custom functions using Erlang)
>  > Not portable across languages
>  >
>  > Thanks,
>  >
>  > Andrew
>  > ___
>  > riak-users mailing list
>  > riak-users@lists.basho.com
>  > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
>  ___
>  riak-users mailing list
>  riak-users@lists.basho.com
>  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> >>>
> >>>
> >>> ___
> >>> riak-users mailing list
> >>> riak-users@lists.basho.com
> >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>
> >>>
> >>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Pros/Cons to not storing JSON

[Jonathan Langevin]

Ah, gotcha. I'll have to stick with JSON then, thanks :-)

- Jon Langevin -- sent from my Android phone
On Jun 14, 2011 12:58 PM, "Will Moss"  wrote:
> You wouldn't be able to do it in Javascript, you'd have to use Erlang.
Then,
> you'd just import the BSON library to parse the values.
>
> Will
>
>
> On Tue, Jun 14, 2011 at 9:50 AM, Jonathan Langevin <
> jlange...@loomlearning.com> wrote:
>
>> How would we tell the m/r that the data type is BSON? Does the js engine
>> already have BSON support included?
>>
>> - Jon Langevin -- sent from my Android phone
>> On Jun 14, 2011 12:47 PM, "Will Moss"  wrote:
>> > I still haven't had a chance to try out using BSON for a map-reduce,
but
>> in
>> > general, parsing BSON is faster than parsing JSON and I'd imagine
that's
>> > true in Erlang as well. So, I'd assume using BSON is faster, but you
know
>> > what they say about assumptions...
>> >
>> > Will
>> >
>> >
>> > On Tue, Jun 14, 2011 at 9:40 AM, Jonathan Langevin <
>> > jlange...@loomlearning.com> wrote:
>> >
>> >> Would the Riak standard map/reduce be able to deal with BSON properly,
>> or
>> >> would map/reduce functionality be less efficient than if it were
>> inspecting
>> >> JSON objects?
>> >>
>> >> 
>> >> * Jonathan Langevin
>>
>> >> Systems Administrator
>> >> Loom Inc.
>> >> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
>> >> www.loomlearning.com - Skype: intel352 *
>> >>
>> >>
>> >>
>> >> On Thu, Jun 9, 2011 at 5:26 PM, Will Moss  wrote:
>> >>
>> >>> Hey Andrew,
>> >>>
>> >>> We're using BSON (bsonspec.org), because it stores binary (and other)
>> >>> data types better than JSON and is also faster and more wire
efficient
>> >>> (sounds like about the same reasons you're considering leaving JSON).
>> There
>> >>> are also libraries to parse BSON it in just about every language.
>> >>>
>> >>> I haven't tried using it in a Erlang map-reduce yet (we don't do
>> >>> map-reduces for any of our production work), but there is a library
out
>> >>> there so it shouldn't be too hard.
>> >>>
>> >>> Will
>> >>>
>> >>>
>> >>> On Thu, Jun 9, 2011 at 2:24 PM, Sean Cribbs  wrote:
>> >>>
>>  Andrew,
>> 
>>  I think you're on the right track here, but I might add that you'll
>> want
>>  to have upgrade paths available if you're using records -- that is,
>> version
>>  them -- so that you can evolve their structure over time. That could
>> be a
>>  little hairy unless done carefully.
>> 
>>  That said, you could use BERT as the serialization format, making
>>  implementing JavaScript M/R functions a little easier, and interop
>> with
>>  other languages.
>> 
>>  Sean Cribbs 
>>  Developer Advocate
>>  Basho Technologies, Inc.
>>  http://basho.com/
>> 
>>  On Jun 9, 2011, at 5:14 PM, Andrew Berman wrote:
>> 
>>  > I am using Riak using the Erlang Client API (PB) and I was storing
>> my
>>  documents as JSON and then converting them to records when I pull
them
>> out
>>  of Riak, but I got to thinking that maybe this isn't the greatest
>> approach.
>>  I'm thinking that maybe it's better to store documents just as the
>> record
>>  itself (Erlang binary) and then just converting the binary back to
the
>>  record when I pull them from Riak. I was wondering what the
pros/cons
>> are
>>  to this approach. Here's my list so far:
>>  >
>>  > Pros:
>>  >
>>  > Native Erlang is stored, so less time to convert to the record
>>  > Better support for nested records
>>  > Smaller storage requirements and hence faster on the wire (?)
>>  >
>>  > Cons:
>>  >
>>  > Not readable through Rekon (or other utils) without modification
>>  > Can't use standard M/R functions which analyze the document (have
to
>>  write all custom functions using Erlang)
>>  > Not portable across languages
>>  >
>>  > Thanks,
>>  >
>>  > Andrew
>>  > ___
>>  > riak-users mailing list
>>  > riak-users@lists.basho.com
>>  > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>>  ___
>>  riak-users mailing list
>>  riak-users@lists.basho.com
>>  http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> >>>
>> >>>
>> >>> ___
>> >>> riak-users mailing list
>> >>> riak-users@lists.basho.com
>> >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >>>
>> >>>
>> >>
>>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Correct way to use pbc/mapreduce to do multiget where keys and bucket names are binary values?

[Jacques]

Does anybody have any insight on my questions below?

Thanks,
Jacques


On Tue, Jun 7, 2011 at 6:12 PM, Jacques  wrote:

> I've been working on this.  I have it working with an anonymous javascript
> function.  I was hoping to move it to using the "map_object_value" built-in
> erlang function.  However, when I attempt to use this function, I get
> failures if any of my keys don't exist.  Is there a way to construct my map
> phase so that it gracefully handles not founds and just returns what it
> finds?
>
> Also, using this function, is there a way to return the bucket and key
> names?  Or can one assume the response order is identical to the order of
> inputs?
>
> I have my current constructions below.
>
> Thanks for any help.
>
> Jacques
>
>
>
> *Working Function Definition (including not founds)*
>   private static final OtpErlangAtom FUNCTION_TYPE = new
> OtpErlangAtom("jsanon");
>   private static final OtpErlangBinary FUNCTION_VALUE = new
> OtpErlangBinary("function(v){ return [v]; }".getBytes());
>   private static final OtpErlangTuple MAP_REDUCE_FUNCTION = new
> OtpErlangTuple(new OtpErlangObject[] { FUNCTION_TYPE, FUNCTION_VALUE });
>
> *Non-working Function Definition (works fine if all input list exists.
>  Fails on not founds.)*
>   private static final OtpErlangAtom FUNCTION_TYPE = new
> OtpErlangAtom("modfun");
>   private static final OtpErlangAtom FUNCTION_MODULE = new
> OtpErlangAtom("riak_kv_mapreduce");
>   private static final OtpErlangAtom FUNCTION_NAME = new
> OtpErlangAtom("map_object_value");
>   private static final OtpErlangTuple MAP_REDUCE_FUNCTION = new
> OtpErlangTuple(new OtpErlangObject[] { FUNCTION_TYPE, FUNCTION_MODULE,
> FUNCTION_NAME });
>
>
> *Final construction of Query object for submission (using either of
> above).*
>   private static final OtpErlangTuple MAP_REDUCE_PHASE = new
> OtpErlangTuple(new OtpErlangObject[]{ATOM_MAP, MAP_REDUCE_FUNCTION,
> ATOM_NONE, KEEP_TRUE});
>   private static final OtpErlangList MAP_REDUCE_PHASES = new
> OtpErlangList(new OtpErlangTuple[]{MAP_REDUCE_PHASE});
>   private static final OtpErlangTuple FULL_QUERY = new OtpErlangTuple(new
> OtpErlangObject[] { ATOM_QUERY, MAP_REDUCE_PHASES });
>
>
>
> On Tue, Jun 7, 2011 at 7:57 AM, Russell Brown wrote:
>
>>
>> On 7 Jun 2011, at 15:29, Jacques wrote:
>>
>> Have you had any success with reading the server response?
>>
>>
>> Yes. Sorry I didn't post a reply.
>>
>> It was trivial (but fraught, see below), I just used the OtpInputStream to
>> deserialize the byte array returned from pbc.MapReduceResponse.
>>
>> Like:
>>
>>  ByteString bs = resp.getContent();
>> OtpInputStream is = new OtpInputStream(bs.toByteArray());
>> OtpErlangObject result = is.read_any();
>> // and then all sorts of looping, sniffing types, unpacking etc
>>
>>
>> Caveats:
>>
>> You have a pain unpacking any reasonably complex result.
>> The ETF specification drops this doozy about "strings":
>> http://www.erlang.org/doc/apps/erts/erl_ext_dist.html#id85596.
>> My first test actually returned a list of [0,1,2,3...200] and Jinterface
>> helpfully turned that into a string for me.
>>
>> That aside it is certainly feasible to use Jinterface to
>> serialize/deserialize Map/Reduce jobs/results.
>>
>> Cheers
>>
>> Russell
>>
>>
>> Thanks,
>> Jacques
>>
>> On Sat, Jun 4, 2011 at 1:19 PM, Russell Brown wrote:
>>
>>>
>>> On 4 Jun 2011, at 18:22, Jacques wrote:
>>>
>>>  I like the sound of option 3 also. I'll have a look at it this weekend
 and get back to you.

>>>
>>> Awesome!  Thanks.  If you can give me a point in the right direction
>>> regarding the correct typing approach and what not, I'm up for giving it a
>>> shot as well.
>>>
>>>
>>> Ok, I have a half working hack. It isn't pretty 'cos the Jinterface API
>>> is verbose. I've hacked the pbc.MapReduceBuilder to encode the job as
>>> "application/x-erlang-binary" and submit that, but really this code should
>>> be in a separate class, maybe using the output from
>>> MapReduceBuilder.getJSON() as input. That way you can get the feature
>>> without patching the client.
>>>
>>> What I haven't done is decode the response from Riak yet. If you want  a
>>> pointer here is a gist of the (unclean) hack. It could use a lot of work,
>>> but it proves the concept:
>>>
>>> https://gist.github.com/1008293
>>>
>>> The gist is just the diff so you can apply it as a patch
>>> to src/main/java/com/basho/riak/pbc/mapreduce/MapReduceBuilder.java if you
>>> want to play with it.
>>>
>>> You'll have to add Jinterface to your pom too.
>>>
>>> 
>>> 
>>>  org.erlang.otp
>>>  jinterface
>>>  1.5.4
>>> 
>>>
>>> I think it is best to put the result decoding outside the library too.
>>> I'm going to hack up a poc for that now, but I thought I'd post what I have
>>> thus far.
>>>
>>> Cheers
>>>
>>> Russell
>>>
>>>
>>> Thanks again,
>>> Jacques
>>>
>>>
>> ___
>> riak-users maili

Re: Riak Recap for June 10 - 12

[Andrew Tunnell-Jones]

On Mon, Jun 13, 2011 at 7:48 PM, Mark Phillips  wrote:
> * Kresten Krab Thorup's "Bring Riak to the Mobile Platform" --->
> http://www.erlang-factory.com/upload/presentations/413/ErlangFactory2011_KrestenKrabThorup_RiakSync.pdf

That looks interesting, is it open-source?

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com