Re: Link walking with a java client

[Deepak Balasubramanyam]

This looks like a bug. The code to walk links via a HTTP client
works perfectly. The same code fails when the PB client is used. The POJO
attached in this email reproduces the problem.

I searched the email archives and existing issues and found no trace of
this problem. Please run the POJO by swapping the clients returned from the
getClient() method to reproduce the problem. I can create a bug report once
someone from the dev team confirms this really is a bug.

Riak client pom:
   
  com.basho.riak
  riak-client
  1.0.5
  

Riak server version - 1.1.2. Built from source.
4 nodes running on 1 machine. OS - Linux mint.

On Sun, May 27, 2012 at 10:05 AM, Deepak Balasubramanyam <
deepak.b...@gmail.com> wrote:

> Hi,
>
> I have a cluster that contains 2 buckets. A bucket named 'usersMine'
> contains the key 'user2', which is linked to several keys (about 10) under
> a bucket named userPreferences. The relationship exists under the name
> 'myPref'. A user and a preference have String values.
>
> I can successfully traverse the link over HTTP using the following URL -
>
> curl -v localhost:8091/riak/usersMine/user2/_,myPref,1
>
>   
> > User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6
> OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
> > Host: localhost:8091
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
> < Expires: Sun, 27 May 2012 04:12:39 GMT
> < Date: Sun, 27 May 2012 04:02:39 GMT
> < Content-Type: multipart/mixed; boundary=IYGfKNqjGdco9ddfyjRP1Utzfi2
> < Content-Length: 3271
> <
>
> --IYGfKNqjGdco9ddfyjRP1Utzfi2
> Content-Type: multipart/mixed; boundary=3YVES0x2tFnUDOdTzfn1OGS6uMt
>
> --3YVES0x2tFnUDOdTzfn1OGS6uMt
> X-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/nvy7mSwZTImMfKUKpodpIvCwA=
> Location: /riak/userPreferences/preference3004
> Content-Type: text/plain; charset=UTF-8
> Link: ; rel="up"
> Etag: 5GucnGSk4TjQc8BO1eNLyI
> Last-Modified: Sun, 27 May 2012 03:54:29 GMT
>
> junk
>
> <<  Truncated  >>
> junk
> --3YVES0x2tFnUDOdTzfn1OGS6uMt--
>
> --IYGfKNqjGdco9ddfyjRP1Utzfi2--
>   
>
> However when I use the java client to walk the link, I get a
> ClassCastException.
>
>   
> Java code:
>   
> private void getAllLinks()
> {
> String user="user2";
> IRiakClient riakClient = null;
> try
> {
> long past = System.currentTimeMillis();
> riakClient = RiakFactory.pbcClient("localhost",8081);
> Bucket userBucket =
> riakClient.fetchBucket("usersMine").execute();
> DefaultRiakObject user1 =(DefaultRiakObject)
> userBucket.fetch(user).execute();
> List links = user1.getLinks();
> System.out.println(links.size());
> WalkResult execute =
> riakClient.walk(user1).addStep("userPreferences", "myPref",true).execute();
> Iterator iterator = execute.iterator();
> while(iterator.hasNext())
> {
> Object next = iterator.next();
> System.out.println(next);
> }
> long now = System.currentTimeMillis();
> System.out.println("Retrieval in " + (now-past) + " ms");
> }
> catch (Exception e)
> {
> e.printStackTrace();
> }
> finally
> {
> if(riakClient != null)
> {
> riakClient.shutdown();
> }
> }
> }
>
>   
> Stack:
>   
> java.lang.ClassCastException: java.lang.String cannot be cast to
> java.util.List
> at
> com.basho.riak.client.raw.pbc.PBClientAdapter.linkWalkSecondPhase(PBClientAdapter.java:380)
>  at
> com.basho.riak.client.raw.pbc.PBClientAdapter.linkWalk(PBClientAdapter.java:325)
> at com.basho.riak.client.query.LinkWalk.execute(LinkWalk.java:63)
>  at com.chatterbox.persistence.riak.RiakTest.getAllLinks(RiakTest.java:81)
> at com.chatterbox.persistence.riak.RiakTest.main(RiakTest.java:25)
>   
>
> At

Re: Link walking with a java client

[Deepak Balasubramanyam]

Brian,

Yes I read about the hack somewhere in your documentation. My understanding
is that the link walking operation will cease to work via HTTP after links
to a node grow beyond a particular number. This happens because a HTTP
header is used to send link related data and there are limits around how
much data a header can hold.

I was interested in exploring if the PB client can overcome this
limitation. The use case I am interested in is this - If a single node
links to 10k nodes, how much time would it take a link walker to visit all
nodes via the PB client ? What sort of client would you recommend for
something like this ?

I know that linking is only meant to be used as a lightweight feature.
Would 10k links to a node be considered lightweight ?

Thanks
Deepak

On Tue, May 29, 2012 at 9:33 PM, Brian Roach  wrote:

> Deepak -
>
> I'll take a look at it this week, but more than likely it's a bug.
>
> Link walking is a REST only operation as far as Riak’s interfaces are
> concerned. Link Walking in the protocol buffers Java client is a hack that
> issues two m/r jobs to the protocol buffers interface (the first constructs
> the inputs to the second by walking the links, the second returns the data).
>
> Thanks,
> Brian Roach
>
> On May 27, 2012, at 7:19 AM, Deepak Balasubramanyam wrote:
>
> > This looks like a bug. The code to walk links via a HTTP client works
> perfectly. The same code fails when the PB client is used. The POJO
> attached in this email reproduces the problem.
> >
> > I searched the email archives and existing issues and found no trace of
> this problem. Please run the POJO by swapping the clients returned from the
> getClient() method to reproduce the problem. I can create a bug report once
> someone from the dev team confirms this really is a bug.
> >
> > Riak client pom:
> >   
> >   com.basho.riak
> >   riak-client
> >   1.0.5
> >   
> >
> > Riak server version - 1.1.2. Built from source.
> > 4 nodes running on 1 machine. OS - Linux mint.
> >
> > On Sun, May 27, 2012 at 10:05 AM, Deepak Balasubramanyam <
> deepak.b...@gmail.com> wrote:
> > Hi,
> >
> > I have a cluster that contains 2 buckets. A bucket named 'usersMine'
> contains the key 'user2', which is linked to several keys (about 10) under
> a bucket named userPreferences. The relationship exists under the name
> 'myPref'. A user and a preference have String values.
> >
> > I can successfully traverse the link over HTTP using the following URL -
> >
> > curl -v localhost:8091/riak/usersMine/user2/_,myPref,1
> >
> >  
> 
> > > User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6
> OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
> > > Host: localhost:8091
> > > Accept: */*
> > >
> > < HTTP/1.1 200 OK
> > < Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
> > < Expires: Sun, 27 May 2012 04:12:39 GMT
> > < Date: Sun, 27 May 2012 04:02:39 GMT
> > < Content-Type: multipart/mixed; boundary=IYGfKNqjGdco9ddfyjRP1Utzfi2
> > < Content-Length: 3271
> > <
> >
> > --IYGfKNqjGdco9ddfyjRP1Utzfi2
> > Content-Type: multipart/mixed; boundary=3YVES0x2tFnUDOdTzfn1OGS6uMt
> >
> > --3YVES0x2tFnUDOdTzfn1OGS6uMt
> > X-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/nvy7mSwZTImMfKUKpodpIvCwA=
> > Location: /riak/userPreferences/preference3004
> > Content-Type: text/plain; charset=UTF-8
> > Link: ; rel="up"
> > Etag: 5GucnGSk4TjQc8BO1eNLyI
> > Last-Modified: Sun, 27 May 2012 03:54:29 GMT
> >
> > junk
> >
> > <<  Truncated  >>
> > junk
> > --3YVES0x2tFnUDOdTzfn1OGS6uMt--
> >
> > --IYGfKNqjGdco9ddfyjRP1Utzfi2--
> >  
> 
> >
> > However when I use the java client to walk the link, I get a
> ClassCastException.
> >
> >  
> 
> > Java code:
> >  
> 
> > private void getAllLinks()
> > {
> > String user="user2";
> > IRiakClient riakClient = null;
> > try
> > {
> > long past = System.currentTimeMillis();
> > riakClient = RiakFactory.pbcClient("localhost",8081);
> > Bucke

Re: Link walking with a java client

[Deepak Balasubramanyam]

Created bug 1341 <https://issues.basho.com/show_bug.cgi?id=1341> to track
this.

Can you help me with the questions below ? I am trying to gauge how
expensive linking is in comparison to using a secondary index to track a
node.

Thanks
Deepak

On Wed, May 30, 2012 at 9:42 AM, Deepak Balasubramanyam <
deepak.b...@gmail.com> wrote:

> Brian,
>
> Yes I read about the hack somewhere in your documentation. My
> understanding is that the link walking operation will cease to work via
> HTTP after links to a node grow beyond a particular number. This happens
> because a HTTP header is used to send link related data and there are
> limits around how much data a header can hold.
>
> I was interested in exploring if the PB client can overcome this
> limitation. The use case I am interested in is this - If a single node
> links to 10k nodes, how much time would it take a link walker to visit all
> nodes via the PB client ? What sort of client would you recommend for
> something like this ?
>
> I know that linking is only meant to be used as a lightweight feature.
> Would 10k links to a node be considered lightweight ?
>
> Thanks
> Deepak
>
> On Tue, May 29, 2012 at 9:33 PM, Brian Roach  wrote:
>
>> Deepak -
>>
>> I'll take a look at it this week, but more than likely it's a bug.
>>
>> Link walking is a REST only operation as far as Riak’s interfaces are
>> concerned. Link Walking in the protocol buffers Java client is a hack that
>> issues two m/r jobs to the protocol buffers interface (the first constructs
>> the inputs to the second by walking the links, the second returns the data).
>>
>> Thanks,
>> Brian Roach
>>
>> On May 27, 2012, at 7:19 AM, Deepak Balasubramanyam wrote:
>>
>> > This looks like a bug. The code to walk links via a HTTP client works
>> perfectly. The same code fails when the PB client is used. The POJO
>> attached in this email reproduces the problem.
>> >
>> > I searched the email archives and existing issues and found no trace of
>> this problem. Please run the POJO by swapping the clients returned from the
>> getClient() method to reproduce the problem. I can create a bug report once
>> someone from the dev team confirms this really is a bug.
>> >
>> > Riak client pom:
>> >   
>> >   com.basho.riak
>> >   riak-client
>> >   1.0.5
>> >   
>> >
>> > Riak server version - 1.1.2. Built from source.
>> > 4 nodes running on 1 machine. OS - Linux mint.
>> >
>> > On Sun, May 27, 2012 at 10:05 AM, Deepak Balasubramanyam <
>> deepak.b...@gmail.com> wrote:
>> > Hi,
>> >
>> > I have a cluster that contains 2 buckets. A bucket named 'usersMine'
>> contains the key 'user2', which is linked to several keys (about 10) under
>> a bucket named userPreferences. The relationship exists under the name
>> 'myPref'. A user and a preference have String values.
>> >
>> > I can successfully traverse the link over HTTP using the following URL -
>> >
>> > curl -v localhost:8091/riak/usersMine/user2/_,myPref,1
>> >
>> >  
>> 
>> > > User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6
>> OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
>> > > Host: localhost:8091
>> > > Accept: */*
>> > >
>> > < HTTP/1.1 200 OK
>> > < Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
>> > < Expires: Sun, 27 May 2012 04:12:39 GMT
>> > < Date: Sun, 27 May 2012 04:02:39 GMT
>> > < Content-Type: multipart/mixed; boundary=IYGfKNqjGdco9ddfyjRP1Utzfi2
>> > < Content-Length: 3271
>> > <
>> >
>> > --IYGfKNqjGdco9ddfyjRP1Utzfi2
>> > Content-Type: multipart/mixed; boundary=3YVES0x2tFnUDOdTzfn1OGS6uMt
>> >
>> > --3YVES0x2tFnUDOdTzfn1OGS6uMt
>> > X-Riak-Vclock: a85hYGBgzGDKBVIcMRuuc/nvy7mSwZTImMfKUKpodpIvCwA=
>> > Location: /riak/userPreferences/preference3004
>> > Content-Type: text/plain; charset=UTF-8
>> > Link: ; rel="up"
>> > Etag: 5GucnGSk4TjQc8BO1eNLyI
>> > Last-Modified: Sun, 27 May 2012 03:54:29 GMT
>> >
>> > junk
>> >
>> > <<  Truncated  >>
>> > junk
>> > --3YVES0x2tFnUDOdTzfn1OGS6uMt--
>> >
>> > --IYGfKNqjGdco9ddfyjRP1Utzfi2--
>> > --

Re: MapReduce - sorting via the Java client

[Deepak Balasubramanyam]

I was researching the same question and found the answer after some
tinkering around. You need a JS source function. You must write the JS
source code for the sort algorithm too.

JSSourceFunction phaseFunction = new JSSourceFunction(
"function(v) {" +
"return v.sort(function(a, b) {" +
"return a.field - b.field ;" +
"}" +
 ");" +
 "}");
IndexQuery iq = new BinValueQuery(BinIndex.named("indx_name"),"bucket",
"value"); // get an index
MapReduceResult execute = client.mapReduce(iq).addMapPhase(new
NamedJSFunction("Riak.mapValuesJson"),
false).addReducePhase(phaseFunction).execute();
String resultRaw = execute.getResultRaw(); // Raw data
Collection result = execute.getResult(YourType .class); //
serialized to a domain object

The result of this operation will depend on what you return in your
reduce() function. The result may be a YourType or YourType[] etc. I wanted
to mention that since Jackson can complain about serialization issues if
you return the wrong values.

Anyone know how to get this into a wiki ?

Thanks
Deepak Bala

On Wed, Aug 29, 2012 at 5:09 PM, Oved Machlev wrote:

>  Hi,
>
> ** **
>
> How can I use the sorting functions over 'real' fields in the stored
> objects? How can I set the field on which I want to sort by?
>
> ** **
>
> It seems that the following refers to the key, in my case it's a UUID, and
> therefore useless to sort by:
>
> ** **
>
> MapReduceResult result = riakClient.
>
> mapReduce("some_bucket").
>
> addMapPhase(*new* NamedJSFunction(
> "Riak.mapValuesJson"), *false*).
>
> *addReducePhase(new NamedErlangFunction(
> "riak_kv_mapreduce", "reduce_sort"), true).*
>
> execute();
>
> ** **
>
> ** **
>
> thanks,
>
> Oved.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: MapReduce - sorting via the Java client

[Deepak Balasubramanyam]

Hi Mark,

Sure, I can help whip up a page when I find the time. Do I get access to
edit the wiki or should I email ro...@basho.com ?

Thanks
Deepak Bala

On Tue, Sep 11, 2012 at 10:58 PM, Mark Phillips  wrote:

> Hi Deepak,
>
> In-line
>
> On Tue, Sep 11, 2012 at 10:16 AM, Deepak Balasubramanyam
>  wrote:
> > I was researching the same question and found the answer after some
> > tinkering around. You need a JS source function. You must write the JS
> > source code for the sort algorithm too.
> >
> > JSSourceFunction phaseFunction = new JSSourceFunction(
> > "function(v) {" +
> > "return v.sort(function(a, b) {" +
> > "return a.field - b.field ;" +
> > "}" +
> >  ");" +
> >  "}");
> > IndexQuery iq = new BinValueQuery(BinIndex.named("indx_name"),"bucket",
> > "value"); // get an index
> > MapReduceResult execute = client.mapReduce(iq).addMapPhase(new
> > NamedJSFunction("Riak.mapValuesJson"),
> > false).addReducePhase(phaseFunction).execute();
> > String resultRaw = execute.getResultRaw(); // Raw data
> > Collection result = execute.getResult(YourType .class); //
> > serialized to a domain object
> >
> > The result of this operation will depend on what you return in your
> reduce()
> > function. The result may be a YourType or YourType[] etc. I wanted to
> > mention that since Jackson can complain about serialization issues if you
> > return the wrong values.
> >
> > Anyone know how to get this into a wiki ?
>
>
> This would probably fit better on the Riak-Java-Client wiki.
>
> https://github.com/basho/riak-java-client/wiki/TODO
>
> There's an open TODO for MapReduce examples and I know Brian would
> love some help with it. Care to take a stab at adding that code?
>
> Mark
>
> >
> > Thanks
> > Deepak Bala
> >
> > On Wed, Aug 29, 2012 at 5:09 PM, Oved Machlev 
> > wrote:
> >>
> >> Hi,
> >>
> >>
> >>
> >> How can I use the sorting functions over 'real' fields in the stored
> >> objects? How can I set the field on which I want to sort by?
> >>
> >>
> >>
> >> It seems that the following refers to the key, in my case it's a UUID,
> and
> >> therefore useless to sort by:
> >>
> >>
> >>
> >> MapReduceResult result = riakClient.
> >>
> >> mapReduce("some_bucket").
> >>
> >> addMapPhase(new
> >> NamedJSFunction("Riak.mapValuesJson"), false).
> >>
> >> addReducePhase(new
> >> NamedErlangFunction("riak_kv_mapreduce", "reduce_sort"), true).
> >>
> >> execute();
> >>
> >>
> >>
> >>
> >>
> >> thanks,
> >>
> >> Oved.
> >>
> >>
> >> ___
> >> riak-users mailing list
> >> riak-users@lists.basho.com
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Java map-reduce and result keys

[Deepak Balasubramanyam]

Hi,

I've written a map reduce query on the riak java client like so...

client.mapReduce(BUCKET).addKeyFilter(keyFilter)
.addLinkPhase(BUCKET, "_", false)
.addMapPhase(new NamedJSFunction("Riak.mapValuesJson"),
false)
.addReducePhase(phaseFunction).execute();
Collection types = result.getResult(MyType.class);

This is the class definition for MyType

public class MyType
{
@RiakKey
private String myKey;
private String property1;
private String property2;

/*  Getters / Setters go here */
}

When the object mapper deserializes the results into Collection,
none of the types have the myKey property populated in them. When I
debugged the calls made by riak I realized that the result of the /mapred
call does not contain any key information in the body. It only contains the
value that each key represents. So that explains why the keys are null in
the result.

On the contrary, a link walk in riak returns the Location header for each
multipart form entry in the response (Location: /riak/bucket/key). So I
guess there is at least some way to tweak a client to parse the location to
get the keys, but you lose out on the map-reduce goodness.

Is there some way a map-reduce query can be formed to allow the resulting
type's RiakKey to be populated ? What are my options ?

Thanks
Deepak Bala
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java map-reduce and result keys

[Deepak Balasubramanyam]

Thanks for the tip Russell. I managed to get this done.

I have one more question and a suggestion.


   - Is the name keyData a misnomer in the mapping function ->  function(
   value, keyData, arg)  ? keyData is a String that carries the bucket
   name. The value object carries more information. value[0].data represents
   the data itself among other properties that represent user metadata; riak
   links; indexes; key; etc etc.
   - If mapValuesJson: function(value, keyData, arg) can take an input
   argument and use it to add a json property whose value is the riak key, a
   custom function would not be necessary. It would be a neat little system
   feature which I take would need amendments to
mapred_builtins.js<https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js>
   .
  - Is riak_kv the repo to make contributions to system built functions
  ? There are quite a few open pull requests on that repo, so I'm not sure
  where this change should go.


Thanks
Deepak Bala

On Fri, Sep 14, 2012 at 7:16 PM, Russell Brown  wrote:

>
> On 14 Sep 2012, at 14:24, Deepak Balasubramanyam wrote:
>
> > Hi,
> >
> > I've written a map reduce query on the riak java client like so...
> >
> > client.mapReduce(BUCKET).addKeyFilter(keyFilter)
> > .addLinkPhase(BUCKET, "_", false)
> > .addMapPhase(new
> NamedJSFunction("Riak.mapValuesJson"), false)
> > .addReducePhase(phaseFunction).execute();
> > Collection types = result.getResult(MyType.class);
> >
> > This is the class definition for MyType
> >
> > public class MyType
> > {
> > @RiakKey
> > private String myKey;
> > private String property1;
> > private String property2;
> >
> > /*  Getters / Setters go here */
> > }
> >
> > When the object mapper deserializes the results into Collection,
> none of the types have the myKey property populated in them. When I
> debugged the calls made by riak I realized that the result of the /mapred
> call does not contain any key information in the body. It only contains the
> value that each key represents. So that explains why the keys are null in
> the result.
>
> The Java client doesn't add the value of the @RiakKey field to the value
> stored in riak.
>
> >
> > On the contrary, a link walk in riak returns the Location header for
> each multipart form entry in the response (Location: /riak/bucket/key). So
> I guess there is at least some way to tweak a client to parse the location
> to get the keys, but you lose out on the map-reduce goodness.
> >
> > Is there some way a map-reduce query can be formed to allow the
> resulting type's RiakKey to be populated ? What are my options ?
>
> A custom Map function may do what you want. Get the Key from the Key Data
> passed to the Map function and add it to the JSON value returned. Jackson
> should then take care of de-serialising it into your values.
>
> Cheers
>
> Russell
>
> >
> > Thanks
> > Deepak Bala
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Java map-reduce and result keys

[Deepak Balasubramanyam]

Thank you Mark. I'll contribute to the repo when I find time.

Thanks
Deepak Bala

On Mon, Sep 17, 2012 at 5:45 AM, Mark Phillips  wrote:

> Addendum:
>
> In the interest of completeness, it's worth noting that the Erlang
> builtins are here:
>
> https://github.com/basho/riak_kv/blob/master/src/riak_kv_mapreduce.erl
>
> Mark
>
> On Sun, Sep 16, 2012 at 5:08 PM, Mark Phillips  wrote:
> > Hi Deepak,
> >
> > I can answer one of two questions here...
> >
> > On Fri, Sep 14, 2012 at 9:42 AM, Deepak Balasubramanyam
> >  wrote:
> >> Thanks for the tip Russell. I managed to get this done.
> >>
> >> I have one more question and a suggestion.
> >>
> >> Is the name keyData a misnomer in the mapping function ->
>  function(value,
> >> keyData, arg)  ? keyData is a String that carries the bucket name. The
> value
> >> object carries more information. value[0].data represents the data
> itself
> >> among other properties that represent user metadata; riak links;
> indexes;
> >> key; etc etc.
> >> If mapValuesJson: function(value, keyData, arg) can take an input
> argument
> >> and use it to add a json property whose value is the riak key, a custom
> >> function would not be necessary. It would be a neat little system
> feature
> >> which I take would need amendments to mapred_builtins.js.
> >>
> >> Is riak_kv the repo to make contributions to system built functions ?
> There
> >> are quite a few open pull requests on that repo, so I'm not sure where
> this
> >> change should go.
> >>
> >
> > The riak_kv repo is indeed the repo you want for this. Specifically,
> > you'll want to start here:
> >
> > https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js
> >
> > As far as the large queue of pull requests go, we're working on
> > merging/addressing the backlog (for this and a few other repos). Send
> > yours our way. It'll get some love when time allows.
> >
> > Thanks.
> >
> > Mark
> >
> >
> >>
> >> Thanks
> >> Deepak Bala
> >>
> >> On Fri, Sep 14, 2012 at 7:16 PM, Russell Brown 
> wrote:
> >>>
> >>>
> >>> On 14 Sep 2012, at 14:24, Deepak Balasubramanyam wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I've written a map reduce query on the riak java client like so...
> >>> >
> >>> > client.mapReduce(BUCKET).addKeyFilter(keyFilter)
> >>> > .addLinkPhase(BUCKET, "_", false)
> >>> > .addMapPhase(new
> >>> > NamedJSFunction("Riak.mapValuesJson"), false)
> >>> > .addReducePhase(phaseFunction).execute();
> >>> > Collection types = result.getResult(MyType.class);
> >>> >
> >>> > This is the class definition for MyType
> >>> >
> >>> > public class MyType
> >>> > {
> >>> > @RiakKey
> >>> > private String myKey;
> >>> > private String property1;
> >>> > private String property2;
> >>> >
> >>> > /*  Getters / Setters go here */
> >>> > }
> >>> >
> >>> > When the object mapper deserializes the results into
> Collection,
> >>> > none of the types have the myKey property populated in them. When I
> debugged
> >>> > the calls made by riak I realized that the result of the /mapred
> call does
> >>> > not contain any key information in the body. It only contains the
> value that
> >>> > each key represents. So that explains why the keys are null in the
> result.
> >>>
> >>> The Java client doesn't add the value of the @RiakKey field to the
> value
> >>> stored in riak.
> >>>
> >>> >
> >>> > On the contrary, a link walk in riak returns the Location header for
> >>> > each multipart form entry in the response (Location:
> /riak/bucket/key). So I
> >>> > guess there is at least some way to tweak a client to parse the
> location to
> >>> > get the keys, but you lose out on the map-reduce goodness.
> >>> >
> >>> > Is there some way a map-reduce query can be formed to allow the
> >>> > resulting type's RiakKey to be populated ? What are my options ?
> >>>
> >>> A custom Map function may do what you want. Get the Key from the Key
> Data
> >>> passed to the Map function and add it to the JSON value returned.
> Jackson
> >>> should then take care of de-serialising it into your values.
> >>>
> >>> Cheers
> >>>
> >>> Russell
> >>>
> >>> >
> >>> > Thanks
> >>> > Deepak Bala
> >>> > ___
> >>> > riak-users mailing list
> >>> > riak-users@lists.basho.com
> >>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>
> >>
> >>
> >> ___
> >> riak-users mailing list
> >> riak-users@lists.basho.com
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Evolving data model

[Deepak Balasubramanyam]

It really depends on your use case. Guido's tips on not-null and
ignore-properties will help. With *JsonIgnoreProperties* you can also
specify which ones you would like to ignore. That helps check ignoring
properties that you know *should* exist.

Converting your documents to a new format is a lot of work. I assume your
buckets are filled with millions of keys that need to take on a new format.
The complexity depends on your Q / R / W values and the format of your keys
(are they sequential / random with a uniform distribution ?). You will need
to take into account the eventual consistency of the system and how the
migration should be done on a live environment. It can get messy.
Performance would be a concern too since you will be iterating through all
keys in a bucket which Riak frowns upon. Not to mention recreating indexes
/ links / metadata (if applicable).

It will be easier to just keep adding attributes and ignore the ones that
you know older models will not understand. Think of the model as a JSON
object that is like a protobuf message (a .proto file if you have come
across one) or a representation of a row on a table. It is easy to add
data, but when you delete something and someone's model was depending on
it, it can get ugly.

Thanks
Deepak Bala

On Wed, Sep 19, 2012 at 10:03 PM, Guido Medina wrote:

>  We have done similar things, but it always depends on the available
> tools, your requirements and needs, I will give you a short example, our
> main application uses a standard SQL, for historical data we use Riak, data
> that is not changing, for example, audit trails, daily chunks from
> different sources and so on.
>
> We make sure our data is 100% JSON compatible, and our tools are the
> available JSON libraries, it is fair easy to add new "columns" to your data
> (I know, columns right?), and keep your fetching still valid by ignoring
> deprecated properties and when writing back just overwriting old data, that
> way, your schema can change all the time without losing old data and
> evolving at the same time.
>
> 2i is fine to stamp and migrate if you wish, since it makes your code
> tedious if you need to be checking for nulls all the time, so for migration
> you can simply use JSON transformers, from this property to another (from
> old to new schema) without even coding, but it will be up to your tools.
>
> In Java for example all that can be aid with Jackson which happens to be
> the defacto JSON Riak Java client provider, here is a list of few
> annotations to accomplish most of the things are you worried about:
>
>
> *@JsonIgnoreProperties(ignoreUnknown=true)* (Say an old property just got
> deprecated and you want your POJO not to throw exceptions while converting
> from JSON to your POJO)
> *@JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)* (Saves lot of
> space)
>
>
>   @Override
>   *@JsonProperty("ranges")* (Say, a property just changed its type, and
> because of that, I need to map it to a new property, in this case, I don't
> have a list of integers anymore but a list of ranges, so a transformation
> is required...)
>   public List getEntries()
>   {
> return intRangeCollection.getRangesAsArray();
>   }
>
>   *@JsonProperty("ranges")*
>   public void setEntries(final List entries)
>   {
>
> this.intRangeCollection=IntRangeCollection.buildIntRangesCollectionFromArrays(entries);
>   }
>
> Well, there is so much I could show you, but, examples are limitless, so
> depending on your use cases, you will figure your own ways to keep your
> code kind of clean and your schema changing constantly.
>
> Hope that helps,
>
> Guido.
>
>
> On 19/09/12 16:08, Pinney Colton wrote:
>
> Wow, that's a pretty big question.
>
>  IMO, it depends upon what you're doing with your data.  Personally, I'm
> storing up to 4 different "versions" of the same data in Riak, each version
> is optimized for different types of analytical operations.
>
>  That's probably not ideal for everybody.  Heck, storing 4 copies of my
> data isn't even optimal  for me from a storage perspective - but it does
> help optimize performance of different queries.  I care more about that
> than disk or memory.
>
> On Tue, Sep 18, 2012 at 5:55 PM, Allen Johnson wrote:
>
>> Hey everyone,
>>
>> I'm beginning to experiment with Riak and I'm trying to better
>> understand how to model my data.  One question I have at the moment is
>> how to evolve my data with a data store such as Riak?  I know that
>> it's schema-less and that I can add new fields as needed but I'm
>> thinking more about the existing documents.
>>
>> For example, say hypothetically, that I have a fairly successful
>> riak-based app.  As the application code continues to evolve over
>> several versions I begin to find a "better" way to model my data.  By
>> this time I have already stored many, many documents.  What is the
>> appropriate path here?  Do I version my documents with metadata and
>> rely on my application code to continue to deal with old-style

HTTP Client bug

[Deepak Balasubramanyam]

Hi,

My servers complain about a leaky apache DateUtils class during hot
deployments. This is a known
bugin http
4.1.x and has been fixed in 4.2.x. Can the next version of
riak-client be updated to include the newer HTTP client (the
current version is 4.1.1 ) ? Is the riak-client compatible with httpclient
4.2.x as it is ?

Thanks
Deepak Bala
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: HTTP Client bug

[Deepak Balasubramanyam]

Guido,

Reading the comments on that pull request, I get the feeling that pcl is
suggesting that upgrading to 4.2.2 is another way to solve the problem. I
dont think that route was taken to resolve the issue. Quoting his comment...

 upgrading your dependency to 4.2.2 once it becomes available will
*also*resolve the issue
>


I dont see a change to pom.xml during this pull. I tried the riak-client
1.0.6 and it still depends on 4.1.x. So the upgrade would be necessary to
fix the memory leak.

Thanks
Deepak Bala

On Tue, Sep 25, 2012 at 12:35 AM, Guido Medina wrote:

>  Deepak,
>
>   I think that was fixed on the following pull request but, you would have
> to compile your own client from github which is easy; or, did you get the
> new 1.0.6? I'm not sure if it is already available at maven repos, anyways,
> there is a lot going on in the riak client in java, I would suggest you to
> keep an eye on the github, many good bugs getting fixed quickly.
>
> https://github.com/basho/riak-java-client/pull/159
>
> Best regards,
>
> Guido
>
>
>
> On 24/09/12 19:45, Deepak Balasubramanyam wrote:
>
> Hi,
>
>  My servers complain about a leaky apache DateUtils class during hot
> deployments. This is a known 
> bug<https://issues.apache.org/jira/browse/HTTPCLIENT-1216>in http 4.1.x and 
> has been fixed in 4.2.x. Can the next version of
> riak-client be updated to include the newer HTTP client (the
> current version is 4.1.1 ) ? Is the riak-client compatible with httpclient
> 4.2.x as it is ?
>
>  Thanks
> Deepak Bala
>
>
> ___
> riak-users mailing 
> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: HTTP Client bug

[Deepak Balasubramanyam]

Yes I noticed that it is a work around. I can bump the version beyond 4.2.x
and re-compile the riak-client, but can riak make a note of this as a
known-issue through a Github-Issue or buzilla (or whatever it is that you
use to track tasks) so you can get around to it later when you find the
time ?

Thank you for your help Guido.

- Deepak

On Tue, Sep 25, 2012 at 12:48 AM, Guido Medina wrote:

>  The pull request doesn't fix the HttpClient issue, it is more like a
> workaround the issue in the code, if you notice, even if they didn't
> upgrade the POM, the issue won't be reproduced by the riak java client.
>
>
> On 24/09/12 20:13, Deepak Balasubramanyam wrote:
>
> Guido,
>
>  Reading the comments on that pull request, I get the feeling that pcl is
> suggesting that upgrading to 4.2.2 is another way to solve the problem. I
> dont think that route was taken to resolve the issue. Quoting his
> comment...
>
>   upgrading your dependency to 4.2.2 once it becomes available will 
> *also*resolve the issue
>>
>
>
>  I dont see a change to pom.xml during this pull. I tried the riak-client
> 1.0.6 and it still depends on 4.1.x. So the upgrade would be necessary to
> fix the memory leak.
>
>  Thanks
> Deepak Bala
>
> On Tue, Sep 25, 2012 at 12:35 AM, Guido Medina 
> wrote:
>
>>  Deepak,
>>
>>   I think that was fixed on the following pull request but, you would
>> have to compile your own client from github which is easy; or, did you get
>> the new 1.0.6? I'm not sure if it is already available at maven repos,
>> anyways, there is a lot going on in the riak client in java, I would
>> suggest you to keep an eye on the github, many good bugs getting fixed
>> quickly.
>>
>> https://github.com/basho/riak-java-client/pull/159
>>
>> Best regards,
>>
>> Guido
>>
>>
>>
>> On 24/09/12 19:45, Deepak Balasubramanyam wrote:
>>
>>  Hi,
>>
>>  My servers complain about a leaky apache DateUtils class during hot
>> deployments. This is a known 
>> bug<https://issues.apache.org/jira/browse/HTTPCLIENT-1216>in http 4.1.x and 
>> has been fixed in 4.2.x. Can the next version of
>> riak-client be updated to include the newer HTTP client (the
>> current version is 4.1.1 ) ? Is the riak-client compatible with httpclient
>> 4.2.x as it is ?
>>
>>  Thanks
>> Deepak Bala
>>
>>
>>  ___
>> riak-users mailing 
>> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Bug on riak-client 1.0.6 - Riak index property not serialized

[Deepak Balasubramanyam]

I switched to the java riak-client 1.0.6 to take it for a spin, and several
test cases of mine failed. Upon further investigation I found that any
member variable that contains the @RiakIndex annotation does not serialize
into Riak anymore. You can reproduce the problem with the following type

@JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
public class MyType
{
public static final String SOME_NAME_STR_REFERENCE = "blah";
@RiakKey
private String myKey;
@RiakIndex(name=SOME_NAME_STR_REFERENCE)
private String indexedProp;
 // Getters and setters go here
}

Make a call to bucket.store(typeRef).execute() followed by a GET
to /riak/myBucket/myKey. The indexedProp element will be missing in the
json for calls made on riak-client version 1.0.6 but will be available when
the call is made from riak-client version 1.0.5.

Thanks
Deepak Bala
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Bug on riak-client 1.0.6 - Riak index property not serialized

[Deepak Balasubramanyam]

I see. All index annotations will need the @JsonProperty annotation to
survive serialization then. I guess another side effect which will break
backward compatibility will be experienced by map-reduce queries on any
entity that contains these indexes. They will not return the indexed value
either since the json value will no longer contain them. Unless of course
you use @JsonProperty.

Thanks for letting me know about the change.

Deepak Bala

On Tue, Sep 25, 2012 at 9:04 PM, Brian Roach  wrote:

> This is not a bug, it's a feature ;) The fact that the index values
> were were being serialized was actually not consistant with our other
> annotated fields, so I made the decision to bring it in line and not
> do so.
>
> I actually highlighted the change in the CHANGELOG:
>
> af12b6c - The default JSONConverter now supports multiple values via a
> @RiakIndex annotated Set<> (Integer or String). Note this also brings
> serialization/deserialization in line with our other annotaded fields
> in that these values will not be present in the resulting JSON. To
> override this behavior the Jackson @JsonProperty annotation can be
> supplied
>
> Thanks,
> Brian Roach
>
> On Tue, Sep 25, 2012 at 1:01 AM, Deepak Balasubramanyam
>  wrote:
> > I switched to the java riak-client 1.0.6 to take it for a spin, and
> several
> > test cases of mine failed. Upon further investigation I found that any
> > member variable that contains the @RiakIndex annotation does not
> serialize
> > into Riak anymore. You can reproduce the problem with the following type
> >
> > @JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
> > public class MyType
> > {
> > public static final String SOME_NAME_STR_REFERENCE = "blah";
> > @RiakKey
> > private String myKey;
> > @RiakIndex(name=SOME_NAME_STR_REFERENCE)
> > private String indexedProp;
> > // Getters and setters go here
> > }
> >
> > Make a call to bucket.store(typeRef).execute() followed by a GET to
> > /riak/myBucket/myKey. The indexedProp element will be missing in the json
> > for calls made on riak-client version 1.0.6 but will be available when
> the
> > call is made from riak-client version 1.0.5.
> >
> > Thanks
> > Deepak Bala
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Bug on riak-client 1.0.6 - Riak index property not serialized

[Deepak Balasubramanyam]

Guido,

I guess that is the other side of the coin. Thank for the tip.

Deepak Bala

On Wed, Sep 26, 2012 at 1:51 PM, Guido Medina wrote:

>  Deepak,
>
>   That's why in our company we have index annotations at method level, for
> us we agreed that an index is not really a POJO property but a
> pseudo/computed property, it was a waste for us to add a field just because
> an index, so we have our own branch of the Riak java client where the
> methods can be annotated as index, I requested this change before but
> something stopped it, I do believe indexes are kind of special and could go
> on method without need to wait for other annotations to behave the same:
>
> Then you would just need:
>
> *@RiakIndex(name="something")**
> **@JsonIgnore *// Optional if you want this index to behave like a pseudo
> property and not be serialized at all*
> *public int getSomething(){
>   return someCalculatedInteger;
> }
>
> Here is a link to our branch:
>
> *git clone https://github.com/guidomedina/riak-java-client.git**
> **git checkout temetra-riak-java-client*
> *mvn clean install*
>
> Then on target you will see riak-java-client-1.0.6t.jar
>
> Best regards,
>
> Guido
>
>
> On 26/09/12 07:10, Deepak Balasubramanyam wrote:
>
> I see. All index annotations will need the @JsonProperty annotation to
> survive serialization then. I guess another side effect which will break
> backward compatibility will be experienced by map-reduce queries on any
> entity that contains these indexes. They will not return the indexed value
> either since the json value will no longer contain them. Unless of course
> you use @JsonProperty.
>
>  Thanks for letting me know about the change.
>
>  Deepak Bala
>
> On Tue, Sep 25, 2012 at 9:04 PM, Brian Roach  wrote:
>
>> This is not a bug, it's a feature ;) The fact that the index values
>> were were being serialized was actually not consistant with our other
>> annotated fields, so I made the decision to bring it in line and not
>> do so.
>>
>> I actually highlighted the change in the CHANGELOG:
>>
>> af12b6c - The default JSONConverter now supports multiple values via a
>> @RiakIndex annotated Set<> (Integer or String). Note this also brings
>> serialization/deserialization in line with our other annotaded fields
>> in that these values will not be present in the resulting JSON. To
>> override this behavior the Jackson @JsonProperty annotation can be
>> supplied
>>
>> Thanks,
>> Brian Roach
>>
>> On Tue, Sep 25, 2012 at 1:01 AM, Deepak Balasubramanyam
>>  wrote:
>> > I switched to the java riak-client 1.0.6 to take it for a spin, and
>> several
>> > test cases of mine failed. Upon further investigation I found that any
>> > member variable that contains the @RiakIndex annotation does not
>> serialize
>> > into Riak anymore. You can reproduce the problem with the following type
>> >
>> > @JsonSerialize(include=JsonSerialize.Inclusion.NON_NULL)
>> > public class MyType
>> > {
>> > public static final String SOME_NAME_STR_REFERENCE = "blah";
>> > @RiakKey
>> > private String myKey;
>> > @RiakIndex(name=SOME_NAME_STR_REFERENCE)
>> > private String indexedProp;
>> > // Getters and setters go here
>> > }
>> >
>> > Make a call to bucket.store(typeRef).execute() followed by a GET to
>> > /riak/myBucket/myKey. The indexedProp element will be missing in the
>> json
>> > for calls made on riak-client version 1.0.6 but will be available when
>> the
>> > call is made from riak-client version 1.0.5.
>> >
>> > Thanks
>> > Deepak Bala
>> >
>>  > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>>
>
>
>
> ___
> riak-users mailing 
> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Indexes on 64 bit values

[Deepak Balasubramanyam]

Hi folks,

I have a query regarding riak indexes. My current understanding is that
riak cannot index numbers whose representation needs more than 32 bits.
There is an issue on the basho repo to address this -
https://github.com/basho/riak-java-client/issues/112 . Do you have any idea
when it will be fixed ?

I've mentioned a way to work around this problem on a comment on the issue.
The work-around would not be efficient since JS code would have to go
through N rows to drop ones that do not fit a particular criteria on the
number. If there is a better solution to the problem, I'd like to hear your
thoughts on that also.

Thank you for your time.
-Deepak
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Dissecting a riak crash

[Deepak Balasubramanyam]

Hi,

I'm running a 4 member riak cluster that failed today for unknown reasons.
At a particular point in time, I noticed that the HTTP riak connector began
refusing connections to my client

java.lang.RuntimeException: com.basho.riak.client.RiakRetryFailedException:
com.basho.riak.client.http.response.RiakIORuntimeException:
org.apache.http.conn.HttpHostConnectException: *Connection to
http://localhost: refused*

I initiated a start command on all clusters just in case some of them were
down. Cluster 1 (main) and cluster 4 accepted the start command. The others
maintained that they were already running.

After the system was back up I began my hunt for error / crash reports and
found nothing substantial. The error.log file on cluster 1 contained this

2012-12-06 08:44:35.322 [error] <0.15539.62> gen_fsm <0.15539.62> in state
wait_pipeline_shutdown terminated with reason: *{sink_died,normal}*
2012-12-06 08:44:35.423 [error] <0.15539.62> CRASH REPORT Process
<0.15539.62> with 0 neighbours crashed with reason: {sink_died,normal}
2012-12-06 08:44:35.428 [error] <0.164.0> Supervisor riak_pipe_builder_sup
had child undefined started with {riak_pipe_builder,start_link,undefined}
at <0.15539.62> exit with reason {sink_died,normal} in context
child_terminated

However I believe this error is normal, as mentioned here -
http://riak-users.197444.n3.nabble.com/Safe-to-ignore-a-sink-died-normal-error-td3824989.html

I ran riaknostic on the failed cluster through the 'riak-admin diag'
command and it did not find any crash reports either. I'm
running riak-1.1.4.

Do you have any pointers / advise that I could use ? Any help is
appreciated.

Thank you
-Deepak
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Indexes on 64 bit values

[Deepak Balasubramanyam]

Bumping this thread back up. Can someone from Basho take a shot at this ?

Thanks
-Deepak


On Wed, Dec 5, 2012 at 6:10 PM, Deepak Balasubramanyam <
deepak.b...@gmail.com> wrote:

> Hi folks,
>
> I have a query regarding riak indexes. My current understanding is that
> riak cannot index numbers whose representation needs more than 32 bits.
> There is an issue on the basho repo to address this -
> https://github.com/basho/riak-java-client/issues/112 . Do you have any
> idea when it will be fixed ?
>
> I've mentioned a way to work around this problem on a comment on the
> issue. The work-around would not be efficient since JS code would have to
> go through N rows to drop ones that do not fit a particular criteria on the
> number. If there is a better solution to the problem, I'd like to hear your
> thoughts on that also.
>
> Thank you for your time.
> -Deepak
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Indexes on 64 bit values

[Deepak Balasubramanyam]

That would work too.

You could always introduce a new method on the interface that
can accommodate a long and deprecate the existing one. That would not break
backward compatibility. I doubt the database would break since it already
supports 64 bit indexes.

Converting the long to its hex equivalent would cause range queries on the
bin index to fail, which is why I needed the feature to begin with. However
building on your idea of using a bin index on the string and from this riak
help page [1], a bin index built on a string representation of a long
number should also retain the capability to run range queries using -
buckets/mybucket/index/field1_bin/val2/val4.

Thank you for your answer Jeremiah. I'll give this a shot sometime and
switch to native 64 bit indexes when they become available.

[1] -
http://docs.basho.com/riak/1.2.0/cookbooks/Secondary-Indexes---Configuration/

Thanks
-Deepak


On Sun, Dec 9, 2012 at 7:58 PM, Jeremiah Peschka  wrote:

> If you need an index on a 64-bit value, you can get creative and store the
> 64-bit number as the hex representation of your 64-bit value in a binary
> index. Seems to me that a bit of lateral thinking is easier than a breaking
> change in either a database client or, worse, a database itself.
>
> Example:
>
> 1619587083804677205 becomes the string 0x1679EDDF2CFD1455.
>
> Using the powers of sorting, everything works out in the end.6
>
>
>
> On Sunday, December 9, 2012, Deepak Balasubramanyam wrote:
>
>> Bumping this thread back up. Can someone from Basho take a shot at this ?
>>
>> Thanks
>> -Deepak
>>
>>
>> On Wed, Dec 5, 2012 at 6:10 PM, Deepak Balasubramanyam <
>> deepak.b...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> I have a query regarding riak indexes. My current understanding is that
>>> riak cannot index numbers whose representation needs more than 32 bits.
>>> There is an issue on the basho repo to address this -
>>> https://github.com/basho/riak-java-client/issues/112 . Do you have any
>>> idea when it will be fixed ?
>>>
>>> I've mentioned a way to work around this problem on a comment on the
>>> issue. The work-around would not be efficient since JS code would have to
>>> go through N rows to drop ones that do not fit a particular criteria on the
>>> number. If there is a better solution to the problem, I'd like to hear your
>>> thoughts on that also.
>>>
>>> Thank you for your time.
>>> -Deepak
>>>
>>
>>
>
> --
> ---
> Jeremiah Peschka
> Founder, Brent Ozar Unlimited
> Microsoft SQL Server MVP
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Production cluster

[Deepak Balasubramanyam]

There is nothing in particular. Reading through loads of documentation and
user experiences on the riak-users list should give you some idea of what
to expect. I've been playing around with a production cluster of late. Here
are some salient points off the top of my head based on load tests I ran
recently.

1. IO performance plays a huge role on how riak performs.
2. The more memory your machine has the better.
3. Prefer dedicated machines over virtualized machines. There is less
competition for IO.
4. Read through the performance tuning tips for linux ; the hard disk and
your riak back end and apply them.
5. Tuning your riak back end is a fine art. Keep a close eye on metrics for
each parameter change. Basho-bench and riak-control are your friends here.
Getting the values right is a process of trial and error. The guide lines
on the riak documentation should reduce the time for convergence.
6. Besides running load tests with basho bench, run one with your
production client against the production database. Your client will execute
a better variety of queries and thus simulate production traffic. Take
map-reduce queries as an example. Running too many of them will cause
errors since riak only allocates N VMs per map / reduce agent.
7, Use basho-bench first and tune riak. Then use your actual client that is
used in application code and tune riak again.
8. Watch out for firewall
configurations
.
9. Go through the best
practices
.
10. Search the riak-user  list for
experiences on setting up a cluster on rackspace. I found some interesting
quips about performance.
11. Take a look at the riak puppet
moduleto
avoid snow flakes.

I'd write more and back it with some data, but it probably deserves a blog
post like you mentioned :)  Good luck with your deployment.

Thanks
-Deepak




On Fri, Dec 14, 2012 at 11:15 PM, Iouri Kostine wrote:

> Hello everyone,
> Can someone share a blog post or some notes about setting up riak
> production cluster at rackspace (RHEL 5)? I was wondering if there is some
> additional steps, best practices etc. when installing from source. Any
> additional information to help us get started would be much appreciated. We
> did read though most of Basho documentation about installing riak, just
> wondering if there are things that are specific to production setups.
>
> Thank you.
>
> iouri
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak and host names

[Deepak Balasubramanyam]

I took the AWS EC2 riak image for a spin today. I have a query regarding
riak nodes and how they behave when the machine reboots.

When an EC2 instance reboots, the internal ip / internal DNS / external DNS
change. This renders the app.config and -name argument on vm.args
incorrect. I was exploring solutions to deal with this problem.

*1. Preventive measures*

Someone on this thread dated May
2011
suggested
using host file entries that point to the local internal IP address. That
does not seem to work. Riak fails with the following error when I add a new
entry to /etc/hosts and configure vm.args with -name riak@riaknode1

Hostname riaknode1 is illegal

I confirmed that riaknode1 pings correctly before starting riak. I guess
erlang tries to match the hostname of the system resulting in this failure
? Can anyone throw some light on this ?

*2. Use -sname*

Is starting the erlang VM with the sname flag an option if it will help
prevent the 'illegal hostname' error ?
Disclaimer: My knowledge of erlang is close to zilch, so sorry if that
option sounded like something you could dismiss easily :)

*3. Use cluster replace
*

a. I understand that the IPs in app.config and vm.args can be replaced with
the correct IP on a restart and using a subsequent 'cluster replace'
command will do. Will executing the 'cluster plan' and 'cluster commit'
commands now produce network chatter ?

b . What happens if 2 nodes go down and one was joined with the other. They
both have 2 different IP addresses on restart. How will 'cluster replace'
work now ?

Do let me know your thoughts.

Thanks
-Deepak
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak and host names

[Deepak Balasubramanyam]

Perhaps I should ask the question differently.

Can someone elaborate on what 'cluster replace' does and how the command is
affected by node names that change after riak starts up ?

Thanks
-Deepak



On Wed, Jan 2, 2013 at 1:13 AM, Deepak Balasubramanyam <
deepak.b...@gmail.com> wrote:

> I took the AWS EC2 riak image for a spin today. I have a query regarding
> riak nodes and how they behave when the machine reboots.
>
> When an EC2 instance reboots, the internal ip / internal DNS / external
> DNS change. This renders the app.config and -name argument on vm.args
> incorrect. I was exploring solutions to deal with this problem.
>
> *1. Preventive measures*
>
> Someone on this thread dated May 
> 2011<http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html>
>  suggested
> using host file entries that point to the local internal IP address. That
> does not seem to work. Riak fails with the following error when I add a new
> entry to /etc/hosts and configure vm.args with -name riak@riaknode1
>
> Hostname riaknode1 is illegal
>
> I confirmed that riaknode1 pings correctly before starting riak. I guess
> erlang tries to match the hostname of the system resulting in this failure
> ? Can anyone throw some light on this ?
>
> *2. Use -sname*
>
> Is starting the erlang VM with the sname flag an option if it will help
> prevent the 'illegal hostname' error ?
> Disclaimer: My knowledge of erlang is close to zilch, so sorry if that
> option sounded like something you could dismiss easily :)
>
> *3. Use cluster replace
> *
>
> a. I understand that the IPs in app.config and vm.args can be replaced
> with the correct IP on a restart and using a subsequent 'cluster replace'
> command will do. Will executing the 'cluster plan' and 'cluster commit'
> commands now produce network chatter ?
>
> b . What happens if 2 nodes go down and one was joined with the other.
> They both have 2 different IP addresses on restart. How will 'cluster
> replace' work now ?
>
> Do let me know your thoughts.
>
> Thanks
> -Deepak
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak and host names

[Deepak Balasubramanyam]

Thanks for all the suggestions. I will take some time to review and try
them out. I'll get back to this thread when I do.

-Deepak


On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw  wrote:

> Hi Matt,
>
> For stopping and starting rather than rebooting, you need to use
> elastic IPs inside of Amazon's virtual private cloud[1] which allows
> for persistent static private addresses.
>
> [1]http://aws.amazon.com/vpc/
>
> Regards
>
> Richard
>
> On Thu, Jan 10, 2013 at 6:09 AM, Matt Black 
> wrote:
> > A quick update on this subject.
> >
> > Using an Elastic IP won't help with AWS since that only binds to the
> public
> > interface - not the internal private one. The hostname command still
> returns
> > the same internal IP address as before, which is what's seen by Riak.
> >
> > In AWS an internal IP address will actually persist across reboots. It
> does
> > not persist across shutdown and startup.
> >
> >
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak and host names

[Deepak Balasubramanyam]

Hi folks,

After trying a couple of configurations, I'd recommend using a VPC on EC2.
I decided to share my experience with a wider audience and wrote about it
on a blog post <http://deepakbala.me/2013/02/08/deploying-riak-on-ec2/>.
The post details (among other things) why VPC can make your life much
easier. Thank you again for all your suggestions.

Is there a Riak wiki that I can contribute this information to ? It would
be easier for users to choose between deployment solutions if they know
what options are available.

Thanks
-Deepak

On Sat, Jan 12, 2013 at 1:02 PM, Deepak Balasubramanyam <
deepak.b...@gmail.com> wrote:

> Thanks for all the suggestions. I will take some time to review and try
> them out. I'll get back to this thread when I do.
>
> -Deepak
>
>
> On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw  wrote:
>
>> Hi Matt,
>>
>> For stopping and starting rather than rebooting, you need to use
>> elastic IPs inside of Amazon's virtual private cloud[1] which allows
>> for persistent static private addresses.
>>
>> [1]http://aws.amazon.com/vpc/
>>
>> Regards
>>
>> Richard
>>
>> On Thu, Jan 10, 2013 at 6:09 AM, Matt Black 
>> wrote:
>> > A quick update on this subject.
>> >
>> > Using an Elastic IP won't help with AWS since that only binds to the
>> public
>> > interface - not the internal private one. The hostname command still
>> returns
>> > the same internal IP address as before, which is what's seen by Riak.
>> >
>> > In AWS an internal IP address will actually persist across reboots. It
>> does
>> > not persist across shutdown and startup.
>> >
>> >
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak and host names

[Deepak Balasubramanyam]

Awesome. I'll make a pull request soon.

Thanks
-Deepak

On Sat, Feb 9, 2013 at 12:08 AM, Mark Phillips  wrote:

> Hi Deepak,
>
> On Fri, Feb 8, 2013 at 3:15 AM, Deepak Balasubramanyam
>  wrote:
> > Hi folks,
> >
> > After trying a couple of configurations, I'd recommend using a VPC on
> EC2. I
> > decided to share my experience with a wider audience and wrote about it
> on a
> > blog post. The post details (among other things) why VPC can make your
> life
> > much easier. Thank you again for all your suggestions.
> >
>
> Nice. Thanks for taking the time to write the post.
>
> > Is there a Riak wiki that I can contribute this information to ? It
> would be
> > easier for users to choose between deployment solutions if they know what
> > options are available.
> >
>
> Absolutely. All our docs are on GitHub [0] and we love (cherish)
> contributions. You'll probably want to add it to the Performance
> Tuning for AWS section [1], and at the very least we should have a
> link to the blog post.
>
> Thanks again.
>
> Mark
> twitter.com/pharkmillups
> A Riak Conference in NYC! ->  ricon.io/east.html
>
> [0] https://github.com/basho/basho_docs
> [1] http://docs.basho.com/riak/latest/cookbooks/Performance-Tuning-AWS/
>
>
> > Thanks
> > -Deepak
> >
> >
> > On Sat, Jan 12, 2013 at 1:02 PM, Deepak Balasubramanyam
> >  wrote:
> >>
> >> Thanks for all the suggestions. I will take some time to review and try
> >> them out. I'll get back to this thread when I do.
> >>
> >> -Deepak
> >>
> >>
> >> On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw 
> wrote:
> >>>
> >>> Hi Matt,
> >>>
> >>> For stopping and starting rather than rebooting, you need to use
> >>> elastic IPs inside of Amazon's virtual private cloud[1] which allows
> >>> for persistent static private addresses.
> >>>
> >>> [1]http://aws.amazon.com/vpc/
> >>>
> >>> Regards
> >>>
> >>> Richard
> >>>
> >>> On Thu, Jan 10, 2013 at 6:09 AM, Matt Black  >
> >>> wrote:
> >>> > A quick update on this subject.
> >>> >
> >>> > Using an Elastic IP won't help with AWS since that only binds to the
> >>> > public
> >>> > interface - not the internal private one. The hostname command still
> >>> > returns
> >>> > the same internal IP address as before, which is what's seen by Riak.
> >>> >
> >>> > In AWS an internal IP address will actually persist across reboots.
> It
> >>> > does
> >>> > not persist across shutdown and startup.
> >>> >
> >>> >
> >>>
> >>> ___
> >>> riak-users mailing list
> >>> riak-users@lists.basho.com
> >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>
> >>
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Expiring keys

[Deepak Balasubramanyam]

A bitcask backend will allow you to set expiry for keys. Using multiple
backends allows expired and persistent keys to co-exist. I searched the
riak user list and fished out a thread discussing
this.
Let us know if that is what you want.

Thanks
-Deepak

On Tue, Feb 26, 2013 at 8:58 PM, Martin Streicher <
martin.streic...@gmail.com> wrote:

>
> We need to purge some cruft from our servers. We want to expire all keys
> older than a certain date. We cannot blanket expire keys, because some are
> necessary in production no matter the age. What is the recommended
> technique for this?
>
> Martin
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com