Ok, here's a link: http://www.fridgebuzz.com/2016/02/23/simple-key-value-java-client-for-riak/ There's a link to a Gist in there with the full source. Hope it helps someone.
On Mon, Feb 22, 2016 at 4:21 PM, Vanessa Williams < vanessa.willi...@thoughtwire.ca> wrote: > Thanks very much for the advice. I'll give it a good test and then write > something. Somewhere. Cheers. > > On Mon, Feb 22, 2016 at 3:42 PM, Alex Moore <amo...@basho.com> wrote: > >> If the contract is "Return true iff the object existed", then the second >> fetch is superfluous + so is the async example I posted. You can use the >> code you had as-is. >> >> Thanks, >> Alex >> >> On Mon, Feb 22, 2016 at 1:23 PM, Vanessa Williams < >> vanessa.willi...@thoughtwire.ca> wrote: >> >>> Hi Alex, would a second fetch just indicate that the object is *still* >>> deleted? Or that this delete operation succeeded? In other words, perhaps >>> what my contract really is is: return true if there was already a value >>> there. In which case would the second fetch be superfluous? >>> >>> Thanks for your help. >>> >>> Vanessa >>> >>> On Mon, Feb 22, 2016 at 11:15 AM, Alex Moore <amo...@basho.com> wrote: >>> >>>> That's the correct behaviour: it should return true iff a value was >>>>> actually deleted. >>>> >>>> >>>> Ok, if that's the case you should do another FetchValue after the >>>> deletion (to update the response.hasValues()) field, or use the async >>>> version of the delete function. I also noticed that we weren't passing the >>>> vclock to the Delete function, so I added that here as well: >>>> >>>> public boolean delete(String key) throws ExecutionException, >>>> InterruptedException { >>>> >>>> // fetch in order to get the causal context >>>> FetchValue.Response response = fetchValue(key); >>>> >>>> if(response.isNotFound()) >>>> { >>>> return ???; // what do we return if it doesn't exist? >>>> } >>>> >>>> DeleteValue deleteValue = new DeleteValue.Builder(new >>>> Location(namespace, key)) >>>> >>>> .withVClock(response.getVectorClock()) >>>> .build(); >>>> >>>> final RiakFuture<Void, Location> deleteFuture = >>>> client.executeAsync(deleteValue); >>>> >>>> deleteFuture.await(); >>>> >>>> if(deleteFuture.isSuccess()) >>>> { >>>> return true; >>>> } >>>> else >>>> { >>>> deleteFuture.cause(); // Cause of failure >>>> return false; >>>> } >>>> } >>>> >>>> >>>> Thanks, >>>> Alex >>>> >>>> On Mon, Feb 22, 2016 at 10:48 AM, Vanessa Williams < >>>> vanessa.willi...@thoughtwire.ca> wrote: >>>> >>>>> See inline: >>>>> >>>>> On Mon, Feb 22, 2016 at 10:31 AM, Alex Moore <amo...@basho.com> wrote: >>>>> >>>>>> Hi Vanessa, >>>>>> >>>>>> You might have a problem with your delete function (depending on it's >>>>>> return value). >>>>>> What does the return value of the delete() function indicate? Right >>>>>> now if an object existed, and was deleted, the function will return true, >>>>>> and will only return false if the object didn't exist or only consisted >>>>>> of >>>>>> tombstones. >>>>>> >>>>> >>>>> >>>>> That's the correct behaviour: it should return true iff a value was >>>>> actually deleted. >>>>> >>>>> >>>>>> If you never look at the object value returned by your fetchValue(key) >>>>>> function, another potential optimization you could make is to only >>>>>> return the HEAD / metadata: >>>>>> >>>>>> FetchValue fv = new FetchValue.Builder(new Location(new Namespace( >>>>>> "some_bucket"), key)) >>>>>> >>>>>> .withOption(FetchValue.Option.HEAD, true) >>>>>> .build(); >>>>>> >>>>>> This would be more efficient because Riak won't have to send you the >>>>>> values over the wire, if you only need the metadata. >>>>>> >>>>>> >>>>> Thanks, I'll clean that up. >>>>> >>>>> >>>>>> If you do write this up somewhere, share the link! :) >>>>>> >>>>> >>>>> Will do! >>>>> >>>>> Regards, >>>>> Vanessa >>>>> >>>>> >>>>>> >>>>>> Thanks, >>>>>> Alex >>>>>> >>>>>> >>>>>> On Mon, Feb 22, 2016 at 6:23 AM, Vanessa Williams < >>>>>> vanessa.willi...@thoughtwire.ca> wrote: >>>>>> >>>>>>> Hi Dmitri, this thread is old, but I read this part of your answer >>>>>>> carefully: >>>>>>> >>>>>>> You can use the following strategies to prevent stale values, in >>>>>>>> increasing order of security/preference: >>>>>>>> 1) Use timestamps (and not pass in vector clocks/causal context). >>>>>>>> This is ok if you're not editing objects, or you're ok with a bit of >>>>>>>> risk >>>>>>>> of stale values. >>>>>>>> 2) Use causal context correctly (which means, read-before-you-write >>>>>>>> -- in fact, the Update operation in the java client does this for you, >>>>>>>> I >>>>>>>> think). And if Riak can't determine which version is correct, it will >>>>>>>> fall >>>>>>>> back on timestamps. >>>>>>>> 3) Turn on siblings, for that bucket or bucket type. That way, >>>>>>>> Riak will still try to use causal context to decide the right value. >>>>>>>> But if >>>>>>>> it can't decide, it will store BOTH values, and give them back to you >>>>>>>> on >>>>>>>> the next read, so that your application can decide which is the >>>>>>>> correct one. >>>>>>> >>>>>>> >>>>>>> I decided on strategy #2. What I am hoping for is some validation >>>>>>> that the code we use to "get", "put", and "delete" is correct in that >>>>>>> context, or if it could be simplified in some cases. Not we are using >>>>>>> delete-mode "immediate" and no duplicates. >>>>>>> >>>>>>> In their shortest possible forms, here are the three methods I'd >>>>>>> like some feedback on (note, they're being used in production and >>>>>>> haven't >>>>>>> caused any problems yet, however we have very few writes in production >>>>>>> so >>>>>>> the lack of problems doesn't support the conclusion that the >>>>>>> implementation >>>>>>> is correct.) Note all argument-checking, exception-handling, and logging >>>>>>> removed for clarity. *I'm mostly concerned about correct use of >>>>>>> causal context and response.isNotFound and response.hasValues. *Is >>>>>>> there anything I could/should have left out? >>>>>>> >>>>>>> public RiakClient(String name, >>>>>>> com.basho.riak.client.api.RiakClient client) >>>>>>> { >>>>>>> this.name = name; >>>>>>> this.namespace = new Namespace(name); >>>>>>> this.client = client; >>>>>>> } >>>>>>> >>>>>>> public byte[] get(String key) throws ExecutionException, >>>>>>> InterruptedException { >>>>>>> >>>>>>> FetchValue.Response response = fetchValue(key); >>>>>>> if (!response.isNotFound()) >>>>>>> { >>>>>>> RiakObject riakObject = >>>>>>> response.getValue(RiakObject.class); >>>>>>> return riakObject.getValue().getValue(); >>>>>>> } >>>>>>> return null; >>>>>>> } >>>>>>> >>>>>>> public void put(String key, byte[] value) throws >>>>>>> ExecutionException, InterruptedException { >>>>>>> >>>>>>> // fetch in order to get the causal context >>>>>>> FetchValue.Response response = fetchValue(key); >>>>>>> RiakObject storeObject = new >>>>>>> >>>>>>> RiakObject().setValue(BinaryValue.create(value)).setContentType("binary/octet-stream"); >>>>>>> StoreValue.Builder builder = >>>>>>> new StoreValue.Builder(storeObject).withLocation(new >>>>>>> Location(namespace, key)); >>>>>>> if (response.getVectorClock() != null) { >>>>>>> builder = >>>>>>> builder.withVectorClock(response.getVectorClock()); >>>>>>> } >>>>>>> StoreValue storeValue = builder.build(); >>>>>>> client.execute(storeValue); >>>>>>> } >>>>>>> >>>>>>> public boolean delete(String key) throws ExecutionException, >>>>>>> InterruptedException { >>>>>>> >>>>>>> // fetch in order to get the causal context >>>>>>> FetchValue.Response response = fetchValue(key); >>>>>>> if (!response.isNotFound()) >>>>>>> { >>>>>>> DeleteValue deleteValue = new DeleteValue.Builder(new >>>>>>> Location(namespace, key)).build(); >>>>>>> client.execute(deleteValue); >>>>>>> } >>>>>>> return !response.isNotFound() || !response.hasValues(); >>>>>>> } >>>>>>> >>>>>>> >>>>>>> Any comments much appreciated. I want to provide a minimally correct >>>>>>> example of simple client code somewhere (GitHub, blog post, >>>>>>> something...) >>>>>>> so I don't want to post this without review. >>>>>>> >>>>>>> Thanks, >>>>>>> Vanessa >>>>>>> >>>>>>> ThoughtWire Corporation >>>>>>> http://www.thoughtwire.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 8, 2015 at 8:45 AM, Dmitri Zagidulin < >>>>>>> dzagidu...@basho.com> wrote: >>>>>>> >>>>>>>> Hi Vanessa, >>>>>>>> >>>>>>>> The thing to keep in mind about read repair is -- it happens >>>>>>>> asynchronously on every GET, but /after/ the results are returned to >>>>>>>> the >>>>>>>> client. >>>>>>>> >>>>>>>> So, when you issue a GET with r=1, the coordinating node only waits >>>>>>>> for 1 of the replicas before responding to the client with a success, >>>>>>>> and >>>>>>>> only afterwards triggers read-repair. >>>>>>>> >>>>>>>> It's true that with notfound_ok=false, it'll wait for the first >>>>>>>> non-missing replica before responding. But if you edit or update your >>>>>>>> objects at all, an R=1 still gives you a risk of stale values being >>>>>>>> returned. >>>>>>>> >>>>>>>> For example, say you write an object with value A. And let's say >>>>>>>> your 3 replicas now look like this: >>>>>>>> >>>>>>>> replica 1: A, replica 2: A, replica 3: notfound/missing >>>>>>>> >>>>>>>> A read with an R=1 and notfound_ok=false is just fine, here. >>>>>>>> (Chances are, the notfound replica will arrive first, but the >>>>>>>> notfound_ok >>>>>>>> setting will force the coordinator to wait for the first non-empty >>>>>>>> value, >>>>>>>> A, and return it to the client. And then trigger read-repair). >>>>>>>> >>>>>>>> But what happens if you edit that same object, and give it a new >>>>>>>> value, B? So, now, there's a chance that your replicas will look like >>>>>>>> this: >>>>>>>> >>>>>>>> replica 1: A, replica 2: B, replica 3: B. >>>>>>>> >>>>>>>> So now if you do a read with an R=1, there's a chance that replica >>>>>>>> 1, with the old value of A, will arrive first, and that's the response >>>>>>>> that >>>>>>>> will be returned to the client. >>>>>>>> >>>>>>>> Whereas, using R=2 eliminates that risk -- well, at least decreases >>>>>>>> it. You still have the issue of -- how does Riak decide whether A or B >>>>>>>> is >>>>>>>> the correct value? Are you using causal context/vclocks correctly? >>>>>>>> (That >>>>>>>> is, reading the object before you update, to get the correct causal >>>>>>>> context?) Or are you relying on timestamps? (This is an ok strategy, >>>>>>>> provided that the edits are sufficiently far apart in time, and you >>>>>>>> don't >>>>>>>> have many concurrent edits, AND you're ok with the small risk of >>>>>>>> occasionally the timestamp being wrong). You can use the following >>>>>>>> strategies to prevent stale values, in increasing order of >>>>>>>> security/preference: >>>>>>>> >>>>>>>> 1) Use timestamps (and not pass in vector clocks/causal context). >>>>>>>> This is ok if you're not editing objects, or you're ok with a bit of >>>>>>>> risk >>>>>>>> of stale values. >>>>>>>> >>>>>>>> 2) Use causal context correctly (which means, read-before-you-write >>>>>>>> -- in fact, the Update operation in the java client does this for you, >>>>>>>> I >>>>>>>> think). And if Riak can't determine which version is correct, it will >>>>>>>> fall >>>>>>>> back on timestamps. >>>>>>>> >>>>>>>> 3) Turn on siblings, for that bucket or bucket type. That way, >>>>>>>> Riak will still try to use causal context to decide the right value. >>>>>>>> But if >>>>>>>> it can't decide, it will store BOTH values, and give them back to you >>>>>>>> on >>>>>>>> the next read, so that your application can decide which is the >>>>>>>> correct one. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 8, 2015 at 1:56 AM, Vanessa Williams < >>>>>>>> vanessa.willi...@thoughtwire.ca> wrote: >>>>>>>> >>>>>>>>> Hi Dmitri, what would be the benefit of r=2, exactly? It isn't >>>>>>>>> necessary to trigger read-repair, is it? If it's important I'd rather >>>>>>>>> try >>>>>>>>> it sooner than later... >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Vanessa >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Oct 7, 2015 at 4:02 PM, Dmitri Zagidulin < >>>>>>>>> dzagidu...@basho.com> wrote: >>>>>>>>> >>>>>>>>>> Glad you sorted it out! >>>>>>>>>> >>>>>>>>>> (I do want to encourage you to bump your R setting to at least 2, >>>>>>>>>> though. Run some tests -- I think you'll find that the difference in >>>>>>>>>> speed >>>>>>>>>> will not be noticeable, but you do get a lot more data resilience >>>>>>>>>> with 2.) >>>>>>>>>> >>>>>>>>>> On Wed, Oct 7, 2015 at 6:24 PM, Vanessa Williams < >>>>>>>>>> vanessa.willi...@thoughtwire.ca> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Dmitri, well...we solved our problem to our satisfaction but >>>>>>>>>>> it turned out to be something unexpected. >>>>>>>>>>> >>>>>>>>>>> The keys were two properties mentioned in a blog post on >>>>>>>>>>> "configuring Riak’s oft-subtle behavioral characteristics": >>>>>>>>>>> http://basho.com/posts/technical/riaks-config-behaviors-part-4/ >>>>>>>>>>> >>>>>>>>>>> notfound_ok= false >>>>>>>>>>> basic_quorum=true >>>>>>>>>>> >>>>>>>>>>> The 2nd one just makes things a little faster, but the first one >>>>>>>>>>> is the one whose default value of true was killing us. >>>>>>>>>>> >>>>>>>>>>> With r=1 and notfound_ok=true (default) the first node to >>>>>>>>>>> respond, if it didn't find the requested key, the authoritative >>>>>>>>>>> answer was >>>>>>>>>>> "this key is not found". Not what we were expecting at all. >>>>>>>>>>> >>>>>>>>>>> With the changed settings, it will wait for a quorum of >>>>>>>>>>> responses and only if *no one* finds the key will "not found" be >>>>>>>>>>> returned. >>>>>>>>>>> Perfect. (Without this setting it would wait for all responses, not >>>>>>>>>>> ideal.) >>>>>>>>>>> >>>>>>>>>>> Now there is only one snag, which is that if the Riak node the >>>>>>>>>>> client connects to goes down, there will be no communication and we >>>>>>>>>>> have a >>>>>>>>>>> problem. This is easily solvable with a load-balancer, though for >>>>>>>>>>> complicated reasons we actually don't need to do that right now. >>>>>>>>>>> It's just >>>>>>>>>>> acceptable for us temporarily. Later, we'll get the load-balancer >>>>>>>>>>> working >>>>>>>>>>> and even that won't be a problem. >>>>>>>>>>> >>>>>>>>>>> I *think* we're ok now. Thanks for your help! >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Vanessa >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 7, 2015 at 9:33 AM, Dmitri Zagidulin < >>>>>>>>>>> dzagidu...@basho.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Yeah, definitely find out what the sysadmin's experience was, >>>>>>>>>>>> with the load balancer. It could have just been a wrong >>>>>>>>>>>> configuration or >>>>>>>>>>>> something. >>>>>>>>>>>> >>>>>>>>>>>> And yes, that's the documentation page I recommend - >>>>>>>>>>>> http://docs.basho.com/riak/latest/ops/advanced/configs/load-balancing-proxy/ >>>>>>>>>>>> Just set up HAProxy, and point your Java clients to its IP. >>>>>>>>>>>> >>>>>>>>>>>> The drawbacks to load-balancing on the java client side (yes, >>>>>>>>>>>> the cluster object) instead of a standalone load balancer like >>>>>>>>>>>> HAProxy, are >>>>>>>>>>>> the following: >>>>>>>>>>>> >>>>>>>>>>>> 1) Adding node means code changes (or at very least, config >>>>>>>>>>>> file changes) rolled out to all your clients. Which turns out to >>>>>>>>>>>> be a >>>>>>>>>>>> pretty serious hassle. Instead, HAProxy allows you to add or >>>>>>>>>>>> remove nodes >>>>>>>>>>>> without changing any java code or config files. >>>>>>>>>>>> >>>>>>>>>>>> 2) Performance. We've ran many tests to compare performance, >>>>>>>>>>>> and client-side load balancing results in significantly lower >>>>>>>>>>>> throughput >>>>>>>>>>>> than you'd have using haproxy (or nginx). (Specifically, you >>>>>>>>>>>> actually want >>>>>>>>>>>> to use the 'leastconn' load balancing algorithm with HAProxy, >>>>>>>>>>>> instead of >>>>>>>>>>>> round robin). >>>>>>>>>>>> >>>>>>>>>>>> 3) The health check on the client side (so that the java load >>>>>>>>>>>> balancer can tell when a remote node is down) is much less >>>>>>>>>>>> intelligent than >>>>>>>>>>>> a dedicated load balancer would provide. With something like >>>>>>>>>>>> HAProxy, you >>>>>>>>>>>> should be able to take down nodes with no ill effects for the >>>>>>>>>>>> client code. >>>>>>>>>>>> >>>>>>>>>>>> Now, if you load balance on the client side and you take a node >>>>>>>>>>>> down, it's not supposed to stop working completely. (I'm not sure >>>>>>>>>>>> why it's >>>>>>>>>>>> failing for you, we can investigate, but it'll be easier to just >>>>>>>>>>>> use a load >>>>>>>>>>>> balancer). It should throw an error or two, but then start working >>>>>>>>>>>> again >>>>>>>>>>>> (on the retry). >>>>>>>>>>>> >>>>>>>>>>>> Dmitri >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Oct 7, 2015 at 2:45 PM, Vanessa Williams < >>>>>>>>>>>> vanessa.willi...@thoughtwire.ca> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Dmitri, thanks for the quick reply. >>>>>>>>>>>>> >>>>>>>>>>>>> It was actually our sysadmin who tried the load balancer >>>>>>>>>>>>> approach and had no success, late last evening. However I haven't >>>>>>>>>>>>> discussed >>>>>>>>>>>>> the gory details with him yet. The failure he saw was at the >>>>>>>>>>>>> application >>>>>>>>>>>>> level (i.e. failure to read a key), but I don't know a) how he >>>>>>>>>>>>> set up the >>>>>>>>>>>>> LB or b) what the Java exception was, if any. I'll find that out >>>>>>>>>>>>> in an hour >>>>>>>>>>>>> or two and report back. >>>>>>>>>>>>> >>>>>>>>>>>>> I did find this article just now: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> http://docs.basho.com/riak/latest/ops/advanced/configs/load-balancing-proxy/ >>>>>>>>>>>>> >>>>>>>>>>>>> So I suppose we'll give those suggestions a try this morning. >>>>>>>>>>>>> >>>>>>>>>>>>> What is the drawback to having the client connect to all 4 >>>>>>>>>>>>> nodes (the cluster client, I assume you mean?) My understanding >>>>>>>>>>>>> from >>>>>>>>>>>>> reading articles I've found is that one of the nodes going away >>>>>>>>>>>>> causes that >>>>>>>>>>>>> client to fail as well. Is that what you mean, or are there other >>>>>>>>>>>>> drawbacks >>>>>>>>>>>>> as well? >>>>>>>>>>>>> >>>>>>>>>>>>> If there's anything else you can recommend, or links other >>>>>>>>>>>>> than the one above you can point me to, it would be much >>>>>>>>>>>>> appreciated. We >>>>>>>>>>>>> expect both node failure and deliberate node removal for upgrade, >>>>>>>>>>>>> repair, >>>>>>>>>>>>> replacement, etc. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Vanessa >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Oct 7, 2015 at 8:29 AM, Dmitri Zagidulin < >>>>>>>>>>>>> dzagidu...@basho.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Vanessa, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Riak is definitely meant to run behind a load balancer. (Or, >>>>>>>>>>>>>> at the worst case, to be load-balanced on the client side. That >>>>>>>>>>>>>> is, all >>>>>>>>>>>>>> clients connect to all 4 nodes). >>>>>>>>>>>>>> >>>>>>>>>>>>>> When you say "we did try putting all 4 Riak nodes behind a >>>>>>>>>>>>>> load-balancer and pointing the clients at it, but it didn't >>>>>>>>>>>>>> help." -- what >>>>>>>>>>>>>> do you mean exactly, by "it didn't help"? What happened when you >>>>>>>>>>>>>> tried >>>>>>>>>>>>>> using the load balancer? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 1:57 PM, Vanessa Williams < >>>>>>>>>>>>>> vanessa.willi...@thoughtwire.ca> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, we are still (for a while longer) using Riak 1.4 and >>>>>>>>>>>>>>> the matching Java client. The client(s) connect to one node in >>>>>>>>>>>>>>> the cluster >>>>>>>>>>>>>>> (since that's all it can do in this client version). The >>>>>>>>>>>>>>> cluster itself has >>>>>>>>>>>>>>> 4 nodes (sorry, we can't use 5 in this scenario). There are 2 >>>>>>>>>>>>>>> separate >>>>>>>>>>>>>>> clients. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We've tried both n_val = 3 and n_val=4. We achieve >>>>>>>>>>>>>>> consistency-by-writes by setting w=all. Therefore, we only >>>>>>>>>>>>>>> require one >>>>>>>>>>>>>>> successful read (r=1). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When all nodes are up, everything is fine. If one node >>>>>>>>>>>>>>> fails, the clients can no longer read any keys at all. There's >>>>>>>>>>>>>>> an exception >>>>>>>>>>>>>>> like this: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> com.basho.riak.client.RiakRetryFailedException: >>>>>>>>>>>>>>> java.net.ConnectException: Connection refused >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Now, it isn't possible that Riak can't operate when one node >>>>>>>>>>>>>>> fails, so we're clearly missing something here. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Note: we did try putting all 4 Riak nodes behind a >>>>>>>>>>>>>>> load-balancer and pointing the clients at it, but it didn't >>>>>>>>>>>>>>> help. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Riak is a high-availability key-value store, so... why are >>>>>>>>>>>>>>> we failing to achieve high-availability? Any suggestions greatly >>>>>>>>>>>>>>> appreciated, and if more info is required I'll do my best to >>>>>>>>>>>>>>> provide it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>>>> Vanessa >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Vanessa Williams >>>>>>>>>>>>>>> ThoughtWire Corporation >>>>>>>>>>>>>>> http://www.thoughtwire.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> riak-users mailing list >>>>>>> riak-users@lists.basho.com >>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com