Re: Riak Recap for August 22 - 25

[David Dawson]

P 



On 27 Aug 2011, at 00:38, Mark Phillips  wrote:

> Afternoon, Evening, Morning to All -
> 
> Here's a great Recap to take you into the weekend: blog posts, slides,
> videos, new code, and more.
> 
> Also, to all our Riak users who happen to be on the East Coast of the
> US this weekend: stay safe.
> 
> Enjoy.
> 
> Mark
> 
> Community Manager
> Basho Technologies
> wiki.basho.com
> twitter.com/pharkmillups
> --
> 
> Riak Recap for August 22 - 26
> ==
> 
> 1) A video of Rusty Klophaus' Secondary Indexing Talk from OSCON made
> it onto Youtube. (The video quality is not the best, but it gets the
> job done.)
> 
> * Watch here ---> http://www.youtube.com/watch?v=uExMXa4t8eg
> 
> 2) Joseph Lambert has two new Riak-related repos up on GitHub that are
> definitely worth keeping an eye on. (He should also get some points
> for one of the better GitHub nicks I've seen in a while.)
> 
> * ---> https://github.com/snoopaloop/Riak-PBC-Pool
> * ---> https://github.com/snoopaloop/riakyo
> 
> 3) Cody Soyland and Sean Cribbs collaborated to put together a Riak
> Post Commit Hook that send writes to ZeroMQ.
> 
> * Details here --->
> http://twitter.com/#!/codysoyland/statuses/106120496629821440
> 
> 4) The slides from the talk I gave on Wednesday night at the Seattle
> Scalability Meetup are now online.
> 
> * Check them out here --->
> http://www.slideshare.net/pharkmillups/riak-seattlemeetupaugust
> 
> 5) Adam Schepis wrote a script to update Riak Config Files (and turned
> it into a blog post). He was also looking for some feedback on it….
> 
> * Check it out here --->
> http://adamschepis.com/blog/2011/08/25/a-script-to-update-riak-config-files/
> 
> 6) One Job opportunity to pass along.
> 
> * Work with Node.js, Linux, Riak, and more
> --->http://twitter.com/#!/peter_schlegel/status/106732657488039936
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Achieving 100% consitency

[Jonathan Langevin]

Even for development-purposes only? Otherwise it seems data would be written
n times to the same machine, which is needless in a dev environment with low
storage specs...*


Jonathan Langevin
Systems Administrator
Loom Inc.
Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com -
www.loomlearning.com - Skype: intel352
*


On Fri, Aug 26, 2011 at 5:01 PM, Ian Plosker  wrote:

> Lukas,
>
> Also, we don't advise that you run single node clusters. Riak is designed
> to be used in clusters of at least 3 nodes. You can run a multi-node cluster
> on a single development machine by downloading the Riak source, and running
> "make devrel". Take a look at the Riak Fast Track (
> http://wiki.basho.com/The-Riak-Fast-Track.html) for more details.
>
> Ian Plosker
> Developer Advocate
> Basho Technologies
>
> On Aug 26, 2011, at 3:17 PM, Lukas Schulze wrote:
>
> I'm doing some simple tests with Riak and tried to build something like an
> index.
> Therefore I created new buckets for some attributes like "name", "street"
> and "city".
> One entry in the index-bucket "name" is for example "Mueller" and the value
> contains all user ids, formatted as an JSON string: "{id:[1,5,8,13,2,7]}"
> The java objects are saved as JSON strings in a separate bucket "users",
> the keys in this bucket are the user-ids, the values are the JSON strings.
>
> If I add 200 users via Java and the RiakPBC client every loop I fetch the
> index, add the new user id and store it again in Riak.
> But java is too fast, so I receive an old version of the bucket.
>
> Because I've only one node I set the n-value to 1, r = 1, w = 1 and dw = 1.
> But I have to wait nearly 2 seconds to be mostly sure to get the correct
> response. (the computer isn't an high-end machine ;-) )
>
> Is it possible to be sure that the data will be saved permanently and I can
> continue adding users?
> Are there any caching methods I can configure?
> Can I set the default n-value to 1 so that every newly created bucket will
> have this value?
> Does Riak have any kind of indexes or is it possible to implement it a
> better way?
>
> In my first version I saved all users in one bucket and iterated over all
> of them to find the correct one. But for every single request from the Java
> Service to Riak it took nearly 200ms. For a huge amount of entries (10,000)
> this isn't practible. Therefore I tried to implement my own indexes.
>
> The main focus of my question is getting rid of the inconsistent reads.
>
> Thank you.
>
> Best Regards
> Lukas
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Slow Riak issue

[Jeremiah Peschka]

I've CC'd in OJ and the riak-users lists on this.

---
Jeremiah Peschka - Founder, Brent Ozar PLF, LLC
Microsoft SQL Server MVP

On Aug 26, 2011, at 11:35 PM, Neil Unadkat wrote:

> 
> Hey Jeremiah,
> 
> I tried using CorrugatedIron, but it used to connect only to the nodes i had 
> started on my machine. As in, i had 2 nodes running on my mac os, and i have 
> a vm running windows 7, through which using CorrugatedIron i could connect 
> and write to those nodes.

I develop CorrugatedIron in a Windows 7 VM on OS X. I had to do a few things to 
get CorrugatedIron in Windows to talk to Riak in OS X:

1) Configure Riak to listen on 0.0.0.0 - this will listen for connections no 
matter which IP they are asking for. 
THIS IS UNSAFE TO DEPLOY INTO PRODUCTION. 
SET THIS UP PROPERLY WHEN YOU MOVE TO PRODUCTION.
2) Set up a loopback adapter that listens on a specific IP - this makes it easy 
to set up a DNS alias on your localhost and VMs.
3) Set up a DNS alias in /etc/hosts
4) Use the host name instead of the IP address. You can see this in all of the 
CorrugatedIron code, we use host names instead of IP addresses.

I have aliases in my .bash_profile to add and remove loopback adapters. It 
looks like this:

alias add_loopback='sudo ifconfig lo0 alias 10.0.0.22'
alias remove_loopback='sudo ifconfig lo0 -alias 10.0.0.22'

/etc/hosts on OS X has this line in it:

127.0.0.1riak-test

and the Windows hosts file (C:\Windows\System32\drivers\etc\hosts) has this 
line in it:

10.0.0.22riak-test

> Just 2 days back i could manage to talk to the other nodes i had setup on 
> another machine through CorrugatedIron. But still everything is quite slow. I 
> have enabled the PBC interface on the nodes. 
> Even if we use the C# native classes for HTTP it should take too long right?

HTTP is going to be orders of magnitude slower than Protocol Buffers. There's 
an exceptional amount of overhead involved in HTTP, especially for small 
messages. CorrugatedIron will use Protocol Buffers for almost all operations. 
There are a few operations  that are not supported by Riak's API so we have to 
drop down to HTTP requests for those.

Riak's performance is going to be bound by the speed of your hard drive. Every 
write that Riak performs goes to disk. There is no buffering like you would see 
with MongoDB (or even buffering of data like you'd see in an RDBMS). If you are 
using a laptop and you're using the HDD that came with it, you have a 5400RPM 
hard drive which is by all standards an incredibly slow drive. You can compound 
that with 3 Riak nodes running on the same drive AND one of those nodes running 
on a virtual environment AND it's a slow laptop HDD with poor caching. I 
suspect you can see how this would not be the fastest thing on earth.

For development purposes, it's advisable to use a single Riak node running on 
bare metal (a non-virtualized OS). Once you move into production you can either 
change the configuration in CorrugatedIron to use multiple nodes or you can use 
a proxy/load balancer to push your writes out to all of your Riak nodes. The 
benefit of using an abstractions we've built into Corrugated Iron is that you 
don't need to worry about the physical implementation - you can a configuration 
and you're good to go.

> 
> Thanks
> Neil 
> 
> On 26 August 2011 20:04, Jeremiah Peschka  wrote:
> Hi Neil,
> 
> Which one of the .NET clients are you using? If it's CorrugatedIron, let me 
> know and I can work with you off list to get things connected (OJ Reeves and 
> I maintain it). I don't think the other two libraries are getting much 
> maintenance and attention at the moment.
> 
> If you can't connect, have you looked into the app.config to change the IPs 
> that Riak will bind to? By default it's set to only bind to 127.0.0.1. You 
> can change that for development purposes to 0.0.0.0 (I'd use the real IP 
> address for production). Since you have two nodes on the same host, make sure 
> that they're not both using the same port.
> 
> Otherwise:
> 
> You'll see slower performance with HTTP requests than you will with Protocol 
> Buffers requests.
> 
> Let me know how I can help more.
> ---
> Jeremiah Peschka - Founder, Brent Ozar PLF, LLC
> Microsoft SQL Server MVP
> 
> On Aug 25, 2011, at 8:32 PM, Neil Unadkat wrote:
> 
> > Hey,
> > We have been testing Riak to use as our backend database, but one of the 
> > issues that we are facing is that Riak is very slow when a call is made 
> > from the c# sharp code.
> > In the code, we have used the basic Web Client class, and make a simple GET 
> > call. It takes like 2 seconds to get one object.
> > We have setup a cluster which has 2 nodes on Mac mini, and one on an Ubuntu 
> > machine.
> > I have tried using the .net client library mentioned on the site, but with 
> > those i cannot even connect to the nodes.
> > Can anyone help? ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://

Re: Achieving 100% consitency

[Ian Plosker]

Lukas,

Yes, even for dev you'd be best advised to develop and test your application 
with the same or similar number of nodes and n, r, and w settings as you would 
in production. It's good practice to develop applications in a dev/test 
environment that mirrors the production environment as much as is 
reasonable/feasible. You can run a single node cluster, but note that this 
isn't a configuration you'll see in a production.

Ian Plosker
Developer Advocate
Basho Technologies



On Aug 27, 2011, at 5:33 AM, Jonathan Langevin wrote:

> Even for development-purposes only? Otherwise it seems data would be written 
> n times to the same machine, which is needless in a dev environment with low 
> storage specs...
> 
> 
> Jonathan Langevin
> Systems Administrator
> Loom Inc.
> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - 
> www.loomlearning.com - Skype: intel352
> 
> 
> 
> On Fri, Aug 26, 2011 at 5:01 PM, Ian Plosker  wrote:
> Lukas,
> 
> Also, we don't advise that you run single node clusters. Riak is designed to 
> be used in clusters of at least 3 nodes. You can run a multi-node cluster on 
> a single development machine by downloading the Riak source, and running 
> "make devrel". Take a look at the Riak Fast Track 
> (http://wiki.basho.com/The-Riak-Fast-Track.html) for more details.
> 
> Ian Plosker
> Developer Advocate
> Basho Technologies
> 
> On Aug 26, 2011, at 3:17 PM, Lukas Schulze wrote:
> 
>> I'm doing some simple tests with Riak and tried to build something like an 
>> index.
>> Therefore I created new buckets for some attributes like "name", "street" 
>> and "city".
>> One entry in the index-bucket "name" is for example "Mueller" and the value 
>> contains all user ids, formatted as an JSON string: "{id:[1,5,8,13,2,7]}"
>> The java objects are saved as JSON strings in a separate bucket "users", the 
>> keys in this bucket are the user-ids, the values are the JSON strings.
>> 
>> If I add 200 users via Java and the RiakPBC client every loop I fetch the 
>> index, add the new user id and store it again in Riak.
>> But java is too fast, so I receive an old version of the bucket.
>> 
>> Because I've only one node I set the n-value to 1, r = 1, w = 1 and dw = 1.
>> But I have to wait nearly 2 seconds to be mostly sure to get the correct 
>> response. (the computer isn't an high-end machine ;-) )
>> 
>> Is it possible to be sure that the data will be saved permanently and I can 
>> continue adding users?
>> Are there any caching methods I can configure?
>> Can I set the default n-value to 1 so that every newly created bucket will 
>> have this value?
>> Does Riak have any kind of indexes or is it possible to implement it a 
>> better way?
>> 
>> In my first version I saved all users in one bucket and iterated over all of 
>> them to find the correct one. But for every single request from the Java 
>> Service to Riak it took nearly 200ms. For a huge amount of entries (10,000) 
>> this isn't practible. Therefore I tried to implement my own indexes.
>> 
>> The main focus of my question is getting rid of the inconsistent reads.
>> 
>> Thank you.
>> 
>> Best Regards
>> Lukas
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Achieving 100% consitency

[Ian Plosker]

Jonathan,

Excuse me, that last message should have been addressed to you.

Ian Plosker
Developer Advocate
Basho Technologies


On Aug 27, 2011, at 11:39 AM, Ian Plosker wrote:

> Lukas,
> 
> Yes, even for dev you'd be best advised to develop and test your application 
> with the same or similar number of nodes and n, r, and w settings as you 
> would in production. It's good practice to develop applications in a dev/test 
> environment that mirrors the production environment as much as is 
> reasonable/feasible. You can run a single node cluster, but note that this 
> isn't a configuration you'll see in a production.
> 
> Ian Plosker
> Developer Advocate
> Basho Technologies
> 
> 
> 
> On Aug 27, 2011, at 5:33 AM, Jonathan Langevin wrote:
> 
>> Even for development-purposes only? Otherwise it seems data would be written 
>> n times to the same machine, which is needless in a dev environment with low 
>> storage specs...
>> 
>> 
>> Jonathan Langevin
>> Systems Administrator
>> Loom Inc.
>> Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com - 
>> www.loomlearning.com - Skype: intel352
>> 
>> 
>> 
>> On Fri, Aug 26, 2011 at 5:01 PM, Ian Plosker  wrote:
>> Lukas,
>> 
>> Also, we don't advise that you run single node clusters. Riak is designed to 
>> be used in clusters of at least 3 nodes. You can run a multi-node cluster on 
>> a single development machine by downloading the Riak source, and running 
>> "make devrel". Take a look at the Riak Fast Track 
>> (http://wiki.basho.com/The-Riak-Fast-Track.html) for more details.
>> 
>> Ian Plosker
>> Developer Advocate
>> Basho Technologies
>> 
>> On Aug 26, 2011, at 3:17 PM, Lukas Schulze wrote:
>> 
>>> I'm doing some simple tests with Riak and tried to build something like an 
>>> index.
>>> Therefore I created new buckets for some attributes like "name", "street" 
>>> and "city".
>>> One entry in the index-bucket "name" is for example "Mueller" and the value 
>>> contains all user ids, formatted as an JSON string: "{id:[1,5,8,13,2,7]}"
>>> The java objects are saved as JSON strings in a separate bucket "users", 
>>> the keys in this bucket are the user-ids, the values are the JSON strings.
>>> 
>>> If I add 200 users via Java and the RiakPBC client every loop I fetch the 
>>> index, add the new user id and store it again in Riak.
>>> But java is too fast, so I receive an old version of the bucket.
>>> 
>>> Because I've only one node I set the n-value to 1, r = 1, w = 1 and dw = 1.
>>> But I have to wait nearly 2 seconds to be mostly sure to get the correct 
>>> response. (the computer isn't an high-end machine ;-) )
>>> 
>>> Is it possible to be sure that the data will be saved permanently and I can 
>>> continue adding users?
>>> Are there any caching methods I can configure?
>>> Can I set the default n-value to 1 so that every newly created bucket will 
>>> have this value?
>>> Does Riak have any kind of indexes or is it possible to implement it a 
>>> better way?
>>> 
>>> In my first version I saved all users in one bucket and iterated over all 
>>> of them to find the correct one. But for every single request from the Java 
>>> Service to Riak it took nearly 200ms. For a huge amount of entries (10,000) 
>>> this isn't practible. Therefore I tried to implement my own indexes.
>>> 
>>> The main focus of my question is getting rid of the inconsistent reads.
>>> 
>>> Thank you.
>>> 
>>> Best Regards
>>> Lukas
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
> 
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com