Invoking Python script as part of a M/R job?

[Jeffrey Eliasen]

I would like to write a job that invokes a python script to execute the
processing of each node in a bucket. I can't find a way to do this using
Javascript and I don't really know Erlang well enough to make this work...
is there a sample piece of code somewhere that demonstrates this?

Thanks in advance!

jeff
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

How to bind Riak to an interface rather than to IP address?

[jeffrey eliasen]

My Riak cluster is on a network that uses dynamic IPs, so each node gets a
different IP address at reboot. How do I indicate in app.config that I want
to bind to an interface (such as 'eth0') regardless of it's IP address
rather than to a specific IP address?

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Does the name in vm.args need to be the same at all times?

[Jeffrey Eliasen]

Does the name in vm.args provided for a node in the cluster matter from one
run of the service to the next? In other words, could I:

   1. name the node 'x' in vm.args
   2. start the service
   3. stop the service
   4. rename the node 'y' in vm.args
   5. start the service again


I would like to dynamically assign the name prior to starting the service,
but if it must remain the same for the life of the node then this will not
work (obviously).

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak dynamic cluster HOWTO

[Jeffrey Eliasen]

I spent the Christmas break setting up a Riak cluster and documenting all
the steps. I've got a howto/tutorial set up at
http://jeff.jke.net/2011/10/02/setting-nosql-cluster that will walk you
through the steps.

In particular, I aimed for low cost and low power. Each node in the cluster
boots off the network (single point of maintenance for all node OS
configuration) and just requires a data drive. Nodes are assigned addresses
using DHCP and automatically update vm.args and execute reip if the node
address changes from boot to boot. The scripts for this might be handy for
anyone else running in an environment where machine addresses change
(Amazon, etc).

I'd love feedback, and more to the point, I'd like to find someone to walk
through it step-by-step and make sure I didn't miss anything.

Oh, and *thanks* to the list for helping me with configuration help over
the last couple weeks!

jeff

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak dynamic cluster HOWTO

[Jeffrey Eliasen]

I haven't, but that's a good idea. I will get that up in a couple days.

jeff

On Fri, Jan 6, 2012 at 9:38 AM, Jeff Thompson  wrote:

> Hey Jeff, (great name!)
>
> Have you posted the specs for the low cost/power nodes you've built? If
> not, would be interested in doing that? (parts, cost, anything interesting)
>
> I noticed in your Future Steps section (
> http://jeff.jke.net/2011/12/14/future-steps ) that you were looking to
> add an administrative dashboard for things like node status. While I assume
> you'd probably like something custom, have you looked at riak_control yet?
> It's still being developed, but does a really good job presenting node and
> ring status right now. It's quite pretty too. :)
>
> You can find the riak_control repo here:
> https://github.com/basho/riak_control It's part of the current master of
> riak, so you probably already have it. I've been using it with the
> distributed mud I'm building on top of riak_core (mostly to learn
> riak_core), and have found it quite useful.
>
> Going to read through your tutorial more carefully now.
>
> Cheers!
>
> Jeff
>
>
>
>
> On Fri, Jan 6, 2012 at 11:23 AM, Jeffrey Eliasen  wrote:
>
>> I spent the Christmas break setting up a Riak cluster and documenting all
>> the steps. I've got a howto/tutorial set up at
>> http://jeff.jke.net/2011/10/02/setting-nosql-cluster that will walk you
>> through the steps.
>>
>> In particular, I aimed for low cost and low power. Each node in the
>> cluster boots off the network (single point of maintenance for all node OS
>> configuration) and just requires a data drive. Nodes are assigned addresses
>> using DHCP and automatically update vm.args and execute reip if the node
>> address changes from boot to boot. The scripts for this might be handy for
>> anyone else running in an environment where machine addresses change
>> (Amazon, etc).
>>
>> I'd love feedback, and more to the point, I'd like to find someone to
>> walk through it step-by-step and make sure I didn't miss anything.
>>
>> Oh, and *thanks* to the list for helping me with configuration help over
>> the last couple weeks!
>>
>> jeff
>>
>> --
>>
>> jeffrey k eliasen
>>
>> Find and follow me on:
>> Blog: http://jeff.jke.net
>> Twitter: http://twitter.com/jeffreyeliasen
>> Facebook: http://facebook.com/jeffrey.eliasen
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>


-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak dynamic cluster HOWTO

[Jeffrey Eliasen]

Yeah, that goal changed over time and I forgot to update the front page. I
realized I didn't have enough resources to handle the overhead of that many
machines (open file handles, RAM resources, etc) and reduced my expectation
to about 10 machines, with a more realistic probability of 5-6. In this
case, the default ring size should be adequate. I agree I need to annotate
that page and/or the configuration pages to reflect the shift.

I *really* appreciate that it's being read, and I hope it can be useful for
people :)

jeff

On Wed, Jan 11, 2012 at 1:03 PM, Mark Phillips  wrote:

> Hey Jeff,
> Sadly I'm just now getting around to working my way through this guide
> on account of the post-holiday catchup game...
> Firstly, great job on this. This clearly wasn't a small undertaking.
> Thanks for taking the time to write and publish it.
> One piece of initial feedback -  On the first page you state one of
> your goals as:
>
> "The cluster can be scaled to at least 128 machines."
>
> This is a great goal (and I hope everyone is building 128 node Riak
> clusters). One thing I wanted to point out (which was brought to my
> attention on twitter by @jdmaturen) is that the stock ring creation
> size of 64 won't be sufficient for this. You'll need something closer
> to 1024. Not sure how you want to note this in the tutorial, but it's
> probably worth including a reference to ring_creation_size setting [1]
> on the Custom Configuration Page [2].
>
> I'll send along anything else that comes up.
>
> Mark
>
> [1] http://wiki.basho.com/Configuration-Files.html#ring_creation_size
> [2] http://jeff.jke.net/2011/12/24/custom-configuration
> On Fri, Jan 6, 2012 at 9:23 AM, Jeffrey Eliasen  wrote:
> > I spent the Christmas break setting up a Riak cluster and documenting all
> > the steps. I've got a howto/tutorial set up at
> > http://jeff.jke.net/2011/10/02/setting-nosql-cluster that will walk you
> > through the steps.
> >
> > In particular, I aimed for low cost and low power. Each node in the
> cluster
> > boots off the network (single point of maintenance for all node OS
> > configuration) and just requires a data drive. Nodes are assigned
> addresses
> > using DHCP and automatically update vm.args and execute reip if the node
> > address changes from boot to boot. The scripts for this might be handy
> for
> > anyone else running in an environment where machine addresses change
> > (Amazon, etc).
> >
> > I'd love feedback, and more to the point, I'd like to find someone to
> walk
> > through it step-by-step and make sure I didn't miss anything.
> >
> > Oh, and *thanks* to the list for helping me with configuration help over
> the
> > last couple weeks!
> >
> > jeff
> >
> > --
> >
> > jeffrey k eliasen
> >
> > Find and follow me on:
> > Blog: http://jeff.jke.net
> > Twitter: http://twitter.com/jeffreyeliasen
> > Facebook: http://facebook.com/jeffrey.eliasen
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>



-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

How to work with JSON objects using Erlang?

[Jeffrey Eliasen]

I 've been playing with Riak for a few
weeks now using Python, and JavaScript, so I have ~8M records in JSON
stored in my cluster. Now I would like to run a map/reduce job that adds
links and index data to every item.

>From what I understand, this can't be done using JavaScript and map/reduce,
so I need to use Erlang. I've been learning Erlang for a couple weeks, but
I'm still pretty new to it overall.

My question is this: what's the easiest way to manipulate this data? I see
three options:

   1. Learn me some Erlang and do it that way...
  1. Where can I find specific help in learning to manipulate JSON data
  using Erlang?
  2. Am I better off converting every node to Erlang data structures
  and just using only Erlang from this point forward?
   2. Use Erlang but call Python code that does the data manipulation...
  1. Any good references for hooking into Python or other external code
  from within Erlang?
   3. Use JavaScript but call Python code that does the re-writes back to
   the cluster...
  1. This appears to be less desireable as M/R jobs will timeout
  quickly and I can't configure this; any other gotcha's I need to
know about?


Thanks!

jeffrey k eliasen
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com