Millions of buckets?

[alexeypro]

I have a need to keep "notifications" for "users", where each user has
folders "a", "b" and "c". Also every note has "read" status
Original thought was to create bucket "notifications" which will have the
key user_id, and the value will be JSON object in some structure like this:
{   
   user_id: "3ds78df7df8d7f8df7",
   folder: "a",
   read: 0,
   note: "my note here",
   timestamp: 18298329329
}
I am using Riak Search, so say, if I need to find all unread notes in folder
"a" for user_id "3ds78df7df8d7f8df7" I am doing:

select?q=..

user_id:3ds78df7df8d7f8df7 AND read:0

This works ugly slow -- takes about 12 secs for my case with only 5mln of
records bringing 5 results back.
My next idea was to create buckets per every user. Say user
"3ds78df7df8d7f8df7" will get bucket: "3ds78df7df8d7f8df7_notes" and we
search within it, which should be faster.

Please advise!

--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Millions-of-buckets-tp2928567p2928567.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Millions of buckets?

[alexeypro]

Generally the problem there that I may end up with N buckets, where N is
number of users. If I have 5 to 10 mln of users -- then it's a lot of
buckets. How Riak will handle it?

--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Millions-of-buckets-tp2928567p2928642.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Millions of buckets?

[Aphyr]

Since buckets are essentially key prefixes, I think buckets will 
probably not make this faster. Maybe one of the riak-search experts 
knows why your search is taking so long.


--Kyle

On 05/11/2011 12:00 PM, alexeypro wrote:

Generally the problem there that I may end up with N buckets, where N is
number of users. If I have 5 to 10 mln of users -- then it's a lot of
buckets. How Riak will handle it?

--
View this message in context: 
http://riak-users.197444.n3.nabble.com/Millions-of-buckets-tp2928567p2928642.html
Sent from the Riak Users mailing list archive at Nabble.com.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Millions of buckets?

[Jared Morrow]

It seems like what you are needing is a lot what the Yammer guys needed for
their streamie application.   They have a video here:
http://vimeo.com/21598799 about how they modeled their data.   It might be
pretty helpful for your application.   If not, no harm done, you still get
to watch a video from some pretty smart people!

Hope it helps,
Jared

On Wed, May 11, 2011 at 12:39 PM, alexeypro  wrote:

> I have a need to keep "notifications" for "users", where each user has
> folders "a", "b" and "c". Also every note has "read" status
> Original thought was to create bucket "notifications" which will have the
> key user_id, and the value will be JSON object in some structure like this:
> {
>   user_id: "3ds78df7df8d7f8df7",
>   folder: "a",
>   read: 0,
>   note: "my note here",
>   timestamp: 18298329329
> }
> I am using Riak Search, so say, if I need to find all unread notes in
> folder
> "a" for user_id "3ds78df7df8d7f8df7" I am doing:
>
> select?q=..
>
> user_id:3ds78df7df8d7f8df7 AND read:0
>
> This works ugly slow -- takes about 12 secs for my case with only 5mln of
> records bringing 5 results back.
> My next idea was to create buckets per every user. Say user
> "3ds78df7df8d7f8df7" will get bucket: "3ds78df7df8d7f8df7_notes" and we
> search within it, which should be faster.
>
> Please advise!
>
> --
> View this message in context:
> http://riak-users.197444.n3.nabble.com/Millions-of-buckets-tp2928567p2928567.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Millions of buckets?

[Alexey Prohorenko]

Well, technically searching within a smaller bucket (with much less keys)
should be faster, correct?

On Wed, May 11, 2011 at 12:19 PM, Aphyr  wrote:

> Since buckets are essentially key prefixes, I think buckets will probably
> not make this faster. Maybe one of the riak-search experts knows why your
> search is taking so long.
>
> --Kyle
>
>
> On 05/11/2011 12:00 PM, alexeypro wrote:
>
>> Generally the problem there that I may end up with N buckets, where N is
>> number of users. If I have 5 to 10 mln of users -- then it's a lot of
>> buckets. How Riak will handle it?
>>
>> --
>> View this message in context:
>> http://riak-users.197444.n3.nabble.com/Millions-of-buckets-tp2928567p2928642.html
>>
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

alphanumeric sort

[Claus Guttesen]

Hi.

I have a function that performs a numeric sort (ascending) as
described at http://siculars.posterous.com/using-riaks-mapreduce-for-sorting.
But how to perform an alphanumeric sort?

I'm using the code found in the link above:

var reduceSort = function (v, args) {
  return v.sort (function(a, b) { return a['key'] - b['key'] });
};

I'm writing a small app using express, riakjs and riak to keep track
of crosscountry-races for kids, and would like to sort by name like:

var reduceSort = function (v, args) {
  return v.sort (function(a, b) { return a['fname'] - b['fname'] });
};

Hopefully I'm not asking dumb questions. :-)

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentler gamester is the soonest winner.

Shakespeare

twitter.com/kometen

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: alphanumeric sort

[Eric Moritz]

Your compare function will need to return a -1, 0 or 1 for less than,
the same or greater than.  Luckily javascript allows you
to strings lexicographically so the function is a nice two liner:

function(a, b) {
   if(a == b) return 0;
   return a > b ? 1 : -1;
}

Application:

var reduceSort = function (v, args) {
 return v.sort (function(a, b) {
   if(a['key'] == b['key']) return 0;
   return a['key'] > b['key'] ? 1 : -1;
 });
};

Eric.

On Wed, May 11, 2011 at 3:54 PM, Claus Guttesen  wrote:
> Hi.
>
> I have a function that performs a numeric sort (ascending) as
> described at http://siculars.posterous.com/using-riaks-mapreduce-for-sorting.
> But how to perform an alphanumeric sort?
>
> I'm using the code found in the link above:
>
> var reduceSort = function (v, args) {
>  return v.sort (function(a, b) { return a['key'] - b['key'] });
> };
>
> I'm writing a small app using express, riakjs and riak to keep track
> of crosscountry-races for kids, and would like to sort by name like:
>
> var reduceSort = function (v, args) {
>  return v.sort (function(a, b) { return a['fname'] - b['fname'] });
> };
>
> Hopefully I'm not asking dumb questions. :-)
>
> --
> regards
> Claus
>
> When lenity and cruelty play for a kingdom,
> the gentler gamester is the soonest winner.
>
> Shakespeare
>
> twitter.com/kometen
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: alphanumeric sort

[Claus Guttesen]

> Your compare function will need to return a -1, 0 or 1 for less than,
> the same or greater than.  Luckily javascript allows you
> to strings lexicographically so the function is a nice two liner:
>
> function(a, b) {
>   if(a == b) return 0;
>   return a > b ? 1 : -1;
> }
>
> Application:
>
> var reduceSort = function (v, args) {
>  return v.sort (function(a, b) {
>   if(a['key'] == b['key']) return 0;
>   return a['key'] > b['key'] ? 1 : -1;
>  });
> };
>
> Eric.

Thank you so much! :-) Worked like a charm.

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentler gamester is the soonest winner.

Shakespeare

twitter.com/kometen

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: alphanumeric sort

[Alexander Sicular]

Iirc, "sort()" is by default lexographic in JavaScript so an added  
function is not necessary. The added function is specifically intended  
for a custom sort, ie. numerical.


You could simply do something like:

return v['key'].sort()

Try that out and lemme know if it works as you expect.

Cheers ,
Alexander

@siculars on twitter
http://siculars.posterous.com

Sent from my iPhone

On May 11, 2011, at 13:10, Claus Guttesen  wrote:


Your compare function will need to return a -1, 0 or 1 for less than,
the same or greater than.  Luckily javascript allows you
to strings lexicographically so the function is a nice two liner:

function(a, b) {
  if(a == b) return 0;
  return a > b ? 1 : -1;
}

Application:

var reduceSort = function (v, args) {
 return v.sort (function(a, b) {
  if(a['key'] == b['key']) return 0;
  return a['key'] > b['key'] ? 1 : -1;
 });
};

Eric.


Thank you so much! :-) Worked like a charm.

--
regards
Claus

When lenity and cruelty play for a kingdom,
the gentler gamester is the soonest winner.

Shakespeare

twitter.com/kometen

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Javascript errors in reduce phases seen as {error, javascript_reduce_timeout}?

[Jeremy Raymond]

Implementing the m/r functions in Erlang did the trick.

- Jeremy

On 5/10/2011 9:45 AM, Sean Cribbs wrote:
You might see an error in the log, but there are a lot of things about 
the JS VM that are too opaque. It's on our list of things to improve, 
for sure.


Sean Cribbs mailto:s...@basho.com>>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On May 10, 2011, at 9:30 AM, Jeremy Raymond wrote:

Okay thanks. I figure something like this was going on. Besides the 
timeout return value is there any way to tell what happened or at 
what phase of some chained map reduce operations things failed? Would 
I see errors in the logs that indicated a crash of the Javascript 
engine or something along those lines?


- Jeremy


On Tue, May 10, 2011 at 9:27 AM, Sean Cribbs > wrote:


For large numbers of objects, your JavaScript functions will
become bottlenecks, largely because the data has to be converted
to and from JSON for every application of the function. If your
reduce function isn't actually coalescing a value (e.g. sort
doesn't), then you will run into this problem.  See if you can
reimplement the functions in Erlang or decrease the quantity of
values being passed to and returned from the reduce functions.

Sean Cribbs mailto:s...@basho.com>>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On May 10, 2011, at 9:20 AM, Jeremy Raymond wrote:


Hello,
In a MapReduce job I have filtering through about 6500 items if
I filter out enough items my m/r job completes successfully. If
I loosen the filtering such that a large number of items would
be included in the results the m/r fails with error
{error,javascript_reduce_timeout}. This error occurs after about
10 seconds of processing. Would Javascript errors such as the
Javascript process running out of memory and crashing result in
this timeout error? I also noticed that after this error is
returned, the riak node I made the request on chugs along at
~100% CPU for about another minute but the other two nodes quite
down almost immediately after the error is returned. How might I
go about debugging this?
- Jeremy
-
This transmission (including any attachments) may contain
confidential information, privileged material (including
material protected by the solicitor-client or other applicable
privileges), or constitute non-public information. Any use of
this information by anyone other than the intended recipient is
prohibited. If you have received this transmission in error,
please immediately reply to the sender and delete this
information from your system. Use, dissemination, distribution,
or reproduction of this transmission by unintended recipients is
not authorized and may be
unlawful.___
riak-users mailing list
riak-users@lists.basho.com 
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



___
riak-users mailing list
riak-users@lists.basho.com 
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: alphanumeric sort

[Ben Tilly]

In JavaScript the default sort order depends on the data types of the
variables in question.  If either is a string, you convert the other
to a string and sort lexicographically, but if both are numbers you
compare numerically.  There is a similar inconsistency around what the
"+" operation does.

Inconsistencies that can arise from string promotion are a common
source of surprise.

On Wed, May 11, 2011 at 1:35 PM, Alexander Sicular  wrote:
> Iirc, "sort()" is by default lexographic in JavaScript so an added function
> is not necessary. The added function is specifically intended for a custom
> sort, ie. numerical.
>
> You could simply do something like:
>
> return v['key'].sort()
>
> Try that out and lemme know if it works as you expect.
>
> Cheers ,
> Alexander
>
> @siculars on twitter
> http://siculars.posterous.com
>
> Sent from my iPhone
>
> On May 11, 2011, at 13:10, Claus Guttesen  wrote:
>
>>> Your compare function will need to return a -1, 0 or 1 for less than,
>>> the same or greater than.  Luckily javascript allows you
>>> to strings lexicographically so the function is a nice two liner:
>>>
>>> function(a, b) {
>>>  if(a == b) return 0;
>>>  return a > b ? 1 : -1;
>>> }
>>>
>>> Application:
>>>
>>> var reduceSort = function (v, args) {
>>>  return v.sort (function(a, b) {
>>>  if(a['key'] == b['key']) return 0;
>>>  return a['key'] > b['key'] ? 1 : -1;
>>>  });
>>> };
>>>
>>> Eric.
>>
>> Thank you so much! :-) Worked like a charm.
>>
>> --
>> regards
>> Claus
>>
>> When lenity and cruelty play for a kingdom,
>> the gentler gamester is the soonest winner.
>>
>> Shakespeare
>>
>> twitter.com/kometen
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak Recap for May 09 - 10

[Mark Phillips]

Evening, Morning, Afternoon to All -

For today's Recap: new wiki content, new code, and more.

Enjoy.

Mark

Community Manager
Basho Technologies
wiki.basho.com
twitter.com/pharkmillups
--

Riak Recap for May 09 - 10



1) We added a new page to the wiki aimed at helping people better
estimate what sizes their Riak clusters should be. It definitely needs
some refining, so try it out and file a GitHub Issue against the wiki
repo if you think there is something that needs improving/fixing.

* Take it for a spin here --->
http://wiki.basho.com/Bitcask-Capacity-Planning.html

2) Some code-related news for Riak Search fans: we just recently made
merge_index, the Riak Search backend, a stand alone piece of code.
(You can thank Ryan Zezeski for this.) Time permitting, we'll be
trying to get some docs out the door for this.

Code here on GitHub ---> https://github.com/basho/merge_index

3) v0.9.4 of Ripple is out!

* All the details here --->
http://twitter.com/#!/seancribbs/statuses/67932699796582400

4) Another reminder about the inaugural Riak Meetup happening in NYC on 5/18:

* Details here http://www.meetup.com/NYC-Riak-Meetup/

It was also just announced that Basho CTO Justin Sheehy will be
speaking at this Meetup...

5) Another Reminder About the Erlang Meetup in Washington DC on 5/17
where Rusty Klophaus will be speaking:

* Details here ---> http://www.meetup.com/erlang/events/17550721/

6) One job for today:

* Ruby on Rails Developer in Washington DC ---> http://bit.ly/jQacOU

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: alphanumeric sort

[Claus Guttesen]

> Iirc, "sort()" is by default lexographic in JavaScript so an added function
> is not necessary. The added function is specifically intended for a custom
> sort, ie. numerical.
>
> You could simply do something like:
>
> return v['key'].sort()
>
> Try that out and lemme know if it works as you expect.

I tried it but it didn't work. And I believe sort expects a list like
[3, 5, 8, 1] but I have an array with json-formatted objects and have
to inspect each element as described in the link I referred to in my
OP.

-- 
regards
Claus

When lenity and cruelty play for a kingdom,
the gentler gamester is the soonest winner.

Shakespeare

twitter.com/kometen

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: alphanumeric sort

[Eric Moritz]

The reason why a custom function is needed is because he wants to sort
on a key in the javascript object.  If it was a list of strings,
calling .sort() would be fine.  We need a

Riak.reduceSortByField(values, name) {
return values.sort(function(x,y) {
  if(a[key] == b[key]) return 0;
  return a[key] > b[key] ? 1 : -1;
 });
}

On Thu, May 12, 2011 at 2:21 AM, Claus Guttesen  wrote:
>> Iirc, "sort()" is by default lexographic in JavaScript so an added function
>> is not necessary. The added function is specifically intended for a custom
>> sort, ie. numerical.
>>
>> You could simply do something like:
>>
>> return v['key'].sort()
>>
>> Try that out and lemme know if it works as you expect.
>
> I tried it but it didn't work. And I believe sort expects a list like
> [3, 5, 8, 1] but I have an array with json-formatted objects and have
> to inspect each element as described in the link I referred to in my
> OP.
>
> --
> regards
> Claus
>
> When lenity and cruelty play for a kingdom,
> the gentler gamester is the soonest winner.
>
> Shakespeare
>
> twitter.com/kometen
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com