Re: Is Riak suitable for s small-record write-intensive billion-records application?

[Yassen Damyanov]

Thanks again everyone (and apologies for junking your mailbox with my
last posts).

So far, I do plan to use Riak for the task. Here's my current vision
on the application design:

The several nodes will be bound to a single "external" IP via CARP.
Thus at any given moment only one of the nodes is going to receive
external requests.

There is a front-end and a back-end part of the application, each of
them running at each node. The current  CARP-active node will receive
the request, do a local lock, check for the PK presence in a local
in-memory hash table (I plan using murmur 32-bit hash*). The check
will return negative result for 99% of the requests, so this will be a
super-fast step almost always. The node front-end then handles the
record to the back-end of any of the nodes in the cluster, doing
load-balancing.

If the hash table returns a positive result, the fron-end requests the
record from Riak and returns it back. (There is a possibility that the
record is not there in which case the front-end inserts it in the
database.) Then the front-end unlocks.

Any given node can be stopped or additional nodes can be added with
almost no interruption. If the active node is taken down, CARP will
appoint a new active node and its front-end will start accepting
requests replacing the gone node. New nodes will announce themselves
to the front-end apps via multicast.

Thus Riak will work on writes most of the time; very seldom for reads,
and the eventual consistency does not seem to bring issues due to the
central PK-checking.

Does this sound reasonable? What Riak back-end would you recommend for
this scenario? Your comments and suggestions are very much
appreciated.

Yassen

[*] murmur 32-bit, if suitable in terms of hash values distribution.
This means 4Gigs RAM for the table ... not awful lot.

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

SV: Is Riak suitable for s small-record write-intensive billion-records application?

[Jens Rantil]

Hi Yassen,

> Any given node can be stopped or additional nodes can be added with almost no 
> interruption. If the active node is taken down, CARP will appoint a new 
> active node and its front-end will start accepting requests replacing the 
> gone node. New nodes will announce themselves to the front-end apps via 
> multicast.

But CARP only handles when the _machine_ goes down, right? Have you planned for 
the scenario when if Riak would go down, but the machine would be responsive? 
If not, haproxy could be an option.

Regards,
Jens


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Logging chat messages

[Shashwat Srivastava]

Hi,

I want to inquire about feasibility of using riak for logging chat
messages. My architecture is as follows: There are multiple sites where a
user can either send message to another user or to a room (on the same
site), and these messages need to be logged. Is riak a good fit for this
use case?

I intend to make an inbox where users would be able to view all their
conversations (with other users or in a specific room per site). The users
would have ability to view messages via pagination starting with the most
recent message being listed first i.e. last 50 messages, and then next last
50, and so on.

I was planning to create a bucket for every pair of user (who chat) and
every room per site. The user base of these sites is huge, and would grow
with time, and also new sites would join in with time. So, the number of
buckets would increase at a very fast pace. Is this going to be a problem?

Now, each bucket would have conversation between two users or of a room of
a site. The conversation rate for (some) rooms is very high, some 20,000 -
30,000 messages per hour. We have observed that users usually don't access
conversations past one week. So, if a bucket has conversation of 3 years,
then mostly users would access the recent conversation upto a week or
month. Can riak handle this easily? Also, would riak use RAM wisely in this
scenario? Would it only keep keys and indexes, corresponding to recent
messages per bucket, in RAM?

Finally, what is the best approach for creating keys in a bucket? Earlier,
I was planning to use timestamp (in milliseconds). But in a room there can
be multiple messages at the same time. As I understand I cannot have
a unique incremental message id per bucket (as riak has write capability in
all nodes in a cluster so consistency is not guareented). Please correct me
if I am wrong. One other way could be to let riak generate key and I use
timestamp as a secondary index. But this seems to be a bad design. Also,
what would be the best way to achieve pagination for this use case?

Please do let me know if this can be implemented in a altogether different
and better manner.

Thanks,
Shashwat
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Logging chat messages

[Rapsey]

On Mon, Oct 22, 2012 at 10:29 AM, Shashwat Srivastava wrote:

>
> Now, each bucket would have conversation between two users or of a room of
> a site. The conversation rate for (some) rooms is very high, some 20,000 -
> 30,000 messages per hour. We have observed that users usually don't access
> conversations past one week. So, if a bucket has conversation of 3 years,
> then mostly users would access the recent conversation upto a week or
> month. Can riak handle this easily? Also, would riak use RAM wisely in this
> scenario? Would it only keep keys and indexes, corresponding to recent
> messages per bucket, in RAM?
>
>
Leveldb backend should.


> Finally, what is the best approach for creating keys in a bucket? Earlier,
> I was planning to use timestamp (in milliseconds). But in a room there can
> be multiple messages at the same time. As I understand I cannot have
> a unique incremental message id per bucket (as riak has write capability in
> all nodes in a cluster so consistency is not guareented). Please correct me
> if I am wrong. One other way could be to let riak generate key and I use
> timestamp as a secondary index. But this seems to be a bad design. Also,
> what would be the best way to achieve pagination for this use case?
>
>
You could use redis for incremental id's.



Sergej
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Logging chat messages

[Guido Medina]

Don't overkill it with technology, you could use Riak with a simple 2i 
index (integer index MMDD for the message date so you can search day 
by day backward), and for the message sequence or identifier you could 
either user ANY SQL database sequence or a UUID generator.


HTH,

Guido.

On 22/10/12 10:04, Rapsey wrote:


On Mon, Oct 22, 2012 at 10:29 AM, Shashwat Srivastava 
mailto:dark...@gmail.com>> wrote:



Now, each bucket would have conversation between two users or of a
room of a site. The conversation rate for (some) rooms is very
high, some 20,000 - 30,000 messages per hour. We
have observed that users usually don't access conversations past
one week. So, if a bucket has conversation of 3 years, then mostly
users would access the recent conversation upto a week or month.
Can riak handle this easily? Also, would riak use RAM wisely in
this scenario? Would it only keep keys and indexes, corresponding
to recent messages per bucket, in RAM?


Leveldb backend should.

Finally, what is the best approach for creating keys in a bucket?
Earlier, I was planning to use timestamp (in milliseconds). But in
a room there can be multiple messages at the same time. As I
understand I cannot have a unique incremental message id per
bucket (as riak has write capability in all nodes in a cluster so
consistency is not guareented). Please correct me if I am wrong.
One other way could be to let riak generate key and I use
timestamp as a secondary index. But this seems to be a bad design.
Also, what would be the best way to achieve pagination for this
use case?


You could use redis for incremental id's.


Sergej


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Is Riak suitable for s small-record write-intensive billion-records application?

[Joshua Muzaaya]

riak can well do this, but you will have to be strong and willing to learn.
If you want a faster solution for the same, have you checked out Couchbase
Server ( http://couchbase.com) , it too, can handle this data, and its
setup is painless, with already finished SDKs and stuff, JSON in-and-out
and built-in memcached Hashing. You can add or remove nodes from the
cluster at run-time.

This is not meant to down-market Riak, but you mentioned Billions of
records. Riak storage is known to have a few issues as data grows to
billions. However, Couchbase 2.0 has been kinda battle tested, using SQLite
at the storage layer. Get more info there.

So, besides riak, give couchbase a test too.

  
Designed with WiseStamp -
Get
yours



On Mon, Oct 22, 2012 at 11:23 AM, Jens Rantil wrote:

> Hi Yassen,
>
> > Any given node can be stopped or additional nodes can be added with
> almost no interruption. If the active node is taken down, CARP will appoint
> a new active node and its front-end will start accepting requests replacing
> the gone node. New nodes will announce themselves to the front-end apps via
> multicast.
>
> But CARP only handles when the _machine_ goes down, right? Have you
> planned for the scenario when if Riak would go down, but the machine would
> be responsive? If not, haproxy could be an option.
>
> Regards,
> Jens
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



-- 
*Muzaaya Joshua
Systems Engineer
+256774115170*
*"Through it all, i have learned to trust in Jesus. To depend upon His Word"
*
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Is Riak suitable for s small-record write-intensive billion-records application?

[Sean Cribbs]

On Mon, Oct 22, 2012 at 9:41 AM, Joshua Muzaaya  wrote:

>
> This is not meant to down-market Riak, but you mentioned Billions of
> records. Riak storage is known to have a few issues as data grows to
> billions. However, Couchbase 2.0 has been kinda battle tested, using SQLite
> at the storage layer.


I'm not sure how you are backing up this assertion (citation?). We have
customers storing terabytes of data with billions of records (using
LevelDB). SQLite as the storage engine does not necessarily make Couchbase
more stable or battle-tested, just better known among the wider community.
On the other hand, we have spent the last eighteen months, in collaboration
with the original authors at Google, hardening and improving the
performance of LevelDB to support large deployments, and it now performs
very well (although we still can do better).

I'm not saying that the OP should not consider Couchbase, but either
product is going to require more than just a casual knowledge of ops to
deploy, manage and maintain. Clustered datastores are not for the
faint-of-heart, so if the OP can do it in a local SQL database, then he
should.

Sean Cribbs 
Software Engineer
Basho Technologies, Inc.
http://basho.com/
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Is Riak suitable for s small-record write-intensive billion-records application?

[Jared Morrow]

If anyone would like a comparison of Riak to Couchbase with citations for
each bullet point, we took a lot of time to make sure our comparisons page
is objective and fact based.

http://docs.basho.com/riak/latest/references/appendices/comparisons/Riak-Compared-to-Couchbase/
is the one for Riak v. Couchbase.

-Jared


On Mon, Oct 22, 2012 at 7:55 AM, Sean Cribbs  wrote:

>
>
> On Mon, Oct 22, 2012 at 9:41 AM, Joshua Muzaaya wrote:
>
>>
>> This is not meant to down-market Riak, but you mentioned Billions of
>> records. Riak storage is known to have a few issues as data grows to
>> billions. However, Couchbase 2.0 has been kinda battle tested, using SQLite
>> at the storage layer.
>
>
> I'm not sure how you are backing up this assertion (citation?). We have
> customers storing terabytes of data with billions of records (using
> LevelDB). SQLite as the storage engine does not necessarily make Couchbase
> more stable or battle-tested, just better known among the wider community.
> On the other hand, we have spent the last eighteen months, in collaboration
> with the original authors at Google, hardening and improving the
> performance of LevelDB to support large deployments, and it now performs
> very well (although we still can do better).
>
> I'm not saying that the OP should not consider Couchbase, but either
> product is going to require more than just a casual knowledge of ops to
> deploy, manage and maintain. Clustered datastores are not for the
> faint-of-heart, so if the OP can do it in a local SQL database, then he
> should.
>
> Sean Cribbs 
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

riak-js pre-release announcement/warning

[Mathias Meyer]

Hey folks, 

I've been putting some work in the riak-js client for Node.js recently, and as 
we're getting pretty close to doing a release, I wanted to keep you posted on 
changes and especially breaking changes. 

About a year ago, Francisco Treacy started rewriting the code (originally 
written in CoffeeScript) in pure JavaScript. First up, thanks for all the great 
work, Frank!

After he sought a new maintainer for the library earlier this year, the fine 
folks of mostlyserious.io took over maintainership [1], and I eventually hopped 
on board and started hacking on the new JavaScript implementation too, to 
finish off missing bits [2].

Along the way, the domain riakjs.org unfortunately expired, but the website 
found a new home at http://riakjs.com, thanks to Vincent Mac [3].

The good news is that things are looking pretty good, and pretty much ready to 
be shipped, pending some updates on documentation, which I'll get to this week. 
The slightly bad news (only slightly, really) is that there are breaking 
changes.

As the client hasn't officially reached v1.0 yet, I figured it's still a good 
opportunity to move things around, make things more explicit and cleaner.

The basic APIs, though, are still the same or at least very similar. The most 
important changes affect Riak Search and MapReduce.

Riak Search is now fully supported, which means documents (i.e. JavaScript 
objects) can be added and removed directly without the need to store them in 
Riak. The functionality is namespace in the client, e.g. to find documents, you 
can use the following snippet:

db.search.find("users", "email:me...@paperplanes.de")

Same for adding and removing documents:

db.search.add("users", {id: "mathias", email: "me...@paperplanes.de"}) or
db.search.add("users", [{id: "roidrage", email: "me...@paperplanes.de"}, {id: 
"mathias", email: "math...@travis-ci.org"}])

db.search.remove("users", [{id: "mathias"}, {query: 
'email:"me...@paperplanes.de"'}])

MapReduce was also moved to its own namespace, with the API being pretty much 
the same as in the CoffeeScript client:

db.mapreduce.add("users").map("Riak.mapValuesJson").run()

Moving these out into separate namespaces is mostly a matter of clarity and 
separating concerns, drawing some inspiration from other Riak clients.

There are other changes, e.g. regarding bucket properties (now db.getBucket() 
and db.saveBucket()), Luwak (getFile(), removeFile(), saveFile()), though Luwak 
would also be a candidate to be moved into a different namespace and having to 
be explicitly enabled because it's been deprecated and is not included in the 
standard distribution anymore.

So we're close to doing a release, but I just wanted to make sure information 
on what things might break is out before we do. If you rely strongly on 
existing APIs, make sure to pin your package.json to 0.4.1.

The next release might still be a bit rough around the edge, and it'll be 
missing support for Protocol Buffers, but one step at a time :)

Let me know if you have any suggestions or feedback!

Cheers, Mathias

[1] https://github.com/mostlyserious/riak-js
[2] https://github.com/basho-labs/riak-js
[3] https://github.com/vincentmac/riakjs.com
-- 
http://riakhandbook.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak Enterprise: can it be used to migrate to a new configuration?

[Dave Brady]

Hi Evan,

Thanks for responding!

We have five nodes now, though that will grow to 15-20.  We are currently 
projecting 10 TB of data, so 30 TB with N=3.

I cannot find the exact page on your docs site that had the 512 recommendation 
for five nodes.  I certainly may have misinterpreted what was written, on 
whatever page I thought I saw it.

Now to try to figure out how we're going to make this change...

- Original Message -
From: "Evan Vigil-McClanahan" 
To: "Rune Skou Larsen" , dbr...@weborama.com
Cc: riak-users@lists.basho.com
Sent: Friday, October 19, 2012 8:16:03 PM
Subject: Re: Riak Enterprise: can it be used to migrate to a new configuration?

Dave,

64 is fine for a 6 node cluster.  Rune gives a great rundown of the
downsides of large rings on small numbers of machines in his post.
Usually our recommendation is for ~10 ring partitions per physical
machine, rounded up to the next power of two.  Where did you see the
recommendation for 512 from us?

Rune,
Basho's replication won't work in the situation that you've described.
 Are you talking about an in-house replication product?  Our full-sync
doesn't work between clusters of different ring sizes.

On Fri, Oct 19, 2012 at 4:50 AM, Rune Skou Larsen  wrote:
> Yes, we have done excatly that. When we migrated from 256 to 128 partitions
> in a live dual-cluster system, we took one cluster down. Wiped the data,
> changed number of partitions, brought it back up and synced all data back
> with a full sync. Then we did the same with the other cluster.
>
> However, I must disagree with the recomendation of 512 partitions for 5
> nodes. You should go for 128 or 256 unless you plan on scaling out to 10+
> nodes pr. cluster.
>
> There are downsides to having many partitions. The price of the higher
> granularity is that the more storage backend processes use more resources
> for housekeeping. If you do multibackend, the ressources used are multiplied
> yet again with the number of backends because each vnode will have a number
> of running backend processes.
>
> Say you go with the 512 partitions and have a multibackend config with 4
> backends, because you need to backup 4 different types of data
> independently. That gives you 2k running backends on each node of which 412
> will be actively in use in normal running scenario and more when you're
> doing handoff. Thats a lot of ressources just to run these, that you might
> otherwise have used for doing business.
>
> When you increase the number of partitions you should consider:
> - Number of open files. Especially when using eleveldb.
> - Late triggering of bitcask compaction. The default is no compaction of any
> file before it hits 2GB. That means up to 2G of dead space per vnode. This
> can however be configured down to a smaller number than the 2 gigs, which is
> crazy high in almost any use case involving delete, expiry or update of
> data.
> - Leveldb cache is pr. vnode, so you need to lower the number, in order to
> not use all memory, which will lead to death by swapping.
> - With a high number of vnodes pr. node, each vnode's leveldb cache will be
> comparatively small leading to (slighty) less effecient cache usage.
>
> Please be in touch if you need onsite or offsite professional assistance
> configuring, testing or running your Riak clusters.
>
> BR Rune Skou Larsen
>
> Trifork
> - We do Riak PS.
>
> --
>
> Best regards / Venlig hilsen
>
> Rune Skou Larsen
> Trifork Public A/S / Team Riak
> Margrethepladsen 4, 8000 Århus C, Denmark
> Phone: +45 3160 2497  Skype: runeskoularsen   twitter: @RuneSkouLarsen
>
>
>
> Den 19-10-2012 12:38, Dave Brady skrev:
>
> Can Riak Enterprise replicate between rings where each ring has a different
> number of partitions?
>
> Our five-node ring was originally configured with 64 partitions, and I saw
> that Basho is recommending 512 for that number of machines.
>
> Any ideas on how to make as-painless-a-migration-as-possible are welcome, of
> course!
>
> --
> Dave Brady
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Newbie question -- how do I turn on web-based search interface for port 8098?

[Felix Terkhorn]

Hi,

I've searched around on the web for this but am coming up short.  How do I turn 
on the web-based search interface which is documented here? 
(http://docs.basho.com/riak/latest/references/Riak-Search---Settings/#Default 
Ports)

Using "netstat -a | grep 8098" I can see that there's definitely nothing 
listening on that port.

I do have search enabled in app.config for each of my four nodes.

Thanks,
Felix
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Newbie question -- how do I turn on web-based search interface for port 8098?

[Sean Cribbs]

Hi Felix,

Riak Search uses the same interface/port as the normal Riak HTTP
operations, so look in your app.config file for the {http, [ ... ]}
line to find the host/port combination.  There's no UI exported from
that interface, but you should be able to use Solr-like URLs to
perform queries and modify the indexes.

On Mon, Oct 22, 2012 at 11:18 AM, Felix Terkhorn
 wrote:
> Hi,
>
>
>
> I’ve searched around on the web for this but am coming up short.  How do I
> turn on the web-based search interface which is documented here?
> (http://docs.basho.com/riak/latest/references/Riak-Search---Settings/#Default
> Ports)
>
>
>
> Using “netstat -a | grep 8098” I can see that there’s definitely nothing
> listening on that port.
>
>
>
> I do have search enabled in app.config for each of my four nodes.
>
>
>
> Thanks,
>
> Felix
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



-- 
Sean Cribbs 
Software Engineer
Basho Technologies, Inc.
http://basho.com/

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Newbie question -- how do I turn on web-based search interface for port 8098?

[Felix Terkhorn]

Hi Sean,

Great.  Thanks a ton.  The documentation in docs.basho.com is a bit out of 
date, but with your help, I was able to quickly fire off a query which I'd 
confirmed as working with the command line utility.  Looking forward to deeper 
dives into Riak's capabilities.

Cheers,
Felix

-Original Message-
From: Sean Cribbs [mailto:s...@basho.com] 
Sent: Monday, October 22, 2012 11:23 AM
To: Felix Terkhorn
Cc: riak-users@lists.basho.com
Subject: Re: Newbie question -- how do I turn on web-based search interface for 
port 8098?

Hi Felix,

Riak Search uses the same interface/port as the normal Riak HTTP operations, so 
look in your app.config file for the {http, [ ... ]} line to find the host/port 
combination.  There's no UI exported from that interface, but you should be 
able to use Solr-like URLs to perform queries and modify the indexes.

On Mon, Oct 22, 2012 at 11:18 AM, Felix Terkhorn  
wrote:
> Hi,
>
>
>
> I've searched around on the web for this but am coming up short.  How 
> do I turn on the web-based search interface which is documented here?
> (http://docs.basho.com/riak/latest/references/Riak-Search---Settings/#
> Default
> Ports)
>
>
>
> Using "netstat -a | grep 8098" I can see that there's definitely 
> nothing listening on that port.
>
>
>
> I do have search enabled in app.config for each of my four nodes.
>
>
>
> Thanks,
>
> Felix
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



--
Sean Cribbs 
Software Engineer
Basho Technologies, Inc.
http://basho.com/


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: MapReduce questions

[Mark Phillips]

Hi C,

On Fri, Oct 19, 2012 at 10:04 AM, Callixte Cauchois
 wrote:
> Hey Mark,
>
> really sorry if I sounded aggressive or whatever. English is not my primary
> language and sometimes I do not sound as I wanted... I just wanted to
> acknowledge that no answer was kind of an answer to my questions.

No worries. And sorry about my tone if it was too strong. Responding
to emails before any coffee is not recommended. :)

> And yes, I will share how I would like M/R to behave for future reference,
> even if it is not going to be implemented short or mid term. Unfortunately,
> I do not speak Erlang, so it would be hard for me to contribute, even if I
> would love to. Maybe, I'd feel extra adventurous and give it a try though.
> Thank you for your response, and I also wanted to say that I love the new
> documentation site, compared to how it can be a struggle to understand how
> some competitors'product can be installed, it really kicks ass.
>

Erlang can be a bit daunting to start, but once you dive in you'll
probably be hooked. (Don't say I didn't warn you.) If you're looking
for some Riak-related ways to test it out, there are a few contributed
erlang functions [1] that might be worth examining. Also, writing
riaknostic diagnostics [2] is an easy way to start learning. We're
also happy to help on this list where possible.

The docs have needed a lot of love for some time. There is a lot more
good stuff coming for docs, resources, etc. Glad you like them. Make
sure to tell us where they need some love.

Best,

Mark

[1] http://contrib.basho.com/map-reduce-functions.html
[2] http://riaknostic.basho.com

> C.
>
>
> On Fri, Oct 19, 2012 at 9:52 AM, Mark Phillips  wrote:
>>
>> On Fri, Oct 19, 2012 at 8:48 AM, Callixte Cauchois
>>  wrote:
>> > So... no answers.
>> > I guess there are no smart minds at Basho working on M/R currently. Too
>> > bad,
>> > but I guess a company has to choose its priority.
>>
>> A lovely "good morning" to you, too.
>>
>> >
>> as part of my evaluation of Riak, I am looking at the M/R capabilities
>> and I have several questions:
>> 1/ the doc states that " Riak MapReduce is intended for batch
>> processing, not real-time querying." But as of now, you always get the
>> results and cannot automatically store them in a bucket like you would
>> do with MongoDB for example. Is taht something that is on the roadmap?
>> >
>>
>> This isn't something that's on the roadmap. That doesn't mean we won't
>> do it, but no one is working on code that does this at the moment.
>> That said, I'm told by one of our smart, busy minds here that works on
>> m/r that it would not be the most complicated addition. It may well
>> end up being part of future work but, again, there's no timeline on
>> it.  In the mean time, any use case details you could share (short of
>> "like you would do with MongoDB") that would help inform the
>> development would be stellar.
>>
>> If you were feeling adventurous and wanted to write some erlang code,
>> this behavior could be written as part of map or reduce job.
>>
>> >
>> 2/ Basho blog has two articles from last year on Hadoop on top of
>> Riak. Is this project still live? It is on github, so I can always dig
>> it up and work on it, but I am wondering if there is something like
>> Yokozuna for Hadoop integration.
>> >
>>
>> The work on this specific code has stalled. That said, there are some
>> people using Riak in production alongside hadoop, and we have plans to
>> build deeper integration between the two at some point.
>>
>> As an aside, if you're looking for lower latency responses on
>> implementation questions, you might want to consider hanging out in
>> #riak on freenode.
>>
>> Mark
>> twitter.com/pharkmillups
>>
>> >
>> >
>> > On Tue, Oct 16, 2012 at 11:03 AM, Callixte Cauchois
>> > 
>> > wrote:
>> >>
>> >> Hi there,
>> >>
>> >> as part of my evaluation of Riak, I am looking at the M/R capabilities
>> >> and
>> >> I have several questions:
>> >> 1/ the doc states that " Riak MapReduce is intended for batch
>> >> processing,
>> >> not real-time querying." But as of now, you always get the results and
>> >> cannot automatically store them in a bucket like you would do with
>> >> MongoDB
>> >> for example. Is taht something that is on the roadmap?
>> >> 2/ Basho blog has two articles from last year on Hadoop on top of Riak.
>> >> Is
>> >> this project still live? It is on github, so I can always dig it up and
>> >> work
>> >> on it, but I am wondering if there is something like Yokozuna for
>> >> Hadoop
>> >> integration.
>> >>
>> >> Thanks.
>> >> C.
>> >
>> >
>> >
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Riak performance problems when LevelDB database grows beyond 16GB

[Matthew Von-Maszewski]

Jan,

I apologize for the delayed response.

1.  Did you realize that the "log_jan.txt" file from #1 below documented a hard 
disk failure?  You mentioned a failed drive once.  I am not sure if this is the 
same drive.


2.  The "sse4_2" tells me that your Intel cpu supports hardware CRC32c 
calculation.  This feature is not useful to you at this moment (unless you want 
to pull the mv-hardware-crc branch from basho/leveldb).  It will bring some 
performance improvements to you in the next release IF we do not decide that 
your problems are hard disk performance limited.


3.  This just confirmed for me that the app.config is not accidentally blowing 
your physical memory.  The app.config file you posted suggested this was not 
the case, but I wanted to verify.

You also discuss a basho_bench failure.  Is this the same test run as the 
log_jan.txt file?  The hard drives had their first failure at:

2012/10/18-02:08:44.136238 7f8297fff700 Compaction error: Corruption: corrupted 
compressed block contents

And things go really bad at:

2012/10/18-06:10:37.657072 7f829effd700 Moving corrupted block to 
lost/BLOCKS.bad (size 1647)


4.  I was looking to see if your data was compressing well.  The answer it that 
it is.  You are achieving 2 to 2.6x compression ratio.  Since you are concerned 
about throughput, I was verifying that the time leveldb spends on block 
compression is worthwhile for you (it is).


The next question from me is whether the drive / disk array problems are your 
only problem at this point.  The data in log_jan.txt looks ok until the 
failures start.  I am willing to work more, but I need to better understand 
your next level of problems.

Matthew


On Oct 19, 2012, at 7:49 AM,  
 wrote:

> Hi Matthew,
> 
> big thanks for responding. I see that you are the main committer to Basho's 
> leveldb code. :-)
> 
>> 1.  Execute the following on one of the servers:
>> sort /home/riak/leveldb/*/LOG* >log_jan.txt
> 
> See the attached file log_jan.txt.gz. It is from the stalled Riak node.
> 
>> 2.  Execute the following on one of the servers:
>>  grep -i flags /proc/cpuinfo
> 
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
> pat 
> pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc 
> aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr 
> pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx lahf_lm arat epb 
> xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
> 
>> 3.  On a running server that is processing data, execute:
>> grep -i swap /proc/meminfo
> 
> I will restart the test and report back when it stalls again. In the meantime,
>  am sending you yesterday's zabbix graph showing memory usage on the node 
> (attached file ZabbixMemory.png). The time when the node stopped responding is
> logged as:
> 
> 2012-10-18 08:28:47.537 [error]  ** Node 'riak@172.16.0.6' not responding **
> 
> I am also attaching the corresponding Basho bench output of the test. The test
> was started on Oct 17 16:38 with an empty database, and it was run on a plain
> ext4 partition (no RAID).
> 
>> 4.  Pick a server, then one directory in /home/riak/leveldb.  Select 3 of 
> the largest *.sst files.  Tar/gzip those and email back.
> 
> I will send them in the next mail. I can also put the entire database 
> somewhere for you for download, if you need it.
> 
> Thanks, Jan<4Riak_2K_2.1RC2_noraid.png>


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Is Riak suitable for s small-record write-intensive billion-records application?

[Les Mikesell]

On Mon, Oct 22, 2012 at 9:00 AM, Jared Morrow  wrote:
> If anyone would like a comparison of Riak to Couchbase with citations for
> each bullet point, we took a lot of time to make sure our comparisons page
> is objective and fact based.
>
> http://docs.basho.com/riak/latest/references/appendices/comparisons/Riak-Compared-to-Couchbase/
> is the one for Riak v. Couchbase.

Do you know how bigcouch fits into this picture?

-- 
   Les Mikesell
 lesmikes...@gmail.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riakc doesn't save metadata properly?

[David Parfitt]

Hello Metin -

  Sorry for the delay. At the moment, all metadata needs to be stored
under the <<"X-Riak-Meta">> dict key. The following example
illustrates how this works:

%% To store metadata:
Object = riakc_obj:new(<<"groceries">>, <<"mine">>, <<"eggs & bacon">>).
MetaData  = dict:from_list([{<<"X-Riak-Meta">>, [{"Foo", "Bar"}]}]),
Object2 = riakc_obj:update_metadata(Object, MetaData).
riakc_pb_socket:put(Pid, Object2).

%% To retrieve metadata:
{ok, O} = riakc_pb_socket:get(Pid, <<"groceries">>, <<"mine">>).
{ok, MD} = dict:find(<<"X-Riak-Meta">>, riakc_obj:get_metadata(O)).

We're kicking around ways to improve this in the future.

Cheers -
Dave

On Fri, Oct 19, 2012 at 8:31 AM, Metin Akat  wrote:
> I am trying to do something like:
>
> Meta1 = dict:store(<<"ver">>, Ver, Meta0),
> Obj1 = riakc_obj:update_metadata(Obj, Meta1),
>
> and then save the object. But on a subsequent read, the metadata k/v pair is
> not there.
>
> Is there anything special that needs to be done for this to work, or is this
> illegal way to use Riak?
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riakc doesn't save metadata properly?

[Metin Akat]

Oh, I see, thanks for this clarification, much appreciated.
I think this should be written somewhere in bold, as 3 days later, I still
wasn't able to find anything on google.

On Tue, Oct 23, 2012 at 12:08 AM, David Parfitt  wrote:

> Hello Metin -
>
>   Sorry for the delay. At the moment, all metadata needs to be stored
> under the <<"X-Riak-Meta">> dict key. The following example
> illustrates how this works:
>
> %% To store metadata:
> Object = riakc_obj:new(<<"groceries">>, <<"mine">>, <<"eggs & bacon">>).
> MetaData  = dict:from_list([{<<"X-Riak-Meta">>, [{"Foo", "Bar"}]}]),
> Object2 = riakc_obj:update_metadata(Object, MetaData).
> riakc_pb_socket:put(Pid, Object2).
>
> %% To retrieve metadata:
> {ok, O} = riakc_pb_socket:get(Pid, <<"groceries">>, <<"mine">>).
> {ok, MD} = dict:find(<<"X-Riak-Meta">>, riakc_obj:get_metadata(O)).
>
> We're kicking around ways to improve this in the future.
>
> Cheers -
> Dave
>
> On Fri, Oct 19, 2012 at 8:31 AM, Metin Akat  wrote:
> > I am trying to do something like:
> >
> > Meta1 = dict:store(<<"ver">>, Ver, Meta0),
> > Obj1 = riakc_obj:update_metadata(Obj, Meta1),
> >
> > and then save the object. But on a subsequent read, the metadata k/v
> pair is
> > not there.
> >
> > Is there anything special that needs to be done for this to work, or is
> this
> > illegal way to use Riak?
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riakc doesn't save metadata properly?

[David Parfitt]

Hello Metin -

I'm trying to get that into the docs right now. Will italics do? :-)
https://github.com/basho/riak-erlang-client/pull/74

Cheers -
Dave

On Mon, Oct 22, 2012 at 5:22 PM, Metin Akat  wrote:
> Oh, I see, thanks for this clarification, much appreciated.
> I think this should be written somewhere in bold, as 3 days later, I still
> wasn't able to find anything on google.
>
>
> On Tue, Oct 23, 2012 at 12:08 AM, David Parfitt  wrote:
>>
>> Hello Metin -
>>
>>   Sorry for the delay. At the moment, all metadata needs to be stored
>> under the <<"X-Riak-Meta">> dict key. The following example
>> illustrates how this works:
>>
>> %% To store metadata:
>> Object = riakc_obj:new(<<"groceries">>, <<"mine">>, <<"eggs & bacon">>).
>> MetaData  = dict:from_list([{<<"X-Riak-Meta">>, [{"Foo", "Bar"}]}]),
>> Object2 = riakc_obj:update_metadata(Object, MetaData).
>> riakc_pb_socket:put(Pid, Object2).
>>
>> %% To retrieve metadata:
>> {ok, O} = riakc_pb_socket:get(Pid, <<"groceries">>, <<"mine">>).
>> {ok, MD} = dict:find(<<"X-Riak-Meta">>, riakc_obj:get_metadata(O)).
>>
>> We're kicking around ways to improve this in the future.
>>
>> Cheers -
>> Dave
>>
>> On Fri, Oct 19, 2012 at 8:31 AM, Metin Akat  wrote:
>> > I am trying to do something like:
>> >
>> > Meta1 = dict:store(<<"ver">>, Ver, Meta0),
>> > Obj1 = riakc_obj:update_metadata(Obj, Meta1),
>> >
>> > and then save the object. But on a subsequent read, the metadata k/v
>> > pair is
>> > not there.
>> >
>> > Is there anything special that needs to be done for this to work, or is
>> > this
>> > illegal way to use Riak?
>> >
>> > ___
>> > riak-users mailing list
>> > riak-users@lists.basho.com
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riakc doesn't save metadata properly?

[Metin Akat]

Haha, well, the bold part was a bit of a joke, but any docs will be
perfect. Thanks :)

On Tue, Oct 23, 2012 at 12:25 AM, David Parfitt  wrote:

> Hello Metin -
>
> I'm trying to get that into the docs right now. Will italics do? :-)
> https://github.com/basho/riak-erlang-client/pull/74
>
> Cheers -
> Dave
>
> On Mon, Oct 22, 2012 at 5:22 PM, Metin Akat  wrote:
> > Oh, I see, thanks for this clarification, much appreciated.
> > I think this should be written somewhere in bold, as 3 days later, I
> still
> > wasn't able to find anything on google.
> >
> >
> > On Tue, Oct 23, 2012 at 12:08 AM, David Parfitt 
> wrote:
> >>
> >> Hello Metin -
> >>
> >>   Sorry for the delay. At the moment, all metadata needs to be stored
> >> under the <<"X-Riak-Meta">> dict key. The following example
> >> illustrates how this works:
> >>
> >> %% To store metadata:
> >> Object = riakc_obj:new(<<"groceries">>, <<"mine">>, <<"eggs & bacon">>).
> >> MetaData  = dict:from_list([{<<"X-Riak-Meta">>, [{"Foo", "Bar"}]}]),
> >> Object2 = riakc_obj:update_metadata(Object, MetaData).
> >> riakc_pb_socket:put(Pid, Object2).
> >>
> >> %% To retrieve metadata:
> >> {ok, O} = riakc_pb_socket:get(Pid, <<"groceries">>, <<"mine">>).
> >> {ok, MD} = dict:find(<<"X-Riak-Meta">>, riakc_obj:get_metadata(O)).
> >>
> >> We're kicking around ways to improve this in the future.
> >>
> >> Cheers -
> >> Dave
> >>
> >> On Fri, Oct 19, 2012 at 8:31 AM, Metin Akat 
> wrote:
> >> > I am trying to do something like:
> >> >
> >> > Meta1 = dict:store(<<"ver">>, Ver, Meta0),
> >> > Obj1 = riakc_obj:update_metadata(Obj, Meta1),
> >> >
> >> > and then save the object. But on a subsequent read, the metadata k/v
> >> > pair is
> >> > not there.
> >> >
> >> > Is there anything special that needs to be done for this to work, or
> is
> >> > this
> >> > illegal way to use Riak?
> >> >
> >> > ___
> >> > riak-users mailing list
> >> > riak-users@lists.basho.com
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Logging chat messages

[Shashwat Srivastava]

Thanks Sergej. I will try this with Leveldb backend and use another db for
incremental ids.

Shashwat

On Mon, Oct 22, 2012 at 2:34 PM, Rapsey  wrote:

>
> On Mon, Oct 22, 2012 at 10:29 AM, Shashwat Srivastava 
> wrote:
>
>>
>> Now, each bucket would have conversation between two users or of a room
>> of a site. The conversation rate for (some) rooms is very high, some 20,000
>> - 30,000 messages per hour. We have observed that users usually don't
>> access conversations past one week. So, if a bucket has conversation of 3
>> years, then mostly users would access the recent conversation upto a week
>> or month. Can riak handle this easily? Also, would riak use RAM wisely in
>> this scenario? Would it only keep keys and indexes, corresponding to recent
>> messages per bucket, in RAM?
>>
>>
> Leveldb backend should.
>
>
>> Finally, what is the best approach for creating keys in a bucket?
>> Earlier, I was planning to use timestamp (in milliseconds). But in a room
>> there can be multiple messages at the same time. As I understand I cannot
>> have a unique incremental message id per bucket (as riak has write
>> capability in all nodes in a cluster so consistency is not guareented).
>> Please correct me if I am wrong. One other way could be to let riak
>> generate key and I use timestamp as a secondary index. But this seems to be
>> a bad design. Also, what would be the best way to achieve pagination for
>> this use case?
>>
>>
> You could use redis for incremental id's.
>
>
>
> Sergej
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Logging chat messages

[Shashwat Srivastava]

Thank you Guido. Yes, a secondary index based on date would be immensely
helpful for me to navigate via date. I will do this. An incremental message
id would be helpful for me to get last 50 messages and so forth. I will use
another db for this. Thanks for all your help.

Shashwat

On Mon, Oct 22, 2012 at 2:46 PM, Guido Medina wrote:

>  Don't overkill it with technology, you could use Riak with a simple 2i
> index (integer index MMDD for the message date so you can search day by
> day backward), and for the message sequence or identifier you could either
> user ANY SQL database sequence or a UUID generator.
>
> HTH,
>
> Guido.
>
>
> On 22/10/12 10:04, Rapsey wrote:
>
>
> On Mon, Oct 22, 2012 at 10:29 AM, Shashwat Srivastava 
> wrote:
>
>>
>>  Now, each bucket would have conversation between two users or of a room
>> of a site. The conversation rate for (some) rooms is very high, some 20,000
>> - 30,000 messages per hour. We have observed that users usually don't
>> access conversations past one week. So, if a bucket has conversation of 3
>> years, then mostly users would access the recent conversation upto a week
>> or month. Can riak handle this easily? Also, would riak use RAM wisely in
>> this scenario? Would it only keep keys and indexes, corresponding to recent
>> messages per bucket, in RAM?
>>
>>
>  Leveldb backend should.
>
>
>> Finally, what is the best approach for creating keys in a bucket?
>> Earlier, I was planning to use timestamp (in milliseconds). But in a room
>> there can be multiple messages at the same time. As I understand I cannot
>> have a unique incremental message id per bucket (as riak has write
>> capability in all nodes in a cluster so consistency is not guareented).
>> Please correct me if I am wrong. One other way could be to let riak
>> generate key and I use timestamp as a secondary index. But this seems to be
>> a bad design. Also, what would be the best way to achieve pagination for
>> this use case?
>>
>>
>  You could use redis for incremental id's.
>
>
>
>  Sergej
>
>
> ___
> riak-users mailing 
> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Logging chat messages

[Rapsey]

There is also another trick you can use. Pick a number. Assign every app
server you have a number between 1 and N. The number assigned to the server
is your starting ID, then increment by N every time you generate an ID from
that server. The only limitation is that you have to know in advance how
big N can get (it has to be larger than the number of your app servers).


Sergej

On Tue, Oct 23, 2012 at 6:00 AM, Shashwat Srivastava wrote:

> Thank you Guido. Yes, a secondary index based on date would be immensely
> helpful for me to navigate via date. I will do this. An incremental message
> id would be helpful for me to get last 50 messages and so forth. I will use
> another db for this. Thanks for all your help.
>
> Shashwat
>
>
> On Mon, Oct 22, 2012 at 2:46 PM, Guido Medina wrote:
>
>>  Don't overkill it with technology, you could use Riak with a simple 2i
>> index (integer index MMDD for the message date so you can search day by
>> day backward), and for the message sequence or identifier you could either
>> user ANY SQL database sequence or a UUID generator.
>>
>> HTH,
>>
>> Guido.
>>
>>
>> On 22/10/12 10:04, Rapsey wrote:
>>
>>
>> On Mon, Oct 22, 2012 at 10:29 AM, Shashwat Srivastava 
>> wrote:
>>
>>>
>>>  Now, each bucket would have conversation between two users or of a
>>> room of a site. The conversation rate for (some) rooms is very high, some
>>> 20,000 - 30,000 messages per hour. We have observed that users usually
>>> don't access conversations past one week. So, if a bucket has conversation
>>> of 3 years, then mostly users would access the recent conversation upto a
>>> week or month. Can riak handle this easily? Also, would riak use RAM wisely
>>> in this scenario? Would it only keep keys and indexes, corresponding to
>>> recent messages per bucket, in RAM?
>>>
>>>
>>  Leveldb backend should.
>>
>>
>>> Finally, what is the best approach for creating keys in a bucket?
>>> Earlier, I was planning to use timestamp (in milliseconds). But in a room
>>> there can be multiple messages at the same time. As I understand I cannot
>>> have a unique incremental message id per bucket (as riak has write
>>> capability in all nodes in a cluster so consistency is not guareented).
>>> Please correct me if I am wrong. One other way could be to let riak
>>> generate key and I use timestamp as a secondary index. But this seems to be
>>> a bad design. Also, what would be the best way to achieve pagination for
>>> this use case?
>>>
>>>
>>  You could use redis for incremental id's.
>>
>>
>>
>>  Sergej
>>
>>
>> ___
>> riak-users mailing 
>> listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: riakc doesn't save metadata properly?

[Christian Dahlqvist]

On 22/10/2012 22:25, David Parfitt wrote:

Hello Metin -

I'm trying to get that into the docs right now. Will italics do? :-)
https://github.com/basho/riak-erlang-client/pull/74

Cheers -
Dave

On Mon, Oct 22, 2012 at 5:22 PM, Metin Akat  wrote:

Oh, I see, thanks for this clarification, much appreciated.
I think this should be written somewhere in bold, as 3 days later, I still
wasn't able to find anything on google.


On Tue, Oct 23, 2012 at 12:08 AM, David Parfitt  wrote:

Hello Metin -

   Sorry for the delay. At the moment, all metadata needs to be stored
under the <<"X-Riak-Meta">> dict key. The following example
illustrates how this works:

%% To store metadata:
Object = riakc_obj:new(<<"groceries">>, <<"mine">>, <<"eggs & bacon">>).
MetaData  =dict:from_list([{<<"X-Riak-Meta">>, [{"Foo", "Bar"}]}]),
Object2 = riakc_obj:update_metadata(Object, MetaData).
riakc_pb_socket:put(Pid, Object2).

%% To retrieve metadata:
{ok, O} = riakc_pb_socket:get(Pid, <<"groceries">>, <<"mine">>).
{ok, MD} =dict:find(<<"X-Riak-Meta">>, riakc_obj:get_metadata(O)).

We're kicking around ways to improve this in the future.

Cheers -
Dave

On Fri, Oct 19, 2012 at 8:31 AM, Metin Akat  wrote:

I am trying to do something like:

 Meta1 =dict:store(<<"ver">>, Ver, Meta0),
 Obj1 = riakc_obj:update_metadata(Obj, Meta1),

and then save the object. But on a subsequent read, the metadata k/v
pair is
not there.

Is there anything special that needs to be done for this to work, or is
this
illegal way to use Riak?

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


Hi,

As far as I can see there seems to be an inconsistency between the 
Erlang and HTTP APIs with respect to how metadata is handled.


When I query the record created in the example through the Erlang API 
(attached to console) I, as expected, get the following:


{ok, RO} = C:get(<<"groceries">>, <<"mine">>).
{ok, MD} = dict:find(<<"X-Riak-Meta">>, riak_object:get_metadata(RO)).
{ok,[{"Foo","Bar"}]}

The metadata parameter is stored in the dictionary exactly as created, 
which is expected.


When I query this record through the HTTP interface the same metadata is 
returned as a header:


curl -v -X HEAD http://localhost:8098/riak/groceries/mine
> HEAD /riak/groceries/mine HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) 
libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5

> Host: localhost:8098
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Riak-Vclock: a85hYGBgzGDKBVIcypz/fga0yrzJYEpkzGNlEG36dZIvCwA=
< Vary: Accept-Encoding
< Server: MochiWeb/1.1 WebMachine/1.9.2 (someone had painted it blue)
< Link: ; rel="up"
*< Foo: Bar*
< ETag: "5hdCr8ayy5j9orqA6nDJfs"
< Content-Type: application/octet-stream
< Content-Length: 12
<

When I tried to create another record with the same metadata as the 
first one through the HTTP interface, I had to prefix the metadata with 
"X-Riak-Meta-" in order for it to be saved on the record. When I did 
this the interface did however not seem to strip the "X-Riak-Meta-" 
prefix from the parameter, and when I query the record through the 
Erlang API, the parameter is stored in the metadata dict as 
"X-Riak-Meta-Foo" instead of just "Foo".


I was expecting the parameter to be stored as "Foo" internally in Riak 
and represented as "X-Riak-Meta-Foo" when getting or setting through the 
HTTP API, but that does not seem to be the case.


Is this an inconsistency or is there something I am doing wrong or have 
missed here?


Best regards,

Christian



___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com