Re: secondary indexes - no multi_backend?
Oh, thanks, Kresten, I'm going to try this out! 30.09.2011 2:33, Kresten Krab Thorup пишет: Artem, Felt enticed, and so I just posted a pull request for this (https://github.com/basho/riak_kv/pull/227) but it looks like they're tagging everything '1.0' as we speak. ... but there will always be another dot release :-) Kresten On Sep 29, 2011, at 11:04 PM, Artem wrote: We are pleased to announce the availability of the second release candidate for Riak 1.0. The RC2 doesn't seem to work with secondary indexes in the multi_backend configuration. Will secondary indexes be supported in 1.0 multi_backend? --- /etc/riak/app.config --- {storage_backend, riak_kv_multi_backend}, {multi_backend_default,<<"leveldb">>}, {multi_backend, [ {<<"leveldb">>, riak_kv_eleveldb_backend, [ {data_root, "/var/lib/riak/leveldb"} ]}, {<<"bitcask">>, riak_kv_bitcask_backend, [ {data_root, "/var/lib/riak/bitcask"} ]} ]}, --- command-line --- curl -v -X PUT -d 'payload' -H 'X-Riak-Index-num_int: 2' http://127.0.0.1:8098/riak/test/doc2 curl -v http://127.0.0.1:8098/buckets/test/index/num_int/1 --- /var/log/riak/console.log --- 2011-09-29 22:56:11.556 [error]<0.514.0> gen_fsm<0.514.0> in state waiting_results terminated with reason: {error,{indexes_not_supported,riak_kv_multi_backend}} 2011-09-29 22:56:11.843 [error]<0.514.0> CRASH REPORT Process<0.514.0> with 0 neighbours crashed with reason: {error,{indexes_not_supported,riak_kv_multi_backend}} 2011-09-29 22:56:12.096 [error]<0.323.0> Supervisor riak_kv_index_fsm_sup had child undefined started with {riak_core_coverage_fsm,start_link,undefined} at<0.514.0> exit with reason {error,{indexes_not_supported,riak_kv_multi_backend}} in context child_terminated 2011-09-29 22:56:12.412 [error]<0.506.0> webmachine error: path="/buckets/test/index/num_int/1" {error,{error,{indexes_not_supported,riak_kv_multi_backend}}} ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: secondary indexes - no multi_backend?
Thanks for the info, David! 30.09.2011 2:36, David Smith wrote: Artem, This is a known issue that didn't make the cutoff for 1.0. We'll be working to address this in the point releases following. D. On Thu, Sep 29, 2011 at 3:04 PM, Artem wrote: We are pleased to announce the availability of the second release candidate for Riak 1.0. The RC2 doesn't seem to work with secondary indexes in the multi_backend configuration. Will secondary indexes be supported in 1.0 multi_backend? --- /etc/riak/app.config --- {storage_backend, riak_kv_multi_backend}, {multi_backend_default,<<"leveldb">>}, {multi_backend, [ {<<"leveldb">>, riak_kv_eleveldb_backend, [ {data_root, "/var/lib/riak/leveldb"} ]}, {<<"bitcask">>, riak_kv_bitcask_backend, [ {data_root, "/var/lib/riak/bitcask"} ]} ]}, --- command-line --- curl -v -X PUT -d 'payload' -H 'X-Riak-Index-num_int: 2' http://127.0.0.1:8098/riak/test/doc2 curl -v http://127.0.0.1:8098/buckets/test/index/num_int/1 --- /var/log/riak/console.log --- 2011-09-29 22:56:11.556 [error]<0.514.0> gen_fsm<0.514.0> in state waiting_results terminated with reason: {error,{indexes_not_supported,riak_kv_multi_backend}} 2011-09-29 22:56:11.843 [error]<0.514.0> CRASH REPORT Process<0.514.0> with 0 neighbours crashed with reason: {error,{indexes_not_supported,riak_kv_multi_backend}} 2011-09-29 22:56:12.096 [error]<0.323.0> Supervisor riak_kv_index_fsm_sup had child undefined started with {riak_core_coverage_fsm,start_link,undefined} at<0.514.0> exit with reason {error,{indexes_not_supported,riak_kv_multi_backend}} in context child_terminated 2011-09-29 22:56:12.412 [error]<0.506.0> webmachine error: path="/buckets/test/index/num_int/1" {error,{error,{indexes_not_supported,riak_kv_multi_backend}}} ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Timeout when storing
IIRC, {error, emfile} indicates that the max # of ports (in the erlang VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args. D. On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler wrote: > Thanks Sean. I added the ulimit -n 10240 to /etc/default/riak, restarted > riak, but that didn't work. > > Fyodor Yarochkin suggested that the bitcask files could be corrupted, but I > wasn't sure which bitcask *.data or *.hint file to delete. Any pointers? > > Here's the /var/log/riak/erlang.log: > > =ERROR REPORT 29-Sep-2011::20:27:42 === > ** State machine <0.369.0> terminating > ** Last event in was {riak_vnode_req_v1, > 94198347718593352149846873983790012612771840, > {fsm,undefined,<0.27704.1>}, > {riak_kv_put_req_v1, > {<<"nodes">>,<<"screen_name-psych_ic-info">>}, > > {r_object,<<"nodes">>,<<"screen_name-psych_ic-info">>, > [{r_content, > {dict,3,16,16,8,80,48, > > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, > {{[],[],[],[],[],[],[],[],[],[], > [[<<"content-type">>,97,112,112,108,105,99,97, > 116,105,111,110,47,106,115,111,110], > > [<<"X-Riak-VTag">>,49,90,120,65,84,100,56,99,48, > 80,86,99,111,122,71,79,108,90,70,97,53,87]], > [],[], > [[<<"X-Riak-Last-Modified">>| > {1317,353201,695471}]], > [],[]}}}, > <<"{DELETED DATA}">>}], > [{<<2,65,205,48>>,{1,63484572401}}], > {dict,1,16,16,8,80,48, > {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, > {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], > [[clean|true]], > []}}}, > undefined}, > 1174401,63484572401, > [{returnbody,true}]}} > ** When State == active > ** Data == {state,94198347718593352149846873983790012612771840, > riak_kv_vnode, > > {state,94198347718593352149846873983790012612771840, > riak_kv_bitcask_backend, > {#Ref<0.0.0.3952>, > > "/var/lib/riak/bitcask/94198347718593352149846873983790012612771840"}, > {dict,0,16,16,8,80,48, > > {[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[],[]}, > > {{[],[],[],[],[],[],[],[],[],[],[],[],[], > [],[],[]}}}, > false}, > undefined,none,6} > ** Reason for termination = > ** {{badmatch,{error,emfile}}, > [{bitcask_fileops,create_file_loop,3}, > {bitcask,put,3}, > {riak_kv_bitcask_backend,put,3}, > {riak_kv_vnode,perform_put,3}, > {riak_kv_vnode,do_put,7}, > {riak_kv_vnode,handle_command,3}, > {riak_core_vnode,vnode_command,3}, > {gen_fsm,handle_msg,7}]} > > > > -Original Message- > From: Sean Cribbs [mailto:s...@basho.com] > Sent: Thu 9/29/2011 3:02 PM > To: Jim Adler > Cc: riak-users@lists.basho.com > Subject: Re: Timeout when storing > > Your environment has too few file handles. Retry starting riak after > setting `ulimit -n 1024` in the shell. Also see our wiki page about this > issue: http://wiki.basho.com/Open-Files-Limit.html You may need to set this > limit specifically for the 'riak' user. > > Cheers, > > -- > Sean Cribbs > Developer Advocate > Basho Technologies, Inc. > http://www.basho.com/ > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > -- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: CorrugatedIron v0.1.3 Released
Jeremiah, really nice job on CorrugatedIron! Some of it is a bit too over-engineered and more complicated than it has to be. For example, you can't just create a RiakClient object and interact with Riak (like you can do with the official Riak client libraries)... To create a RiakClient (in CorrugatedIron) you need to have a RiakCluster object to pass to the RiakClient object constructor. And you can't easily create a RiakCluster object either. You can only create it from a config section (there's no way to explicitly provide the config values directly in the API). Not every app uses .NET app config files and it's also not uncommon for the app config code to cause exceptions because it can be brittle... On Thu, Sep 22, 2011 at 3:33 PM, Jeremiah Peschka wrote: > I just finished up v0.1.3 of CorrugatedIron, everyone's favorite .NET 4.0 > Riak client. There is no new functionality in this release, just a fix from > Matthew Whitfield: > https://github.com/DistributedNonsense/CorrugatedIron/issues/18 > > Thanks Matt for helping us make CorrugatedIron better! > > Binaries have been pushed to NuGet > (http://nuget.org/List/Packages/CorrugatedIron) and source is available on > github > (https://github.com/DistributedNonsense/CorrugatedIron/commit/e9c0af10465176f117bbbf04724087063f9b7fd2) > > Enjoy! > > --- > Jeremiah Peschka - Founder, Brent Ozar PLF, LLC > Microsoft SQL Server MVP > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: CorrugatedIron v0.1.3 Released
Thanks. I can't take all the credit/blame - OJ Reeves has written massive amounts of code, too. We're aware that many people like to complain about our use of config files. When you write a piece of software, you make the choice between shipping software or adding more features. We chose to ship CorrugatedIron and see then what people liked and what they didn't like. People didn't like our use of .NET's crusty old config system that they even created a github issue about it: https://github.com/DistributedNonsense/CorrugatedIron/issues/4 We hear you and will be moving to get this out as soon as we can after we get the Riak 1.0 feature compatibility ironed out. Is our use of .NET configuration files blocking adoption of CorrugatedIron in your organization? --- Jeremiah Peschka - Founder, Brent Ozar PLF, LLC Microsoft SQL Server MVP On Sep 30, 2011, at 1:40 PM, Kyle Quest wrote: > Jeremiah, really nice job on CorrugatedIron! Some of it is a bit too > over-engineered and more complicated than it has to be. For example, > you can't just create a RiakClient object and interact with Riak (like > you can do with the official Riak client libraries)... To create a > RiakClient (in CorrugatedIron) you need to have a RiakCluster object > to pass to the RiakClient object constructor. And you can't easily > create a RiakCluster object either. You can only create it from a > config section (there's no way to explicitly provide the config values > directly in the API). Not every app uses .NET app config files and > it's also not uncommon for the app config code to cause exceptions > because it can be brittle... > > On Thu, Sep 22, 2011 at 3:33 PM, Jeremiah Peschka > wrote: >> I just finished up v0.1.3 of CorrugatedIron, everyone's favorite .NET 4.0 >> Riak client. There is no new functionality in this release, just a fix from >> Matthew Whitfield: >> https://github.com/DistributedNonsense/CorrugatedIron/issues/18 >> >> Thanks Matt for helping us make CorrugatedIron better! >> >> Binaries have been pushed to NuGet >> (http://nuget.org/List/Packages/CorrugatedIron) and source is available on >> github >> (https://github.com/DistributedNonsense/CorrugatedIron/commit/e9c0af10465176f117bbbf04724087063f9b7fd2) >> >> Enjoy! >> >> --- >> Jeremiah Peschka - Founder, Brent Ozar PLF, LLC >> Microsoft SQL Server MVP >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak security
This is a pretty common situation with the NoSQL databases. They have no security and the standard answer is that it's your job to do with firewalls and proxies. This is a good indication that the NoSQL world is still in its infancy. Security features will get there eventually and Accumulo is an example of progress in terms of security capabilities, but it's going to take a while... a long while :-) Now in this case you can do something :-) One option is to use a web proxy that would expose two different ports for GET and PUT requests and then have the appropriate HTTP method filters for each of those ports. However, this doesn't really do much for security because these GET and PUT requests will still be sent to the same Riak node. A better solution is to have separate Riak nodes for reads and writes. You would still need a web proxy to do the HTTP method filtering... to allow only GET HTTP methods for the first node requests and allow only PUT HTTP methods for the second node requests. A much easier solution would involve configuring MochiWeb to do the HTTP method filtering, so you wouldn 't need any proxies; however, I'm not a MochiWeb expert and I don't know if you can define the allowed HTTP methods for MochiWeb in Riak configs. On Sat, Sep 10, 2011 at 12:06 AM, Mark Turner wrote: > On Friday, September 9, 2011 at 11:09 PM, raghwani sohil wrote: > > Hi , > > currently we are using port 8098 for both GET and PUT request for riak . So > I want to achieve security in riak . > > 1> Is there any way to use seperate port for GET and PUT request for > riak ?? > > 2> If there is no way to use seperate port for GET and PUT request then > please can any one explain me how should I achieve security in riak .?? > > I don't know about changing the ports. > As far as the security goes, that responsibility falls to you at the > network/firewall/proxy/router level. > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: CorrugatedIron v0.1.3 Released
Looks like OJ has mostly finished up the work on that, anyway. We'll just need to integrate that back in to develop for the imminent 0.2.0 release. In closing: I suspect that our use of .NET config files is not blocking the use of CorrugatedIron within Microsoft. --- Jeremiah Peschka - Founder, Brent Ozar PLF, LLC Microsoft SQL Server MVP On Sep 30, 2011, at 1:45 PM, Jeremiah Peschka wrote: > Thanks. I can't take all the credit/blame - OJ Reeves has written massive > amounts of code, too. > > We're aware that many people like to complain about our use of config files. > When you write a piece of software, you make the choice between shipping > software or adding more features. We chose to ship CorrugatedIron and see > then what people liked and what they didn't like. > > People didn't like our use of .NET's crusty old config system that they even > created a github issue about it: > https://github.com/DistributedNonsense/CorrugatedIron/issues/4 > > We hear you and will be moving to get this out as soon as we can after we get > the Riak 1.0 feature compatibility ironed out. > > Is our use of .NET configuration files blocking adoption of CorrugatedIron in > your organization? > --- > Jeremiah Peschka - Founder, Brent Ozar PLF, LLC > Microsoft SQL Server MVP > > On Sep 30, 2011, at 1:40 PM, Kyle Quest wrote: > >> Jeremiah, really nice job on CorrugatedIron! Some of it is a bit too >> over-engineered and more complicated than it has to be. For example, >> you can't just create a RiakClient object and interact with Riak (like >> you can do with the official Riak client libraries)... To create a >> RiakClient (in CorrugatedIron) you need to have a RiakCluster object >> to pass to the RiakClient object constructor. And you can't easily >> create a RiakCluster object either. You can only create it from a >> config section (there's no way to explicitly provide the config values >> directly in the API). Not every app uses .NET app config files and >> it's also not uncommon for the app config code to cause exceptions >> because it can be brittle... >> >> On Thu, Sep 22, 2011 at 3:33 PM, Jeremiah Peschka >> wrote: >>> I just finished up v0.1.3 of CorrugatedIron, everyone's favorite .NET 4.0 >>> Riak client. There is no new functionality in this release, just a fix from >>> Matthew Whitfield: >>> https://github.com/DistributedNonsense/CorrugatedIron/issues/18 >>> >>> Thanks Matt for helping us make CorrugatedIron better! >>> >>> Binaries have been pushed to NuGet >>> (http://nuget.org/List/Packages/CorrugatedIron) and source is available on >>> github >>> (https://github.com/DistributedNonsense/CorrugatedIron/commit/e9c0af10465176f117bbbf04724087063f9b7fd2) >>> >>> Enjoy! >>> >>> --- >>> Jeremiah Peschka - Founder, Brent Ozar PLF, LLC >>> Microsoft SQL Server MVP >>> >>> >>> ___ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: CorrugatedIron v0.1.3 Released
I looked at a lot of different Riak libraries in different languages for my work with Riak and CorrugatedIron was one of them. This gave me an opportunity to compare how each of them worked. CorrugatedIron was one of the more complicated ones :-) It was also one of the hardest to make it work because of the app config design (the library generated exception when it tried to load the config data). My suggestion for anybody trying to design a new client library is to mimic the API of the existing official libraries first and then add fancy features :-) This will help the adoption. If it's too complicated (to understand and to setup) then people will simply give up and go somewhere else. On Fri, Sep 30, 2011 at 10:45 AM, Jeremiah Peschka wrote: > Thanks. I can't take all the credit/blame - OJ Reeves has written massive > amounts of code, too. > > We're aware that many people like to complain about our use of config files. > When you write a piece of software, you make the choice between shipping > software or adding more features. We chose to ship CorrugatedIron and see > then what people liked and what they didn't like. > > People didn't like our use of .NET's crusty old config system that they even > created a github issue about it: > https://github.com/DistributedNonsense/CorrugatedIron/issues/4 > > We hear you and will be moving to get this out as soon as we can after we get > the Riak 1.0 feature compatibility ironed out. > > Is our use of .NET configuration files blocking adoption of CorrugatedIron in > your organization? > --- > Jeremiah Peschka - Founder, Brent Ozar PLF, LLC > Microsoft SQL Server MVP > > On Sep 30, 2011, at 1:40 PM, Kyle Quest wrote: > >> Jeremiah, really nice job on CorrugatedIron! Some of it is a bit too >> over-engineered and more complicated than it has to be. For example, >> you can't just create a RiakClient object and interact with Riak (like >> you can do with the official Riak client libraries)... To create a >> RiakClient (in CorrugatedIron) you need to have a RiakCluster object >> to pass to the RiakClient object constructor. And you can't easily >> create a RiakCluster object either. You can only create it from a >> config section (there's no way to explicitly provide the config values >> directly in the API). Not every app uses .NET app config files and >> it's also not uncommon for the app config code to cause exceptions >> because it can be brittle... >> >> On Thu, Sep 22, 2011 at 3:33 PM, Jeremiah Peschka >> wrote: >>> I just finished up v0.1.3 of CorrugatedIron, everyone's favorite .NET 4.0 >>> Riak client. There is no new functionality in this release, just a fix from >>> Matthew Whitfield: >>> https://github.com/DistributedNonsense/CorrugatedIron/issues/18 >>> >>> Thanks Matt for helping us make CorrugatedIron better! >>> >>> Binaries have been pushed to NuGet >>> (http://nuget.org/List/Packages/CorrugatedIron) and source is available on >>> github >>> (https://github.com/DistributedNonsense/CorrugatedIron/commit/e9c0af10465176f117bbbf04724087063f9b7fd2) >>> >>> Enjoy! >>> >>> --- >>> Jeremiah Peschka - Founder, Brent Ozar PLF, LLC >>> Microsoft SQL Server MVP >>> >>> >>> ___ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak security
Kyle, This is something I have been trying to figure the best approach for as well. One possibility, and the one I am trying to build a solution for is to have a proxy in the front that accepts and API key or some other form of credential, validates, then passes through to Riak on whatever port is specified. The goal is that in addition to your IP filtering and access rules, you also have a layer of authorization at what ever level you want through the proxy. Hope that helps, Jeff On Fri, Sep 30, 2011 at 2:00 PM, Kyle Quest wrote: > This is a pretty common situation with the NoSQL databases. They have > no security and the standard answer is that it's your job to do with > firewalls and proxies. This is a good indication that the NoSQL world > is still in its infancy. Security features will get there eventually > and Accumulo is an example of progress in terms of security > capabilities, but it's going to take a while... a long while :-) > > Now in this case you can do something :-) One option is to use a web > proxy that would expose two different ports for GET and PUT requests > and then have the appropriate HTTP method filters for each of those > ports. However, this doesn't really do much for security because these > GET and PUT requests will still be sent to the same Riak node. A > better solution is to have separate Riak nodes for reads and writes. > You would still need a web proxy to do the HTTP method filtering... to > allow only GET HTTP methods for the first node requests and allow only > PUT HTTP methods for the second node requests. A much easier solution > would involve configuring MochiWeb to do the HTTP method filtering, so > you wouldn 't need any proxies; however, I'm not a MochiWeb expert and > I don't know if you can define the allowed HTTP methods for MochiWeb > in Riak configs. > > > > On Sat, Sep 10, 2011 at 12:06 AM, Mark Turner wrote: > > On Friday, September 9, 2011 at 11:09 PM, raghwani sohil wrote: > > > > Hi , > > > > currently we are using port 8098 for both GET and PUT request for riak . > So > > I want to achieve security in riak . > > > > 1> Is there any way to use seperate port for GET and PUT request for > > riak ?? > > > > 2> If there is no way to use seperate port for GET and PUT request then > > please can any one explain me how should I achieve security in riak .?? > > > > I don't know about changing the ports. > > As far as the security goes, that responsibility falls to you at the > > network/firewall/proxy/router level. > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak security
We've been over this several times on riak-users, which suggests to me a blog post might help. I'll try to draft something. On 09/30/2011 11:00 AM, Kyle Quest wrote: This is a pretty common situation with the NoSQL databases. They have no security and the standard answer is that it's your job to do with firewalls and proxies. This is a good indication that the NoSQL world is still in its infancy. Security features will get there eventually and Accumulo is an example of progress in terms of security capabilities, but it's going to take a while... a long while :-) If you have a sensible, flexible, comprehensive proposal for access control in an eventually consistent distributed key-value store, I would love to hear about it. Thus far, every attempt I've encountered rapidly approaches the proverbial "blatant layering violation", or is a subset of {inconsistent, inadequate, overly specific, complex, slow}. Accumulo (IIUC, docs are scarce) tags data with authorizations; restricting access is up to the client. You can do that in Riak. You can front Riak with a layer which allows puts/gets (globally, to buckets, to keys, at various times, etc.) on the basis of HTTP auth, sessions, IP, certificate, etc. You can also implement access control by writing valid state transitions on an object to a statebox and invalidating concurrent modifications that introduce policy conflicts. Authentication provided by cryptographic hashes on transitions, or by a layer above. You could introduce a lock service and use it to enforce certain classes of sequential access, simplifying consistency. Hopefully this suggests why it's not as simple as "adding security to the database". There are a wide variety of security semantics, and many can be layered on top of Riak (or any other datastore) without changing the database itself. Now in this case you can do something :-) One option is to use a web proxy that would expose two different ports for GET and PUT requests and then have the appropriate HTTP method filters for each of those ports. Writes require reads beforehand, but this will do the trick. However, this doesn't really do much for security because these GET and PUT requests will still be sent to the same Riak node. ... A better solution is to have separate Riak nodes for reads and writes. Riak forwards requests to the appropriate vnode for a key. Doing this would have no effect on operational semantics. --Kyle ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Problem with Reduce Phase Sorting (Javascript)
hello all, we are having issues sorting in Reduce. this is from within a java app. we are passing in an anonymous js function, but the values are not being sorted. here is the data structure: [{"score":77.88765180825125,"key":"{\"@class\"[cid:A1AB6387-083F-41D4-8F63-89D3A73EA8C6@sharecare.local]"name1\"}, {"score":167.33742092903964,"key":"{\"@class\"[cid:A1AB6387-083F-41D4-8F63-89D3A73EA8C6@sharecare.local]"name2\" }] so it is an array of maps, with two keys ('score' & 'key').. the 'score' key is an integer that we would like to sort on. the 'key' key is a json object that we have removed from this example. here is the java code, that executes, but does not sort: builder = builder.reduce(JavascriptFunction.anon("" + "function(values){" + "var flattened = values.reduce(function(a,b){" + "return a.concat(b);" + "},[]);" + "return flattened.sort(function(a,b){ " + "return b[\"score\"] -a[\"score\"]" + "});" + "}"),true); MapReduceResponse response = builder.submit(); advTHANKSanced, barron <>___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak security
Having separate nodes for reads and writes provides an opportunity for better isolation and control even when the requests are forwarded to different vnodes... Just like with anything else you can always build something on top... The difference in maturity is determined by what you have to build yourself vs what's already available and integrated into a unified solution. Yes, there are different and unique aspects to how NoSQL databases operate, but it's no excuse for not having any integrated security. It's going to take time for integrated solutions to emerge, but this is exactly what I was saying about the maturity stage the NoSQL databases are in. There won't be a perfect security solution right away. And it's not just one thing... there are many aspects to security and it'll take time to develop and go through various stages of evolution. And by definition, it will make things slower :-) I did not give Accumulo as an example of a NoSQL database that figured out this whole "security" thing. Either way saying, "you customer go take care of our database product security" is not the answer :-) On Fri, Sep 30, 2011 at 12:00 PM, Aphyr wrote: > We've been over this several times on riak-users, which suggests to me a > blog post might help. I'll try to draft something. > > On 09/30/2011 11:00 AM, Kyle Quest wrote: >> >> This is a pretty common situation with the NoSQL databases. They have >> no security and the standard answer is that it's your job to do with >> firewalls and proxies. This is a good indication that the NoSQL world >> is still in its infancy. Security features will get there eventually >> and Accumulo is an example of progress in terms of security >> capabilities, but it's going to take a while... a long while :-) > > If you have a sensible, flexible, comprehensive proposal for access control > in an eventually consistent distributed key-value store, I would love to > hear about it. Thus far, every attempt I've encountered rapidly approaches > the proverbial "blatant layering violation", or is a subset of > {inconsistent, inadequate, overly specific, complex, slow}. > > Accumulo (IIUC, docs are scarce) tags data with authorizations; restricting > access is up to the client. You can do that in Riak. You can front Riak with > a layer which allows puts/gets (globally, to buckets, to keys, at various > times, etc.) on the basis of HTTP auth, sessions, IP, certificate, etc. You > can also implement access control by writing valid state transitions on an > object to a statebox and invalidating concurrent modifications that > introduce policy conflicts. Authentication provided by cryptographic hashes > on transitions, or by a layer above. You could introduce a lock service and > use it to enforce certain classes of sequential access, simplifying > consistency. > > Hopefully this suggests why it's not as simple as "adding security to the > database". There are a wide variety of security semantics, and many can be > layered on top of Riak (or any other datastore) without changing the > database itself. > >> Now in this case you can do something :-) One option is to use a web >> proxy that would expose two different ports for GET and PUT requests >> and then have the appropriate HTTP method filters for each of those >> ports. > > Writes require reads beforehand, but this will do the trick. > >> However, this doesn't really do much for security because these >> GET and PUT requests will still be sent to the same Riak node. > > ... > >> A better solution is to have separate Riak nodes for reads and writes. > > Riak forwards requests to the appropriate vnode for a key. Doing this would > have no effect on operational semantics. > > --Kyle > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak 1.0
Hi All, I am very pleased to announce that Riak 1.0 is tagged and officially ready for downloading. Pre-built installations and source tarballs are available at: http://downloads.basho.com/riak/CURRENT Release notes are here: https://github.com/basho/riak/blob/1.0/RELEASE-NOTES.org There is also a blog post about the release here: http://blog.basho.com/2011/09/30/Riak-1-dot-0-is-Official/ 1.0 has been two years and 15 releases in the making. Thank you all for being a part of this project and wonderful community. We couldn't have done it without you. Sincerely, D. -- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Timeout when storing
Thanks David. I bumped ERL_MAX_PORTS to 16384 in vm.args, restarted riak, but still get the timeout. Jim Sent from my phone. Please forgive the typos. On Sep 30, 2011, at 9:56 AM, "David Smith" wrote: > IIRC, {error, emfile} indicates that the max # of ports (in the erlang > VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args. > > D. > > On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler wrote: >> Thanks Sean. I added the ulimit -n 10240 to /etc/default/riak, restarted >> riak, but that didn't work. >> >> Fyodor Yarochkin suggested that the bitcask files could be corrupted, but I >> wasn't sure which bitcask *.data or *.hint file to delete. Any pointers? >> >> Here's the /var/log/riak/erlang.log: >> >> =ERROR REPORT 29-Sep-2011::20:27:42 === >> ** State machine <0.369.0> terminating >> ** Last event in was {riak_vnode_req_v1, >> 94198347718593352149846873983790012612771840, >> {fsm,undefined,<0.27704.1>}, >> {riak_kv_put_req_v1, >>{<<"nodes">>,<<"screen_name-psych_ic-info">>}, >> >> {r_object,<<"nodes">>,<<"screen_name-psych_ic-info">>, >> [{r_content, >> {dict,3,16,16,8,80,48, >> >> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, >>{{[],[],[],[],[],[],[],[],[],[], >> [[<<"content-type">>,97,112,112,108,105,99,97, >>116,105,111,110,47,106,115,111,110], >> >> [<<"X-Riak-VTag">>,49,90,120,65,84,100,56,99,48, >>80,86,99,111,122,71,79,108,90,70,97,53,87]], >> [],[], >> [[<<"X-Riak-Last-Modified">>| >>{1317,353201,695471}]], >> [],[]}}}, >> <<"{DELETED DATA}">>}], >> [{<<2,65,205,48>>,{1,63484572401}}], >> {dict,1,16,16,8,80,48, >> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, >> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], >>[[clean|true]], >>[]}}}, >> undefined}, >>1174401,63484572401, >>[{returnbody,true}]}} >> ** When State == active >> ** Data == {state,94198347718593352149846873983790012612771840, >> riak_kv_vnode, >> >> {state,94198347718593352149846873983790012612771840, >>riak_kv_bitcask_backend, >>{#Ref<0.0.0.3952>, >> >> "/var/lib/riak/bitcask/94198347718593352149846873983790012612771840"}, >>{dict,0,16,16,8,80,48, >> >> {[],[],[],[],[],[],[],[],[],[],[],[],[], >> [],[],[]}, >> >> {{[],[],[],[],[],[],[],[],[],[],[],[],[], >>[],[],[]}}}, >>false}, >> undefined,none,6} >> ** Reason for termination = >> ** {{badmatch,{error,emfile}}, >> [{bitcask_fileops,create_file_loop,3}, >> {bitcask,put,3}, >> {riak_kv_bitcask_backend,put,3}, >> {riak_kv_vnode,perform_put,3}, >> {riak_kv_vnode,do_put,7}, >> {riak_kv_vnode,handle_command,3}, >> {riak_core_vnode,vnode_command,3}, >> {gen_fsm,handle_msg,7}]} >> >> >> >> -Original Message- >> From: Sean Cribbs [mailto:s...@basho.com] >> Sent: Thu 9/29/2011 3:02 PM >> To: Jim Adler >> Cc: riak-users@lists.basho.com >> Subject: Re: Timeout when storing >> >> Your environment has too few file handles. Retry starting riak after >> setting `ulimit -n 1024` in the shell. Also see our wiki page about this >> issue: http://wiki.basho.com/Open-Files-Limit.html You may need to set this >> limit specifically for the 'riak' user. >> >> Cheers, >> >> -- >> Sean Cribbs >> Developer Advocate >> Basho Technologies, Inc. >> http://www.basho.com/ >> >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > > > -- > Dave Smith > Director, Engineering > Basho Technologies, Inc. > diz...@basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.0
Congratulations! On Fri, Sep 30, 2011 at 4:31 PM, David Smith wrote: > Hi All, > > I am very pleased to announce that Riak 1.0 is tagged and officially > ready for downloading. > > Pre-built installations and source tarballs are available at: > > http://downloads.basho.com/riak/CURRENT > > Release notes are here: > > https://github.com/basho/riak/blob/1.0/RELEASE-NOTES.org > > There is also a blog post about the release here: > > http://blog.basho.com/2011/09/30/Riak-1-dot-0-is-Official/ > > 1.0 has been two years and 15 releases in the making. Thank you all > for being a part of this project and wonderful community. We couldn't > have done it without you. > > Sincerely, > > D. > > -- > Dave Smith > Director, Engineering > Basho Technologies, Inc. > diz...@basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.0
Awesome! Already got it in place on my mac and ubuntu dev boxes. Smooth. -Alexander Sicular @siculars http://siculars.posterous.com On Sep 30, 2011, at 4:31 PM, David Smith wrote: > Hi All, > > I am very pleased to announce that Riak 1.0 is tagged and officially > ready for downloading. > > Pre-built installations and source tarballs are available at: > > http://downloads.basho.com/riak/CURRENT > > Release notes are here: > > https://github.com/basho/riak/blob/1.0/RELEASE-NOTES.org > > There is also a blog post about the release here: > > http://blog.basho.com/2011/09/30/Riak-1-dot-0-is-Official/ > > 1.0 has been two years and 15 releases in the making. Thank you all > for being a part of this project and wonderful community. We couldn't > have done it without you. > > Sincerely, > > D. > > -- > Dave Smith > Director, Engineering > Basho Technologies, Inc. > diz...@basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Java Client 1.0rc1
Hi, In case you want to try all the juicy new Riak 1.0 features from Java, I pushed a release candidate of the RJC to sonatype[1]. It will find its way to Maven Central as and when they sync. Just update your pom to use 1.0rc1. Not _all_ features are fully supported (head_only/return_head on PB fetch/store are not available yet) and some of the ORM code needs to catch-up to. But the bulk of stuff is there. I'll be pushing a 1.0 next week and a point release with the ORM stuff soon after. Cheers Russell [1] https://oss.sonatype.org/content/repositories/releases/com/basho/riak/riak-client/1.0rc1/___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: CorrugatedIron v0.1.3 Released
Hi Kyle, Thanks for the feedback on our .NET client. I'm sorry to hear that you've had issues with it. I'm very keen on hearing more about the issues and details of the level of complexity you're struggling with so that Jeremiah and I can work on making it better. Configuration is one of those things that annoys everyone :) It's incredibly hard to provide power, flexibility and simplicity all at the same time when dealing with configuration. One thing that I think is important to bear in mind is the target of the library is .NET. For many years this kind of configuration has be done using app.config or web.config settings when building .NET applications and that it something that we didn't want to fly in the face of. While I do feel it's important to be able to configure the library through code (in case you're using a REPL like FSI), I felt it was more important to support the idiomatic .NET approach to configuration. Yes that does come with a bit of XML configuration (which I also hate) but is in many ways a necessary evil. We had feedback on github and on HN from people indicating that they had different preferences for the configuration and we sat up and listened. We have since made some changes to the way you can configure CI, which can completely do away with using the XML configuration if you so wish. This does, however, mean you have to configure the client in code. If that's what you decide to do you may have issues as you deploy to different environments. I would still recommend using the XML configuration if you can as managing that configuration for multiple environments is much easier. YMMV :) I'm not sure about the state of other clients, but CI supports talking to a cluster both directly or via a proxy/loadbalancer and also has built-in connection pooling. Having support for all of these things adds complexity. This is the typical features vs simplicity tradeoff. Our feelings on the topic were that we didn't want to limit .NET developers in their options. We also felt that, at least in the short term, that .NET developers might not want to deal with deploying a load balancer in front of their development clusters, and hence built-in support for talking to a cluster was a must. I'm fairly certain that there is at least one other client (for another language) that doesn't have cluster/connection pooling support built-in and the people in that community have asked for on a few occasions. A classic case of damned if you do, damned if you don't :) May I ask what issues in particular you have with getting the configuration to work? Do you feel that the documentation on the topic is lacking? Are the sample applications and code snippets not clear enough? Are the required fields confusing in some way? Error handling is certainly something that we can put a bit more work into. As far as the API or object model is concerned, could you please elaborate on what you found difficult or complicated? Is this another case of poor documentation and bad examples? Do you feel that it doesn't do a good job of being a .NET library and using the features of the languages where appropriate? Jeremiah and I are obviously very keen to remove unnecessary complexity and reduce the friction that people feel when using the lib, so any detail here would be greatly appreciated. Have you had a look at the Riak clients written in strongly-typed languages? How do they compare for complexity? One of the "pains" when dealing with the likes of .NET is dealing with types. The nature of the platform adds constraints that languages like Ruby and dynamic languages like Erlang/Python don't have. This is another thing that's hard to juggle while building a Riak client. Any thoughts that you might have on this topic would be well received. > My suggestion for anybody trying to design a new client library is to mimic the API of > the existing official libraries first and then add fancy features :-) While this sounds sensible and nice in theory, in practice it's actually not that simple (or the right approach) for a few reasons. Languages vary a great deal and not just in syntax. Attempting to "port" an existing client to a language doesn't do the target language justice and can often fly in the face of convention for that language. I would be rather disturbed if I saw a direct port of the Erlang API in the .NET world. Why? Because they're vastly different beasts and the .NET version of the API would be horrible and nothing like any other .NET API. Also, it just wouldn't be possible given the nature of Erlang's dynamic typing vs .NET strong typing. In short: a library should fit the technical ecosystem of the language it's written in. That, in my humble opinion, would be more of a negative influence on the uptake of the library than anything else. I feel that a library should respect the platform it's written for. Happy to hear your thoughts on this too! > If it's too complicated (to understand and to setup) then > people will s
Re: Riak security
On 09/30/2011 01:28 PM, Kyle Quest wrote: Having separate nodes for reads and writes provides an opportunity for better isolation and control even when the requests are forwarded to different vnodes... I humbly suggest this is a bad idea. Varying behavior between nodes a.) is a headache to configure and maintain b.) creates fault-tolerance problems c.) creates unbalanced loads It's at odds with the symmetric layout of a dynamo system. Best case, you'll fail more often. In addition, splitting reads and writes will require you to work harder to handle client IDs. If you aren't careful, you'll lose data. Just like with anything else you can always build something on top... The difference in maturity is determined by what you have to build yourself vs what's already available and integrated into a unified solution. Yes, there are different and unique aspects to how NoSQL databases operate, but it's no excuse for not having any integrated security. It's going to take time for integrated solutions to emerge, but this is exactly what I was saying about the maturity stage the NoSQL databases are in. What was the last datastore you *didn't* have to wrap in your own security layer? I've built... I dunno, twenty or thirty of them for various applications. Because trust is complicated, sufficiently general security systems require almost as much configuration and integration glue as the code you'd write to do it from existing primitives. That said, if you have a proposal for a security model I'd like to see it. There is a dearth of pluggable access control layers for datastores. I suspect there's a good reason for that. Either way saying, "you customer go take care of our database product security" is not the answer :-) It is an entirely reasonable answer; access control is almost totally orthogonal to robust data storage. You're asking Ikea to control who puts things in your cabinets. I'm guessing the reason your posts appear so confused is because you don't have a clear idea about what properties a "database security" system would have. It might be worth writing down your specific requirements, and asking "What percentage of use cases will this satisfy efficiently?" --Kyle Kingsbury ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak security
I'm not here to define a perfect infrastructure for securing NoSQL databases and Riak and go into implementation details... It's not my intention because I simply don't have time to dedicate to this big project and it's impossible to come up with a perfect solution right away. Either way asking customers to be security experts is asking for trouble... And I base this statement on the actual real world experience in security, which I have quite a bit. I'll leave it on this note :-) And let's talk in 10 or 15 years :-) A quick advice to the Riak team... If you want to go beyond startups as your customers you'll need to have a good story around security and an integrated solution people can use. Good luck! On Fri, Sep 30, 2011 at 2:23 PM, Aphyr wrote: > On 09/30/2011 01:28 PM, Kyle Quest wrote: >> >> Having separate nodes for reads and writes provides an opportunity for >> better isolation and control even when the requests are forwarded to >> different vnodes... > > I humbly suggest this is a bad idea. Varying behavior between nodes > > a.) is a headache to configure and maintain > b.) creates fault-tolerance problems > c.) creates unbalanced loads > > It's at odds with the symmetric layout of a dynamo system. Best case, you'll > fail more often. In addition, splitting reads and writes will require you to > work harder to handle client IDs. If you aren't careful, you'll lose data. > >> Just like with anything else you can always build something on top... >> The difference in maturity is determined by what you have to build >> yourself vs what's already available and integrated into a unified >> solution. Yes, there are different and unique aspects to how NoSQL >> databases operate, but it's no excuse for not having any integrated >> security. It's going to take time for integrated solutions to emerge, >> but this is exactly what I was saying about the maturity stage the >> NoSQL databases are in. > > What was the last datastore you *didn't* have to wrap in your own security > layer? I've built... I dunno, twenty or thirty of them for various > applications. Because trust is complicated, sufficiently general security > systems require almost as much configuration and integration glue as the > code you'd write to do it from existing primitives. > > That said, if you have a proposal for a security model I'd like to see it. > There is a dearth of pluggable access control layers for datastores. I > suspect there's a good reason for that. > >> Either way saying, "you customer go take care of our database >> product security" is not the answer :-) > > It is an entirely reasonable answer; access control is almost totally > orthogonal to robust data storage. You're asking Ikea to control who puts > things in your cabinets. > > I'm guessing the reason your posts appear so confused is because you don't > have a clear idea about what properties a "database security" system would > have. It might be worth writing down your specific requirements, and asking > "What percentage of use cases will this satisfy efficiently?" > > --Kyle Kingsbury > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak security
On 09/30/2011 02:50 PM, Kyle Quest wrote: I'm not here to define a perfect infrastructure for securing NoSQL databases and Riak and go into implementation details... It's not my intention because I simply don't have time to dedicate to this big project and it's impossible to come up with a perfect solution right away. Either way asking customers to be security experts is asking for trouble... And I base this statement on the actual real world experience in security, which I have quite a bit. I'll leave it on this note :-) And let's talk in 10 or 15 years :-) Let's skip the ad hominem. I'm gay. You are *not* going to win a bitchiness contest. I want to help people build robust, secure systems. What little you've proposed is not only useless but dangerous. I can't risk someone implementing it. I'm volunteering my time to answer questions on building secure applications in general: with Riak, with MySQL, with HTTP, with Active Directory, whatever. Not an expert on everything, but I can provide pointers to more comprehensive sources. Feel free to contact me off-list if it doesn't pertain to Riak. I'll also try to write an introductory blog post on application security this weekend. If you'd like to contribute, or just want to see some topic covered, let me know. --Kyle Kingsbury ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Custom Extractor Syntax
Hi, Somewhere along the line, it looks like the mechanism for setting a custom extractor changed slightly. The documentation now says to set a property called search_extractor with the fields mod, fun and arg. It used to be a property called rs_extractfun with the fields language, function and module. The old rs_extractfun syntax still seems to work with 1.0, but I'm wondering if I should switch over the new style to stay compatible with future releases. This also brings up an interesting question - how do I unset a bucket property? -- Greg Clipboard ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak 1.0
Congratulations! Can you give me an example of how the 0.14.2(bitcask) upgrade to 1.0(levelDB)? 2011/10/1 Alexander Sicular > Awesome! > > Already got it in place on my mac and ubuntu dev boxes. Smooth. > > -Alexander Sicular > > @siculars > http://siculars.posterous.com > > On Sep 30, 2011, at 4:31 PM, David Smith wrote: > > > Hi All, > > > > I am very pleased to announce that Riak 1.0 is tagged and officially > > ready for downloading. > > > > Pre-built installations and source tarballs are available at: > > > > http://downloads.basho.com/riak/CURRENT > > > > Release notes are here: > > > > https://github.com/basho/riak/blob/1.0/RELEASE-NOTES.org > > > > There is also a blog post about the release here: > > > > http://blog.basho.com/2011/09/30/Riak-1-dot-0-is-Official/ > > > > 1.0 has been two years and 15 releases in the making. Thank you all > > for being a part of this project and wonderful community. We couldn't > > have done it without you. > > > > Sincerely, > > > > D. > > > > -- > > Dave Smith > > Director, Engineering > > Basho Technologies, Inc. > > diz...@basho.com > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- - http://www.54chen.com http://twitter.com/54chen ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com