Basically, I meant sibling reconciliation. Best regards, Andrey Martyanov
On Fri, Mar 2, 2012 at 8:01 PM, Shuhao Wu <ad...@thekks.net> wrote: > Also, what do you mean by conflict resolution? Synchronization or > conflicts between references such as deleting references to a document when > that document is deleted? > > Shuhao > > > > On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <ad...@thekks.net> wrote: > >> Actually it's okay for no pull request for now.. I'm in the process of >> rethinking the whole framework as well as I'm having some of the issues >> that Andrey is describing as well. >> >> Shuhao >> >> >> >> On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <ja...@basho.com> wrote: >> >>> Andrey, >>> >>> First off, you are indeed sending two emails to the Riak Users list each >>> time you email, not just one, so I think gmail is doing the right thing the >>> first time. >>> >>> Second, I'm sure Shuhao would love some pull requests to help with >>> Riakkit since you seem to have a lot of ideas to make it better. >>> >>> -Jared >>> >>> >>> On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote: >>> >>> Also, I think we can add RoR-like migrations on top of that with >>> automatic schema versioning and all the stuff. >>> >>> Best regards, >>> Andrey Martyanov >>> >>> On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov >>> <reald...@gmail.com>wrote: >>> >>>> I'm sorry for the repost, but my gmail client by default sends an email >>>> to a specific subscriber, not to riak-users. >>>> >>>> >>>> Hi Shuhao, >>>> >>>> I don't have enough time to write my own implementation and to see how >>>> it works in practice. But I share my thoughts about that. >>>> It's possible to save the current API, it's common now and used in >>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit >>>> different. The simple workflow: >>>> 1. Define the models with specific fields and properties; >>>> 2. Generate the JSON Schema [1] based on these fields; >>>> 3. Create an instances of our models; >>>> 4. Validate against the schema; >>>> 5. Generate the optimal sequence of actions we need to do to store the >>>> data, based on the meta information of our models; >>>> 6. Store the data, resolve the conflicts if occur. >>>> >>>> Also it should be possible to deserialize the JSON objects into python >>>> objects. >>>> >>>> [1] - http://json-schema.org/ >>>> >>>> Best regards, >>>> Andrey Martyanov >>>> >>>> On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <reald...@gmail.com >>>> > wrote: >>>> >>>>> Hi Shuhao, >>>>> >>>>> I don't have enough time to write my own implementation and to see how >>>>> it works in practice. But I share my thoughts about that. >>>>> It's possible to save the current API, it's common now and used in >>>>> SQLAlchemy and Django ORM. But underlying implementation I see a bit >>>>> different. The simple workflow: >>>>> 1. Define the models with specific fields and properties; >>>>> 2. Generate the JSON Schema [1] based on these fields; >>>>> 3. Create an instances of our models; >>>>> 4. Validate against the schema; >>>>> 5. Generate the optimal sequence of actions we need to do to store the >>>>> data, based on the meta information of our models; >>>>> 6. Store the data, resolve the conflicts if occur. >>>>> >>>>> Also it should be possible to deserialize the JSON objects into python >>>>> objects. >>>>> >>>>> [1] - http://json-schema.org/ >>>>> >>>>> Best regards, >>>>> Andrey Martyanov >>>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <ad...@thekks.net> wrote: >>>>>> >>>>>>> Yeah. You're looking at something like dictshield. As riakkit is >>>>>>> more a RAD tool in its current state. You bring in an excellent point, >>>>>>> nonetheless, which gave me some intuition and ideas about the next step. >>>>>>> >>>>>>> Cheers, >>>>>>> Shuhao >>>>>>> Sent from my phone. >>>>>>> On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <reald...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Riakkit in its current implementation has a lot of overhead. But >>>>>>>> what I get for this price? Almost nothing. Only field validation make >>>>>>>> sense. Simple key-value mapping is about 50-70 lines of code, not a big >>>>>>>> deal, even for small project. Riak used mostly in web-based projects. >>>>>>>> Almost all modern projects use JSON. The simple workflow is get JSON >>>>>>>> from >>>>>>>> the client, validate, store in the database, query the database, send >>>>>>>> JSON >>>>>>>> to the client. If our mapper doesn't understand the JSON and cannot >>>>>>>> operate >>>>>>>> on it then it's not useful. We can use jsonpickle with a lot more >>>>>>>> profit. >>>>>>>> Just jsonpickle.encode the object and store it in Riak. I'm sorry, >>>>>>>> this is >>>>>>>> my position! >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Andrey Martyanov >>>>>>>> >>>>>>>> On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov < >>>>>>>> reald...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I'm just saying that to efficiently store and get the data user >>>>>>>>> should have ability to customize the relationship. Riak has several >>>>>>>>> options >>>>>>>>> to get related data: secondary indexes, map-reduce etc. I think that >>>>>>>>> the >>>>>>>>> most important task of mapper is to validate the data and to track >>>>>>>>> these >>>>>>>>> dependencies. When you have enough meta information about objects >>>>>>>>> storing >>>>>>>>> these objects is not a big problem. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <ad...@thekks.net>wrote: >>>>>>>>> >>>>>>>>>> The thing about riak-python-client is that it's already pretty >>>>>>>>>> high level, but not high level enough to be that simple. >>>>>>>>>> >>>>>>>>>> From what I understand, tracking relationship between models will >>>>>>>>>> still involve a certain level of magic. Right now the magic is that >>>>>>>>>> when >>>>>>>>>> you save any document, all the related document will be saved into >>>>>>>>>> the >>>>>>>>>> database as well. You're saying that just tracking those relationship >>>>>>>>>> change is good enough and saving will be up to the user? >>>>>>>>>> >>>>>>>>>> Shuhao >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov < >>>>>>>>>> reald...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> I'm not sure about forking the riak-python-client. The the whole >>>>>>>>>>> idea of separation the low-level client library from the high-level >>>>>>>>>>> mapper >>>>>>>>>>> and the like is quite good. Some sort of SQLAlchemy does. >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> Andrey Martyanov >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <ad...@thekks.net>wrote: >>>>>>>>>>> >>>>>>>>>>>> Another thing I was considering is forking riak-python-client. >>>>>>>>>>>> Some of the stuff in there could use some fixing as well, and >>>>>>>>>>>> maybe just >>>>>>>>>>>> take out the core transports and such and plug that right into >>>>>>>>>>>> riakkit and >>>>>>>>>>>> skip the notion of RiakObject all together as those API is also >>>>>>>>>>>> somewhat >>>>>>>>>>>> weird. (such as add_link and remove_link) >>>>>>>>>>>> >>>>>>>>>>>> Shuhao >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <ad...@thekks.net>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi! >>>>>>>>>>>>> >>>>>>>>>>>>> First off, the code is *LGPL* licensed, not GPL. So you're >>>>>>>>>>>>> free to put this into any code you want, it's just that when you >>>>>>>>>>>>> want to >>>>>>>>>>>>> modify riakkit, you have to release your modified version. >>>>>>>>>>>>> >>>>>>>>>>>>> I just went over your idea and it would be a valid solution to >>>>>>>>>>>>> some of the problems I'm having right now with my own projects >>>>>>>>>>>>> with >>>>>>>>>>>>> riakkit. When I initially started writing riakkit, I needed >>>>>>>>>>>>> something to do >>>>>>>>>>>>> RAD with, which would make sense to have an all-in-one, very >>>>>>>>>>>>> simple to use >>>>>>>>>>>>> framework, rather than working with serialization and what not. >>>>>>>>>>>>> However, >>>>>>>>>>>>> now I'm beginning to see problems in this approach when it comes >>>>>>>>>>>>> to >>>>>>>>>>>>> scalability as I'm slowing moving out of the RAD phase. I haven't >>>>>>>>>>>>> done >>>>>>>>>>>>> anything as I have not encountered any major issues yet (though I >>>>>>>>>>>>> probably >>>>>>>>>>>>> will). >>>>>>>>>>>>> >>>>>>>>>>>>> So here's what I'm thinking for the next branch ( I still have >>>>>>>>>>>>> to maintain the master branch due to some code i've written and >>>>>>>>>>>>> need ): I'm >>>>>>>>>>>>> going to break riakkit into 2 pieces. The first piece is the >>>>>>>>>>>>> underlying >>>>>>>>>>>>> piece, which will do what you said, serialize/deserialize objects, >>>>>>>>>>>>> validation, track relationships and what not, however, it doesn't >>>>>>>>>>>>> load it >>>>>>>>>>>>> those references up nor does it try to accomplish everything all >>>>>>>>>>>>> in one >>>>>>>>>>>>> shot, and a RAD layer that's compatible with the current API that >>>>>>>>>>>>> will >>>>>>>>>>>>> allow RAD to work with it, but still reserve the flexibility of >>>>>>>>>>>>> having that >>>>>>>>>>>>> underlying API that's relatively easy (as oppose to dead simple) >>>>>>>>>>>>> to work >>>>>>>>>>>>> with and not as messy as what's going on right now. >>>>>>>>>>>>> >>>>>>>>>>>>> I do agree that the code is messy.. Most of the time it's me >>>>>>>>>>>>> fixing bugs I didn't anticipate as I go along and hacking >>>>>>>>>>>>> everything into >>>>>>>>>>>>> place.. >>>>>>>>>>>>> >>>>>>>>>>>>> One thing I'm not too clear about is what you mean by "Track >>>>>>>>>>>>> dependencies between models". I'm not sure how you think I should >>>>>>>>>>>>> go about >>>>>>>>>>>>> approaching this (which right now is a pretty big disaster.. >>>>>>>>>>>>> though >>>>>>>>>>>>> relatively stable) >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> >>>>>>>>>>>>> Shuhao >>>>>>>>>>>>> >>>>>>>>>>>>> P.S. I don't want to go into the whole PEP8 discussion and 4 >>>>>>>>>>>>> spaces vs whatever.. I just like the 2 spaces and the style guide >>>>>>>>>>>>> that >>>>>>>>>>>>> Google uses. Is that really a big deal if you're using it as a >>>>>>>>>>>>> library? >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov < >>>>>>>>>>>>> reald...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Two things force me to avoid using Riakkit: the code is GPL >>>>>>>>>>>>>> licensed and non pep8 compliant. So, I cannot fork it and make >>>>>>>>>>>>>> changes. The >>>>>>>>>>>>>> code needs clean up, for example, json module is imported in >>>>>>>>>>>>>> document.py, >>>>>>>>>>>>>> but not used. Riakkit has no conflict resolution logic, it's >>>>>>>>>>>>>> basically >>>>>>>>>>>>>> delegate the work to underlying python client. If I'd like to >>>>>>>>>>>>>> implement my >>>>>>>>>>>>>> save method I need to copy and rewrite more than 150 lines of >>>>>>>>>>>>>> code mess. >>>>>>>>>>>>>> What I think the mapper should do: >>>>>>>>>>>>>> 1. Serialize the objects to valid json; >>>>>>>>>>>>>> 2. Deserialize the data back to class instances; >>>>>>>>>>>>>> 3. Validate the fields data; >>>>>>>>>>>>>> 4. Track dependencies between models; >>>>>>>>>>>>>> >>>>>>>>>>>>>> Simple example using Riakkit-like syntax; >>>>>>>>>>>>>> >>>>>>>>>>>>>> class Post(Document): >>>>>>>>>>>>>> >>>>>>>>>>>>>> bucket_name = 'posts' >>>>>>>>>>>>>> >>>>>>>>>>>>>> title = types.StringProperty(required=True) >>>>>>>>>>>>>> content = types.StringProperty(required=True) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> class Comment(Document): >>>>>>>>>>>>>> >>>>>>>>>>>>>> bucket_name = 'comments' >>>>>>>>>>>>>> >>>>>>>>>>>>>> text = types.StringProperty(required=True) >>>>>>>>>>>>>> post = types.IndexReferenceProperty(reference_class=Post, >>>>>>>>>>>>>> collection_name='comments') >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> post = Post('title', 'content') >>>>>>>>>>>>>> >>> comment1 = ('comment1', post=post) >>>>>>>>>>>>>> >>> comment2 = ('comment2', post=post) >>>>>>>>>>>>>> >>> post.is_valid() >>>>>>>>>>>>>> True >>>>>>>>>>>>>> >> post.serialize() >>>>>>>>>>>>>> { >>>>>>>>>>>>>> "key": "post_key", >>>>>>>>>>>>>> "title": "title", >>>>>>>>>>>>>> "content": "content", >>>>>>>>>>>>>> "comments": [ >>>>>>>>>>>>>> { >>>>>>>>>>>>>> "key": "comment1_key", >>>>>>>>>>>>>> "text": "comment1" >>>>>>>>>>>>>> }, >>>>>>>>>>>>>> { >>>>>>>>>>>>>> "key": "comment2_key", >>>>>>>>>>>>>> "text": "comment2" >>>>>>>>>>>>>> }, >>>>>>>>>>>>>> ] >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>> post.store() >>>>>>>>>>>>>> True >>>>>>>>>>>>>> >>> post = Post.get_with_key('post_key') >>>>>>>>>>>>>> >>> post.comments >>>>>>>>>>>>>> ['Post comment1', 'Post comment2'] >>>>>>>>>>>>>> >>> post_json_data = get_json_data_from_request() >>>>>>>>>>>>>> >>> post = Post.deserialize(post_json_data) >>>>>>>>>>>>>> >>> post.title >>>>>>>>>>>>>> 'title' >>>>>>>>>>>>>> >>> post.comments >>>>>>>>>>>>>> ['Post comment1', 'Post comment2'] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Something like that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>> Andrey Martyanov >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey >>>>>>>>>>>>>> <ca...@basho.com>wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Keep up the great work Shuhao! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mar 1, 2012, at 7:36 PM, Shuhao Wu <ad...@thekks.net> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hey guys, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I started writing riakkit approximately 3 months ago and >>>>>>>>>>>>>>> announced it here. In case you missed that, Riakkit is >>>>>>>>>>>>>>> basically an object >>>>>>>>>>>>>>> mapper for python and riak, which seemed to be lacking in at >>>>>>>>>>>>>>> the time. Now >>>>>>>>>>>>>>> that 3 months has passed and I have developed this project into >>>>>>>>>>>>>>> much more >>>>>>>>>>>>>>> than what I had envisioned to begin with. It's currently also >>>>>>>>>>>>>>> being used >>>>>>>>>>>>>>> and tested in a project of mine (web application) as well and >>>>>>>>>>>>>>> functioning >>>>>>>>>>>>>>> quite nicely. I would say that currently this is the only >>>>>>>>>>>>>>> object mapper >>>>>>>>>>>>>>> that can compete with their counterparts in other languages for >>>>>>>>>>>>>>> Riak in >>>>>>>>>>>>>>> Python, although a lot of work still needs to be done before it >>>>>>>>>>>>>>> could fully >>>>>>>>>>>>>>> compete. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So today I felt that riakkit is "stable" and "clean" enough >>>>>>>>>>>>>>> to deserve the first beta release. So I whipped up a website >>>>>>>>>>>>>>> and it's >>>>>>>>>>>>>>> released to pypi. Here's (again), a demo (the full tutorial can >>>>>>>>>>>>>>> be found on >>>>>>>>>>>>>>> the links below) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> class User(Document): >>>>>>>>>>>>>>> client = riak.RiakClient() >>>>>>>>>>>>>>> bucket_name = "users" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> name = StringProperty(required=True) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>> user = User(name="foo") >>>>>>>>>>>>>>> >>> user.save() >>>>>>>>>>>>>>> >>> print user.name >>>>>>>>>>>>>>> foo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If anyone is interested, you can find the website here: >>>>>>>>>>>>>>> http://ultimatebuster.github.com/riakkit and the code at >>>>>>>>>>>>>>> http://github.com/ultimatebuster/riakkit >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Shuhao >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> riak-users mailing list >>>>>>>>>>>>>> riak-users@lists.basho.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> riak-users mailing list >>>>>>>> riak-users@lists.basho.com >>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>> >>>>>>>> >>>>>> >>>>> >>>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
