Thanks Jonathan. The only reason I suggested JSON was because it already has support for lists. Native support for lists in Cassandra would more than satisfy me. Are there any existing proposals or a bug I can follow? I'm not familiar with the Cassandra codebase, so I'm not entirely sure how helpful I can be, but I'd certainly be interested in taking a look to see what's required.
-Ben On Thu, Mar 29, 2012 at 12:19 PM, Brian O'Neill <b...@alumni.brown.edu>wrote: > Jonathan, > > I was actually going to take this up with Nate McCall a few weeks back. I > think it might make sense to get the client development community together > (Netflix w/ Astyanax, Hector, Pycassa, Virgil, etc.) > > I agree whole-heartedly that it shouldn't go into the database for all the > reasons you point out. > > If we can all decide on some standards for data storage (e.g. composite > types), indexing strategies, etc. We can provide higher-level functions > through the client libraries and also provide interoperability between > them. (without bloating Cassandra) > > CCing Nate. Nate, thoughts? > I wouldn't mind coordinating/facilitating the conversation. If we know > who should be involved. > > -brian > > ---- > Brian O'Neill > Lead Architect, Software Development > Health Market Science | 2700 Horizon Drive | King of Prussia, PA 19406 > p: 215.588.6024blog: http://weblogs.java.net/blog/boneill42/ > blog: http://brianoneill.blogspot.com/ > > > > > > > > On 3/29/12 3:06 PM, "Ben McCann" <b...@benmccann.com> wrote: > > >Jonathan, I asked Brian about his REST > >API< > https://groups.google.com/forum/?fromgroups#!topic/virgil-users/oncBas > >9C8Us>and > >he said he does not take the json objects and split them because the > >client libraries do not agree on implementations. This was exactly my > >concern as well with this solution. I would be perfectly happy to do it > >this way instead of using JSON if it were standardized. The reason I > >suggested JSON is that it is standardized. As far as I can tell, > >Cassandra > >doesn't support maps and lists in a standardized way today, which is the > >root of my problem. > > > >-Ben > > > > > >On Thu, Mar 29, 2012 at 11:30 AM, Drew Kutcharian <d...@venarc.com> > wrote: > > > >> Yes, I meant the "row header index". What I have done is that I'm > >>storing > >> an object (i.e. UserProfile) where you read or write it as a whole (a > >>user > >> updates their user details in a single page in the UI). So I serialize > >>that > >> object into a binary JSON using SMILE format. I then compress it using > >> Snappy on the client side. So as far as Cassandra cares it's storing a > >> byte[]. > >> > >> Now on the client side, I'm using cassandra-cli with a custom type that > >> knows how to turn a byte[] into a JSON text and back. The only issue was > >> CASSANDRA-4081 where "assume" doesn't work with custom types. If > >> CASSANDRA-4081 gets fixed, I'll get the best of both worlds. > >> > >> Also advantages of this vs. the thrift based Super Column families are: > >> > >> 1. Saving extra CPU usage on the Cassandra nodes. Since > >> serialize/deserialize and compression/decompression happens on the > >>client > >> nodes where there is plenty idle CPU time > >> > >> 2. Saving network bandwidth since I'm sending over a compressed byte[] > >> > >> > >> -- Drew > >> > >> > >> > >> On Mar 29, 2012, at 11:16 AM, Jonathan Ellis wrote: > >> > >> > On Thu, Mar 29, 2012 at 1:11 PM, Drew Kutcharian <d...@venarc.com> > >> wrote: > >> >>> I think this is a much better approach because that gives you the > >> >>> ability to update or retrieve just parts of objects efficiently, > >> >>> rather than making column values just blobs with a bunch of special > >> >>> case logic to introspect them. Which feels like a big step > >>backwards > >> >>> to me. > >> >> > >> >> Unless your access pattern involves reading/writing the whole > >>document > >> each time. In that case you're better off serializing the whole document > >> and storing it in a column as a byte[] without incurring the overhead of > >> column indexes. Right? > >> > > >> > Hmm, not sure what you're thinking of there. > >> > > >> > If you mean the "index" that's part of the row header for random > >> > access within a row, then no, serializing to byte[] doesn't save you > >> > anything. > >> > > >> > If you mean secondary indexes, don't declare any if you don't want > >>any. > >> :) > >> > > >> > Just telling C* to store a byte[] *will* be slightly lighter-weight > >> > than giving it named columns, but we're talking negligible compared to > >> > the overhead of actually moving the data on or off disk in the first > >> > place. Not even close to being worth giving up being able to deal > >> > with your data from standard tools like cqlsh, IMO. > >> > > >> > -- > >> > Jonathan Ellis > >> > Project Chair, Apache Cassandra > >> > co-founder of DataStax, the source for professional Cassandra support > >> > http://www.datastax.com > >> > >> > > >
