yes, I was looking at intravert last nite. For the kinds of reports my customers ask us to do, joins and subqueries are important. Having tried to do a simple join in PIG, the level of pain is high. I'm a masochist, so I don't mind breaking a simple join into multiple MR tasks, though I do find myself asking "why the hell does it need to be so painful in PIG?" Many of my friends say "what is this crap!" or "this is better than writing sql queries to run reports?"
Plus, using ETL techniques to extract summaries only works for cases where the data is small enough. Once it gets beyond a certain size, it's not practical, which means we're back to crappy reporting languages that make life painful. Lots of big healthcare companies have thousands of MOLAP cubes on dozens of mainframes. The old OLTP -> DW/OLAP creates it's own set of management headaches. being able to report directly on the raw data avoids many of the issues, but that's my bias perspective. On Wed, Mar 12, 2014 at 8:15 AM, DuyHai Doan <doanduy...@gmail.com> wrote: > "I would love to see Cassandra get to the point where users can define > complex queries with subqueries, like, group by and joins" --> Did you have > a look at Intravert ? I think it does union & intersection on server side > for you. Not sure about join though.. > > > On Wed, Mar 12, 2014 at 12:44 PM, Peter Lin <wool...@gmail.com> wrote: > >> >> Hi Ed, >> >> I agree Solr is deeply integrated into DSE. I've looked at Solandra in >> the past and studied the code. >> >> My understanding is DSE uses Cassandra for storage and the user has both >> API available. I do think it can be integrated further to make moderate to >> complex queries easier and probably faster. That's why we built our own >> JPA-like object query API. I would love to see Cassandra get to the point >> where users can define complex queries with subqueries, like, group by and >> joins. Clearly lots of people want these features and even google built >> their own tools to do these types of queries. >> >> I see lots of people trying to improve this with Presto, Impala, drill, >> etc. To me, it's a natural progression as NoSql databases mature. For most >> people, at some point you want to be able to report/analyze the data. Today >> some people use MapReduce to summarize the data and ETL it into a >> relational database or OLAP database for reporting. Even though I don't >> need CAS or atomic batch for what I do in cassandra today, I'm sure in the >> future it will be handy. From my experience in the financial and insurance >> sector, features like CAS and "select for update" are important for the >> kinds of transactions they handle. I'm bias, these kinds of features are >> useful and good addition to cassandra. >> >> These are interesting times in database land! >> >> >> >> >> On Tue, Mar 11, 2014 at 10:57 PM, Edward Capriolo >> <edlinuxg...@gmail.com>wrote: >> >>> Peter, >>> Solr is deeply integrated into DSE. Seemingly this can not efficiently >>> be done client side (CQL/Thrift whatever) but the Solandra approach was to >>> embed Solr in Cassandra. I think that is actually the future client dev, >>> allowing users to embedded custom server side logic into there own API. >>> >>> Things like this take a while. Back in the day no one wanted cassandra >>> to be heavy-weight and rejected ideas like read-before write operations. >>> The common advice was "do them client side". Now in the case of collections >>> sometimes they do read-before-write and it is the "stuff users want". >>> >>> >>> >>> On Tue, Mar 11, 2014 at 10:07 PM, Peter Lin <wool...@gmail.com> wrote: >>> >>>> >>>> I'll give you a concrete example. >>>> >>>> One of the things we often need to do is do a keyword search on >>>> unstructured text. What we did in our tooling is we combined solr with >>>> cassandra, but we put an Object API infront of it. The API is inspired by >>>> JPA, but designed specifically to fit our needs. >>>> >>>> the user can do queries with like %blah% and behind the scenes we >>>> issues a query to solr to find the keys and then query cassandra for the >>>> records. >>>> >>>> With plain Cassandra, the developer has to manually do all of this >>>> stuff and integrate solr. Then they have to know which system to query and >>>> in what order. Our tooling lets the user define the schema in a modeler. >>>> Once the model is done, it compiles the classes, configuration files, data >>>> access objects and unit tests. >>>> >>>> when the application makes a call, our query classes handle the details >>>> behind the scene. I know lots of people would like to see Solr integrated >>>> more deeply into Cassandra and CQL. I hope it happens in the future. If >>>> DataStax accepts my talk, we will be showing our temporal database and >>>> modeler in september. >>>> >>>> >>>> >>>> >>>> On Tue, Mar 11, 2014 at 9:54 PM, Steven A Robenalt < >>>> srobe...@stanford.edu> wrote: >>>> >>>>> I should add that I'm not trying to ignite a flame war. Just trying to >>>>> understand your intentions. >>>>> >>>>> >>>>> On Tue, Mar 11, 2014 at 6:50 PM, Steven A Robenalt < >>>>> srobe...@stanford.edu> wrote: >>>>> >>>>>> Okay, I'm officially lost on this thread. If you plan on forking >>>>>> Cassandra to preserve and continue to enhance the Thrift interface, you >>>>>> would also want to add a bunch of relational features to CQL as part of >>>>>> that same fork? >>>>>> >>>>>> >>>>>> On Tue, Mar 11, 2014 at 6:20 PM, Edward Capriolo < >>>>>> edlinuxg...@gmail.com> wrote: >>>>>> >>>>>>> "one of the things I'd like to see happen is for Cassandra to >>>>>>> support queries with disjunction, exist, subqueries, joins and like. In >>>>>>> theory CQL could support these features in the future. Cassandra would >>>>>>> need >>>>>>> a new query compiler and query planner. I don't see how the current >>>>>>> design >>>>>>> could do these things without a significant redesign/enhancement. In a >>>>>>> past >>>>>>> life, I implemented an inference rule engine, so I've spent over decade >>>>>>> studying and implementing query optimizers. All of these things can be >>>>>>> done, it's just a matter of people finding the time to do it." >>>>>>> >>>>>>> I see what your saying. CQL started as a way to make slice easier >>>>>>> but it is not even a query language, retrofitting these things is going >>>>>>> to >>>>>>> be very hard. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 11, 2014 at 7:45 PM, Peter Lin <wool...@gmail.com>wrote: >>>>>>> >>>>>>>> >>>>>>>> I have no problems maintain my own fork :) or joining others >>>>>>>> forking cassandra. >>>>>>>> >>>>>>>> I'd be happy to work with you or anyone else to add features to >>>>>>>> thrift. That's the great thing about open source. Each person can >>>>>>>> scratch a >>>>>>>> technical itch and do what they love. I see lots of potential for >>>>>>>> Cassandra >>>>>>>> and many of them include improving thrift to make it happen. Some of >>>>>>>> the >>>>>>>> features in theory "could" be done in CQL, but not with the current >>>>>>>> design. >>>>>>>> >>>>>>>> one of the things I'd like to see happen is for Cassandra to >>>>>>>> support queries with disjunction, exist, subqueries, joins and like. In >>>>>>>> theory CQL could support these features in the future. Cassandra would >>>>>>>> need >>>>>>>> a new query compiler and query planner. I don't see how the current >>>>>>>> design >>>>>>>> could do these things without a significant redesign/enhancement. In a >>>>>>>> past >>>>>>>> life, I implemented an inference rule engine, so I've spent over decade >>>>>>>> studying and implementing query optimizers. All of these things can be >>>>>>>> done, it's just a matter of people finding the time to do it. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Mar 11, 2014 at 6:17 PM, Edward Capriolo < >>>>>>>> edlinuxg...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Peter, >>>>>>>>> >>>>>>>>> My advice. Do not bother. I have become very active recently in >>>>>>>>> attempting to add features to thrift. I had 4 open tickets I was >>>>>>>>> actively >>>>>>>>> working on. (I even found two bugs in the Cassandra in the process). >>>>>>>>> >>>>>>>>> People were aware of this and still called this vote. Several >>>>>>>>> commit people have voted in a +1 and my -1 vote is non binding. It is >>>>>>>>> a >>>>>>>>> clear message: The committers are unwilling to accept new thrift >>>>>>>>> features >>>>>>>>> even if said features are contributed by others. >>>>>>>>> >>>>>>>>> Edward >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Mar 11, 2014 at 5:51 PM, Peter Lin <wool...@gmail.com>wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> My bias opinion, just because some member of cassandra develop >>>>>>>>>> want to abandon Thrift, I see benefits of continuing to improve it. >>>>>>>>>> >>>>>>>>>> The great thing about open source is that as long as some people >>>>>>>>>> want to keep working on it and improve it, it can happen. I plan to >>>>>>>>>> do my >>>>>>>>>> best to keep Thrift going, since it gives me fine grain control that >>>>>>>>>> I want >>>>>>>>>> and need. If the ultimate goal of Cassandra is to be "as close to >>>>>>>>>> SQL" as >>>>>>>>>> practical, my bias take is use a NewSQL database that gives you the >>>>>>>>>> full >>>>>>>>>> power of subqueries, like, exists and disjunction. >>>>>>>>>> >>>>>>>>>> When customers ask me which database to choose and they really >>>>>>>>>> want Relational model, I tell them use NewSql. I love that Cassandra >>>>>>>>>> sits >>>>>>>>>> between NoSql and NewSql. There are things I do in Cassandra today >>>>>>>>>> that are >>>>>>>>>> much harder in NewSql or NoSql document databases. NewSql database >>>>>>>>>> can >>>>>>>>>> scale to similar sizes, so the "big" part of big data won't be a >>>>>>>>>> significant advantage forever. Looking at some of the recent NewSql >>>>>>>>>> performance numbers, it's clear the gap is closing. >>>>>>>>>> >>>>>>>>>> peter >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 11, 2014 at 3:59 PM, Tyler Hobbs >>>>>>>>>> <ty...@datastax.com>wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 11, 2014 at 2:41 PM, Shao-Chuan Wang < >>>>>>>>>>> shaochuan.w...@bloomreach.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> So, does anyone know how to do "describing the splits" and >>>>>>>>>>>> "describing the local rings" using native protocol? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> For a ring description, you would do something like "select >>>>>>>>>>> peer, tokens from system.peers". I'm not sure about >>>>>>>>>>> describe_splits(). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Also, cqlsh uses python client, which is talking via thrift >>>>>>>>>>>> protocol too. Does it mean that it will be migrated to native >>>>>>>>>>>> protocol soon >>>>>>>>>>>> as well? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes: https://issues.apache.org/jira/browse/CASSANDRA-6307 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Tyler Hobbs >>>>>>>>>>> DataStax <http://datastax.com/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Steve Robenalt >>>>>> Software Architect >>>>>> HighWire | Stanford University >>>>>> 425 Broadway St, Redwood City, CA 94063 >>>>>> >>>>>> srobe...@stanford.edu >>>>>> http://highwire.stanford.edu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Steve Robenalt >>>>> Software Architect >>>>> HighWire | Stanford University >>>>> 425 Broadway St, Redwood City, CA 94063 >>>>> >>>>> srobe...@stanford.edu >>>>> http://highwire.stanford.edu >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
