I don't plan to abandon HiveQL compatibility, but I'd like to see us move towards something with more SQL compliance (perhaps just newer versions of the HiveQL parser). Exactly which parser will do that for us is under investigation.
On Wed, Dec 9, 2015 at 11:02 AM, Xiao Li <gatorsm...@gmail.com> wrote: > Hi, Michael, > > Does that mean SqlContext will be built on HiveQL in the near future? > > Thanks, > > Xiao Li > > > 2015-12-09 10:36 GMT-08:00 Michael Armbrust <mich...@databricks.com>: > >> I think that it is generally good to have parity when the functionality >> is useful. However, in some cases various features are there just to >> maintain compatibility with other system. For example CACHE TABLE is eager >> because Shark's cache table was. df.cache() is lazy because Spark's cache >> is. Does that mean that we need to add some eager caching mechanism to >> dataframes to have parity? Probably not, users can just call .count() if >> they want to force materialization. >> >> Regarding the differences between HiveQL and the SQLParser, I think we >> should get rid of the SQL parser. Its kind of a hack that I built just so >> that there was some SQL story for people who didn't compile with Hive. >> Moving forward, I'd like to see the distinction between the HiveContext and >> SQLContext removed and we can standardize on a single parser. For this >> reason I'd be opposed to spending a lot of dev/reviewer time on adding >> features there. >> >> On Wed, Dec 9, 2015 at 8:34 AM, Cristian O < >> cristian.b.op...@googlemail.com> wrote: >> >>> Hi, >>> >>> I was wondering what the "official" view is on feature parity between >>> SQL and DF apis. Docs are pretty sparse on the SQL front, and it seems that >>> some features are only supported at various times in only one of Spark SQL >>> dialect, HiveQL dialect and DF API. DF.cube(), DISTRIBUTE BY, CACHE LAZY >>> are some examples >>> >>> Is there an explicit goal of having consistent support for all features >>> in both DF and SQL ? >>> >>> Thanks, >>> Cristian >>> >> >> >
