dropping user list Yup I just took a look -- you are right.
What's the reason you'd need a HiveContext? The only method that HiveContext has and SQLContext does not have is refreshTable. Given this is meant for helping code transition, it might be easier to just use SQLContext and change the places that use refreshTable? In order for SparkSession.sqlContext to return an actual HiveContext, we'd need to use reflection to create a HiveContext, which is pretty hacky. On Tue, Jul 19, 2016 at 10:58 AM, Michael Allman <mich...@videoamp.com> wrote: > Sorry Reynold, I want to triple check this with you. I'm looking at the > `SparkSession.sqlContext` field in the latest 2.0 branch, and it appears > that that val is set specifically to an instance of the `SQLContext` class. > A cast to `HiveContext` will fail. Maybe there's a misunderstanding here. > This is what I'm looking at: > > > https://github.com/apache/spark/blob/24ea875198ffcef4a4c3ba28aba128d6d7d9a395/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala#L122 > > Michael > > > > On Jul 19, 2016, at 10:01 AM, Reynold Xin <r...@databricks.com> wrote: > > Yes. But in order to access methods available only in HiveContext a user > cast is required. > > On Tuesday, July 19, 2016, Maciej Bryński <mac...@brynski.pl> wrote: > >> @Reynold Xin, >> How this will work with Hive Support ? >> SparkSession.sqlContext return HiveContext ? >> >> 2016-07-19 0:26 GMT+02:00 Reynold Xin <r...@databricks.com>: >> > Good idea. >> > >> > https://github.com/apache/spark/pull/14252 >> > >> > >> > >> > On Mon, Jul 18, 2016 at 12:16 PM, Michael Armbrust < >> mich...@databricks.com> >> > wrote: >> >> >> >> + dev, reynold >> >> >> >> Yeah, thats a good point. I wonder if SparkSession.sqlContext should >> be >> >> public/deprecated? >> >> >> >> On Mon, Jul 18, 2016 at 8:37 AM, Koert Kuipers <ko...@tresata.com> >> wrote: >> >>> >> >>> in my codebase i would like to gradually transition to SparkSession, >> so >> >>> while i start using SparkSession i also want a SQLContext to be >> available as >> >>> before (but with a deprecated warning when i use it). this should be >> easy >> >>> since SQLContext is now a wrapper for SparkSession. >> >>> >> >>> so basically: >> >>> val session = SparkSession.builder.set(..., ...).getOrCreate() >> >>> val sqlc = new SQLContext(session) >> >>> >> >>> however this doesnt work, the SQLContext constructor i am trying to >> use >> >>> is private. SparkSession.sqlContext is also private. >> >>> >> >>> am i missing something? >> >>> >> >>> a non-gradual switch is not very realistic in any significant >> codebase, >> >>> and i do not want to create SparkSession and SQLContext independendly >> (both >> >>> from same SparkContext) since that can only lead to confusion and >> >>> inconsistent settings. >> >> >> >> >> > >> >> >> >> -- >> Maciek Bryński >> > >
