Hi Ajay, When a Spark SQL statement references a table, that table has to be "registered" first. Usually the way this is done is by reading in a DataFrame, then calling the createOrReplaceTempView (or one of a few other functions) on that data frame, with the argument being the name under which you'd like to register that table. You can then use the table in SQL statements. As far as I know, you cannot directly refer to any external data store without reading it in first.
Jerry On Thu, Jul 11, 2019 at 1:27 PM infa elance <infa.ela...@gmail.com> wrote: > Sorry, i guess i hit the send button too soon.... > > This question is regarding a spark stand-alone cluster. My understanding > is spark is an execution engine and not a storage layer. > Spark processes data in memory but when someone refers to a spark table > created through sparksql(df/rdd) what exactly are they referring to? > > Could it be a Hive table? If yes, is it the same hive store that spark > uses? > Is it a table in memory? If yes, how can an external app access this > in-memory table? if JDBC what driver ? > > On a databricks cluster -- could they be referring spark table created > through sparksql(df/rdd) as hive or deltalake table? > > Spark version with hadoop : spark-2.0.2-bin-hadoop2.7 > > Thanks and appreciate your help!! > Ajay. > > > > On Thu, Jul 11, 2019 at 12:19 PM infa elance <infa.ela...@gmail.com> > wrote: > >> This is stand-alone spark cluster. My understanding is spark is an >> execution engine and not a storage layer. >> Spark processes data in memory but when someone refers to a spark table >> created through sparksql(df/rdd) what exactly are they referring to? >> >> Could it be a Hive table? If yes, is it the same hive store that spark >> uses? >> Is it a table in memory? If yes, how can an external app >> >> Spark version with hadoop : spark-2.0.2-bin-hadoop2.7 >> >> Thanks and appreciate your help!! >> Ajay. >> > -- http://www.google.com/profiles/grapesmoker
