As Vincent mentioned, Alluxio helps with sharing data across different Spark contexts. This blog post about Spark dataframes and Alluxio discusses that use case <https://alluxio.com/blog/effective-spark-dataframes-with-alluxio>.
Thanks, Gene On Sat, Apr 22, 2017 at 2:14 AM, vincent gromakowski < [email protected]> wrote: > Look at alluxio for sharing across drivers or spark jobserver > > Le 22 avr. 2017 10:24 AM, "Hemanth Gudela" <[email protected]> a > écrit : > >> Thanks for your reply. >> >> >> >> Creating a table is an option, but such approach slows down reads & >> writes for a real-time analytics streaming use case that I’m currently >> working on. >> >> If at all global temporary view could have been accessible across >> sessions/spark contexts, that would have simplified my usecase a lot. >> >> >> >> But yeah, thanks for explaining the behavior of global temporary view, >> now it’s clear J >> >> >> >> -Hemanth >> >> >> >> *From: *Felix Cheung <[email protected]> >> *Date: *Saturday, 22 April 2017 at 11.05 >> *To: *Hemanth Gudela <[email protected]>, "[email protected]" >> <[email protected]> >> *Subject: *Re: Spark SQL - Global Temporary View is not behaving as >> expected >> >> >> >> Cross session is this context is multiple spark sessions from the same >> spark context. Since you are running two shells, you are having different >> spark context. >> >> >> >> Do you have to you a temp view? Could you create a table? >> >> >> >> _____________________________ >> From: Hemanth Gudela <[email protected]> >> Sent: Saturday, April 22, 2017 12:57 AM >> Subject: Spark SQL - Global Temporary View is not behaving as expected >> To: <[email protected]> >> >> >> >> Hi, >> >> >> >> According to documentation >> <http://spark.apache.org/docs/latest/sql-programming-guide.html#global-temporary-view>, >> global temporary views are cross-session accessible. >> >> >> >> But when I try to query a global temporary view from another spark shell >> like thisà >> >> *Instance 1 of spark-shell* >> >> ---------------------------------- >> >> scala> spark.sql("select 1 as col1").createGlobalTempView("gView1") >> >> >> >> *Instance 2 of spark-shell *(while Instance 1 of spark-shell is still >> alive) >> >> --------------------------------- >> >> scala> spark.sql("select * from global_temp.gView1").show() >> >> org.apache.spark.sql.AnalysisException: Table or view not found: >> `global_temp`.`gView1` >> >> 'Project [*] >> >> +- 'UnresolvedRelation `global_temp`.`gView1` >> >> >> >> I am expecting that global temporary view created in shell 1 should be >> accessible in shell 2, but it isn’t! >> >> Please correct me if I missing something here. >> >> >> >> Thanks (in advance), >> >> Hemanth >> >> >> >
