I ran:
sqlContext.cacheTable("product")
var df = sqlContext.sql("...complex query...")
df.explain(true)
...and obtained: http://pastebin.com/k9skERsr
...where "[...]" corresponds therein to huge lists of records from the
addressed table (product)
The query is of the following form:
"SELECT d
Can you obtain output from explain(true) on the query after cacheTable()
call ?
Potentially related JIRA:
[SPARK-13657] [SQL] Support parsing very long AND/OR expressions
On Thu, Mar 24, 2016 at 12:55 PM, Mohamed Nadjib MAMI
wrote:
> Here is the stack trace: http://pastebin.com/ueHqiznH
>
> H
Here is the stack trace: http://pastebin.com/ueHqiznH
Here's the code:
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val table = sqlContext.read.parquet("hdfs://...parquet_table")
table.registerTempTable("table")
sqlContext.sql("...complex query...").show() /** works */
Can you pastebin the stack trace ?
If you can show snippet of your code, that would help give us more clue.
Thanks
> On Mar 24, 2016, at 2:43 AM, Mohamed Nadjib MAMI wrote:
>
> Hi all,
> I'm running SQL queries (sqlContext.sql()) on Parquet tables and facing a
> problem with table caching (s
