Instead of running from sqlline, I created a sample program to run the
program.I added the generated classpath folder into by eclipse project's
classpath.But I got a NullPointerExceptionCaused by:
java.lang.NullPointerException at
org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:74) at
CalciteProgram151003.bind(Unknown Source) at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:327)
I debugged it and found that in org.apache.calcite.jdbc.createDataContext , for
spark it creates SlimDataContext which ignores the parameterValues passed to
createDataContext method.
Due to this in the generated code org.apache.calcite.rel.RelNode v0stashed =
(org.apache.calcite.rel.RelNode) root.get("v0stashed");
the v0stashed would be null.
So I though let me modify createDataContext to return DataContextImpl(this,
parameterValues), even for spark.
Doing so did make the code run, but looking at the generated code and my
eclipse's console, nothing is actually getting executed on spark. I can
contribute to the spark-calcite project to make it work.
I want to use spark as the engine for processing anything that I am not able to
push down to my custom adapter so that in case of large data things don't go
out of memory.
Thanks & Regards,Dharmendra
From: Dharmendra Chouhan <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Tuesday, 29 November 2016 7:43 PM
Subject: Using Spark engine for processing SQL syntax that cannot be pushed to
the source system
Hi,
In the calcite connection property document , I saw a property SPARK. Its
description is - "Specifies whether Spark should be used as the engine for
processing that cannot be pushed to the source system"So I thought of testing
it over the CSV adapter. i.e anything which is not implemented by csv adapter ,
use spark for running them. For doing this I followed below steps
1. In pom.xml of csv adapter I added dependency of calcite-spark2. Deleted the
target\dependencies folder3. Started sqline.bat4. Used connection string as
!connect jdbc:calcite:model=target/test-classes/model.json;spark=true admin
admin5. Ran sql query select * from emps;
Is my understanding of this property correct ? Am I setting things up correctly
?Why am I getting the below exception and how to resolve it ?
java.sql.SQLException: Error while executing SQL "select * from emps": null
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at
org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:209)
at sqlline.Commands.execute(Commands.java:822) at
sqlline.Commands.sql(Commands.java:732) at
sqlline.SqlLine.dispatch(SqlLine.java:813) at
sqlline.SqlLine.begin(SqlLine.java:686) at
sqlline.SqlLine.start(SqlLine.java:398) at
sqlline.SqlLine.main(SqlLine.java:291)Caused by: java.lang.NullPointerException
at
org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:74)
at Baz.bind(Unknown Source) at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:327)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:295)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:599)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
