The command "list jar" doesn't seem accepted in beeline with Spark's ThriftServer in both Spark 1.3.1 and Spark1.4.
0: jdbc:hive2://localhost:10000> list jar; Error: org.apache.spark.sql.AnalysisException: cannot recognize input near 'list' 'jar' '<EOF>'; line 1 pos 0 (state=,code=0) Thanks On Tue, Jul 14, 2015 at 8:46 PM, prosp4300 <prosp4...@163.com> wrote: > > > > What's the result of "list jar" in both 1.3.1 and 1.4.0, please check if > there is any difference > > > > At 2015-07-15 08:10:44, "ogoh" <oke...@gmail.com> wrote: >>Hello, >>I am using SparkSQL along with ThriftServer so that we can access using >> Hive >>queries. >>With Spark 1.3.1, I can register UDF function. But, Spark 1.4.0 doesn't >> work >>for that. The jar of the udf is same. >>Below is logs: >>I appreciate any advice. >> >> >>== With Spark 1.4 >>Beeline version 1.4.0 by Apache Hive >> >>0: jdbc:hive2://localhost:10000> add jar >>hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar; >> >>0: jdbc:hive2://localhost:10000> create temporary function parse_trace as >>'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation'; >> >>15/07/14 23:49:43 DEBUG transport.TSaslTransport: writing data length: 206 >> >>15/07/14 23:49:43 DEBUG transport.TSaslTransport: CLIENT: reading data >>length: 201 >> >>Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: >>Execution Error, return code 1 from >>org.apache.hadoop.hive.ql.exec.FunctionTask (state=,code=0) >> >> >>== With Spark 1.3.1: >> >>Beeline version 1.3.1 by Apache Hive >> >>0: jdbc:hive2://localhost:10001> add jar >>hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar; >> >>+---------+ >> >>| Result | >> >>+---------+ >> >>+---------+ >> >>No rows selected (1.313 seconds) >> >>0: jdbc:hive2://localhost:10001> create temporary function parse_trace as >>'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation'; >> >>+---------+ >> >>| result | >> >>+---------+ >> >>+---------+ >> >>No rows selected (0.999 seconds) >> >> >>=== The logs of ThriftServer of Spark 1.4.0 >> >>15/07/14 23:49:43 INFO SparkExecuteStatementOperation: Running query >> 'create >>temporary function parse_trace as >>'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'' >> >>15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary >>function parse_trace as >>'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation' >> >>15/07/14 23:49:43 INFO ParseDriver: Parse Completed >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.run >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=TimeToSubmit >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO Driver: Concurrency mode is disabled, not creating a >>lock manager >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=compile >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=parse >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary >>function parse_trace as >>'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation' >> >>15/07/14 23:49:43 INFO ParseDriver: Parse Completed >> >>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=parse >>start=1436917783106 end=1436917783106 duration=0 >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=semanticAnalyze >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO HiveMetaStore: 2: get_database: default >> >>15/07/14 23:49:43 INFO audit: ugi=anonymous ip=unknown-ip-addr >>cmd=get_database: default >> >>15/07/14 23:49:43 INFO HiveMetaStore: 2: Opening raw store with >>implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >> >>15/07/14 23:49:43 INFO ObjectStore: ObjectStore, initialize called >> >>15/07/14 23:49:43 INFO MetaStoreDirectSql: MySQL check failed, assuming we >>are not on mysql: Lexical error at line 1, column 5. Encountered: "@" >> (64), >>after : "". >> >>15/07/14 23:49:43 INFO Query: Reading in results for query >>"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is >>closing >> >>15/07/14 23:49:43 INFO ObjectStore: Initialized ObjectStore >> >>15/07/14 23:49:43 INFO FunctionSemanticAnalyzer: analyze done >> >>15/07/14 23:49:43 INFO Driver: Semantic Analysis Completed >> >>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=semanticAnalyze >>start=1436917783106 end=1436917783114 duration=8 >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO Driver: Returning Hive schema: >>Schema(fieldSchemas:null, properties:null) >> >>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=compile >>start=1436917783106 end=1436917783114 duration=8 >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.execute >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO Driver: Starting command: create temporary function >>parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation' >> >>15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=TimeToSubmit >>start=1436917783105 end=1436917783115 duration=10 >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=runTasks >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=task.FUNCTION.Stage-0 >>from=org.apache.hadoop.hive.ql.Driver> >> >>15/07/14 23:49:43 ERROR Task: FAILED: Class >>com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation not found >> >>15/07/14 23:49:43 INFO FunctionTask: create function: >>java.lang.ClassNotFoundException: >>com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:372) >> >> at java.net.URLClassLoader$1.run( > URLClassLoader.java:361) >> >> >> >>-- >>View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-4-can-t-accept-registration-of-UDF-tp23840.html >>Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >>--------------------------------------------------------------------- >>To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>For additional commands, e-mail: user-h...@spark.apache.org >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
