I created permanent functions(rather than temp functions) in Hive to use it
across different sessions. It all works fine until I actually restart the hive
server or cluster for any reason.
So is this the intended functionality of Permanent functions?
Here is the hive doc link for Permanent functions.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/Drop/ReloadFunction
1) Placed my utils jar for hive in hdfs location
hdfs:///opt/myUtiljars/myUtil.jar
2) Create function schemaName.myFunctName as 'com.myclass.name' using
'hdfs:///opt/myUtiljars/myUtil.jar'
3) Select schemaName.myFunctName() from tableName;
This is what I did to create a permanent function through beeline. And this is
working fine in other beeline sessions as well.
Now after I restart the servers I was able to see the functions name in "show
functions;" command but I cannot use this function in any of my queries.
When I issue the command in 3) Error: Error while compiling statement: FAILED:
SemanticException Line 0:-1 Invalid function schemaName.myFunctName '
(state=42000,code=40000)
Please find the stacktrace attached with this mail as well..
I would like to create Permanent functions as mentioned above and I don't want
to deal with them every time I restart.
Any corrections(if I am missing anything) or suggestion are greatly appreciated.
Thanks in advance
Prasanth Chagarlamudi
________________________________
This e-mail and files transmitted with it are confidential, and are intended
solely for the use of the individual or entity to whom this e-mail is
addressed. If you are not the intended recipient, or the employee or agent
responsible to deliver it to the intended recipient, you are hereby notified
that any dissemination, distribution or copying of this communication is
strictly prohibited. If you are not one of the named recipient(s) or otherwise
have reason to believe that you received this message in error, please
immediately notify sender by e-mail, and destroy the original message. Thank
You.
2016-02-05 13:34:30,098 INFO org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG
method=compile from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,099 INFO org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG
method=parse from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,099 INFO hive.ql.parse.ParseDriver: Parsing command: select
now_perm() from view_upecds_mul_9_1454018694068
2016-02-05 13:34:30,101 INFO hive.ql.parse.ParseDriver: Parse Completed
2016-02-05 13:34:30,102 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
</PERFLOG method=parse start=1454700870099 end=1454700870102 duration=3
from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,102 INFO org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG
method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,104 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Starting Semantic Analysis
2016-02-05 13:34:30,105 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Completed phase 1 of Semantic Analysis
2016-02-05 13:34:30,105 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Get metadata for source tables
2016-02-05 13:34:30,139 INFO hive.ql.parse.ParseDriver: Parsing command: SELECT
`upecds_mul_9_1454018694068`.`customerkey_1454015023296`,`upecds_mul_9_1454018694068`.`emailaddress_1454015023297`,`upecds_mul_9_1454018694068`.`clientemailaddressdeliverystatus_1454015023297`,`upecds_mul_9_1454018694068`.`cds_sub_org_id`,`upecds_mul_9_1454018694068`.`cds_updated_date`,`upecds_mul_9_1454018694068`.`cds_updated_by`,`upecds_mul_9_1454018694068`.`cds_client_corelation_id`,`upecds_mul_9_1454018694068`.`cds_pk`,`upecds_mul_9_1454018694068`.`cds_import_job_id`
FROM `MUL`.`UPECDS_MUL_9_1454018694068` WHERE
`upecds_mul_9_1454018694068`.`cds_updated_date` IS NOT NULL
2016-02-05 13:34:30,146 INFO hive.ql.parse.ParseDriver: Parse Completed
2016-02-05 13:34:30,149 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Get metadata for subqueries
2016-02-05 13:34:30,149 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Get metadata for source tables
2016-02-05 13:34:30,237 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Get metadata for subqueries
2016-02-05 13:34:30,237 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Get metadata for destination tables
2016-02-05 13:34:30,237 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Get metadata for destination tables
2016-02-05 13:34:30,252 INFO hive.ql.Context: New scratch dir is
hdfs://localhost.localdomain:8020/tmp/hive/cds_user/287b8cbf-4374-4e56-a1d2-1cdd73c968ce/hive_2016-02-05_13-34-30_099_3284786619543371524-1
2016-02-05 13:34:30,252 INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:
Completed getting MetaData in Semantic Analysis
2016-02-05 13:34:30,266 ERROR org.apache.hadoop.hive.ql.Driver: FAILED:
NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:232)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:958)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1168)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:192)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:145)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10567)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10523)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3720)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3499)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9037)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8992)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9849)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9742)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10178)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10189)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10106)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:423)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1106)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:171)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy23.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-02-05 13:34:30,266 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
</PERFLOG method=compile start=1454700870098 end=1454700870266 duration=168
from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,266 INFO org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG
method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,266 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
</PERFLOG method=releaseLocks start=1454700870266 end=1454700870266 duration=0
from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,273 INFO org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG
method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,273 INFO org.apache.hadoop.hive.ql.log.PerfLogger:
</PERFLOG method=releaseLocks start=1454700870273 end=1454700870273 duration=0
from=org.apache.hadoop.hive.ql.Driver>
2016-02-05 13:34:30,274 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:cds_user (auth:SIMPLE)
cause:org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: NullPointerException null
2016-02-05 13:34:30,274 WARN
org.apache.hive.service.cli.thrift.ThriftCLIService: Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement:
FAILED: NullPointerException null
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:102)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:171)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:398)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:385)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy23.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:271)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:490)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:232)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:958)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1168)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:192)
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:145)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10567)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10523)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3720)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3499)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9037)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8992)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9849)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9742)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10178)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10189)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10106)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:423)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1112)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1106)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100)
... 27 more
