大家后,

   我们项目目前 集成了HiveModule,遇到一些问题。请教下大家。

在集成  Hive Module 之前,substr,split 都是无法使用的。集成后,验证都是可以的。
比如:select split('1,2,2,4',',')

但是在使用 explode 函数,select explode(split('1,2,2,4',','));
有如下错误:


The main method caused an error: SQL validation failed. From line 1, column
8 to line 1, column 36: *No match found for function signature explode(*
*)  *

*完整堆栈:*
The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: SQL validation failed. From line 1, column 8 to line 1,
column 36: No match found for function signature explode()
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
at
org.apache.flink.client.cli.CliFrontend$$Lambda$38/1205406622.call(Unknown
Source)
at
org.apache.flink.runtime.security.HadoopSecurityContext$$Lambda$39/293907205.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
Caused by: org.apache.flink.table.api.ValidationException: SQL validation
failed. From line 1, column 8 to line 1, column 36: No match found for
function signature explode()
at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org
$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:130)
at
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:105)
at
org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:127)
at
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:66)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:464)
at com.sui.bigdata.PlatformEngine$.$anonfun$main$4(PlatformEngine.scala:88)
at
com.sui.bigdata.PlatformEngine$.$anonfun$main$4$adapted(PlatformEngine.scala:87)
at com.sui.bigdata.PlatformEngine$$$Lambda$765/623545006.apply(Unknown
Source)
at scala.collection.immutable.List.foreach(List.scala:388)
at com.sui.bigdata.PlatformEngine$.main(PlatformEngine.scala:87)
at com.sui.bigdata.PlatformEngine.main(PlatformEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
... 13 more
Caused by: org.apache.calcite.runtime.CalciteContextException: From line 1,
column 8 to line 1, column 36: No match found for function signature
explode()
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:834)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:819)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:4841)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1773)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:293)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:219)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5600)
at
org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:5587)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:139)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1691)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1676)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:479)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4105)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3389)
at
org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
at
org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1008)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:968)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:943)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:650)
at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org
$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:126)
... 28 more
Caused by: org.apache.calcite.sql.validate.SqlValidatorException: No match
found for function signature explode()
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
... 50 more


简单看了 Hive Functions 加载函数代码,
1、都进行了加载
system.registerGenericUDF("split", GenericUDFSplit.class);
system.registerGenericUDTF("explode", GenericUDTFExplode.class);

2、FunctionInfo 返回是否 builtIn 逻辑也是一致的
 this.functionType = isNative ? FunctionType.BUILTIN :
FunctionType.TEMPORARY;,  其中  isNative  是单例对象的属性,
// registry for system functions
  private static final Registry system = new Registry(true);   // isNative
 === true

3、不在 Hive Module 黑名单中,附黑名单列表:
"dense_rank",
 "first_value",
"lag",
"last_value",
"lead",
 "rank",
 "row_number",
"hop",
 "hop_end",
 "hop_proctime",
 "hop_rowtime",
 "hop_start",
"session",
 "session_end",
 "session_proctime",
"session_rowtime",
"session_start",
"tumble",
 "tumble_end",
"tumble_proctime",
"tumble_rowtime",
"tumble_start"

不知道是不是还有什么地方使用方式不对?

谢谢解答。

回复