Hi Arjun,
Linkis is compatible with hive on tez and requires the following steps:
- You need to copy Tez-related dependencies to
{LINKIS_HOME}/lib/linkis-engineconn-plugins/hive/dist/3.1.3/lib is the dist
not plugin directory [1]. You can also modify hive ec pom to add tez
dependency compile
- vim
{LINKIS_HOME}/lib/linkis-engineconn-plugins/hive/dist/3.1.3/conf/linkis-engineconn.properties
and update linkis.hive.engine.type=tez
- sh {LINKIS_HOME}/sbin/linkis-daemon.sh restart linkis-cg-manager
1. https://linkis.apache.org/docs/latest/deployment/install-engineconn/
Best Regards,
Peace Wong.
arjun s <arjunjoice...@gmail.com> 于2023年8月15日周二 02:40写道：
> Hi team,
> I'm new to Apache Linkis. I've successfully set up Apache Linkis version
> 1.4.1 and I'm currently attempting to integrate it with Hive version 3.1.3.
> However, I've encountered an issue while executing the following command:
> sh ./bin/linkis-cli -engineType hive-3.1.3 -codeType hql -code "select *
> from load1" -submitUser techuser -proxyUser techuser
>
> I am getting below error
> 2023-08-14 11:53:20.053 INFO Background is starting a new engine for
> you,execId TaskID_10_otJobId_astJob_9_codeExec_9 mark id is mark_12, it may
> take several seconds, please wait
> 2023-08-14 11:53:42.053 ERROR jobRequest(LINKISCLI_techuser_hive_7)
> execute failed,21304, Task is Failed,errorMsg: errCode: 12003 ,desc:
> linkis-instance:9101_16 Failed to async get EngineNode AMErrorException:
> errCode: 30002 ,desc: ServiceInstance(linkis-cg-engineconn,
> linkis-instance:41969) ticketID: e04ac7a2-fc39-43fb-87a6-d88d4e084f13
> Failed to initialize engine, reason: ServiceInstance(linkis-cg-engineconn,
> linkis-instance:41969): log dir:
>
> /appcom/tmp/techuser/20230814/hive/e04ac7a2-fc39-43fb-87a6-d88d4e084f13/logs,java.lang.NoClassDefFoundError:
> org/apache/tez/dag/api/TezConfiguration
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:661)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:586)
> at
>
> org.apache.linkis.engineplugin.hive.creation.HiveEngineConnFactory.getSessionState(HiveEngineConnFactory.scala:135)
> at
>
> org.apache.linkis.engineplugin.hive.creation.HiveEngineConnFactory.doCreateHiveSession(HiveEngineConnFactory.scala:120)
> at
>
> org.apache.linkis.engineplugin.hive.creation.HiveEngineConnFactory.createEngineConnSession(HiveEngineConnFactory.scala:105)
> at
>
> org.apache.linkis.engineplugin.hive.creation.HiveEngineConnFactory.createEngineConnSession(HiveEngineConnFactory.scala:60)
> at
>
> org.apache.linkis.manager.engineplugin.common.creation.AbstractEngineConnFactory.createEngineConn(EngineConnFactory.scala:47)
> at
>
> org.apache.linkis.manager.engineplugin.common.creation.AbstractEngineConnFactory.createEngineConn$(EngineConnFactory.scala:45)
> at
>
> org.apache.linkis.engineplugin.hive.creation.HiveEngineConnFactory.createEngineConn(HiveEngineConnFactory.scala:60)
> at
>
> org.apache.linkis.engineconn.core.engineconn.DefaultEngineConnManager.createEngineConn(EngineConnManager.scala:47)
> at
>
> org.apache.linkis.engineconn.launch.EngineConnServer$.main(EngineConnServer.scala:88)
> at
>
> org.apache.linkis.engineconn.launch.EngineConnServer.main(EngineConnServer.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.tez.dag.api.TezConfiguration
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 12 more
> ,ip: linkis-instance ,port: 9101 ,serviceKind: linkis-cg-linkismanager
> ,ip: linkis-instance ,port: 9104 ,serviceKind: linkis-cg-entrance
>
>
> To provide context, I'm using Hive on Tez, and I've copied all the Tez JAR
> files to {LINKIS_HOME}/lib/linkis-engineconn-plugins/hive/plugin/3.1.3.
> However, despite this, the error persists. My concern is whether the new
> version of Linkis supports Hive on Tez, or if there are any additional
> steps I need to follow to enable Hive on Tez integration.
>
> Thanks in Advance
>
