各位大神
Your environment
Linkis version used: release-1.1.3-rc1
Environment name and version:
hdp-3.1.5
hive-3.1.0.3.1.4.0-315
hadoop-3.1.1.3.1.4.0-315
scala-2.11.12
jdk 1.8.0_121
spark-2.3.2.3.1.4.0-315
Describe your questions
单独安装 linkis平台和管理页面，完成之后验证基础功能，使用
sh bin/linkis-cli -submitUser ... spark 命令验证失败
包括使用 hive提交的时候也会爆出相同的错误
stdout.log:
1b3/tmp/blockmgr-1098b279-2df6-4fc0-8c3a-5ecb57b2f5ab
2022-09-03 11:57:10.746 [INFO ] [main ]
o.a.s.s.m.MemoryStore (54) [logInfo] - MemoryStore started with capacity 434.4
MB
2022-09-03 11:57:10.813 [INFO ] [main ]
o.a.s.SparkEnv (54) [logInfo] - Registering OutputCommitCoordinator
2022-09-03 11:57:11.142 [INFO ] [main ]
o.a.s.u.Utils (54) [logInfo] - Successfully started service 'SparkUI' on port
4040.
2022-09-03 11:57:11.333 [INFO ] [main ]
o.a.s.u.SparkUI (54) [logInfo] - Bound SparkUI to 0.0.0.0, and started at
http://******:4040
2022-09-03 11:57:11.349 [INFO ] [main ]
o.a.s.SparkContext (54) [logInfo] - Added JAR
file:/appcom/tmp/hadoop/20220903/spark/ff2cebf5-72d7-4ef2-9254-d43ca498c1b3/lib/linkis-engineplugin-spark-1.1.3.jar
at spark://******:6476/jars/linkis-engineplugin-spark-1.1.3.jar with timestamp
1662177431347
2022-09-03 11:57:11.433 [WARN ] [main ]
o.a.s.s.FairSchedulableBuilder (66) [logWarning] - Fair Scheduler configuration
file not found so jobs will be scheduled in FIFO order. To use fair scheduling,
configure pools in fairscheduler.xml or set spark.scheduler.allocation.file to
a file that contains the configuration.
2022-09-03 11:57:11.439 [INFO ] [main ]
o.a.s.s.FairSchedulableBuilder (54) [logInfo] - Created default pool: default,
schedulingMode: FIFO, minShare: 0, weight: 1
2022-09-03 11:57:12.634 [INFO ] [main ]
o.a.h.y.c.ConfiguredRMFailoverProxyProvider (100) [performFailover] - Failing
over to rm2
2022-09-03 11:57:12.673 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Requesting a new application from cluster
with 6 NodeManagers
2022-09-03 11:57:12.755 [INFO ] [main ]
o.a.h.c.Configuration (2757) [getConfResourceAsInputStream] - found resource
resource-types.xml at file:/etc/hadoop/3.1.4.0-315/0/resource-types.xml
2022-09-03 11:57:12.784 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Verifying our application has not requested
more than the maximum memory capability of the cluster (73728 MB per container)
2022-09-03 11:57:12.786 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Will allocate AM container, with 896 MB
memory including 384 MB overhead
2022-09-03 11:57:12.787 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Setting up container launch context for our AM
2022-09-03 11:57:12.795 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Setting up the launch environment for our AM
container
2022-09-03 11:57:12.806 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Preparing resources for our AM container
2022-09-03 11:57:14.556 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Use hdfs cache file as spark.yarn.archive for
HDP,
hdfsCacheFile:hdfs://******/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-yarn-archive.tar.gz
2022-09-03 11:57:14.563 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Source and destination file systems are the
same. Not copying
hdfs://******/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-yarn-archive.tar.gz
2022-09-03 11:57:14.672 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Distribute hdfs cache file as
spark.sql.hive.metastore.jars for HDP,
hdfsCacheFile:hdfs://******/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-hive-archive.tar.gz
2022-09-03 11:57:14.673 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Source and destination file systems are the
same. Not copying
hdfs://******/hdp/apps/3.1.4.0-315/spark2/spark2-hdp-hive-archive.tar.gz
2022-09-03 11:57:14.690 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Source and destination file systems are the
same. Not copying hdfs://******/home/spark_conf/hive-site.xml
2022-09-03 11:57:14.713 [WARN ] [main ]
o.a.s.d.y.Client (66) [logWarning] - Same path resource
hdfs://******/home/spark_conf/hive-site.xml added multiple times to distributed
cache.
2022-09-03 11:57:14.723 [INFO ] [main ]
o.a.s.d.y.Client (54) [logInfo] - Deleted staging directory
hdfs://******/user/hadoop/.sparkStaging/application_1661773015716_0041
2022-09-03 11:57:14.726 [ERROR] [main ]
o.a.s.SparkContext (91) [logError] - Error initializing SparkContext.
java.lang.IllegalArgumentException: Attempt to add
(hdfs://******/home/spark_conf/hive-site.xml) multiple times to the distributed
cache.
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:660)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.immutable.List.foreach(List.scala:392)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:921)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.Option.getOrElse(Option.scala:121)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createSparkSession(SparkEngineConnFactory.scala:117)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
:
2022-09-03 11:57:14.752 [INFO ] [main ]
o.a.s.u.SparkUI (54) [logInfo] - Stopped Spark web UI at http://******:4040
2022-09-03 11:57:14.773 [WARN ] [dispatcher-event-loop-9 ]
o.a.s.s.c.YarnSchedulerBackend$YarnSchedulerEndpoint (66) [logWarning] -
Attempted to request executors before the AM has registered!
2022-09-03 11:57:14.781 [INFO ] [main ]
o.a.s.s.c.YarnClientSchedulerBackend (54) [logInfo] - Stopped
2022-09-03 11:57:14.797 [INFO ] [dispatcher-event-loop-11 ]
o.a.s.MapOutputTrackerMasterEndpoint (54) [logInfo] -
MapOutputTrackerMasterEndpoint stopped!
2022-09-03 11:57:14.810 [INFO ] [main ]
o.a.s.s.m.MemoryStore (54) [logInfo] - MemoryStore cleared
2022-09-03 11:57:14.812 [INFO ] [main ]
o.a.s.s.BlockManager (54) [logInfo] - BlockManager stopped
2022-09-03 11:57:14.832 [INFO ] [main ]
o.a.s.s.BlockManagerMaster (54) [logInfo] - BlockManagerMaster stopped
2022-09-03 11:57:14.834 [WARN ] [main ]
o.a.s.m.MetricsSystem (66) [logWarning] - Stopping a MetricsSystem that is not
running
2022-09-03 11:57:14.850 [INFO ] [dispatcher-event-loop-16 ]
o.a.s.s.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint (54) [logInfo]
- OutputCommitCoordinator stopped!
2022-09-03 11:57:14.876 [INFO ] [main ]
o.a.s.SparkContext (54) [logInfo] - Successfully stopped SparkContext
2022-09-03 11:57:14.877 [ERROR] [main ]
o.a.l.e.l.EngineConnServer$ (58) [error] - EngineConnServer Start Failed.
java.lang.IllegalArgumentException: Attempt to add
(hdfs://******/home/spark_conf/hive-site.xml) multiple times to the distributed
cache.
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:660)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.immutable.List.foreach(List.scala:392)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:921)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.Option.getOrElse(Option.scala:121)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createSparkSession(SparkEngineConnFactory.scala:117)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createEngineConnSession(SparkEngineConnFactory.scala:74)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
at
org.apache.linkis.manager.engineplugin.common.creation.AbstractEngineConnFactory$class.createEngineConn(EngineConnFactory.scala:48)
~[linkis-engineconn-plugin-core-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createEngineConn(SparkEngineConnFactory.scala:42)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineconn.core.engineconn.DefaultEngineConnManager.createEngineConn(EngineConnManager.scala:45)
~[linkis-engineconn-core-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineconn.launch.EngineConnServer$.main(EngineConnServer.scala:64)
~[linkis-engineconn-core-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineconn.launch.EngineConnServer.main(EngineConnServer.scala)
~[linkis-engineconn-core-1.1.3.jar:1.1.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_202]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_202]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_202]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
2022-09-03 11:57:14.891 [ERROR] [main ]
o.a.l.e.c.s.EngineConnAfterStartCallback (46) [callback] - protocol will send
to em: EngineConnStatusCallback(ServiceInstance(linkis-cg-engineconn,
******:8681),ff2cebf5-72d7-4ef2-9254-d43ca498c1b3,Failed,ServiceInstance(linkis-cg-engineconn,
******:8681): log dir:
/appcom/tmp/hadoop/20220903/spark/ff2cebf5-72d7-4ef2-9254-d43ca498c1b3/logs,IllegalArgumentException:
Attempt to add (hdfs://******/home/spark_conf/hive-site.xml) multiple times to
the distributed cache.)
2022-09-03 11:57:14.912 [ERROR] [main ]
o.a.l.e.c.e.h.ComputationEngineConnHook (58) [error] - EngineConnSever start
failed! now exit. java.lang.IllegalArgumentException: Attempt to add
(hdfs://******/home/spark_conf/hive-site.xml) multiple times to the distributed
cache.
:
2022-09-03 11:57:14.891 [ERROR] [main ]
o.a.l.e.c.s.EngineConnAfterStartCallback (46) [callback] - protocol will send
to em: EngineConnStatusCallback(ServiceInstance(linkis-cg-
engineconn,
******:8681),ff2cebf5-72d7-4ef2-9254-d43ca498c1b3,Failed,ServiceInstance(linkis-cg-engineconn,
******:8681): log dir:
/appcom/tmp/hadoop/20220903/spark/ff2cebf5-72d7-4ef2-9254-d43ca
498c1b3/logs,IllegalArgumentException: Attempt to add
(hdfs://******/home/spark_conf/hive-site.xml) multiple times to the distributed
cache.)
2022-09-03 11:57:14.912 [ERROR] [main ]
o.a.l.e.c.e.h.ComputationEngineConnHook (58) [error] - EngineConnSever start
failed! now exit. java.lang.IllegalArgumentException: Attem
pt to add (hdfs://******/home/spark_conf/hive-site.xml) multiple times to the
distributed cache.
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:660)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.immutable.List.foreach(List.scala:392)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:921)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:934)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:925)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.Option.getOrElse(Option.scala:121)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:925)
~[spark-sql_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createSparkSession(SparkEngineConnFactory.scala:117)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createEngineConnSession(SparkEngineConnFactory.scala:74)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
at
org.apache.linkis.manager.engineplugin.common.creation.AbstractEngineConnFactory$class.createEngineConn(EngineConnFactory.scala:48)
~[linkis-engineconn-plugin-core-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineplugin.spark.factory.SparkEngineConnFactory.createEngineConn(SparkEngineConnFactory.scala:42)
~[linkis-engineplugin-spark-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineconn.core.engineconn.DefaultEngineConnManager.createEngineConn(EngineConnManager.scala:45)
~[linkis-engineconn-core-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineconn.launch.EngineConnServer$.main(EngineConnServer.scala:64)
~[linkis-engineconn-core-1.1.3.jar:1.1.3]
at
org.apache.linkis.engineconn.launch.EngineConnServer.main(EngineConnServer.scala)
~[linkis-engineconn-core-1.1.3.jar:1.1.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_202]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_202]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_202]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
2022-09-03 11:57:14.917 [ERROR] [main ]
o.a.l.e.c.h.ShutdownHook (58) [error] - process exit reason:
java.lang.IllegalArgumentException: Attempt to add (hdfs://******/home/
spark_conf/hive-site.xml) multiple times to the distributed cache.
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:660)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17$$anonfun$apply$6.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:651)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$17.apply(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at scala.collection.immutable.List.foreach(List.scala:392)
~[scala-library-2.11.12.jar:?]
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:650)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:921)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
~[spark-yarn_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
~[spark-core_2.11-2.3.2.3.1.4.0-315.jar:2.3.2.3.1.4.0-315]
能麻烦帮忙看下什么问题吗
15000338...@163.com
