The error message is clear, you didn't set the right configuration. You enable the dynamic allocation, and set max executor as 2 but you set the initial executor as 3
Caused by: java.lang.IllegalArgumentException: requirement failed: initial executor number 3 must between min executor number 1 and max executor number 2 Иван Шаповалов <shapovalov.iva...@gmail.com>于2017年6月26日周一 下午7:53写道： > Here is the failure from resource manager > > 17/06/26 05:55:43 ERROR ApplicationMaster: Uncaught exception: > org.apache.spark.SparkException: Exception thrown in awaitResult: > at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194) > at > org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:401) > at > org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:254) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:766) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) > at > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764) > at > org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.cloudera.livy.repl.SparkContextInitializer$class.spark2CreateContext(SparkContextInitializer.scala:94) > at > com.cloudera.livy.repl.SparkContextInitializer$class.createSparkContext(SparkContextInitializer.scala:34) > at > com.cloudera.livy.repl.SparkInterpreter.createSparkContext(SparkInterpreter.scala:36) > at > com.cloudera.livy.repl.SparkInterpreter$$anonfun$start$1.apply$mcV$sp(SparkInterpreter.scala:89) > at > com.cloudera.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:68) > at > com.cloudera.livy.repl.SparkInterpreter$$anonfun$start$1.apply(SparkInterpreter.scala:68) > at > com.cloudera.livy.repl.AbstractSparkInterpreter.restoreContextClassLoader(AbstractSparkInterpreter.scala:256) > at > com.cloudera.livy.repl.SparkInterpreter.start(SparkInterpreter.scala:68) > at com.cloudera.livy.repl.Session$$anonfun$1.apply(Session.scala:76) > at com.cloudera.livy.repl.Session$$anonfun$1.apply(Session.scala:74) > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: requirement failed: initial > executor number 3 must between min executor number 1 and max executor number 2 > at scala.Predef$.require(Predef.scala:224) > at > org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.getInitialTargetExecutorNumber(YarnSparkHadoopUtil.scala:304) > at > org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend.start(YarnClusterSchedulerBackend.scala:37) > at > org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320) > at > org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) > at > org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) > at scala.Option.getOrElse(Option.scala:121) > at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) > ... 19 more > 17/06/26 05:55:43 INFO ApplicationMaster: Unregistering ApplicationMaster > with FAILED (diag message: User class threw exception: > java.lang.reflect.InvocationTargetException) > 17/06/26 05:55:43 INFO ApplicationMaster: Deleting staging directory > > > thank you > > > 2017-06-26 14:16 GMT+03:00 Jeff Zhang <zjf...@gmail.com>: > >> >> Could you check the yarn app log ? >> >> >> >> Иван Шаповалов <shapovalov.iva...@gmail.com>于2017年6月26日周一 下午6:40写道： >> >>> Hi all, >>> I am trying to connect and run note via livy >>> >>> *created interpreter setting:* >>> livy.spark.driver.cores 1 >>> livy.spark.driver.memory 1g >>> livy.spark.dynamicAllocation.cachedExecutorIdleTimeout 600 >>> livy.spark.dynamicAllocation.enabled true >>> livy.spark.dynamicAllocation.initialExecutors 1 >>> livy.spark.dynamicAllocation.maxExecutors 2 >>> livy.spark.dynamicAllocation.minExecutors 1 >>> livy.spark.executor.cores 1 >>> livy.spark.executor.instances 3 >>> livy.spark.executor.memory 1g >>> livy.spark.jars.packages >>> zeppelin.livy.concurrentSQL false >>> zeppelin.livy.displayAppInfo false >>> zeppelin.livy.keytab >>> zeppelin.livy.principal zeppelin >>> zeppelin.livy.pull_status.interval.millis 1000 >>> zeppelin.livy.session.create_timeout 600 >>> zeppelin.livy.spark.sql.maxResult 1000 >>> zeppelin.livy.url http://correct-path:correct-port >>> >>> *and have following livy server config:* >>> livy.impersonation.enabled = false >>> livy.repl.enableHiveContext = true >>> ivy.server.csrf_protection.enabled = true >>> livy.spark.master = yarn.cluster >>> >>> *Trying to run paragraph drives me to:* >>> >>> INFO [2017-06-26 11:39:22,036] ({pool-2-thread-2} >>> BaseLivyInterprereter.java[createSession]:204) - Session 3 is in state >>> shutting_down, appId application_1498447942743_0011 >>> INFO [2017-06-26 11:39:23,184] ({pool-2-thread-2} >>> BaseLivyInterprereter.java[createSession]:204) - Session 3 is in state >>> dead, appId application_1498447942743_0011 >>> ERROR [2017-06-26 11:39:23,185] ({pool-2-thread-2} >>> BaseLivyInterprereter.java[createSession]:214) - Error when creating livy >>> session for user anonymous >>> org.apache.zeppelin.livy.LivyException: Session 3 is finished, appId: >>> application_1498447942743_0011, log: [YARN Diagnostics:, AM container is >>> launched, waiting for AM container to Register with RM] >>> at >>> org.apache.zeppelin.livy.BaseLivyInterprereter.createSession(BaseLivyInterprereter.java:209) >>> at >>> org.apache.zeppelin.livy.BaseLivyInterprereter.initLivySession(BaseLivyInterprereter.java:98) >>> at >>> org.apache.zeppelin.livy.BaseLivyInterprereter.open(BaseLivyInterprereter.java:80) >>> at >>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) >>> at >>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483) >>> at org.apache.zeppelin.scheduler.Job.run(Job.java:175) >>> at >>> org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Could anyone please help to understand what am I doing wrong ? >>> Thanks >>> -- >>> Ivan Shapovalov >>> Kharkov, Ukraine >>> >>> > > > -- > Ivan Shapovalov > Kharkov, Ukraine > >
