Try to start aws EMR 4.2.0 with hadoop and spark applications on spot instances. Then look at how hadoop and spark configured. Try to configure your hadoop and spark similar way On Dec 17, 2015 6:09 PM, "Saisai Shao" <sai.sai.s...@gmail.com> wrote:
> Please check the Yarn AM log to see why AM is failed to start. That's the > reason why using `sc` will get such complaint. > > On Fri, Dec 18, 2015 at 4:25 AM, Eran Witkon <eranwit...@gmail.com> wrote: > >> Hi, >> I am trying to install spark 1.5.2 on Apache hadoop 2.6 and Hive and yarn >> >> spark-env.sh >> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop >> >> bash_profile >> #HADOOP VARIABLES START >> export JAVA_HOME=/usr/lib/jvm/java-8-oracle/ >> export HADOOP_INSTALL=/usr/local/hadoop >> export PATH=$PATH:$HADOOP_INSTALL/bin >> export PATH=$PATH:$HADOOP_INSTALL/sbin >> export HADOOP_MAPRED_HOME=$HADOOP_INSTALL >> export HADOOP_COMMON_HOME=$HADOOP_INSTALL >> export HADOOP_HDFS_HOME=$HADOOP_INSTALL >> export YARN_HOME=$HADOOP_INSTALL >> export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native >> export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib" >> export HADOOP_USER_CLASSPATH_FIRST=true >> export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop >> export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop >> #HADOOP VARIABLES END >> >> export SPARK_HOME=/usr/local/spark >> export HIVE_HOME=/usr/local/hive >> export PATH=$PATH:$HIVE_HOME/bin >> >> >> When I run spark-shell >> ./bin/spark-shell --master yarn-client >> >> Output: >> 15/12/17 22:22:07 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing view acls to: >> hduser >> 15/12/17 22:22:07 INFO spark.SecurityManager: Changing modify acls to: >> hduser >> 15/12/17 22:22:07 INFO spark.SecurityManager: SecurityManager: >> authentication disabled; ui acls disabled; users with view permissions: >> Set(hduser); users with modify permissions: Set(hduser) >> 15/12/17 22:22:07 INFO spark.HttpServer: Starting HTTP Server >> 15/12/17 22:22:07 INFO server.Server: jetty-8.y.z-SNAPSHOT >> 15/12/17 22:22:08 INFO server.AbstractConnector: Started >> SocketConnector@0.0.0.0:38389 >> 15/12/17 22:22:08 INFO util.Utils: Successfully started service 'HTTP >> class server' on port 38389. >> Welcome to >> ____ __ >> / __/__ ___ _____/ /__ >> _\ \/ _ \/ _ `/ __/ '_/ >> /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 >> /_/ >> >> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java >> 1.8.0_66) >> Type in expressions to have them evaluated. >> Type :help for more information. >> 15/12/17 22:22:11 WARN util.Utils: Your hostname, eranw-Lenovo-Yoga-2-Pro >> resolves to a loopback address: 127.0.1.1; using 10.0.0.1 instead (on >> interface wlp1s0) >> 15/12/17 22:22:11 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind >> to another address >> 15/12/17 22:22:11 INFO spark.SparkContext: Running Spark version 1.5.2 >> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing view acls to: >> hduser >> 15/12/17 22:22:11 INFO spark.SecurityManager: Changing modify acls to: >> hduser >> 15/12/17 22:22:11 INFO spark.SecurityManager: SecurityManager: >> authentication disabled; ui acls disabled; users with view permissions: >> Set(hduser); users with modify permissions: Set(hduser) >> 15/12/17 22:22:11 INFO slf4j.Slf4jLogger: Slf4jLogger started >> 15/12/17 22:22:11 INFO Remoting: Starting remoting >> 15/12/17 22:22:12 INFO util.Utils: Successfully started service >> 'sparkDriver' on port 36381. >> 15/12/17 22:22:12 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://sparkDriver@10.0.0.1:36381] >> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering MapOutputTracker >> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering BlockManagerMaster >> 15/12/17 22:22:12 INFO storage.DiskBlockManager: Created local directory >> at /tmp/blockmgr-139fac31-5f21-4c61-9575-3110d5205f7d >> 15/12/17 22:22:12 INFO storage.MemoryStore: MemoryStore started with >> capacity 530.0 MB >> 15/12/17 22:22:12 INFO spark.HttpFileServer: HTTP File server directory >> is >> /tmp/spark-955ef002-a802-49c6-b440-0656861f737c/httpd-2127cbe1-97d7-40a5-a96f-75216f115f00 >> 15/12/17 22:22:12 INFO spark.HttpServer: Starting HTTP Server >> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT >> 15/12/17 22:22:12 INFO server.AbstractConnector: Started >> SocketConnector@0.0.0.0:36760 >> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'HTTP >> file server' on port 36760. >> 15/12/17 22:22:12 INFO spark.SparkEnv: Registering OutputCommitCoordinator >> 15/12/17 22:22:12 INFO server.Server: jetty-8.y.z-SNAPSHOT >> 15/12/17 22:22:12 INFO server.AbstractConnector: Started >> SelectChannelConnector@0.0.0.0:4040 >> 15/12/17 22:22:12 INFO util.Utils: Successfully started service 'SparkUI' >> on port 4040. >> 15/12/17 22:22:12 INFO ui.SparkUI: Started SparkUI at >> http://10.0.0.1:4040 >> 15/12/17 22:22:12 WARN metrics.MetricsSystem: Using default name >> DAGScheduler for source because spark.app.id is not set. >> 15/12/17 22:22:12 INFO client.RMProxy: Connecting to ResourceManager at / >> 0.0.0.0:8032 >> 15/12/17 22:22:12 INFO yarn.Client: Requesting a new application from >> cluster with 1 NodeManagers >> 15/12/17 22:22:12 INFO yarn.Client: Verifying our application has not >> requested more than the maximum memory capability of the cluster (8192 MB >> per container) >> 15/12/17 22:22:12 INFO yarn.Client: Will allocate AM container, with 896 >> MB memory including 384 MB overhead >> 15/12/17 22:22:12 INFO yarn.Client: Setting up container launch context >> for our AM >> 15/12/17 22:22:12 INFO yarn.Client: Setting up the launch environment for >> our AM container >> 15/12/17 22:22:12 INFO yarn.Client: Preparing resources for our AM >> container >> 15/12/17 22:22:13 INFO yarn.Client: Uploading resource >> file:/usr/local/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar -> >> hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/spark-assembly-1.5.2-hadoop2.6.0.jar >> 15/12/17 22:22:15 INFO yarn.Client: Uploading resource >> file:/tmp/spark-955ef002-a802-49c6-b440-0656861f737c/__spark_conf__3605678380745697769.zip >> -> >> hdfs://localhost:54310/user/hduser/.sparkStaging/application_1450383657924_0003/__spark_conf__3605678380745697769.zip >> 15/12/17 22:22:15 INFO spark.SecurityManager: Changing view acls to: >> hduser >> 15/12/17 22:22:15 INFO spark.SecurityManager: Changing modify acls to: >> hduser >> 15/12/17 22:22:15 INFO spark.SecurityManager: SecurityManager: >> authentication disabled; ui acls disabled; users with view permissions: >> Set(hduser); users with modify permissions: Set(hduser) >> 15/12/17 22:22:15 INFO yarn.Client: Submitting application 3 to >> ResourceManager >> 15/12/17 22:22:15 INFO impl.YarnClientImpl: Submitted application >> application_1450383657924_0003 >> 15/12/17 22:22:16 INFO yarn.Client: Application report for >> application_1450383657924_0003 (state: ACCEPTED) >> 15/12/17 22:22:16 INFO yarn.Client: >> client token: N/A >> diagnostics: N/A >> ApplicationMaster host: N/A >> ApplicationMaster RPC port: -1 >> queue: default >> start time: 1450383735512 >> final status: UNDEFINED >> tracking URL: >> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/ >> user: hduser >> 15/12/17 22:22:17 INFO yarn.Client: Application report for >> application_1450383657924_0003 (state: ACCEPTED) >> 15/12/17 22:22:18 INFO yarn.Client: Application report for >> application_1450383657924_0003 (state: ACCEPTED) >> 15/12/17 22:22:19 INFO yarn.Client: Application report for >> application_1450383657924_0003 (state: ACCEPTED) >> 15/12/17 22:22:20 INFO yarn.Client: Application report for >> application_1450383657924_0003 (state: ACCEPTED) >> 15/12/17 22:22:21 INFO >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster >> registered as AkkaRpcEndpointRef(Actor[akka.tcp:// >> sparkYarnAM@10.0.0.1:34719/user/YarnAM#-313490512]) >> 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Add WebUI >> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, >> Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES -> >> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003), >> /proxy/application_1450383657924_0003 >> 15/12/17 22:22:21 INFO ui.JettyUtils: Adding filter: >> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter >> 15/12/17 22:22:21 INFO yarn.Client: Application report for >> application_1450383657924_0003 (state: RUNNING) >> 15/12/17 22:22:21 INFO yarn.Client: >> client token: N/A >> diagnostics: N/A >> ApplicationMaster host: 10.0.0.1 >> ApplicationMaster RPC port: 0 >> queue: default >> start time: 1450383735512 >> final status: UNDEFINED >> tracking URL: >> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003/ >> user: hduser >> 15/12/17 22:22:21 INFO cluster.YarnClientSchedulerBackend: Application >> application_1450383657924_0003 has started running. >> 15/12/17 22:22:21 INFO util.Utils: Successfully started service >> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33155. >> 15/12/17 22:22:21 INFO netty.NettyBlockTransferService: Server created on >> 33155 >> 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Trying to register >> BlockManager >> 15/12/17 22:22:21 INFO storage.BlockManagerMasterEndpoint: Registering >> block manager 10.0.0.1:33155 with 530.0 MB RAM, BlockManagerId(driver, >> 10.0.0.1, 33155) >> 15/12/17 22:22:21 INFO storage.BlockManagerMaster: Registered BlockManager >> 15/12/17 22:22:22 WARN >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has >> disassociated: 10.0.0.1:34719 >> 15/12/17 22:22:22 WARN remote.ReliableDeliverySupervisor: Association >> with remote system [akka.tcp://sparkYarnAM@10.0.0.1:34719] has failed, >> address is now gated for [5000] ms. Reason: [Disassociated] >> 15/12/17 22:22:22 INFO scheduler.EventLoggingListener: Logging events to >> hdfs://localhost:54310/spark-log/application_1450383657924_0003 >> 15/12/17 22:22:22 WARN >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has >> disassociated: 10.0.0.1:34719 >> 15/12/17 22:22:25 INFO >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster >> registered as AkkaRpcEndpointRef(Actor[akka.tcp:// >> sparkYarnAM@10.0.0.1:39353/user/YarnAM#1861872417]) >> 15/12/17 22:22:25 INFO cluster.YarnClientSchedulerBackend: Add WebUI >> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, >> Map(PROXY_HOSTS -> eranw-Lenovo-Yoga-2-Pro, PROXY_URI_BASES -> >> http://eranw-Lenovo-Yoga-2-Pro:8088/proxy/application_1450383657924_0003), >> /proxy/application_1450383657924_0003 >> 15/12/17 22:22:25 INFO ui.JettyUtils: Adding filter: >> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter >> 15/12/17 22:22:30 WARN >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has >> disassociated: 10.0.0.1:39353 >> 15/12/17 22:22:30 WARN remote.ReliableDeliverySupervisor: Association >> with remote system [akka.tcp://sparkYarnAM@10.0.0.1:39353] has failed, >> address is now gated for [5000] ms. Reason: [Disassociated] >> 15/12/17 22:22:30 WARN >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster has >> disassociated: 10.0.0.1:39353 >> 15/12/17 22:22:30 ERROR cluster.YarnClientSchedulerBackend: Yarn >> application has already exited with state FINISHED! >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/metrics/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/stage/kill,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/api,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/static,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors/threadDump,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/executors,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/environment/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/environment,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage/rdd/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage/rdd,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/storage,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/pool/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/pool,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/stage/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/stage,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/stages,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs/job/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs/job,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs/json,null} >> 15/12/17 22:22:30 INFO handler.ContextHandler: stopped >> o.s.j.s.ServletContextHandler{/jobs,null} >> 15/12/17 22:22:31 INFO ui.SparkUI: Stopped Spark web UI at >> http://10.0.0.1:4040 >> 15/12/17 22:22:31 INFO scheduler.DAGScheduler: Stopping DAGScheduler >> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Shutting down >> all executors >> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Asking each >> executor to shut down >> 15/12/17 22:22:31 INFO cluster.YarnClientSchedulerBackend: Stopped >> 15/12/17 22:22:36 INFO spark.MapOutputTrackerMasterEndpoint: >> MapOutputTrackerMasterEndpoint stopped! >> 15/12/17 22:22:36 INFO storage.MemoryStore: MemoryStore cleared >> 15/12/17 22:22:36 INFO storage.BlockManager: BlockManager stopped >> 15/12/17 22:22:36 INFO storage.BlockManagerMaster: BlockManagerMaster >> stopped >> 15/12/17 22:22:36 INFO spark.SparkContext: Successfully stopped >> SparkContext >> 15/12/17 22:22:36 INFO >> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: >> OutputCommitCoordinator stopped! >> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: >> Shutting down remote daemon. >> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: >> Remote daemon shut down; proceeding with flushing remote transports. >> 15/12/17 22:22:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: >> Remoting shut down. >> 15/12/17 22:22:42 INFO cluster.YarnClientSchedulerBackend: >> SchedulerBackend is ready for scheduling beginning after waiting >> maxRegisteredResourcesWaitingTime: 30000(ms) >> 15/12/17 22:22:42 INFO repl.SparkILoop: Created spark context.. >> Spark context available as sc. >> 15/12/17 22:22:49 INFO hive.HiveContext: Initializing execution hive, >> version 1.2.1 >> 15/12/17 22:22:49 INFO client.ClientWrapper: Inspected Hadoop version: >> 2.6.0 >> 15/12/17 22:22:49 INFO client.ClientWrapper: Loaded >> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 >> 15/12/17 22:22:49 INFO metastore.HiveMetaStore: 0: Opening raw store with >> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >> 15/12/17 22:22:49 INFO metastore.ObjectStore: ObjectStore, initialize >> called >> 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property >> hive.metastore.integral.jdo.pushdown unknown - will be ignored >> 15/12/17 22:22:50 INFO DataNucleus.Persistence: Property >> datanucleus.cache.level2 unknown - will be ignored >> 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not >> present in CLASSPATH (or one of dependencies) >> 15/12/17 22:22:50 WARN DataNucleus.Connection: BoneCP specified but not >> present in CLASSPATH (or one of dependencies) >> 15/12/17 22:22:59 INFO metastore.ObjectStore: Setting MetaStore object >> pin classes with >> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >> 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:00 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:01 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:02 INFO metastore.MetaStoreDirectSql: Using direct SQL, >> underlying DB is DERBY >> 15/12/17 22:23:02 INFO metastore.ObjectStore: Initialized ObjectStore >> 15/12/17 22:23:02 WARN metastore.ObjectStore: Version information not >> found in metastore. hive.metastore.schema.verification is not enabled so >> recording the schema version 1.2.0 >> 15/12/17 22:23:02 WARN metastore.ObjectStore: Failed to get database >> default, returning NoSuchObjectException >> 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added admin role in >> metastore >> 15/12/17 22:23:02 INFO metastore.HiveMetaStore: Added public role in >> metastore >> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: No user is added in admin >> role, since config is empty >> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_all_databases >> 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr >> cmd=get_all_databases >> 15/12/17 22:23:03 INFO metastore.HiveMetaStore: 0: get_functions: >> db=default pat=* >> 15/12/17 22:23:03 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr >> cmd=get_functions: >> db=default pat=* >> 15/12/17 22:23:03 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:09 INFO session.SessionState: Created local directory: >> /tmp/eae419f8-7dad-417b-bc3e-77c048c88337_resources >> 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory: >> /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337 >> 15/12/17 22:23:09 INFO session.SessionState: Created local directory: >> /tmp/hduser/eae419f8-7dad-417b-bc3e-77c048c88337 >> 15/12/17 22:23:09 INFO session.SessionState: Created HDFS directory: >> /tmp/hive/hduser/eae419f8-7dad-417b-bc3e-77c048c88337/_tmp_space.db >> 15/12/17 22:23:09 INFO hive.HiveContext: default warehouse location is >> /user/hive/warehouse >> 15/12/17 22:23:09 INFO hive.HiveContext: Initializing >> HiveMetastoreConnection version 1.2.1 using Spark classes. >> 15/12/17 22:23:09 INFO client.ClientWrapper: Inspected Hadoop version: >> 2.6.0 >> 15/12/17 22:23:09 INFO client.ClientWrapper: Loaded >> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0 >> 15/12/17 22:23:10 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> 15/12/17 22:23:10 INFO metastore.HiveMetaStore: 0: Opening raw store with >> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >> 15/12/17 22:23:10 INFO metastore.ObjectStore: ObjectStore, initialize >> called >> 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property >> hive.metastore.integral.jdo.pushdown unknown - will be ignored >> 15/12/17 22:23:10 INFO DataNucleus.Persistence: Property >> datanucleus.cache.level2 unknown - will be ignored >> 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not >> present in CLASSPATH (or one of dependencies) >> 15/12/17 22:23:10 WARN DataNucleus.Connection: BoneCP specified but not >> present in CLASSPATH (or one of dependencies) >> 15/12/17 22:23:11 INFO metastore.ObjectStore: Setting MetaStore object >> pin classes with >> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:12 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:12 INFO DataNucleus.Query: Reading in results for query >> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used >> is closing >> 15/12/17 22:23:12 INFO metastore.MetaStoreDirectSql: Using direct SQL, >> underlying DB is DERBY >> 15/12/17 22:23:12 INFO metastore.ObjectStore: Initialized ObjectStore >> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added admin role in >> metastore >> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: Added public role in >> metastore >> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: No user is added in admin >> role, since config is empty >> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_all_databases >> 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr >> cmd=get_all_databases >> 15/12/17 22:23:13 INFO metastore.HiveMetaStore: 0: get_functions: >> db=default pat=* >> 15/12/17 22:23:13 INFO HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr >> cmd=get_functions: >> db=default pat=* >> 15/12/17 22:23:13 INFO DataNucleus.Datastore: The class >> "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as >> "embedded-only" so does not have its own datastore table. >> 15/12/17 22:23:18 INFO session.SessionState: Created local directory: >> /tmp/7f892b9d-38ef-4fab-bc44-c1afaf07f50f_resources >> 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory: >> /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f >> 15/12/17 22:23:18 INFO session.SessionState: Created local directory: >> /tmp/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f >> 15/12/17 22:23:18 INFO session.SessionState: Created HDFS directory: >> /tmp/hive/hduser/7f892b9d-38ef-4fab-bc44-c1afaf07f50f/_tmp_space.db >> 15/12/17 22:23:18 INFO repl.SparkILoop: Created sql context (with Hive >> support).. >> SQL context available as sqlContext. >> >> scala> >> >> When I am running the following code: >> val data = sc.wholeTextFiles("hdfs://localhost:54310/spark-log/*") >> >> I am getting exception: >> java.lang.IllegalStateException: Cannot call methods on a stopped >> SparkContext >> at org.apache.spark.SparkContext.org >> $apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:104) >> at >> org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2063) >> at >> org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2076) >> at >> org.apache.spark.SparkContext.wholeTextFiles$default$2(SparkContext.scala:864) >> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:21) >> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26) >> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28) >> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30) >> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:32) >> at $iwC$$iwC$$iwC.<init>(<console>:34) >> at $iwC$$iwC.<init>(<console>:36) >> at $iwC.<init>(<console>:38) >> at <init>(<console>:40) >> at .<init>(<console>:44) >> at .<clinit>(<console>) >> at .<init>(<console>:7) >> at .<clinit>(<console>) >> at $print(<console>) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >> at >> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) >> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >> at >> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >> at >> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) >> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) >> at org.apache.spark.repl.SparkILoop.org >> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> at org.apache.spark.repl.SparkILoop.org >> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >> at org.apache.spark.repl.Main$.main(Main.scala:31) >> at org.apache.spark.repl.Main.main(Main.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> *What is wrong here? how should I configure spark to use yarn and hive >> correctly?* >> >> Eran >> >> >
