Do you have any clue how to get his fixed? On Mon, Nov 23, 2015 at 4:27 PM, Dasun Hegoda <dasunheg...@gmail.com> wrote:
> I get this now. It's different than what you get > > hduser@master:~/spark-1.5.1-bin-hadoop2.6/bin$ ./spark-shell > 15/11/23 05:56:13 INFO spark.SecurityManager: Changing view acls to: hduser > 15/11/23 05:56:13 INFO spark.SecurityManager: Changing modify acls to: > hduser > 15/11/23 05:56:13 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > 15/11/23 05:56:13 INFO spark.HttpServer: Starting HTTP Server > 15/11/23 05:56:13 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/11/23 05:56:13 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:34334 > 15/11/23 05:56:13 INFO util.Utils: Successfully started service 'HTTP > class server' on port 34334. > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 1.5.1 > /_/ > > Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java > 1.7.0_55) > Type in expressions to have them evaluated. > Type :help for more information. > 15/11/23 05:56:17 INFO spark.SparkContext: Running Spark version 1.5.1 > 15/11/23 05:56:17 WARN spark.SparkConf: > SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411'). > This is deprecated in Spark 1.0+. > > Please instead use: > - ./spark-submit with conf/spark-defaults.conf to set defaults for an > application > - ./spark-submit with --driver-java-options to set -X options for a driver > - spark.executor.extraJavaOptions to set -X options for executors > - SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons > (master or worker) > > 15/11/23 05:56:17 WARN spark.SparkConf: Setting > 'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a > work-around. > 15/11/23 05:56:17 WARN spark.SparkConf: Setting > 'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a > work-around. > 15/11/23 05:56:17 INFO spark.SecurityManager: Changing view acls to: hduser > 15/11/23 05:56:17 INFO spark.SecurityManager: Changing modify acls to: > hduser > 15/11/23 05:56:17 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(hduser); users with modify permissions: Set(hduser) > 15/11/23 05:56:18 INFO slf4j.Slf4jLogger: Slf4jLogger started > 15/11/23 05:56:18 INFO Remoting: Starting remoting > 15/11/23 05:56:18 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkDriver@192.168.7.87:53411] > 15/11/23 05:56:18 INFO util.Utils: Successfully started service > 'sparkDriver' on port 53411. > 15/11/23 05:56:18 INFO spark.SparkEnv: Registering MapOutputTracker > 15/11/23 05:56:18 INFO spark.SparkEnv: Registering BlockManagerMaster > 15/11/23 05:56:18 INFO storage.DiskBlockManager: Created local directory > at /tmp/blockmgr-0232975c-c76b-444d-b7f7-1ef2f28e388c > 15/11/23 05:56:18 INFO storage.MemoryStore: MemoryStore started with > capacity 530.3 MB > 15/11/23 05:56:18 INFO spark.HttpFileServer: HTTP File server directory is > /tmp/spark-2413b536-c845-4964-a96d-973e5ec02593/httpd-311975ea-ac22-493d-8fd5-0f48b562a9a5 > 15/11/23 05:56:18 INFO spark.HttpServer: Starting HTTP Server > 15/11/23 05:56:18 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/11/23 05:56:18 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:60477 > 15/11/23 05:56:18 INFO util.Utils: Successfully started service 'HTTP file > server' on port 60477. > 15/11/23 05:56:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator > 15/11/23 05:56:18 INFO server.Server: jetty-8.y.z-SNAPSHOT > 15/11/23 05:56:18 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 15/11/23 05:56:18 INFO util.Utils: Successfully started service 'SparkUI' > on port 4040. > 15/11/23 05:56:18 INFO ui.SparkUI: Started SparkUI at > http://192.168.7.87:4040 > 15/11/23 05:56:18 WARN metrics.MetricsSystem: Using default name > DAGScheduler for source because spark.app.id is not set. > 15/11/23 05:56:18 INFO client.AppClient$ClientEndpoint: Connecting to > master spark://master:7077... > 15/11/23 05:56:38 ERROR util.SparkUncaughtExceptionHandler: Uncaught > exception in thread Thread[appclient-registration-retry-thread,5,main] > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.FutureTask@236f0e3a rejected from > java.util.concurrent.ThreadPoolExecutor@500f1402[Running, pool size = 1, > active threads = 0, queued tasks = 0, completed tasks = 1] > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96) > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:95) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(AppClient.scala:95) > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121) > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(AppClient.scala:132) > at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119) > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.scala:124) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 15/11/23 05:56:38 INFO storage.DiskBlockManager: Shutdown hook called > 15/11/23 05:56:38 INFO util.ShutdownHookManager: Shutdown hook called > 15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-2413b536-c845-4964-a96d-973e5ec02593/httpd-311975ea-ac22-493d-8fd5-0f48b562a9a5 > 15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-8fefb39a-09b5-443c-b7b4-9c54bce6e245 > 15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-2413b536-c845-4964-a96d-973e5ec02593/userFiles-b593fc93-c23a-4a9e-aede-ed051f149fcb > 15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory > /tmp/spark-2413b536-c845-4964-a96d-973e5ec02593 > > On Mon, Nov 23, 2015 at 4:19 PM, Mich Talebzadeh <m...@peridale.co.uk> > wrote: > >> As example shows all set in hive-core.xml >> >> >> >> <property> >> >> <name>hive.execution.engine</name> >> >> *<value>spark</value>* >> >> <description> >> >> Expects one of [mr, tez, spark]. >> >> Chooses execution engine. Options are: mr (Map reduce, default) or >> tez (hadoop 2 only) >> >> </description> >> >> </property> >> >> >> >> <property> >> >> <name> spark.eventLog.enabled</name> >> >> *<value>true</value>* >> >> <description> >> >> Spark event log setting >> >> </description> >> >> </property> >> >> >> >> >> >> Mich Talebzadeh >> >> >> >> *Sybase ASE 15 Gold Medal Award 2008* >> >> A Winning Strategy: Running the most Critical Financial Data on ASE 15 >> >> >> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf >> >> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE >> 15", ISBN 978-0-9563693-0-7*. >> >> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >> 978-0-9759693-0-4* >> >> *Publications due shortly:* >> >> *Complex Event Processing in Heterogeneous Environments*, ISBN: >> 978-0-9563693-3-8 >> >> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume >> one out shortly >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> NOTE: The information in this email is proprietary and confidential. This >> message is for the designated recipient only, if you are not the intended >> recipient, you should destroy it immediately. Any information in this >> message shall not be understood as given or endorsed by Peridale Technology >> Ltd, its subsidiaries or their employees, unless expressly so stated. It is >> the responsibility of the recipient to ensure that this email is virus >> free, therefore neither Peridale Ltd, its subsidiaries nor their employees >> accept any responsibility. >> >> >> >> *From:* Dasun Hegoda [mailto:dasunheg...@gmail.com] >> *Sent:* 23 November 2015 10:40 >> >> *To:* user@hive.apache.org >> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu >> >> >> >> Thank you very much. This is very informative. Do you know how to set >> these in hive-site.xml? >> >> >> >> hive> set spark.master=<Spark Master URL> >> >> hive> set spark.eventLog.enabled=true; >> >> hive> set spark.eventLog.dir=<Spark event log folder (must exist)> >> >> hive> set spark.executor.memory=512m; >> >> hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer; >> >> >> >> If these set these in hive-site I think we will be able to get through >> >> >> >> On Mon, Nov 23, 2015 at 3:05 PM, Mich Talebzadeh <m...@peridale.co.uk> >> wrote: >> >> Hi, >> >> >> >> I am looking at the set up here >> >> >> >> >> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started >> . >> >> >> >> First this is about configuration of Hive to work with Spark. These are >> my understanding >> >> >> >> 1. Hive uses Yarn as its resource manager regardless >> >> 2. Hive uses MapReduce as its execution engine by default >> >> 3. Changing the execution engine to that of Spark at the >> configuration level. If you look at Hive configuration file -> >> $HIVE_HOME/conf/hive-site.xml, you will see that default is mr MapReduce >> >> <property> >> >> <name>hive.execution.engine</name> >> >> *<value>mr</value>* >> >> <description> >> >> Expects one of [mr, tez]. >> >> Chooses execution engine. Options are: mr (Map reduce, default) or >> tez (hadoop 2 only) >> >> </description> >> >> </property> >> >> >> >> 4. If you change that to *spark and restart Hive, *you will force >> Hive to use spark as its engine. So the choice is either do it at the >> configuration level or session level (i.e set set >> hive.execution.engine=spark;). For the rest of parameters you can do the >> same. i.e. at hive-core.xml or at session level. Personally I would still >> want hive to use MR engine so I will create spark-defaults.conf as >> mentioned. >> >> 5. I then start spark as standalone that works fine >> >> *hduser@rhes564::/usr/lib/spark> ./sbin/start-master.sh* >> >> starting org.apache.spark.deploy.master.Master, logging to >> /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out >> >> hduser@rhes564::/usr/lib/spark> more >> /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out >> >> Spark Command: /usr/java/latest/bin/java -cp >> /usr/lib/spark/sbin/../conf/:/usr/lib/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark/lib/datanucleus-ap >> >> i-jdo-3.2.6.jar:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar -Xms1g >> -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip >> rhes564 --port 7077 --webui-port 8080 >> >> ======================================== >> >> Using Spark's default log4j profile: >> org/apache/spark/log4j-defaults.properties >> >> 15/11/21 21:41:58 INFO Master: Registered signal handlers for [TERM, HUP, >> INT] >> >> 15/11/21 21:41:58 WARN Utils: Your hostname, rhes564 resolves to a >> loopback address: 127.0.0.1; using 50.140.197.217 instead (on interface >> eth0) >> >> 15/11/21 21:41:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to >> another address >> >> 15/11/21 21:41:59 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> >> 15/11/21 21:41:59 INFO SecurityManager: Changing view acls to: hduser >> >> 15/11/21 21:41:59 INFO SecurityManager: Changing modify acls to: hduser >> >> 15/11/21 21:41:59 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(hduser); users >> with modify permissions: Set(hduser) >> >> 15/11/21 21:41:59 INFO Slf4jLogger: Slf4jLogger started >> >> 15/11/21 21:42:00 INFO Remoting: Starting remoting >> >> 15/11/21 21:42:00 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://sparkMaster@rhes564:7077] >> >> 15/11/21 21:42:00 INFO Utils: Successfully started service 'sparkMaster' >> on port 7077. >> >> 15/11/21 21:42:00 INFO Master: Starting Spark master at >> spark://rhes564:7077 >> >> 15/11/21 21:42:00 INFO Master: Running Spark version 1.5.2 >> >> 15/11/21 21:42:00 INFO Utils: Successfully started service 'MasterUI' on >> port 8080. >> >> 15/11/21 21:42:00 INFO MasterWebUI: Started MasterWebUI at >> http://50.140.197.217:8080 >> >> 15/11/21 21:42:00 INFO Utils: Successfully started service on port 6066. >> >> 15/11/21 21:42:00 INFO StandaloneRestServer: Started REST server for >> submitting applications on port 6066 >> >> 15/11/21 21:42:00 INFO Master: I have been elected leader! New state: >> ALIVE >> >> 6. Then I try to start interactive spark-shell and it fails with an >> error that I reported before >> >> *hduser@rhes564::/usr/lib/spark/bin> ./spark-shell --master >> spark://rhes564:7077* >> >> log4j:WARN No appenders could be found for logger >> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >> >> log4j:WARN Please initialize the log4j system properly. >> >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for >> more info. >> >> Using Spark's repl log4j profile: >> org/apache/spark/log4j-defaults-repl.properties >> >> To adjust logging level use sc.setLogLevel("INFO") >> >> Welcome to >> >> ____ __ >> >> / __/__ ___ _____/ /__ >> >> _\ \/ _ \/ _ `/ __/ '_/ >> >> /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 >> >> /_/ >> >> >> >> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java >> 1.7.0_25) >> >> Type in expressions to have them evaluated. >> >> Type :help for more information. >> >> 15/11/23 09:33:56 WARN Utils: Your hostname, rhes564 resolves to a >> loopback address: 127.0.0.1; using 50.140.197.217 instead (on interface >> eth0) >> >> 15/11/23 09:33:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to >> another address >> >> 15/11/23 09:33:57 WARN MetricsSystem: Using default name DAGScheduler for >> source because spark.app.id is not set. >> >> Spark context available as sc. >> >> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name >> hive.server2.thrift.http.min.worker.threads does not exist >> >> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name >> hive.mapjoin.optimized.keys does not exist >> >> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name >> hive.mapjoin.lazy.hashtable does not exist >> >> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name >> hive.server2.thrift.http.max.worker.threads does not exist >> >> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name >> hive.server2.logging.operation.verbose does not exist >> >> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name >> hive.optimize.multigroupby.common.distincts does not exist >> >> *java.lang.RuntimeException: java.lang.RuntimeException: The root scratch >> dir: /tmp/hive on HDFS should be writable. Current permissions are: >> rwx------* >> >> >> >> That is where I am now and I have reported this spark user group but no >> luck yet. >> >> >> >> >> >> Mich Talebzadeh >> >> >> >> *Sybase ASE 15 Gold Medal Award 2008* >> >> A Winning Strategy: Running the most Critical Financial Data on ASE 15 >> >> >> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf >> >> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE >> 15", ISBN 978-0-9563693-0-7*. >> >> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >> 978-0-9759693-0-4* >> >> *Publications due shortly:* >> >> *Complex Event Processing in Heterogeneous Environments*, ISBN: >> 978-0-9563693-3-8 >> >> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume >> one out shortly >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> NOTE: The information in this email is proprietary and confidential. This >> message is for the designated recipient only, if you are not the intended >> recipient, you should destroy it immediately. Any information in this >> message shall not be understood as given or endorsed by Peridale Technology >> Ltd, its subsidiaries or their employees, unless expressly so stated. It is >> the responsibility of the recipient to ensure that this email is virus >> free, therefore neither Peridale Ltd, its subsidiaries nor their employees >> accept any responsibility. >> >> >> >> *From:* Dasun Hegoda [mailto:dasunheg...@gmail.com] >> *Sent:* 23 November 2015 07:05 >> *To:* user@hive.apache.org >> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu >> >> >> >> Anyone???? >> >> >> >> On Sat, Nov 21, 2015 at 1:32 PM, Dasun Hegoda <dasunheg...@gmail.com> >> wrote: >> >> Thank you very much but I would like to do the integration of these >> components myself rather than using a packaged distribution. I think I have >> come to right place. Can you please kindly tell me the configuration >> steps run Hive on Spark? >> >> >> >> At least someone please elaborate these steps. >> >> >> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started >> . >> >> >> >> Because at the latter part of the guide configurations are set in the >> Hive runtime shell which is not permanent according to my knowledge. >> >> >> >> Please help me to get this done. Also I'm planning write a detailed guide >> with configuration steps to run Hive on Spark. So others can benefited from >> it and not troubled like me. >> >> >> >> Can someone please kindly tell me the configuration steps run Hive on >> Spark? >> >> >> >> >> >> On Sat, Nov 21, 2015 at 12:28 PM, Sai Gopalakrishnan < >> sai.gopalakrish...@aspiresys.com> wrote: >> >> Hi everyone, >> >> >> >> Thank you for your responses. I think Mich's suggestion is a great one, >> will go with it. As Alan suggested, using compactor in Hive should help out >> with managing the delta files. >> >> >> >> @Dasun, pardon me for deviating from the topic. Regarding configuration, >> you could try a packaged distribution (Hortonworks , Cloudera or MapR) >> like Jörn Franke said. I use Hortonworks, its open-source and compatible >> with Linux and Windows, provides detailed documentation for installation >> and can be installed in less than a day provided you're all set with the >> hardware. http://hortonworks.com/hdp/downloads/ >> >> [image: Image removed by sender.] <http://hortonworks.com/hdp/downloads/> >> >> Download Hadoop - Hortonworks >> >> Download Apache Hadoop for the enterprise with Hortonworks Data Platform. >> Data access, storage, governance, security and operations across Linux and >> Windows >> >> Read more... <http://hortonworks.com/hdp/downloads/> >> >> >> >> >> >> Regards, >> >> Sai >> >> >> ------------------------------ >> >> *From:* Dasun Hegoda <dasunheg...@gmail.com> >> *Sent:* Saturday, November 21, 2015 8:00 AM >> *To:* user@hive.apache.org >> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu >> >> >> >> Hi Mich, Hi Sai, Hi Jorn, >> >> Thank you very much for the information. I think we are deviating from >> the original question. Hive on Spark on Ubuntu. Can you please kindly tell >> me the configuration steps? >> >> >> >> >> >> >> >> On Fri, Nov 20, 2015 at 11:10 PM, Jörn Franke <jornfra...@gmail.com> >> wrote: >> >> I think the most recent versions of cloudera or Hortonworks should >> include all these components - try their Sandboxes. >> >> >> On 20 Nov 2015, at 12:54, Dasun Hegoda <dasunheg...@gmail.com> wrote: >> >> Where can I get a Hadoop distribution containing these technologies? >> Link? >> >> >> >> On Fri, Nov 20, 2015 at 5:22 PM, Jörn Franke <jornfra...@gmail.com> >> wrote: >> >> I recommend to use a Hadoop distribution containing these technologies. I >> think you get also other useful tools for your scenario, such as Auditing >> using sentry or ranger. >> >> >> On 20 Nov 2015, at 10:48, Mich Talebzadeh <m...@peridale.co.uk> wrote: >> >> Well >> >> >> >> “I'm planning to deploy Hive on Spark but I can't find the installation >> steps. I tried to read the official '[Hive on Spark][1]' guide but it has >> problems. As an example it says under 'Configuring Yarn' >> `yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` >> but does not imply where should I do it. Also as per the guide >> configurations are set in the Hive runtime shell which is not permanent >> according to my knowledge.” >> >> >> >> You can do that in yarn-site.xml file which is normally under >> $HADOOP_HOME/etc/hadoop. >> >> >> >> >> >> HTH >> >> >> >> >> >> >> >> Mich Talebzadeh >> >> >> >> *Sybase ASE 15 Gold Medal Award 2008* >> >> A Winning Strategy: Running the most Critical Financial Data on ASE 15 >> >> >> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf >> >> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE >> 15", ISBN 978-0-9563693-0-7*. >> >> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >> 978-0-9759693-0-4* >> >> *Publications due shortly:* >> >> *Complex Event Processing in Heterogeneous Environments*, ISBN: >> 978-0-9563693-3-8 >> >> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, >> volume one out shortly >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> NOTE: The information in this email is proprietary and confidential. This >> message is for the designated recipient only, if you are not the intended >> recipient, you should destroy it immediately. Any information in this >> message shall not be understood as given or endorsed by Peridale Technology >> Ltd, its subsidiaries or their employees, unless expressly so stated. It is >> the responsibility of the recipient to ensure that this email is virus >> free, therefore neither Peridale Ltd, its subsidiaries nor their employees >> accept any responsibility. >> >> >> >> *From:* Dasun Hegoda [mailto:dasunheg...@gmail.com >> <dasunheg...@gmail.com>] >> *Sent:* 20 November 2015 09:36 >> *To:* user@hive.apache.org >> *Subject:* Hive on Spark - Hadoop 2 - Installation - Ubuntu >> >> >> >> Hi, >> >> >> >> What I'm planning to do is develop a reporting platform using existing >> data. I have an existing RDBMS which has large number of records. So I'm >> using. ( >> http://stackoverflow.com/questions/33635234/hadoop-2-7-spark-hive-jasperreports-scoop-architecuture >> ) >> >> >> >> - Scoop - Extract data from RDBMS to Hadoop >> >> - Hadoop - Storage platform -> *Deployment Completed* >> >> - Hive - Datawarehouse >> >> - Spark - Read time processing -> *Deployment Completed* >> >> >> >> I'm planning to deploy Hive on Spark but I can't find the installation >> steps. I tried to read the official '[Hive on Spark][1]' guide but it has >> problems. As an example it says under 'Configuring Yarn' >> `yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` >> but does not imply where should I do it. Also as per the guide >> configurations are set in the Hive runtime shell which is not permanent >> according to my knowledge. >> >> >> >> Given that I read [this][2] but it does not have any steps. >> >> >> >> Please provide me the steps to run Hive on Spark on Ubuntu as a >> production system? >> >> >> >> >> >> [1]: >> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started >> >> [2]: >> http://stackoverflow.com/questions/26018306/how-to-configure-hive-to-use-spark >> >> >> >> -- >> >> Regards, >> >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunheg...@gmail.com >> >> >> >> >> >> -- >> >> Regards, >> >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunheg...@gmail.com >> >> >> >> >> >> -- >> >> Regards, >> >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunheg...@gmail.com >> >> [image: Image removed by sender. Aspire Systems] >> >> This e-mail message and any attachments are for the sole use of the >> intended recipient(s) and may contain proprietary, confidential, trade >> secret or privileged information. Any unauthorized review, use, disclosure >> or distribution is prohibited and may be a violation of law. If you are not >> the intended recipient, please contact the sender by reply e-mail and >> destroy all copies of the original message. >> >> >> >> >> >> -- >> >> Regards, >> >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunheg...@gmail.com >> >> >> >> >> >> -- >> >> Regards, >> >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunheg...@gmail.com >> >> >> >> >> >> -- >> >> Regards, >> >> Dasun Hegoda, Software Engineer >> www.dasunhegoda.com | dasunheg...@gmail.com >> > > > > -- > Regards, > Dasun Hegoda, Software Engineer > www.dasunhegoda.com | dasunheg...@gmail.com > -- Regards, Dasun Hegoda, Software Engineer www.dasunhegoda.com | dasunheg...@gmail.com
