I get this now. It's different than what you get
hduser@master:~/spark-1.5.1-bin-hadoop2.6/bin$ ./spark-shell
15/11/23 05:56:13 INFO spark.SecurityManager: Changing view acls to: hduser
15/11/23 05:56:13 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/11/23 05:56:13 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/11/23 05:56:13 INFO spark.HttpServer: Starting HTTP Server
15/11/23 05:56:13 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/11/23 05:56:13 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:34334
15/11/23 05:56:13 INFO util.Utils: Successfully started service 'HTTP class
server' on port 34334.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.1
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.7.0_55)
Type in expressions to have them evaluated.
Type :help for more information.
15/11/23 05:56:17 INFO spark.SparkContext: Running Spark version 1.5.1
15/11/23 05:56:17 WARN spark.SparkConf:
SPARK_JAVA_OPTS was detected (set to '-Dspark.driver.port=53411').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with conf/spark-defaults.conf to set defaults for an
application
- ./spark-submit with --driver-java-options to set -X options for a driver
- spark.executor.extraJavaOptions to set -X options for executors
- SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons
(master or worker)
15/11/23 05:56:17 WARN spark.SparkConf: Setting
'spark.executor.extraJavaOptions' to '-Dspark.driver.port=53411' as a
work-around.
15/11/23 05:56:17 WARN spark.SparkConf: Setting
'spark.driver.extraJavaOptions' to '-Dspark.driver.port=53411' as a
work-around.
15/11/23 05:56:17 INFO spark.SecurityManager: Changing view acls to: hduser
15/11/23 05:56:17 INFO spark.SecurityManager: Changing modify acls to:
hduser
15/11/23 05:56:17 INFO spark.SecurityManager: SecurityManager:
authentication disabled; ui acls disabled; users with view permissions:
Set(hduser); users with modify permissions: Set(hduser)
15/11/23 05:56:18 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/11/23 05:56:18 INFO Remoting: Starting remoting
15/11/23 05:56:18 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkDriver@192.168.7.87:53411]
15/11/23 05:56:18 INFO util.Utils: Successfully started service
'sparkDriver' on port 53411.
15/11/23 05:56:18 INFO spark.SparkEnv: Registering MapOutputTracker
15/11/23 05:56:18 INFO spark.SparkEnv: Registering BlockManagerMaster
15/11/23 05:56:18 INFO storage.DiskBlockManager: Created local directory at
/tmp/blockmgr-0232975c-c76b-444d-b7f7-1ef2f28e388c
15/11/23 05:56:18 INFO storage.MemoryStore: MemoryStore started with
capacity 530.3 MB
15/11/23 05:56:18 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-2413b536-c845-4964-a96d-973e5ec02593/httpd-311975ea-ac22-493d-8fd5-0f48b562a9a5
15/11/23 05:56:18 INFO spark.HttpServer: Starting HTTP Server
15/11/23 05:56:18 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/11/23 05:56:18 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:60477
15/11/23 05:56:18 INFO util.Utils: Successfully started service 'HTTP file
server' on port 60477.
15/11/23 05:56:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/11/23 05:56:18 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/11/23 05:56:18 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4040
15/11/23 05:56:18 INFO util.Utils: Successfully started service 'SparkUI'
on port 4040.
15/11/23 05:56:18 INFO ui.SparkUI: Started SparkUI at
http://192.168.7.87:4040
15/11/23 05:56:18 WARN metrics.MetricsSystem: Using default name
DAGScheduler for source because spark.app.id is not set.
15/11/23 05:56:18 INFO client.AppClient$ClientEndpoint: Connecting to
master spark://master:7077...
15/11/23 05:56:38 ERROR util.SparkUncaughtExceptionHandler: Uncaught
exception in thread Thread[appclient-registration-retry-thread,5,main]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@236f0e3a rejected from
java.util.concurrent.ThreadPoolExecutor@500f1402[Running, pool size = 1,
active threads = 0, queued tasks = 0, completed tasks = 1]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:96)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1.apply(AppClient.scala:95)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters(AppClient.scala:95)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deploy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1.apply$mcV$sp(AppClient.scala:132)
at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119)
at
org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient.scala:124)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15/11/23 05:56:38 INFO storage.DiskBlockManager: Shutdown hook called
15/11/23 05:56:38 INFO util.ShutdownHookManager: Shutdown hook called
15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-2413b536-c845-4964-a96d-973e5ec02593/httpd-311975ea-ac22-493d-8fd5-0f48b562a9a5
15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-8fefb39a-09b5-443c-b7b4-9c54bce6e245
15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-2413b536-c845-4964-a96d-973e5ec02593/userFiles-b593fc93-c23a-4a9e-aede-ed051f149fcb
15/11/23 05:56:38 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-2413b536-c845-4964-a96d-973e5ec02593
On Mon, Nov 23, 2015 at 4:19 PM, Mich Talebzadeh <m...@peridale.co.uk>
wrote:
> As example shows all set in hive-core.xml
>
>
>
> <property>
>
> <name>hive.execution.engine</name>
>
> *<value>spark</value>*
>
> <description>
>
> Expects one of [mr, tez, spark].
>
> Chooses execution engine. Options are: mr (Map reduce, default) or
> tez (hadoop 2 only)
>
> </description>
>
> </property>
>
>
>
> <property>
>
> <name> spark.eventLog.enabled</name>
>
> *<value>true</value>*
>
> <description>
>
> Spark event log setting
>
> </description>
>
> </property>
>
>
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Dasun Hegoda [mailto:dasunheg...@gmail.com]
> *Sent:* 23 November 2015 10:40
>
> *To:* user@hive.apache.org
> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu
>
>
>
> Thank you very much. This is very informative. Do you know how to set
> these in hive-site.xml?
>
>
>
> hive> set spark.master=<Spark Master URL>
>
> hive> set spark.eventLog.enabled=true;
>
> hive> set spark.eventLog.dir=<Spark event log folder (must exist)>
>
> hive> set spark.executor.memory=512m;
>
> hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;
>
>
>
> If these set these in hive-site I think we will be able to get through
>
>
>
> On Mon, Nov 23, 2015 at 3:05 PM, Mich Talebzadeh <m...@peridale.co.uk>
> wrote:
>
> Hi,
>
>
>
> I am looking at the set up here
>
>
>
>
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
> .
>
>
>
> First this is about configuration of Hive to work with Spark. These are my
> understanding
>
>
>
> 1. Hive uses Yarn as its resource manager regardless
>
> 2. Hive uses MapReduce as its execution engine by default
>
> 3. Changing the execution engine to that of Spark at the configuration
> level. If you look at Hive configuration file ->
> $HIVE_HOME/conf/hive-site.xml, you will see that default is mr MapReduce
>
> <property>
>
> <name>hive.execution.engine</name>
>
> *<value>mr</value>*
>
> <description>
>
> Expects one of [mr, tez].
>
> Chooses execution engine. Options are: mr (Map reduce, default) or
> tez (hadoop 2 only)
>
> </description>
>
> </property>
>
>
>
> 4. If you change that to *spark and restart Hive, *you will force Hive
> to use spark as its engine. So the choice is either do it at the
> configuration level or session level (i.e set set
> hive.execution.engine=spark;). For the rest of parameters you can do the
> same. i.e. at hive-core.xml or at session level. Personally I would still
> want hive to use MR engine so I will create spark-defaults.conf as
> mentioned.
>
> 5. I then start spark as standalone that works fine
>
> *hduser@rhes564::/usr/lib/spark> ./sbin/start-master.sh*
>
> starting org.apache.spark.deploy.master.Master, logging to
> /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
>
> hduser@rhes564::/usr/lib/spark> more
> /usr/lib/spark/sbin/../logs/spark-hduser-org.apache.spark.deploy.master.Master-1-rhes564.out
>
> Spark Command: /usr/java/latest/bin/java -cp
> /usr/lib/spark/sbin/../conf/:/usr/lib/spark/lib/spark-assembly-1.5.2-hadoop2.6.0.jar:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar:/usr/lib/spark/lib/datanucleus-ap
>
> i-jdo-3.2.6.jar:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar -Xms1g
> -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master --ip
> rhes564 --port 7077 --webui-port 8080
>
> ========================================
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> 15/11/21 21:41:58 INFO Master: Registered signal handlers for [TERM, HUP,
> INT]
>
> 15/11/21 21:41:58 WARN Utils: Your hostname, rhes564 resolves to a
> loopback address: 127.0.0.1; using 50.140.197.217 instead (on interface
> eth0)
>
> 15/11/21 21:41:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
>
> 15/11/21 21:41:59 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 15/11/21 21:41:59 INFO SecurityManager: Changing view acls to: hduser
>
> 15/11/21 21:41:59 INFO SecurityManager: Changing modify acls to: hduser
>
> 15/11/21 21:41:59 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(hduser); users
> with modify permissions: Set(hduser)
>
> 15/11/21 21:41:59 INFO Slf4jLogger: Slf4jLogger started
>
> 15/11/21 21:42:00 INFO Remoting: Starting remoting
>
> 15/11/21 21:42:00 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkMaster@rhes564:7077]
>
> 15/11/21 21:42:00 INFO Utils: Successfully started service 'sparkMaster'
> on port 7077.
>
> 15/11/21 21:42:00 INFO Master: Starting Spark master at
> spark://rhes564:7077
>
> 15/11/21 21:42:00 INFO Master: Running Spark version 1.5.2
>
> 15/11/21 21:42:00 INFO Utils: Successfully started service 'MasterUI' on
> port 8080.
>
> 15/11/21 21:42:00 INFO MasterWebUI: Started MasterWebUI at
> http://50.140.197.217:8080
>
> 15/11/21 21:42:00 INFO Utils: Successfully started service on port 6066.
>
> 15/11/21 21:42:00 INFO StandaloneRestServer: Started REST server for
> submitting applications on port 6066
>
> 15/11/21 21:42:00 INFO Master: I have been elected leader! New state: ALIVE
>
> 6. Then I try to start interactive spark-shell and it fails with an
> error that I reported before
>
> *hduser@rhes564::/usr/lib/spark/bin> ./spark-shell --master
> spark://rhes564:7077*
>
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> Using Spark's repl log4j profile:
> org/apache/spark/log4j-defaults-repl.properties
>
> To adjust logging level use sc.setLogLevel("INFO")
>
> Welcome to
>
> ____ __
>
> / __/__ ___ _____/ /__
>
> _\ \/ _ \/ _ `/ __/ '_/
>
> /___/ .__/\_,_/_/ /_/\_\ version 1.5.2
>
> /_/
>
>
>
> Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.7.0_25)
>
> Type in expressions to have them evaluated.
>
> Type :help for more information.
>
> 15/11/23 09:33:56 WARN Utils: Your hostname, rhes564 resolves to a
> loopback address: 127.0.0.1; using 50.140.197.217 instead (on interface
> eth0)
>
> 15/11/23 09:33:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
>
> 15/11/23 09:33:57 WARN MetricsSystem: Using default name DAGScheduler for
> source because spark.app.id is not set.
>
> Spark context available as sc.
>
> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name
> hive.server2.thrift.http.min.worker.threads does not exist
>
> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name
> hive.mapjoin.optimized.keys does not exist
>
> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name
> hive.mapjoin.lazy.hashtable does not exist
>
> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name
> hive.server2.thrift.http.max.worker.threads does not exist
>
> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name
> hive.server2.logging.operation.verbose does not exist
>
> 15/11/23 09:34:00 WARN HiveConf: HiveConf of name
> hive.optimize.multigroupby.common.distincts does not exist
>
> *java.lang.RuntimeException: java.lang.RuntimeException: The root scratch
> dir: /tmp/hive on HDFS should be writable. Current permissions are:
> rwx------*
>
>
>
> That is where I am now and I have reported this spark user group but no
> luck yet.
>
>
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Dasun Hegoda [mailto:dasunheg...@gmail.com]
> *Sent:* 23 November 2015 07:05
> *To:* user@hive.apache.org
> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu
>
>
>
> Anyone????
>
>
>
> On Sat, Nov 21, 2015 at 1:32 PM, Dasun Hegoda <dasunheg...@gmail.com>
> wrote:
>
> Thank you very much but I would like to do the integration of these
> components myself rather than using a packaged distribution. I think I have
> come to right place. Can you please kindly tell me the configuration
> steps run Hive on Spark?
>
>
>
> At least someone please elaborate these steps.
>
>
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
> .
>
>
>
> Because at the latter part of the guide configurations are set in the
> Hive runtime shell which is not permanent according to my knowledge.
>
>
>
> Please help me to get this done. Also I'm planning write a detailed guide
> with configuration steps to run Hive on Spark. So others can benefited from
> it and not troubled like me.
>
>
>
> Can someone please kindly tell me the configuration steps run Hive on
> Spark?
>
>
>
>
>
> On Sat, Nov 21, 2015 at 12:28 PM, Sai Gopalakrishnan <
> sai.gopalakrish...@aspiresys.com> wrote:
>
> Hi everyone,
>
>
>
> Thank you for your responses. I think Mich's suggestion is a great one,
> will go with it. As Alan suggested, using compactor in Hive should help out
> with managing the delta files.
>
>
>
> @Dasun, pardon me for deviating from the topic. Regarding configuration,
> you could try a packaged distribution (Hortonworks , Cloudera or MapR)
> like Jörn Franke said. I use Hortonworks, its open-source and compatible
> with Linux and Windows, provides detailed documentation for installation
> and can be installed in less than a day provided you're all set with the
> hardware. http://hortonworks.com/hdp/downloads/
>
> [image: Image removed by sender.] <http://hortonworks.com/hdp/downloads/>
>
> Download Hadoop - Hortonworks
>
> Download Apache Hadoop for the enterprise with Hortonworks Data Platform.
> Data access, storage, governance, security and operations across Linux and
> Windows
>
> Read more... <http://hortonworks.com/hdp/downloads/>
>
>
>
>
>
> Regards,
>
> Sai
>
>
> ------------------------------
>
> *From:* Dasun Hegoda <dasunheg...@gmail.com>
> *Sent:* Saturday, November 21, 2015 8:00 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Hive on Spark - Hadoop 2 - Installation - Ubuntu
>
>
>
> Hi Mich, Hi Sai, Hi Jorn,
>
> Thank you very much for the information. I think we are deviating from the
> original question. Hive on Spark on Ubuntu. Can you please kindly tell me
> the configuration steps?
>
>
>
>
>
>
>
> On Fri, Nov 20, 2015 at 11:10 PM, Jörn Franke <jornfra...@gmail.com>
> wrote:
>
> I think the most recent versions of cloudera or Hortonworks should include
> all these components - try their Sandboxes.
>
>
> On 20 Nov 2015, at 12:54, Dasun Hegoda <dasunheg...@gmail.com> wrote:
>
> Where can I get a Hadoop distribution containing these technologies? Link?
>
>
>
> On Fri, Nov 20, 2015 at 5:22 PM, Jörn Franke <jornfra...@gmail.com> wrote:
>
> I recommend to use a Hadoop distribution containing these technologies. I
> think you get also other useful tools for your scenario, such as Auditing
> using sentry or ranger.
>
>
> On 20 Nov 2015, at 10:48, Mich Talebzadeh <m...@peridale.co.uk> wrote:
>
> Well
>
>
>
> “I'm planning to deploy Hive on Spark but I can't find the installation
> steps. I tried to read the official '[Hive on Spark][1]' guide but it has
> problems. As an example it says under 'Configuring Yarn'
> `yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`
> but does not imply where should I do it. Also as per the guide
> configurations are set in the Hive runtime shell which is not permanent
> according to my knowledge.”
>
>
>
> You can do that in yarn-site.xml file which is normally under
> $HADOOP_HOME/etc/hadoop.
>
>
>
>
>
> HTH
>
>
>
>
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4,
> volume one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Dasun Hegoda [mailto:dasunheg...@gmail.com <dasunheg...@gmail.com>]
>
> *Sent:* 20 November 2015 09:36
> *To:* user@hive.apache.org
> *Subject:* Hive on Spark - Hadoop 2 - Installation - Ubuntu
>
>
>
> Hi,
>
>
>
> What I'm planning to do is develop a reporting platform using existing
> data. I have an existing RDBMS which has large number of records. So I'm
> using. (
> http://stackoverflow.com/questions/33635234/hadoop-2-7-spark-hive-jasperreports-scoop-architecuture
> )
>
>
>
> - Scoop - Extract data from RDBMS to Hadoop
>
> - Hadoop - Storage platform -> *Deployment Completed*
>
> - Hive - Datawarehouse
>
> - Spark - Read time processing -> *Deployment Completed*
>
>
>
> I'm planning to deploy Hive on Spark but I can't find the installation
> steps. I tried to read the official '[Hive on Spark][1]' guide but it has
> problems. As an example it says under 'Configuring Yarn'
> `yarn.resourcemanager.scheduler.class=org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`
> but does not imply where should I do it. Also as per the guide
> configurations are set in the Hive runtime shell which is not permanent
> according to my knowledge.
>
>
>
> Given that I read [this][2] but it does not have any steps.
>
>
>
> Please provide me the steps to run Hive on Spark on Ubuntu as a production
> system?
>
>
>
>
>
> [1]:
> https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
>
> [2]:
> http://stackoverflow.com/questions/26018306/how-to-configure-hive-to-use-spark
>
>
>
> --
>
> Regards,
>
> Dasun Hegoda, Software Engineer
> www.dasunhegoda.com | dasunheg...@gmail.com
>
>
>
>
>
> --
>
> Regards,
>
> Dasun Hegoda, Software Engineer
> www.dasunhegoda.com | dasunheg...@gmail.com
>
>
>
>
>
> --
>
> Regards,
>
> Dasun Hegoda, Software Engineer
> www.dasunhegoda.com | dasunheg...@gmail.com
>
> [image: Image removed by sender. Aspire Systems]
>
> This e-mail message and any attachments are for the sole use of the
> intended recipient(s) and may contain proprietary, confidential, trade
> secret or privileged information. Any unauthorized review, use, disclosure
> or distribution is prohibited and may be a violation of law. If you are not
> the intended recipient, please contact the sender by reply e-mail and
> destroy all copies of the original message.
>
>
>
>
>
> --
>
> Regards,
>
> Dasun Hegoda, Software Engineer
> www.dasunhegoda.com | dasunheg...@gmail.com
>
>
>
>
>
> --
>
> Regards,
>
> Dasun Hegoda, Software Engineer
> www.dasunhegoda.com | dasunheg...@gmail.com
>
>
>
>
>
> --
>
> Regards,
>
> Dasun Hegoda, Software Engineer
> www.dasunhegoda.com | dasunheg...@gmail.com
>
--
Regards,
Dasun Hegoda, Software Engineer
www.dasunhegoda.com | dasunheg...@gmail.com
