try this link
http://letstalkspark.blogspot.co.uk/2016/02/getting-started-with-spark-on-window-64.html
it helped me when i had similar problems with windows...........
hth
On Wed, Jun 7, 2017 at 3:46 PM, Curtis Burkhalter <
curtisburkhal...@gmail.com> wrote:
> Thanks Doc I saw this on another board yesterday so I've tried this by
> first going to the directory where I've stored the wintutils.exe and then
> as an admin running the command that you suggested and I get this
> exception when checking the permissions:
>
> C:\winutils\bin>winutils.exe ls -F C:\tmp\hive
> FindFileOwnerAndPermission error (1789): The trust relationship between
> this workstation and the primary domain failed.
>
> I'm fairly new to the command line and determining what the different
> exceptions mean. Do you have any advice what this error means and how I
> might go about fixing this?
>
> Thanks again
>
>
> On Wed, Jun 7, 2017 at 9:51 AM, Doc Dwarf <doc.dwar...@gmail.com> wrote:
>
>> Hi Curtis,
>>
>> I believe in windows, the following command needs to be executed: (will
>> need winutils installed)
>>
>> D:\winutils\bin\winutils.exe chmod 777 D:\tmp\hive
>>
>>
>>
>> On 6 June 2017 at 09:45, Curtis Burkhalter <curtisburkhal...@gmail.com>
>> wrote:
>>
>>> Hello all,
>>>
>>> I'm new to Spark and I'm trying to interact with it using Pyspark. I'm
>>> using the prebuilt version of spark v. 2.1.1 and when I go to the command
>>> line and use the command 'bin\pyspark' I have initialization problems and
>>> get the following message:
>>>
>>> C:\spark\spark-2.1.1-bin-hadoop2.7> bin\pyspark
>>> Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41)
>>> [MSC v.1900 64 bit (AMD64)] on win32
>>> Type "help", "copyright", "credits" or "license" for more information.
>>> Using Spark's default log4j profile: org/apache/spark/log4j-default
>>> s.properties
>>> Setting default log level to "WARN".
>>> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
>>> setLogLevel(newLevel).
>>> 17/06/06 10:30:14 WARN NativeCodeLoader: Unable to load native-hadoop
>>> library for your platform... using builtin-java classes where applicable
>>> 17/06/06 10:30:21 WARN ObjectStore: Version information not found in
>>> metastore. hive.metastore.schema.verification is not enabled so
>>> recording the schema version 1.2.0
>>> 17/06/06 10:30:21 WARN ObjectStore: Failed to get database default,
>>> returning NoSuchObjectException
>>> Traceback (most recent call last):
>>> File "C:\spark\spark-2.1.1-bin-hadoop2.7\python\pyspark\sql\utils.py",
>>> line 63, in deco
>>> return f(*a, **kw)
>>> File
>>> "C:\spark\spark-2.1.1-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py",
>>> line 319, in get_return_value
>>> py4j.protocol.Py4JJavaError: An error occurred while calling
>>> o22.sessionState.
>>> : java.lang.IllegalArgumentException: Error while instantiating
>>> 'org.apache.spark.sql.hive.HiveSessionState':
>>> at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$Spar
>>> kSession$$reflect(SparkSession.scala:981)
>>> at org.apache.spark.sql.SparkSession.sessionState$lzycompute(Sp
>>> arkSession.scala:110)
>>> at org.apache.spark.sql.SparkSession.sessionState(SparkSession.
>>> scala:109)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.jav
>>> a:357)
>>> at py4j.Gateway.invoke(Gateway.java:280)
>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.j
>>> ava:132)
>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>> at py4j.GatewayConnection.run(GatewayConnection.java:214)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>>> ConstructorAccessorImpl.java:62)
>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>>> legatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:4
>>> 23)
>>> at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$Spar
>>> kSession$$reflect(SparkSession.scala:978)
>>> ... 13 more
>>> Caused by: java.lang.IllegalArgumentException: Error while
>>> instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
>>> at org.apache.spark.sql.internal.SharedState$.org$apache$spark$
>>> sql$internal$SharedState$$reflect(SharedState.scala:169)
>>> at org.apache.spark.sql.internal.SharedState.<init>(SharedState
>>> .scala:86)
>>> at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.app
>>> ly(SparkSession.scala:101)
>>> at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.app
>>> ly(SparkSession.scala:101)
>>> at scala.Option.getOrElse(Option.scala:121)
>>> at org.apache.spark.sql.SparkSession.sharedState$lzycompute(Spa
>>> rkSession.scala:101)
>>> at org.apache.spark.sql.SparkSession.sharedState(SparkSession.s
>>> cala:100)
>>> at org.apache.spark.sql.internal.SessionState.<init>(SessionSta
>>> te.scala:157)
>>> at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessio
>>> nState.scala:32)
>>> ... 18 more
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>>> ConstructorAccessorImpl.java:62)
>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>>> legatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:4
>>> 23)
>>> at org.apache.spark.sql.internal.SharedState$.org$apache$spark$
>>> sql$internal$SharedState$$reflect(SharedState.scala:166)
>>> ... 26 more
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>>> ConstructorAccessorImpl.java:62)
>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>>> legatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:4
>>> 23)
>>> at org.apache.spark.sql.hive.client.IsolatedClientLoader.create
>>> Client(IsolatedClientLoader.scala:264)
>>> at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(Hi
>>> veUtils.scala:358)
>>> at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(Hi
>>> veUtils.scala:262)
>>> at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExt
>>> ernalCatalog.scala:66)
>>> ... 31 more
>>> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The
>>> root scratch dir: /tmp/hive on HDFS should be writable. Current permissions
>>> are: rw-rw-rw-
>>> at org.apache.hadoop.hive.ql.session.SessionState.start(Session
>>> State.java:522)
>>> at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveC
>>> lientImpl.scala:188)
>>> ... 39 more
>>> Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive
>>> on HDFS should be writable. Current permissions are: rw-rw-rw-
>>> at org.apache.hadoop.hive.ql.session.SessionState.createRootHDF
>>> SDir(SessionState.java:612)
>>> at org.apache.hadoop.hive.ql.session.SessionState.createSession
>>> Dirs(SessionState.java:554)
>>> at org.apache.hadoop.hive.ql.session.SessionState.start(Session
>>> State.java:508)
>>> ... 40 more
>>>
>>>
>>> During handling of the above exception, another exception occurred:
>>>
>>> Traceback (most recent call last):
>>> File "C:\spark\spark-2.1.1-bin-hadoop2.7\bin\..\python\pyspark\shell.py",
>>> line 43, in <module>
>>> spark = SparkSession.builder\
>>> File "C:\spark\spark-2.1.1-bin-hadoop2.7\python\pyspark\sql\session.py",
>>> line 179, in getOrCreate
>>> session._jsparkSession.sessionState().conf().setConfString(key,
>>> value)
>>> File
>>> "C:\spark\spark-2.1.1-bin-hadoop2.7\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py",
>>> line 1133, in __call__
>>> File "C:\spark\spark-2.1.1-bin-hadoop2.7\python\pyspark\sql\utils.py",
>>> line 79, in deco
>>> raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
>>> pyspark.sql.utils.IllegalArgumentException: "Error while instantiating
>>> 'org.apache.spark.sql.hive.HiveSessionState':"
>>> >>>
>>>
>>> Any help with what might be going wrong here would be greatly
>>> appreciated.
>>>
>>> Best
>>> --
>>> Curtis Burkhalter
>>> Postdoctoral Research Associate, National Audubon Society
>>>
>>> https://sites.google.com/site/curtisburkhalter/
>>>
>>
>>
>
>
> --
> Curtis Burkhalter
> Postdoctoral Research Associate, National Audubon Society
>
> https://sites.google.com/site/curtisburkhalter/
>
