I do, and this is what I will fall back to if nobody has a better idea :)
I was just hoping to get this working as it is much more convenient for my
testing pipeline.
Thanks again for the help
> On Apr 13, 2018, at 11:33 AM, Geoff Von Allmen <ge...@ibleducation.com> wrote:
>
> Ok - `LOCAL` makes sense now.
>
> Do you have the option to still use `spark-submit` in this scenario, but
> using the following options:
>
> ```bash
> --master "local[*]" \
> --deploy-mode "client" \
> ...
> ```
>
> I know in the past, I have setup some options using `.config("Option",
> "value")` when creating the spark session, and then other runtime options as
> you describe above with `spark.conf.set`. At this point though I've just
> moved everything out into a `spark-submit` script.
>
> On Fri, Apr 13, 2018 at 8:18 AM, Jason Boorn <jbo...@gmail.com
> <mailto:jbo...@gmail.com>> wrote:
> Hi Geoff -
>
> Appreciate the help here - I do understand what you’re saying below. And I
> am able to get this working when I submit a job to a local cluster.
>
> I think part of the issue here is that there’s ambiguity in the terminology.
> When I say “LOCAL” spark, I mean an instance of spark that is created by my
> driver program, and is not a cluster itself. It means that my master node is
> “local”, and this mode is primarily used for testing.
>
> https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-local.html
>
> <https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-local.html>
>
> While I am able to get alluxio working with spark-submit, I am unable to get
> it working when using local mode. The mechanisms for setting class paths
> during spark-submit are not available in local mode. My understanding is
> that all one is able to use is:
>
> spark.conf.set(“”)
>
> To set any runtime properties of the local instance. Note that it is
> possible (and I am more convinced of this as time goes on) that alluxio
> simply does not work in spark local mode as described above.
>
>
>> On Apr 13, 2018, at 11:09 AM, Geoff Von Allmen <ge...@ibleducation.com
>> <mailto:ge...@ibleducation.com>> wrote:
>>
>> I fought with a ClassNotFoundException for quite some time, but it was for
>> kafka.
>>
>> The final configuration that got everything working was running spark-submit
>> with the following options:
>>
>> --jars "/path/to/.ivy2/jars/package.jar" \
>> --driver-class-path "/path/to/.ivy2/jars/package.jar" \
>> --conf "spark.executor.extraClassPath=/path/to/.ivy2/package.jar" \
>> --packages org.some.package:package_name:version
>> While this was needed for me to run in cluster mode, it works equally well
>> for client mode as well.
>>
>> One other note when needing to supplied multiple items to these args -
>> --jars and --packages should be comma separated, --driver-class-path and
>> extraClassPath should be : separated
>>
>> HTH
>>
>>
>> On Fri, Apr 13, 2018 at 4:28 AM, jb44 <jbo...@gmail.com
>> <mailto:jbo...@gmail.com>> wrote:
>> Haoyuan -
>>
>> As I mentioned below, I've been through the documentation already. It has
>> not helped me to resolve the issue.
>>
>> Here is what I have tried so far:
>>
>> - setting extraClassPath as explained below
>> - adding fs.alluxio.impl through sparkconf
>> - adding spark.sql.hive.metastore.sharedPrefixes (though I don't believe
>> this matters in my case)
>> - compiling the client from source
>>
>> Do you have any other suggestions on how to get this working?
>>
>> Thanks
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>> <http://apache-spark-user-list.1001560.n3.nabble.com/>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>> <mailto:user-unsubscr...@spark.apache.org>
>>
>>
>
>
