The --jars flag should be available for PySpark as well (I could be wrong,
I've only used Spark 1.4 onward). Take, for example, the command I'm using
to stark a PySpark shell for a Jupyter Notebook:
"--jars hdfs://{our namenode}/tmp/postgresql-9.4-1204.jdbc42.jar
--driver-class-path
/usr/local/share/jupyter/kernels/postgres/postgresql-9.4-1204.jdbc42.jar
--executor-memory 1G --total-executor-cores 15 pyspark-shell"
James
On Fri, Nov 13, 2015 at 12:12 PM Krishna Sangeeth KS <
kskrishnasange...@gmail.com> wrote:
>
> Hi,
>
> I have been trying to do this today at work with impala as the data source
> . I have been getting the same error as well.
>
> I am using PySpark APIs with Spark 1.3 version and I was wondering if
> there is any workaround for Pyspark. I don't think we can use --jars option
> in PySpark.
>
> Cheers,
> Sangeeth
>
>
> On Fri, Nov 13, 2015, 9:04 PM James Nowell <jnowell...@gmail.com> wrote:
>
>> I recently had this same issue. Though I didn't find the cause, I was
>> able to work around it by loading the JAR into hdfs. Once in HDFS, I used
>> the --jars flag with the full hdfs path: --jars hdfs://{our
>> namenode}/tmp/postgresql-9.4-1204-jdbc42.jar
>>
>> James
>>
>> On Fri, Nov 13, 2015 at 10:14 AM satish chandra j <
>> jsatishchan...@gmail.com> wrote:
>>
>>> HI All,
>>> Currently using Spark 1.4.1, my Spark job has to fetche data from
>>> PostgreSQL database using JdbcRDD
>>> I am submitting my spark job using --jars to pass PostgreSQL JDBC driver
>>> but still getting error as mentioned below:
>>>
>>> "java.sql.SQLException: No suitable driver found for PostgreSQL JDBC"
>>>
>>> when the same is given through Spark Shell it is working fine
>>>
>>> In several blogs it is mentioned that it is fixed in Spark 1.4.1 by just
>>> passing JDBC Driver through --jars option but still i am stuck
>>>
>>> I have tried below options:
>>>
>>>
>>> 1. SPARK_CLASSPATH= /path/postgresql.jar in
>>> spark/conf/spark-defaults.conf
>>>
>>> 2. -driver-class-path /path/postgresql.jar and -conf
>>> spark.executor.extraClassPath = /path/postgreSQL.jar
>>>
>>> 3. --jars /path/postgreSQL,jar
>>>
>>> 4. Currently trying to add SPARK_CLASSPATH in file
>>> "compute_classpath.sh" for each node of cluster
>>>
>>> Please let me know if any inputs on the same to proceed further
>>>
>>> Regards
>>> Satish Chandra
>>>
>>
