[
https://issues.apache.org/jira/browse/SPARK-20551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991929#comment-15991929
]
Hyukjin Kwon commented on SPARK-20551:
--------------------------------------
It sounds you are importing jars in Python without Py4J support. I tested to
import custom Python package and it looks working okay in my local. It sounds
like a rather question to me.
> ImportError adding custom class from jar in pyspark
> ---------------------------------------------------
>
> Key: SPARK-20551
> URL: https://issues.apache.org/jira/browse/SPARK-20551
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Shell
> Affects Versions: 2.1.0
> Reporter: Sergio Monteiro
>
> the flowwing imports are failing in PySpark, even when I set the --jars or
> --driver-class-path:
> import net.ripe.hadoop.pcap.io.PcapInputFormat
> import net.ripe.hadoop.pcap.io.CombinePcapInputFormat
> import net.ripe.hadoop.pcap.packet.Packet
> Using Python version 2.7.12 (default, Nov 19 2016 06:48:10)
> SparkSession available as 'spark'.
> >>> import net.ripe.hadoop.pcap.io.PcapInputFormat
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named net.ripe.hadoop.pcap.io.PcapInputFormat
> >>> import net.ripe.hadoop.pcap.io.CombinePcapInputFormat
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named net.ripe.hadoop.pcap.io.CombinePcapInputFormat
> >>> import net.ripe.hadoop.pcap.packet.Packet
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named net.ripe.hadoop.pcap.packet.Packet
> >>>
> The same works great in spark-shell.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
