[
https://issues.apache.org/jira/browse/SPARK-20551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992491#comment-15992491
]
Nick Pentreath commented on SPARK-20551:
----------------------------------------
Yes I agree that it appears you're trying to import Java or Scala classes in
Python which won't work. I suggest you post a question to the Spark user list
asking for help: [email protected]. Please indicate what you are trying to
do and provide some example code (it appears that you're trying to read a
custom Hadoop {{InputFormat}} in PySpark?
> ImportError adding custom class from jar in pyspark
> ---------------------------------------------------
>
> Key: SPARK-20551
> URL: https://issues.apache.org/jira/browse/SPARK-20551
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Shell
> Affects Versions: 2.1.0
> Reporter: Sergio Monteiro
>
> the flowwing imports are failing in PySpark, even when I set the --jars or
> --driver-class-path:
> import net.ripe.hadoop.pcap.io.PcapInputFormat
> import net.ripe.hadoop.pcap.io.CombinePcapInputFormat
> import net.ripe.hadoop.pcap.packet.Packet
> Using Python version 2.7.12 (default, Nov 19 2016 06:48:10)
> SparkSession available as 'spark'.
> >>> import net.ripe.hadoop.pcap.io.PcapInputFormat
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named net.ripe.hadoop.pcap.io.PcapInputFormat
> >>> import net.ripe.hadoop.pcap.io.CombinePcapInputFormat
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named net.ripe.hadoop.pcap.io.CombinePcapInputFormat
> >>> import net.ripe.hadoop.pcap.packet.Packet
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named net.ripe.hadoop.pcap.packet.Packet
> >>>
> The same works great in spark-shell.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
