An example (adding a package to the spark submit): bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.10:1.6.0 spark_v3.py
From: Bhaarat Sharma [mailto:bhaara...@gmail.com] Sent: 30 July 2016 06:38 To: ayan guha <guha.a...@gmail.com> Cc: user <user@spark.apache.org> Subject: Re: PySpark 1.6.1: 'builtin_function_or_method' object has no attribute '__code__' in Pickles I'm very new to Spark. Im running it on a single CentOS7 box. How would I add a test.py to spark submit? Point to any resources would be great. Thanks for your help. On Sat, Jul 30, 2016 at 1:28 AM, ayan guha <guha.a...@gmail.com<mailto:guha.a...@gmail.com>> wrote: I think you need to add test.py in spark submit so that it gets shipped to all executors On Sat, Jul 30, 2016 at 3:24 PM, Bhaarat Sharma <bhaara...@gmail.com<mailto:bhaara...@gmail.com>> wrote: I am using PySpark 1.6.1. In my python program I'm using ctypes and trying to load the liblept library via the liblept.so.4.0.2 file on my system. While trying to load the library via cdll.LoadLibrary("liblept.so.4.0.2") I get an error : 'builtin_function_or_method' object has no attribute '__code__' Here are my files test.py from ctypes import * class FooBar: def __init__(self, options=None, **kwargs): if options is not None: self.options = options def read_image_from_bytes(self, bytes): return "img" def text_from_image(self, img): self.leptonica = cdll.LoadLibrary("liblept.so.4.0.2") return "test from foobar" spark.py from pyspark import SparkContext import test import numpy as np sc = SparkContext("local", "test") foo = test.FooBar() def file_bytes(rawdata): return np.asarray(bytearray(rawdata),dtype=np.uint8) def do_some_with_bytes(bytes): return foo.do_something_on_image(foo.read_image_from_bytes(bytes)) images = sc.binaryFiles("/myimages/*.jpg") image_to_text = lambda rawdata: do_some_with_bytes(file_bytes(rawdata)) print images.values().map(image_to_text).take(1) #this gives an error What is the way to load this library? -- Best Regards, Ayan Guha This email is confidential and may be subject to privilege. If you are not the intended recipient, please do not copy or disclose its content but contact the sender immediately upon receipt.
