Hello again, Philip, I really appreciate you sticking with me. Hopefully this will help someone else, too. I've done some more reading, and will offer some minimal code below.
I've known about this page for a while, and it describes some of the unconventional things one needs to consider when subclassing numpy.ndarray: http://www.scipy.org/Subclasses Now, section 11.1 of the Python documentation says the following concerning pickling: "Classes can further influence how their instances are pickled; if the class defines the method __getstate__(), it is called and the return state is pickled as the contents for the instance, instead of the contents of the instance’s dictionary. If there is no __getstate__() method, the instance’s __dict__ is pickled." http://docs.python.org/release/2.6.6/library/pickle.html That being said, I'm having problems there! I've found this minimal example of pickling with overridden __getstate__ and __setstate__ methods: http://stackoverflow.com/questions/1939058/please-give-me-a-simple-example-about-setstate-and-getstate-thanks I'll put all of the thoughts generated by these links together after responding to a few things you wrote. On Apr 5, 10:43 am, Philip Semanchuk <phi...@semanchuk.com> wrote: > But as Dan Stromberg pointed out, there are some pickle-free ways to > communicate between processes using multiprocessing. I only see your reply to Dan Stromberg in this thread, but not Dan Stromberg's original post. I am reading this through Google Groups. Perhaps Dan's post failed to make it through a gateway for some reason? > As a side note, you should always use "new style" classes, particularly since > you're exploring the details of Python class construction. "New" is a bit a > of misnomer now, as "new" style classes were introduced in Python 2.2. They > have been the status quo in Python 2.x for a while now and are the only > choice in Python 3.x. Sorry, that was an oversight on my part. Normally I do remember that, but I've been doing a lot of subclassing rather than defining top- level classes, and therefore it slipped my mind. > One can pickle user-defined classes: OK, I got that working, I'll show an example farther down. (I never tried to pickle a minimal class until now, I went straight for a hard one.) > And as Robert Kern pointed out, numpy arrays are also pickle-able. OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as I would expect. I already have lots of code that is based on such subclasses, and they do everything I want EXCEPT pickle correctly. I may have to direct this line of questions to the numpy discussion group after all. This is going to be a longer exposition. So let me thank you in advance for your patience, and ask you to get comfortable. ========== begin code ========== from numpy import array, ndarray from pickle import dumps, loads ##=== class Foo(object): def __init__(self, contents): self.contents = contents def __str__(self): return str(type(self)) + "\n" + str(self.__dict__) ##=== class SuperFoo(Foo): def __getstate__(self): print "__getstate__" duplicate = dict(self.__dict__) duplicate["bonus"] = "added during pickling" return duplicate def __setstate__(self, d): print "__setstate__" self.__dict__ = d self.finale = "added during unpickling" ##=== class ArraySubclass(ndarray): """ See http://www.scipy.org/Subclasses, this class is very similar. """ def __new__(subclass, data, extra=None, dtype=None, copy=False): print " __new__" arr = array(data, dtype=dtype, copy=copy) arr = arr.view(subclass) if extra is not None: arr.extra = extra elif hasattr(data, "extra"): arr.extra = data.extra return arr def __array_finalize__(self, other): print " __array_finalize__" self.__dict__ = getattr(other, "__dict__", {}) def __str__(self): return str(type(self)) + "\n" + ndarray.__str__(self) + \ "\n__dict__ : " + str(self.__dict__) ##=== class PicklableArraySubclass(ArraySubclass): def __getstate__(self): print "__getstate__" return self.__dict__ def __setstate__(self, d): print "__setstate__" self.__dict__ = d ##=== print "\n\n** Create a Foo object, then create a copy via pickling." original = Foo("pickle me please") print original clone = loads(dumps(original)) print clone print "\n\n** Create a SuperFoo object, just to watch __setstate__ and __getstate__." original = SuperFoo("pickle me too, please") print original clone = loads(dumps(original)) print clone print "\n\n** Create a numpy ndarray, then create a copy via pickling." original = array(((1,2,3),(4,5,6))) print original clone = loads(dumps(original)) print clone print "\n\n** Create an ArraySubclass object, with meta-data..." original = ArraySubclass(((9,8,7),(6,5,4)), extra = "pickle me PLEASE!") print original print "\n...now attempt to create a copy via pickling." clone = loads(dumps(original)) print clone print "\n\n** That failed, try a PicklableArraySubclass..." original = PicklableArraySubclass(((1,2),(3,4)), extra = "pickle, dangit!!!") print original print "\n...now try to create a copy of the PicklableArraySubclass via pickling." clone = loads(dumps(original)) print clone ========== end code, begin output ========== ** Create a Foo object, then create a copy via pickling. <class '__main__.Foo'> {'contents': 'pickle me please'} <class '__main__.Foo'> {'contents': 'pickle me please'} ** Create a SuperFoo object, just to watch __setstate__ and __getstate__. <class '__main__.SuperFoo'> {'contents': 'pickle me too, please'} __getstate__ __setstate__ <class '__main__.SuperFoo'> {'bonus': 'added during pickling', 'finale': 'added during unpickling', 'contents': 'pickle me too, please'} ** Create a numpy ndarray, then create a copy via pickling. [[1 2 3] [4 5 6]] [[1 2 3] [4 5 6]] ** Create an ArraySubclass object, with meta-data... __new__ __array_finalize__ __array_finalize__ __array_finalize__ <class '__main__.ArraySubclass'> [[9 8 7] [6 5 4]] __dict__ : {'extra': 'pickle me PLEASE!'} ...now attempt to create a copy via pickling. __array_finalize__ __array_finalize__ __array_finalize__ <class '__main__.ArraySubclass'> [[9 8 7] [6 5 4]] __dict__ : {} ** That failed, try a PicklableArraySubclass... __new__ __array_finalize__ __array_finalize__ __array_finalize__ <class '__main__.PicklableArraySubclass'> [[1 2] [3 4]] __dict__ : {'extra': 'pickle, dangit!!!'} ...now try to create a copy of the PicklableArraySubclass via pickling. __array_finalize__ __setstate__ Traceback (most recent call last): File "minimal ndarray subclass example for posting.py", line 109, in <module> clone = loads(dumps(original)) File "/usr/lib/python2.6/pickle.py", line 1374, in loads return Unpickler(file).load() File "/usr/lib/python2.6/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib/python2.6/pickle.py", line 1217, in load_build setstate(state) File "minimal ndarray subclass example for posting.py", line 75, in __setstate__ self.__dict__ = d TypeError: __dict__ must be set to a dictionary, not a 'tuple' >Exit code: 1 ========== end output ========== Observations: My first three examples work fine. The second example, my SuperFoo class, shows that I can intercept pickling attempts and add data to __dict__. Problems appear in the fourth example, when I try to pickle an ArraySubclass object. It gets the array itself, but __dict__ is not copied, despite the fact that that is supposed to be Python's default behavior, and my first example did indeed show this default behavior. So in my fifth and final example, the PickleableArraySublcass object, I tried to override __getstate__ and __setstate__ just as I did with SuperFoo. Here's what I see: 1) __getstate__ is NEVER called. 2) __setstate__ is called, but it's expecting a dictionary as an argument, and that dictionary is absent. What's up with that? -- http://mail.python.org/mailman/listinfo/python-list
