On 06/03/2019 20:24, Peter Otten wrote:
> duncan smith wrote:
>
>> On 06/03/2019 16:14, duncan smith wrote:
>>> Hello,
>>> I've been trying to figure out why one of my classes can be
>>> pickled but not unpickled. (I realise the problem is probably with the
>>> pickling, but I get the error when I attempt to unpickle.)
>>>
>>> A relatively minimal example is pasted below.
>>>
>>>
>>>>>> import pickle
>>>>>> class test(dict):
>>> def __init__(self, keys, shape=None):
>>> self.shape = shape
>>> for key in keys:
>>> self[key] = None
>>>
>>> def __setitem__(self, key, val):
>>> print (self.shape)
>>> dict.__setitem__(self, key, val)
>>>
>>>
>>>>>> x = test([1,2,3])
>>> None
>>> None
>>> None
>>>>>> s = pickle.dumps(x)
>>>>>> y = pickle.loads(s)
>>> Traceback (most recent call last):
>>> File "<pyshell#114>", line 1, in <module>
>>> y = pickle.loads(s)
>>> File "<pyshell#111>", line 8, in __setitem__
>>> print (self.shape)
>>> AttributeError: 'test' object has no attribute 'shape'
>>>
>>>
>>> I have DUCkDuckGo'ed the issue and have tinkered with __getnewargs__ and
>>> __getnewargs_ex__ without being able to figure out exactly what's going
>>> on. If I comment out the print call, then it seems to be fine. I'd
>>> appreciate any pointers to the underlying problem. I have one or two
>>> other things I can do to try to isolate the issue further, but I think
>>> the example is perhaps small enough that someone in the know could spot
>>> the problem at a glance. Cheers.
>>>
>>> Duncan
>>>
>>
>> OK, this seems to be a "won't fix" bug dating back to 2003
>> (https://bugs.python.org/issue826897). The workaround,
>>
>>
>> class DictPlus(dict):
>> def __init__(self, *args, **kwargs):
>> self.extra_thing = ExtraThingClass()
>> dict.__init__(self, *args, **kwargs)
>> def __setitem__(self, k, v):
>> try:
>> do_something_with(self.extra_thing, k, v)
>> except AttributeError:
>> self.extra_thing = ExtraThingClass()
>> do_something_with(self.extra_thing, k, v)
>> dict.__setitem__(self, k, v)
>> def __setstate__(self, adict):
>> pass
>>
>>
>> doesn't work around the problem for me because I need the actual value
>> of self.shape from the original instance. But I only need it for sanity
>> checking, and under the assumption that the original instance was valid,
>> I don't need to do this when unpickling. I haven't managed to find a
>> workaround that exploits that (yet?). Cheers.
>
> I've been playing around with __getnewargs__(), and it looks like you can
> get it to work with a custom __new__(). Just set the shape attribute there
> rather than in __init__():
>
> $ cat pickle_dict_subclass.py
> import pickle
>
>
> class A(dict):
> def __new__(cls, keys=(), shape=None):
> obj = dict.__new__(cls)
> obj.shape = shape
> return obj
>
> def __init__(self, keys=(), shape=None):
> print("INIT")
> for key in keys:
> self[key] = None
> print("EXIT")
>
> def __setitem__(self, key, val):
> print(self.shape, ": ", key, " <-- ", val, sep="")
> super().__setitem__(key, val)
>
> def __getnewargs__(self):
> print("GETNEWARGS")
> return ("xyz", self.shape)
>
> x = A([1, 2, 3], shape="SHAPE")
> x["foo"] = "bar"
> print("pickling:")
> s = pickle.dumps(x)
> print("unpickling:")
> y = pickle.loads(s)
> print(y)
> $ python3 pickle_dict_subclass.py
> INIT
> SHAPE: 1 <-- None
> SHAPE: 2 <-- None
> SHAPE: 3 <-- None
> EXIT
> SHAPE: foo <-- bar
> pickling:
> GETNEWARGS
> unpickling:
> SHAPE: 1 <-- None
> SHAPE: 2 <-- None
> SHAPE: 3 <-- None
> SHAPE: foo <-- bar
> {1: None, 2: None, 3: None, 'foo': 'bar'}
>
> It's not clear to me how the dict items survive when they are not included
> in the __getnewargs__() result, but apparently they do.
>
>
>
Thanks Peter. The docs for pickle say "When a class instance is
unpickled, its __init__() method is usually not invoked. The default
behaviour first creates an uninitialized instance and then restores the
saved attributes." Your outputs above seem to confirm that it isn't
calling my __init__(), but it *is* calling my __setitem__() when it is
restoring the dict items. I actually have about 50-60 lines of code in
my real __setitem__(), but I probably don't need it if I'm unpickling.
My __setitem__() actually updates a secondary data structure which was
why I thought I needed to do more than just restore the items, but if
that's going to be restored in the same way I just need the dict items
to be restored. The following seems to achieve that.
def __setitem__(self, key, val):
try:
self.shape
except AttributeError:
dict.__setitem__(self, key, val)
else:
# do other stuff
A bit close to original workaround that I didn't think would work
(although that used __setstate__() for some reason). I'm not sure I
prefer it to defining __new__() though. Ideally I'd have something like,
def __setitem__(self, key, val):
if unpickling:
dict.__setitem__(self, key, val)
else:
# do other stuff
rather than potentially masking a bug. Anyway, thanks for your solution
and the outputs that pointed me to another possible solution. Cheers.
Duncan
--
https://mail.python.org/mailman/listinfo/python-list
