On Jul 21, 5:20 pm, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > On Jul 21, 12:08 pm, castironpi <[EMAIL PROTECTED]> wrote: > > > Some time ago, I was asking about the feasibility of a persistent > > deque, a double-ended queue. > > > It runs into the typical space allocation problems. > > Try starting with a dict-based implementation of a double-ended queue > (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/259179) > and then replace the dict with a shelf. Presto, you've got a > persistent deque. > > Raymond
We have the cookbook recipe Raymond brings up, and an automatic pickler that inhahe brought up in May. I've initiated the approach to the solution I have in mind, but the dilemma comes early and isn't pleasant. I can either (i) recompile the whole interpreter, replacing malloc with a custom function that allocates from disk; or (ii) duplicate every class I want to be persistable, making every change to the allocations by hand. Disk-resident objects can clearly not contain references to any objects not on disk (arguably at finalization-stage only), and references to ones that are need to be wrapped by offset, and 'made live' at initialization-time, so that references to the mapped file resume succeeding after termination and restart--- the mapped memory segment won't be the same from run to run. Further, if I can't hijack the existing arena-pool model native in Python, it would take writing a dedicated memory manager. Assuming the whole memory isn't moved to disk, which is feasible, the usage would look like this, to ensure that 'disk-resident' objects don't refer to 'live' ones. listA= perst( [ perst( 0 ), perst( 1 ), perst( "abc" ) ] ) to create the list [ 0, 1, "abc" ]. Modifications, e.g. append "def", would look like this: listA.append( perst( "def" ) ) and would execute the addition directly to mapped memory. The memory file would then have five objects: 0, 1, "abc", "def", and this list. It is not clear that normal assignments would still work, self.b= c, even for disk-resident c, since on a later run, 'self.b' would refer to a different place in memory. It might take either trapping updates to self.__dict__, or something deeper at the level of the ast and Assm't statements. Or an extra level of indirection in accessing it. >>> self.b <C instance at index 12 in mem-mapped file 'data.dat'> The indirection would consist of an offset-finder and a delegation layer. >>> self.b.append( perst( set( ) ) ) It would first have to look-up 'self.b', as opposed to accessing it directly by memory address. What I want from the group is to know if I've assessed the options for a model of persistence accurately. Or if I haven't depicted what I'm going for clearly. It seems enormous and if it is this hard, I doubt I'll produce anything other than a few correct persistent types. -- http://mail.python.org/mailman/listinfo/python-list
