> > > once upon a time, long long, slave boss was happily digging a hole to
> > > china by piling dirt in a small pile, when suddenly extraterrestrial
> > > warships and robot drones swarmed the skies
> > >
> > > slave boss looked at the skies and realized things had changed,
> > >
> > > rather than dig a hole to china, he was going to have to convince
> > > these extraterrestrial warships that he was the king of earth
> >
> > things seem unstable.
> >
> > i'm interesting in upgrading rep/i.py so as to provide for reuse of
> > storage. this would let it structure a temporary cache.
> > i thought i'd do a traditional linked-list-of-deallocated-regions strategy
>
> slave boss was beamed up into an extraterrestrial warship
>
> in the warship everything was like a kaleidoscope because the
> extraterrestrials used densely complex refraction of light to
> communicate
planning
so currently it has a quadword at the start that references the offset
of the next available region, in quadwords.
meanwhile at the start of each allocated region is a quadword that
contains the exact length in bytes of the allocation.
so what's confusing to me is how the aligned length of the dedicated
region differs from the length of the allocation -- it can be a little
more, so that the next region aligns to a quadword.
it would make it clear to make a function or snippet that clearly
converts a length in bytes to an aligned length.
~1625
here's this:
def alloc(self, data): #, replacing=[]):
addr8 = self.q[0]
id = self.w[0:self.q.itemsize].tobytes()
l8 = (len(data)+self.q.itemsize-1)//self.q.itemsize + 1
if addr8 + l8 > self.yq:
y2 = self.yq *2*self.q.itemsize
if (addr8+l8)*self.q.itemsize > y2:
y2 = ((addr+l8*2-1)*self.q.itemsize // mmap.PAGESIZE +
1) * mmap.PAGESIZE
os.truncate(self.n, y2)
self.w = memoryview(mmap.mmap(self.d, y2))
self.q = self.w.cast('Q')
self.yq = len(self.q)
self.q[addr8] = len(data)
addr = (addr8 + 1) * self.q.itemsize
self.w[addr:addr+len(data)] = data
assert self.fetch(id) == data
self.q[0] = addr8 + l8
return id
it looks like l8 is the number of quadwords that the region takes up
~1627 ... but 1 is added to it in the actual calculation. right,
because there's an additional length field at the start.
ok so ummmmmmm if i had a linked list, i'd want to walk it before
decided to grow the store. i'd want a variable of what address to
place it in. i'd also need to create a new deallocated region for the
remaining part (umm and update the linked list).
~1629
ummmm subproblem: what to do about zero-sized allocs. general code
will encounter zero-sized allocs when user data is passed to alloc.
one could either demand the general code special case zero-sized data,
or find some way of handling it. simplest would be allocate 2
quadwords. ummmm ummmm so we could also find a way to fit the linked
list inside a single quadword, but i guess that makes more edgecases
when i'm already being quite confused
~1630
ummmmmmmmmmmmmmmmmmmm ok or uhhhh ummmmmm i think basically we'd
allocate an extra quadword for zero-sized data. and to keep storage
efficient one would only do that if the data is zero-sized. that could
be done with the min()[edit 1633: should be max()] function or a
condition!
ok maybe hum ~1631
so there's l8 that says how many quadwords are needed.
the linked list structure could contain the length in quadwords and
the offset of the next item. ~1632 ~1633
~1633
ok how to find ummm when the list is over. what goes there?
note: the structure also knows the size of the whole file.
and there is a deallocated region at the end.
so, that deallocated region at the end would somehow indicate that it
is the last.
it could contain its own address, or 0 (the first address), or the end
of the file, as next stop
i guess i like 0, which technically makes the list circular maybe i
guess maybe not really
~1635
maybe i like have it contain its own address better, unsure
anyway basically it has some way of indicating it's at the end.
an alternative to find it quickly would be to make it the first one.
1635
so what does it look like to allocate here
each hole points to the next
so if we use a hole, or replace it with a smaller one, we'd need to
update the _last_ pointer to either point to the next one or the
replacing[edited from 'updated'] one.
~1636 /~1636
~1640
here's kind of what i have so far:
def alloc(self, data, replacing=[]):
id = self.w[0:self.q.itemsize].tobytes() # [this should be
moved down in file and reference addr8]
l8 = max((len(data)+self.q.itemsize-1)//self.q.itemsize,1) + 1
addr8 = 0
while self.q[addr8] != addr8:
addr8 = self.q[addr8]
# - [later, in constructor,] initialize data with an empty
region that points to itself and has a given length
if self.q[addr8+1] >= l8:
# found region
break
else:
# no regions were found that data fits in. expand the last one.
assert self.q[addr8+!] == self.yq
# [then i can copy in most of the old allocation code here]
