> > > > > > here's the crash now:
> > > > > >
> > > > > > Traceback (most recent call last):
> > > > > > File
> > > > > > "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/pdb.py",
> > > > > > line 1960, in main
> > > > > > pdb._run(target)
> > > > > > File
> > > > > > "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/pdb.py",
> > > > > > line 1754, in _run
> > > > > > self.run(target.code)
> > > > > > File
> > > > > > "/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/bdb.py",
> > > > > > line 627, in run
> > > > > > exec(cmd, globals, locals)
> > > > > > File "/home/karl3/projects/rep/rep/dict.py", line 394, in <module>
> > > > > > doc.update([[val,val]])
> > > > > > File "/home/karl3/projects/rep/rep/dict.py", line 349, in update
> > > > > > super().update(keyhashitems())
> > > > > > File "/home/karl3/projects/rep/rep/dict.py", line 164, in update
> > > > > > assert int.from_bytes(keyhash[:hashbytes], 'big') >> hashshift
> > > > > > == newidx
> > > > > >
> > > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > > > AssertionError
> > > > > >
> > > > > >
> > > > > >
> > > > > > for keyhash, item in keyhashitems:
> > > > > >
> > > > > > assert
> > > > > > item
> > > > > > != self._sentinel
> > > > > >
> > > > > > byteidx =
> > > > > > int.from_bytes(keyhash[:hashbytes], 'big')
> > > > > >
> > > > > > newidx = byteidx >> hashshift
> > > > > >
> > > > > >
> > > > > > if self._capacity > 0:
> > > > > >
> > > > > > # this
> > > > > > block checks for collision with previous stored values
> > > > > >
> > > > > > if capacity > self._capacity:
> > > > > >
> > > > > >
> > > > > > superidx =
> > > > > > int.from_bytes(keyhash[:self._hashbytes],
> > > > > > 'big') >> self._hashshift
> > > > > > else:
> > > > > >
> > > > > >
> > > > > > superidx = newidx
> > > > > >
> > > > > >
> > > > > > place = self.array[superidx]
> > > > > >
> > > > > > if place !=
> > > > > > self._sentinel:
> > > > > >
> > > > > > collision = self._key(place)
> > > > > >
> > > > > >
> > > > > > if collision != keyhash:
> > > > > >
> > > > > >
> > > > > > assert
> > > > > > superidx == int.from_bytes(collision[:self._hashbytes], 'big') >>
> > > > > > self._hashshift
> > > > > > updates[newidx] =
> > > > > > [collision, place, False]
> > > > > >
> > > > > > # this separated approach to checking for collisions
> > > > > > allows for accepting
> > > > > > #
> > > > > > batched data that ends up containing hash collisions solely within
> > > > > > itself
> > > > > > placing =
> > > > > > updates.get(newidx)
> > > > > > if placing is not None:
> > > > > > collision, place, is_new = placing
> > > > > > while newidx ==
> > > > > > int.from_bytes(collision[:hashbytes],
> > > > > > 'big') >> hashshift:
> > > > > > capacity <<= 1
> > > > > > expansion <<= 1
> > > > > > #spread += 1
> > > > > > #hashbits = self._hashbits + spread
> > > > > > hashbits += 1
> > > > > > hashbytes = (hashbits+7) >> 3
> > > > > > hashshift = (hashbytes << 3) - hashbits
> > > > > > byteidx = int.from_bytes(keyhash[:hashbytes],
> > > > > > 'big')
> > > > > > newidx = byteidx >> hashshift
> > > > > > assert capacity == (1 << hashbits)
> > > > > > new_updates = {}
> > > > > > for keyhash, item, is_new in updates.values():
> > > > > > if is_new:
> > > > > > newnewidx =
> > > > > > int.from_bytes(keyhash[:hashbytes], 'big') >> hashshift
> > > > > > assert newnewidx not in new_updates
> > > > > > new_updates[newnewidx] = [keyhash, item,
> > > > > > True]
> > > > > >
> > > > > >
> > > > > > updates
> > > > > > = new_updates
> > > > > > assert newidx not in updates
> > > > > > assert int.from_bytes(keyhash[:hashbytes], 'big') >>
> > > > > > hashshift == newidx
> > > > > > updates[newidx] = [keyhash, item, True]
> > > > > >
> > > > > >
> > > > > > it looks like the problem is that keyhash is shadowed in the loop
> > > > > > at the end :D
> > > >
> > > > shadowing is when a variable is used in an inner scope with the same
> > > > name as an outer scope. many languages have strictly scoped blocks but
> > > > python is not one of those languages allowing occasional mistakes like
> > > > this. keyhash is used in an outer scope. i then use it as a loop
> > > > variable, which changes its value in the outer scope.
> > >
> > > gotta acknowledge here that the source paste is illegible because of
> > > all the wrapping and whitespace !
> > >
> > > also, how is this like use of the word "shadow" in corruption? the
> > > people causing the corruption are (a) _hidden_ like the old value of
> > > the variable, where people see the one that is presented rather than
> > > the one acted and (b) cause _wrong events_ like the new value of the
> > > variable overridding its correct one
> >
> > it's because shadowed objects aren't lit because light is cast on the
> > object shadowing them instead
> > and then they are less noticed from something else receiving the light
> > cast on them, and so people don't include them and anything they do in
> > their decisions as much :/
>
> industrialized humans are rare in that we basically just cast light
> everywhere, with flashlights and streetlights and infrared cameras and
> sleeping at night and locking doors at night
>
> most living beings just learn to see in the dark and treat incidental
> weirdnesses as serious each time
it's funny
y'know white people stick out like a sore thumb at night the moon
glances off their skin almost like glow-in-the-dark . i like to think
it relates to industrialization somehow
> >
> > >
> > > >
> > > > usually a shadowed variable would make the opposite crash -- where in
> > > > the inner loop, the inner value is used as if it were the outer. here,
> > > > since it's python, the crash is from the outer value being used after
> > > > the inner loop has finished. [...
> > > >
> > > > >
> > > > > now i've got this:
> > > > > File "/home/karl3/projects/rep/rep/dict.py", line 220, in
> > > > > content_generator
> > > > > assert superidx * expansion + subidx ==
> > > > > int.from_bytes(dbg_keyhash[:hashbytes], 'big') >> hashshift
> > > > >
> > > > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > > AssertionError
> > > > > Uncaught exception. Entering post mortem debugging
> > > > > Running 'cont' or 'step' will restart the program
> > > > > > /home/karl3/projects/rep/rep/dict.py(220)content_generator()
> > > > > -> assert superidx * expansion + subidx ==
> > > > > int.from_bytes(dbg_keyhash[:hashbytes], 'big') >> hashshift
> > > > > (Pdb) p item
> > > > > b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
> > > > >
> > > > > but it's just because item is a sentinel which shouldn't hash to its
> > > > > index, can check for that
