Oh I just created the project, you can follow it here: https://gitlab.com/tampe/stis-supervectors
On Thu, Sep 2, 2021 at 5:45 PM Stefan Israelsson Tampe < stefan.ita...@gmail.com> wrote: > Hi guilers! > > My next project is to explore a more advanced bytevector structure than > today's bytevectors. > I think that having this infrastructure in guile and core code taken > advantage of it like having strings otop of it and allowing our string > library to use those (the difficult case is probably to get regexps working > properly) > > Anyhow this is my initial comment in the code: > > #| > The idea of this data structure is to note that when we employ full > large datastructure scans we can allow for a much more rich and featurefull > datastructure then a simple bytevector. The reason is that we can divide > the data in byte chunks and spend most of the time scanning copying maping > those areas with usual methis, even optimized C - code taken advantage of > advanced cpu opcodes are possible here. ANd by dividing it in chunks we get > a lot of new features with essentiually no cose with more than complexity > which we manage mostly in scheme. We gain many things, > > 1. Many new features that can vary on the pages > > 2. less memory hogs as > a. it can use copy ion write semantics > b. it does not need to fins 1GB continuous blocks > > 3. potentially faster operations as we can fast foorward the zeroe on write > pages compared to pure bytevectors > > 4. we can allow for swaping and refferential datastructures to speed up > copying > even further > > 5. can get better fiber features of C programs that spends seconds or > minutes on > performing an operation because it will just spend a microsecond or such > in C-land and then swap back to Scheme. CTRL-C will also work nicely. > > 6. We could potentially build a string library untop of these > datastructures and > also we could add features to pages that lead to a much richer > interface. > > 7. resizing is much faster end efficient > > 8. reversing is super quick > > 9. queues and stacks of byte data can have a very efficient > implementations > > Drawback: > 1. More complex as we need to consider boudaries > 2. Slower one off operations like bytevector-u8-get as guile compiles the > core operatoins to quite effective JIT CPU encodings. But maybe we can > disign some caching to make those operations much faster and even have > suporting JIT operations. > > |# > > WDYT ? > >
