Hi, I'm trying to build something like an embedded data base system around arrow with some properties which I will list below. The question is if someone can point me to existing solutions (or attempts) for such kind of problem. I'm trying to develop a data layout such that * typically reading is not for single value but for a range of keys/fields * this is all in-memory * it provides low overhead for reading and writing * multithreaded reading supported, but updates are single threaded * 99% of updates are primarily adding a new data point to the end of column (a-la log) * return data in arrow tables with minimal overhead (better not to copy data from internal storage but do slicing whenever possible)
On a high level I had in mind something like SSTables with level compaction when the data is accessed for reading. But before doing this I wanted to check if there is a solution for such a problem leveraging an arrow or if there are all the bricks to build such a solution. Any input (including papers) is welcomed. -- Best regards, Kirill Lykov
