On Fri, 2021-04-30 at 10:50 -0700, Peter Geoghegan wrote: > I don't know. This conversation is still too abstract for me to be > able to take a firm position. ISTM that we tend to talk about the > table AM in excessively abstract terms. It would be a lot easier if > we > had clear fixed goals for a small number of additional table AMs.
https://github.com/citusdata/citus/tree/master/src/backend/columnar My colleagues and I have been working on a "columnar" table AM. It doesn't currently support indexes, but it would be useful to support them. The basic idea is we have "stripes" of ~150000 tuples that are rearranged and compressed, and stored in an smgr-controlled file that goes through the buffer cache and uses generic WAL. To support indexes, we could do our own lookups from a "row number" to a particular offset where we can find and decompress the stripe that holds that row number, and then scan forward in the stripe to find the particular row. This will be terrible for random access, but [*waves hands*] we will keep state and use a few optimizations so that this is not terribly slow for clustered access. Granted, TID lookup on columnar will be much slower than for a heap (and we can use a CustomScan so that the costs reflect that). But it will satisfy important use cases: 1. Indexes defined on partition parent tables. Even if the index is never used for queries, we don't want to throw an error when defining the partitioned parent index. 2. Unique indexes and exclusion constraints. 3. Clustered index scans can still be reasonably fast. 4. Could be used for UPDATE/DELETE as well. > More generally, it seems like a good idea to try to make new table > AMs > reasonably close to heapam insofar as possible. The reality is that > everything evolved around heapam, and that that's likely to matter in > all kinds of ways that nobody fully understands just yet. Agreed. I think of this as an evolving situation where we take steps toward a better abstraction. One (hopefully reasonable) step I'd like to take is a well-specified TID. Regards, Jeff Davis
