On Fri, Mar 15, 2019 at 7:37 AM Bruno Hass <bruno_h...@live.com> wrote: > This idea is what I was hoping to achieve. Would we be able to make > optimizations on deTOASTing just by storing the chunk lengths in chunk 0?
I don't know. I guess we could also NOT store the chunk lengths and just say that if you don't know which chunk you want by chunk number, your only other alternative is to read the chunks in order. The problem with that is that it you can no longer index by byte-position without fetching every chunk prior to that byte position, but maybe that's not important enough to justify the overhead of a list of chunk lengths. Or maybe it depends on what you want to do with it. Again, stuff like what you are suggesting here has been suggested before. I think the problem is if someone did the work to invent such an infrastructure, that wouldn't actually do anything by itself. We'd then need to find an application of it where it brought us some clear advantage. As I said in my previous email, jsonb seems like a promising candidate, but I don't think it's a slam dunk. What would the design look like, exactly? Which operations would get faster, and could we really make them work? The existing format is, I think, designed with a byte-oriented format in mind, and a chunk-oriented format might have different design constraints. It seems like an idea with potential, but there's a lot of daylight between a directional idea with potential and a specific idea accompanied by a high-quality implementation thereof. > Also, wouldn't it break existing functions by dedicating a whole chunk > (possibly more) to such metadata? Anybody writing such a patch would have to be prepared to fix any such breakage that occurred, at least as regards core code. I would guess that this could be done without breaking too much third-party code, but I guess it depends on exactly what the author of this hypothetical patch ends up changing. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
