Hi, Aleksander, I'm interested in extending TOAST pointer in various ways. 64-bit TOAST value ID allows to resolve very complex issue for production systems with large tables and heavy update rate.
I agree with Matthias that there should not be processing of TOAST pointer internals outside TOAST macros. Currently, TOASTed value is distinguished as VARATT_IS_EXTERNAL_ONDISK, and it should stay this way. Adding compression requires another implementation (extension) of VARATT_EXTERNAL because current supports only 2 compression methods - it has only 1 bit responsible for compression method, and there is a safe way to do so, without affecting default TOAST mechanics - we must keep it this way for compatibility issues and not to break DB upgrade. Also, I must remind that we should not forget about field alignment inside the TOAST pointer. As it was already mentioned, it seems not very reasonable trying to save a byte or two while we are storing out-of-line values of at least 2 kb in size. On Mon, May 22, 2023 at 4:47 PM Aleksander Alekseev < aleksan...@timescale.com> wrote: > Hi, > > > I see your point, but I do think we should also think about why we do > > the change. > > Personally at the moment I care only about implementing compression > dictionaries on top of this, as is discussed in the corresponding > thread [1]. I'm going to need new fields in the TOAST pointer's > including (but not necessarily limited to) dictionary id. > > As I understand, Nikita is interested in implementing 64-bit TOAST > pointers [2]. I must admit I didn't follow that thread too closely but > I can imagine the needs are similar. > > Last but not least I remember somebody on the mailing list suggested > adding ZSTD compression support for TOAST, besides LZ4. Assuming I > didn't dream it, the proposal was rejected due to the limited amount > of free bits in ToastCompressionId. It was argued that two possible > combinations that are left should be treated with care and ZSTD will > not bring enough value to the users compared to LZ4. > > These are 3 recent cases I could recall. This being said I think our > solution should be generic enough to cover possible future cases > and/or cases unknown to us yet. > > [1]: > https://postgr.es/m/CAJ7c6TM7%2BsTvwREeL74Y5U91%2B5ymNobRbOmnDRfdTonq9trZyQ%40mail.gmail.com > [2]: https://commitfest.postgresql.org/43/4296/ > > -- > Best regards, > Aleksander Alekseev > -- Regards, Nikita Malakhov Postgres Professional The Russian Postgres Company https://postgrespro.ru/
