Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Jul 22, 2025 at 02:56:23PM +0200, Hannu Krosing wrote: > The high-level idea would be to any actual rewrite -- as opposed to > plain vacuum which frees empty space within the TOAST relation -- as > part of the vacuum of the main relation. > Another option would be to store a back-pointer to

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Jul 22, 2025 at 1:24 PM Nikita Malakhov wrote: > > Hi Michael! > > Yes, I know about relation rewrite and have already thought about how > we can avoid excessive storage of toastrelid and do not spoil rewrite, > still do not have a good enough solution. The high-level idea would be to any

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi Michael! Yes, I know about relation rewrite and have already thought about how we can avoid excessive storage of toastrelid and do not spoil rewrite, still do not have a good enough solution. > You have some interesting points around > detoast_external_attr() and detoast_attr_slice(), as far a

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Mon, Jul 21, 2025 at 02:06:03PM +0300, Nikita Malakhov wrote: > While it is very fast on small data - I see several disadvantages: > - first of all, VACUUM should be revised to work with such tables; > - problematic batch insertion due to necessity to store TID chain. > > It is just a POC imple

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Mon, Jul 21, 2025 at 02:20:31PM +0300, Nikita Malakhov wrote: > I agree that storing reltoastrelid in each Toast pointer seems to be > a waste of disk space since the current Postgres state does not > allow multiple TOAST tables per relation. va_toastrelid is a central part of the current syste

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi! I agree that storing reltoastrelid in each Toast pointer seems to be a waste of disk space since the current Postgres state does not allow multiple TOAST tables per relation. But if we consider this as a viable option it could bring additional advantages. I've successfully tried to use multipl

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi! Michael and Hannu, here's a POC patch with direct TIDs TOAST. The simplest implementation where we store a chain of TIDs, each chunk stores the next TID to be fetched. Patch applies on top of commit 998b0b51d5ea763be081804434f177082ba6772b (origin/toast_64bit_v2) Author: Michael Paquier Date:

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

I have been evolving details for Direct TOAST design in https://wiki.postgresql.org/wiki/DirectTOAST The top level goals are * 8-byte TOAST pointer - just (header:1, tag:1 and TID:6) * all other info moved from toast pointer to actual toast record(s), so heap rows are smaller and faster. * all ex

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi, > v26-0014-Design-to-extend-the-varattrib_4b-and-toast-poin.patch: > Design proposal covering varattrib_4b, TOAST pointer layouts, and > related macro updates. > v26-0015-Implement-Zstd-compression-no-dictionary-support.patch: Plain > ZSTD (non dict) support and few basic tests. Sending v27 p

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi Michael, I'm reviewing your patches(toast_64bit_v2 branch) and have prepared two incremental patches that add ZSTD compression support. While doing this I made a small refactoring in heaptoast.c (around ToastTupleContext) to make adding additional TOAST pointer formats easier. Added two new on

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Fri, Jul 18, 2025 at 10:24:12PM +0300, Nikita Malakhov wrote: > I'm currently debugging POC direct tids TOAST patch (on top of your branch), > will mail it in a day or two. Interesting. Of course I may be wrong because I have no idea of how you have shaped things, still I suspect that for the

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Mon, Jul 14, 2025 at 8:15 AM Nikita Malakhov wrote: > ... > Have to mention though that we encountered issues in logical replication > when we made toast values updatable. This seems to indicate that Logical Decoding does not honour visibility checks in TOAST. This works fine if the TOAST visi

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Fri, Jul 18, 2025 at 9:24 PM Nikita Malakhov wrote: > > Hi Michael, > > I'm currently debugging POC direct tids TOAST patch (on top of your branch), > will mail it in a day or two. Great! I also just started looking at it, starting from 0003 as recommended by Michael. Will be interesting to

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi Michael, I'm currently debugging POC direct tids TOAST patch (on top of your branch), will mail it in a day or two. On Tue, Jul 15, 2025 at 3:56 AM Michael Paquier wrote: > On Mon, Jul 14, 2025 at 09:01:28AM +0300, Nikita Malakhov wrote: > > Cool, toast tables are subject to bloating in upda

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Mon, Jul 14, 2025 at 09:01:28AM +0300, Nikita Malakhov wrote: > Cool, toast tables are subject to bloating in update-heavy scenarios > and it's a big problem in production systems, it seems there is a promising > way to solve it once and for all! > > Have to mention though that we encountered i

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi! On this - > Non-FULL vacuum should already work. Only commands like VACUUM FULL > and CLUSTER which move tuples around should be disabled on TOAST > tables. Cool, toast tables are subject to bloating in update-heavy scenarios and it's a big problem in production systems, it seems there is a p

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Jul 08, 2025 at 12:58:26PM -0400, Burd, Greg wrote: > All that aside, I think you're right to tackle this one step at a > time and try not to boil too much of the ocean at once (the patch > set is already large enough). With that in mind I've read once or > twice over your changes and have

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Jul 08, 2025 at 09:31:29PM +0300, Nikita Malakhov wrote: > Michael, one more thing forgot to mention yesterday - > #define TOAST_EXTERNAL_INFO_SIZE (VARTAG_ONDISK_OID + 1) > static const toast_external_info > toast_external_infos[TOAST_EXTERNAL_INFO_SIZE] > VARTAG_ONDISK_OID historically ha

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Tue, Jul 08, 2025 at 08:54:33PM +0200, Hannu Krosing wrote: > I still think we should go with direct toast tid pointers in varlena > and not some kind of oid. > > It will remove the need for any oid management and also will be > many-many orders of magnitude faster for large tables (just 2x fast

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

chunk_ On Tue, Jul 8, 2025 at 9:37 PM Álvaro Herrera wrote: > > On 2025-Jul-08, Hannu Krosing wrote: > > > I still think we should go with direct toast tid pointers in varlena > > and not some kind of oid. > > I think this can be made to work, as long as we stop seeing the toast > table just like

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi, Hannu, we have some thoughts on direct tids storage, it was some time ago and done by another developer, so I have to look. I'll share it as soon as I find it, if you are interested. -- Regards, Nikita Malakhov Postgres Professional The Russian Postgres Company https://postgrespro.ru/

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On 2025-Jul-08, Hannu Krosing wrote: > I still think we should go with direct toast tid pointers in varlena > and not some kind of oid. I think this can be made to work, as long as we stop seeing the toast table just like a normal heap table containing normal tuples. A lot to reimplement though

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

I still think we should go with direct toast tid pointers in varlena and not some kind of oid. It will remove the need for any oid management and also will be many-many orders of magnitude faster for large tables (just 2x faster for in-memory small tables) I plan to go over Michael's patch set he

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi! Greg, thanks for the interest in our work! Michael, one more thing forgot to mention yesterday - #define TOAST_EXTERNAL_INFO_SIZE (VARTAG_ONDISK_OID + 1) static const toast_external_info toast_external_infos[TOAST_EXTERNAL_INFO_SIZE] VARTAG_ONDISK_OID historically has a value of 18 and here w

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

> On Jul 7, 2025, at 7:38 PM, Michael Paquier wrote: > > I have also pushed this v2 on this branch, so feel free to grab it if > that makes your life easier: > https://github.com/michaelpq/postgres/tree/toast_64bit_v2 > -- > Michael Thank you for spending time digging into this and for the we

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi! I'm reviewing code at toast_64bit_v2. Michael, isn't there a typo? static const toast_external_info toast_external_infos[TOAST_EXTERNAL_INFO_SIZE] = { [VARTAG_ONDISK_INT8] = { .toast_pointer_size = TOAST_POINTER_INT8_SIZE, .maximum_chunk_size = TOAST_MAX_CHUNK_SIZE_INT8, .to_external_data = on

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

On Fri, Jul 04, 2025 at 02:38:34PM +0300, Nikita Malakhov wrote: > Hannu, we'd already made an attempt to extract the TOAST functionality as > API and make it extensible and usable by other AMs in [1], the patch > set was met calmly but we still have some hopes on it. Yeah, it's one of these I hav

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi! Hannu, we'd already made an attempt to extract the TOAST functionality as API and make it extensible and usable by other AMs in [1], the patch set was met calmly but we still have some hopes on it. Michael, glad you continue this work! Took patch set for review. [1] Pluggable TOAST

Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi Michael I'll take a look at the patch set. While digging around in the TOAST code did you have any ideas on how one could extract the TOAST APIs in a way that they can be added in Table Access Method definition ? Not all TAMs need TOAST, but the ones that do could also be the ones that still