Hi,
Here's the PoC for a custom TOAST pointer. The main idea is that custom
pointer
provides data space allowing to store custom metadata (i.e. TOAST method,
relation
OIDs, advanced compression information, etc, and even keep part of the data
inline.
Any feedback would be greatly appreciated.
--
On Wed, May 24, 2023 at 11:50:21AM +0200, Matthias van de Meent wrote:
> I think you might not have picked up on what I was arguing for, but I
> agree with what you just said.
>
> My comment on not needing to invent a new ToastCompressionId was on
> the topic of adding capabilities^ to toast point
On Tue, 23 May 2023 at 18:34, Robert Haas wrote:
>
> On Thu, May 18, 2023 at 8:06 AM Matthias van de Meent
> wrote:
> > This enum still has many options to go before it exceeds the maximum
> > of the uint8 va_tag field. Therefore, I don't think we have no disk
> > representations left, nor do I t
Hi!
I've made a WIP patch that uses 64-bit TOAST value ID instead of 32-bit,
and sent it as a part of discussion, but there was no feedback on such a
solution. There was a link to that discussion at the top of this thread.
Also, I have to note that, based on our work on Pluggable TOAST - extendin
On Tue, May 23, 2023 at 12:33:50PM -0400, Robert Haas wrote:
> For projects like this, the details matter a lot. If the goal is to
> add a new compression type that behaves like the existing compression
> types, more or less, then I think we should allocate the last
> ToastCompressionId bit to mean
On Thu, May 18, 2023 at 8:06 AM Matthias van de Meent
wrote:
> This enum still has many options to go before it exceeds the maximum
> of the uint8 va_tag field. Therefore, I don't think we have no disk
> representations left, nor do I think we'll need to add another option
> to the ToastCompressio
On Mon, May 22, 2023 at 6:47 AM Aleksander Alekseev
wrote:
> Last but not least I remember somebody on the mailing list suggested
> adding ZSTD compression support for TOAST, besides LZ4. Assuming I
> didn't dream it, the proposal was rejected due to the limited amount
> of free bits in ToastCompr
On Mon, 22 May 2023 at 18:13, Nikita Malakhov wrote:
>
> Hi,
Could you please not top-post.
> Aleksander, I'm interested in extending TOAST pointer in various ways.
> 64-bit TOAST value ID allows to resolve very complex issue for production
> systems with large tables and heavy update rate.
Co
Hi,
Aleksander, I'm interested in extending TOAST pointer in various ways.
64-bit TOAST value ID allows to resolve very complex issue for production
systems with large tables and heavy update rate.
I agree with Matthias that there should not be processing of TOAST pointer
internals outside TOAST
Hi,
> I see your point, but I do think we should also think about why we do
> the change.
Personally at the moment I care only about implementing compression
dictionaries on top of this, as is discussed in the corresponding
thread [1]. I'm going to need new fields in the TOAST pointer's
including
On Sun, 21 May 2023, 15:39 Aleksander Alekseev,
wrote:
>
> Hi,
>
> > We'd need to stop using the va_tag as length indicator, but I don't
> > think it's currently assumed to be a length indicator anyway (see
> > VARSIZE_EXTERNAL(ptr)). By not using the varatt_external struct
> > currently in use, w
Hi,
> We'd need to stop using the va_tag as length indicator, but I don't
> think it's currently assumed to be a length indicator anyway (see
> VARSIZE_EXTERNAL(ptr)). By not using the varatt_external struct
> currently in use, we could be able to get down to <18B toast pointers
> as well, though
Hi!
Matthias, in the Pluggable TOAST thread we proposed additional pointer
definition, without modification
of the original varatt_external - we have to keep it untouched for
compatibility issues. The following extension
for the TOAST pointer was proposed:
typedef struct varatt_custom
{
uint16 va
On Thu, 18 May 2023 at 15:12, Aleksander Alekseev
wrote:
>
> Hi,
>
> > I agree that va_tag can have another use. But since we are going to
> > make varatt_external variable in size (otherwise I don't see how it
> > could be really **extendable**) I don't think this is the right
> > approach.
Why
Hi,
> I agree that va_tag can have another use. But since we are going to
> make varatt_external variable in size (otherwise I don't see how it
> could be really **extendable**) I don't think this is the right
> approach.
On second thought, perhaps we are talking more or less about the same thing
Hi,
> No, that's inaccurate. The complete on-disk representation of a varatt is
>
> {
> uint8 va_header; /* Always 0x80 or 0x01 */
> uint8 va_tag; /* Type of datum */
> charva_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-dependent
> data, for toasted values tha
On Thu, 18 May 2023 at 12:52, Aleksander Alekseev
wrote:
>
> Hi Nikita,
>
> > this part of the PostgreSQL screams to be revised and improved
>
> I completely agree. The problem with TOAST pointers is that they are
> not extendable at the moment which prevents adding new compression
> algorithms (e
Hi Nikita,
> this part of the PostgreSQL screams to be revised and improved
I completely agree. The problem with TOAST pointers is that they are
not extendable at the moment which prevents adding new compression
algorithms (e.g. ZSTD), new features like compression dictionaries
[1], etc. I sugges
18 matches
Mail list logo
