Re: Proposal: Conflict log history table for Logical Replication

[shveta malik]

On Sun, Aug 30, 2026 at 8:43 PM Dilip Kumar <dilipbal...@gmail.com> wrote:
>
> On Wed, Aug 26, 2026 at 8:01 PM Dilip Kumar <dilipbal...@gmail.com> wrote:
> >
> Updated version of 0002, based on offlist testing by Nisha, revealed
> that the assumption that a column's maximum size could become 6x its
> original size during JSON conversion is incorrect. One edge case is a
> column of type array (int[]). A huge array but mostly empty(all NULL),
> can have a small storage size but produce a much larger JSON since
> each element is serialized. For example
>  - A 4096 byte text column becomes ~24KB json (6x), which stays under
> the budget even accumulated across all columns (24,554 × 1600 × 3 ≈
> 118 MB).
>  - But a 4096 byte all-NULL int[] column can produce ~480KB(120x) of
> json in a edge case.
> Based on this, Amit suggested a offlist POC of the patch which Nisha
> and I further modified. It still needs more review, testing, and logic
> validation, but I am sharing it here so we can review and provide
> feedback.
>

Thanks. I found an issue where worker-statics size-limit flags cause
re-entrant inner JSON operations to prematurely trigger the limit,
resulting in false-positive column omissions. This is demonstrated in
the attached test.

This can be addressed by saving the size-limits before invoking
internal type and cast related functions. I've attached the patch.

thanks
Shveta

On Pub and Sub:

CREATE TYPE inner_bad_enum AS ENUM ('v');

CREATE FUNCTION inner_bad_enum_to_json(inner_bad_enum) RETURNS json AS $f$
DECLARE
    big_arr int[];
    nested_result json;
BEGIN
    -- A large, all-NULL array: small storage footprint, huge JSON output
    big_arr := array_fill(NULL::int, ARRAY[130500,1,1,1,1,1]);

    -- UNRELATED call into json.c. Output is discraded
    nested_result := array_to_json(big_arr);

    -- Return something tiny; the large intermediate result is discarded.
    RETURN json_build_object('note', 'nested call above was large internally');
END;
$f$ LANGUAGE plpgsql IMMUTABLE;

CREATE CAST (inner_bad_enum AS json)
    WITH FUNCTION inner_bad_enum_to_json(inner_bad_enum) AS IMPLICIT;

CREATE TYPE outer_composite AS (field1 inner_bad_enum, field2 int);

CREATE TABLE leak_reentry_tab (a int PRIMARY KEY, b outer_composite);



Pub:
INSERT INTO leak_reentry_tab VALUES (1, ROW('v'::inner_bad_enum, 
42)::outer_composite);

Sub:
DELETE FROM leak_reentry_tab WHERE a = 1;

Pub:
UPDATE leak_reentry_tab SET a = 1 WHERE a = 1;


I get:
postgres=# select * from pg_conflict.pg_conflict_log_16400;
 relid | schemaname |     relname      | conflict_type  | remote_xid | 
remote_commit_lsn |         remote_commit_ts         | remote_origin | 
replica_identity_full | replica_identity |               remote_tuple           
 
   | local_conflicts | has_omitted_values 
-------+------------+------------------+----------------+------------+-------------------+----------------------------------+---------------+-----------------------+------------------+---------------------------------------
---+-----------------+--------------------
 16392 | public     | leak_reentry_tab | update_missing |        701 | 
0/017D5920        | 2026-09-01 10:53:08.752213+05:30 | pg_16400      | f        
             | {"a":1}          | {"a":1,"b":{"omitted":true,"length":28
}} |                 | t
(1 row)

See 'b' got ommited even though length is way below threshold.

While 'b' was nothing but : 'note', 'nested call above was large internally'. 
Everything above json_build_object in inner_bad_enum_to_json() was a 
discaradable output.

If I change code to do this:
json_size_would_exceed(int currentlen, Size addlen)
{
        if (json_size_limit && (Size) currentlen + addlen > json_size_limit)
        {
//              json_size_limit_hit = true;
                json_size_limit_hit = false;
                elog (LOG, "HIT THE LIMIT but not setting limit_hit");

                return true;
        }
        return false;
}

I get remote_tuple correctly reported as:
{"a":1,"b":{"field1":{"note" : "nested call above was large 
internally"},"field2":42}}
This is to show that, there is nothing wrong with 'b' as such.

~~

Also I noticed that log shows:
DETAIL:  Could not find the row to be updated: remote row (1, (v,42)), replica 
identity (a)=(1).

While CLT shows 
{"a":1,"b":{"field1":{"note" : "nested call above was large 
internally"},"field2":42}}

Value of 'b' is different in both.

0001-reentrant-json-fix.patch
Description: Binary data