Hello!
We encountered an issue related to internal memory allocation limit:
ERROR: invalid memory alloc request size
In the documentation on limits:
https://www.postgresql.org/docs/17/limits.html
the description suggests that only a single field is limited to 1GB,
which could imply that the total tuple size can be larger. So as we
planned to store three columns of 1GB each in a table and attempted to
insert this data, we got the error.
Our research showed that the limit is imposed by the palloc() function,
regardless of whether it is a tuple or not, and if the data is
serialized or dumped, the effective limit can be even lower, typically
around 512MB per row. So for allocations exceeding 1GB, the
palloc_extended() function can be used. Please correct me if I'm wrong.
I prepared a small patch for master, if it's worth clarifying, could you
please review the attachment?
--
Ekaterina Kiryanova
Technical Writer
Postgres Professional
the Russian PostgreSQL Company
diff --git a/doc/src/sgml/limits.sgml b/doc/src/sgml/limits.sgml
index f26f4466719..89c383d1db7 100644
--- a/doc/src/sgml/limits.sgml
+++ b/doc/src/sgml/limits.sgml
@@ -71,7 +71,8 @@
<row>
<entry>field size</entry>
<entry>1 GB</entry>
- <entry></entry>
+ <entry>limited by <function>palloc()</function> restriction; see note
+ below</entry>
</row>
<row>
@@ -146,4 +147,14 @@
Typically, this is only an issue for tables containing many terabytes
of data; partitioning is a possible workaround.
</para>
+
+ <para>
+ The 1 GB field size limit is imposed by the memory-allocating
+ <function>palloc()</function> function. The practical limit is less than
+ 1 GB, especially if the data is serialized or dumped. In cases where
+ larger allocations are needed, <function>palloc_extended()</function>
+ can be used, as it bypasses this restriction by invoking
+ <function>malloc()</function> for large memory blocks, which allows
+ requesting memory directly from the operating system.
+ </para>
</appendix>
