On Wed, 2026-08-19 at 10:44 +0300, Heikki Linnakangas wrote: > Looking at pg_strfold() (and other similar functions), there's no > indication that there's an upper bound on the destination size like > this. Does UNICODE_CASEMAP_BUFSZ work for all collation providers and > locales? How about adding a comment in pg_strfold() and friends about > that?
Attached two patches. The first one is the same and backpatches through 18. The second defines some new macros and uses those, which is a better place to document the limits, and I'll only commit that one to master. Along with the Case Mapping Complexities comment I'm adding here: https://www.postgresql.org/message-id/[email protected] (which will be backpatched to 18), I think that should be sufficient, but I could add another note there about the byte limits if you think it would be worthwhile. Regards, Jeff Davis
From eb7a9414f0b8a8c6c64e7249b5ed4aab1177d633 Mon Sep 17 00:00:00 2001 From: Jeff Davis <[email protected]> Date: Tue, 18 Aug 2026 14:57:32 -0700 Subject: [PATCH v2 1/2] ltree/crc32.c: fix fragile code. Explicitly make space for the NUL when casefolding. No known bug in the previous code, because the previous buffer (size 12) was more than large enough for folding any codepoint with enough room left for a NUL. The builtin provider's limit is 7; ICU's limit seems to be 7 also; and libc always does 1:1 mappings so the real limit is MAX_MULTIBYTE_CHAR_LEN + 1 (size 5). Reviewed-by: Heikki Linnakangas <[email protected]> Discussion: https://postgr.es/m/[email protected] Backpatch-through: 18 --- contrib/ltree/crc32.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contrib/ltree/crc32.c b/contrib/ltree/crc32.c index d21bed31fdd..617cc652a7a 100644 --- a/contrib/ltree/crc32.c +++ b/contrib/ltree/crc32.c @@ -32,13 +32,13 @@ ltree_crc32_sz(const char *buf, int size) INIT_TRADITIONAL_CRC32(crc); while (size > 0) { - char foldstr[UNICODE_CASEMAP_BUFSZ]; + char foldstr[UNICODE_CASEMAP_BUFSZ + 1]; int srclen = pg_mblen_range(p, end); size_t foldlen; /* fold one codepoint at a time */ - foldlen = pg_strfold(foldstr, UNICODE_CASEMAP_BUFSZ, p, srclen, - locale); + foldlen = pg_strfold(foldstr, sizeof(foldstr), p, srclen, locale); + Assert(foldlen < sizeof(foldstr)); COMP_TRADITIONAL_CRC32(crc, foldstr, foldlen); -- 2.43.0
From 5013ed3d59403d943aae471c1033f43976cfc11c Mon Sep 17 00:00:00 2001 From: Jeff Davis <[email protected]> Date: Tue, 18 Aug 2026 14:57:32 -0700 Subject: [PATCH v2 2/2] pg_locale.h: define casemapping limits. Clearly define casemapping upper bounds that cover all providers, locales, and encodings in pg_locale.h. Also redefine UTF8_CASEMAP_BUFSZ to include room for a NUL for consistency with PG_CASEMAP_BUFSZ. Use PG_CASEMAP_BUFSZ for the static buffer in ltree/crc32.c. Reviewed-by: Heikki Linnakangas <[email protected]> Discussion: https://postgr.es/m/[email protected] --- contrib/ltree/crc32.c | 2 +- .../unicode/generate-unicode_case_table.pl | 4 +-- src/include/common/unicode_limits.h | 4 +-- src/include/utils/pg_locale.h | 35 ++++++++++++++----- 4 files changed, 31 insertions(+), 14 deletions(-) diff --git a/contrib/ltree/crc32.c b/contrib/ltree/crc32.c index 617cc652a7a..2082b9ef851 100644 --- a/contrib/ltree/crc32.c +++ b/contrib/ltree/crc32.c @@ -32,7 +32,7 @@ ltree_crc32_sz(const char *buf, int size) INIT_TRADITIONAL_CRC32(crc); while (size > 0) { - char foldstr[UNICODE_CASEMAP_BUFSZ + 1]; + char foldstr[PG_CASEMAP_BUFSZ]; int srclen = pg_mblen_range(p, end); size_t foldlen; diff --git a/src/common/unicode/generate-unicode_case_table.pl b/src/common/unicode/generate-unicode_case_table.pl index d1dd3a5e8f2..059fd1231cf 100644 --- a/src/common/unicode/generate-unicode_case_table.pl +++ b/src/common/unicode/generate-unicode_case_table.pl @@ -280,9 +280,9 @@ print $LIMITS <<"EOS"; /* * The maximum number of UTF8 bytes needed to store the result of case - * mapping a single code point. + * mapping a single code point, including terminating NUL. */ -#define UTF8_CASEMAP_BUFSZ $UTF8_CASEMAP_BUFSZ +#define UTF8_CASEMAP_BUFSZ ($UTF8_CASEMAP_BUFSZ + 1) #endif EOS diff --git a/src/include/common/unicode_limits.h b/src/include/common/unicode_limits.h index 59e307857ed..2307141c823 100644 --- a/src/include/common/unicode_limits.h +++ b/src/include/common/unicode_limits.h @@ -32,8 +32,8 @@ /* * The maximum number of UTF8 bytes needed to store the result of case - * mapping a single code point. + * mapping a single code point, including terminating NUL. */ -#define UTF8_CASEMAP_BUFSZ 6 +#define UTF8_CASEMAP_BUFSZ (6 + 1) #endif diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index fcd508f5dd6..6f82075c61d 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -13,23 +13,40 @@ #define _PG_LOCALE_ #include "mb/pg_wchar.h" +#include "common/unicode_limits.h" /* use for libc locale names */ #define LOCALE_NAME_BUFLEN 128 /* - * Maximum number of bytes needed to map a single codepoint. Useful for - * mapping and processing a single input codepoint at a time with a - * statically-allocated buffer. + * Expansion factor of string length, not including terminating NUL. That is, + * the upper bound of the number of multibyte characters in the result string + * per multibyte character in the input string. * - * With full case mapping, an input codepoint may be mapped to as many as - * three output codepoints. See Unicode 16.0.0, section 5.18.2, "Change in - * Length": + * NB: assumes no provider exceeds the Unicode-defined maximum. + */ +#define PG_MAX_CASEMAP_MBCHARS UNICODE_MAX_CASEMAP_CODEPOINTS + +/* + * Expansion factor of a string in bytes, not including terminating NUL. + * + * This is a conservative upper bound, assuming that each character in the + * input string is a 1-byte character that maps to PG_MAX_CASEMAP_MBCHARS + * other characters, all requiring MAX_MULTIBYTE_CHAR_LEN bytes. + */ +#define PG_MAX_CASEMAP_EXPANSION (PG_MAX_CASEMAP_MBCHARS * \ + MAX_MULTIBYTE_CHAR_LEN) + +/* + * The maximum number of bytes needed to store the result of case mapping a + * single multibyte character, including terminating NUL. * - * https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-5/#G29675 + * This is a conservative upper bound, assuming that a single multibyte + * character can expand into PG_MAX_CASEMAP_MBCHARS other multibyte + * characters, each requiring MAX_MULTIBYTE_CHAR_LEN bytes. */ -#define UNICODE_CASEMAP_LEN 3 -#define UNICODE_CASEMAP_BUFSZ (UNICODE_CASEMAP_LEN * MAX_MULTIBYTE_CHAR_LEN) +#define PG_CASEMAP_BUFSZ ((PG_MAX_CASEMAP_MBCHARS * \ + MAX_MULTIBYTE_CHAR_LEN) + 1) /* GUC settings */ extern PGDLLIMPORT char *locale_messages; -- 2.43.0
