Hi,
Sehrope Sarkuni <[email protected]> wrote:
> On this machine, block sizes from 16 kB up to 4 MB performed the same
> within noise, while 16 MB was consistently slower for large results,
> much like v4's pure doubling. So I don't see a reason to go bigger
> than 16 kB.
I ran v5 on a different x86 machine: an AMD Ryzen AI 9 HX 370 (Zen 5),
glibc 2.43, where x86_non_temporal_threshold is 6 MiB. Release builds
of master 2c10c2ce4d7, with your method: pgbench -c 1, rounds
alternating between builds, median of 5. This CPU mixes Zen 5 and
Zen 5c cores, so the server was pinned to one Zen 5 core. I also
included Chenhui's v2 from the repeat() thread (Ready for Committer),
since 0003 replaces the same loop.
1. Block size
The repeat() thread found the best block to be the first one at or
above glibc's non-temporal threshold, on three Intel machines. To
test that here, I built v5 with REPEAT_BYTES_BLOCK = 8 MB, above the
6 MiB threshold, and ran that build a second time with non-temporal
stores disabled through GLIBC_TUNABLES.
At 100 MB the three stay within the round-to-round spread (about 10%)
for repeat() and rpad() with 1-, 10- and 100-byte sources. For
example, repeat() with a 10-byte source took 46.8 ms with 16 kB,
50.4 ms with 8 MB, and 46.7 ms with 8 MB and no non-temporal stores.
So the effect seen on the Intel machines does not show up on this
Zen 5, and 16 kB costs nothing here.
2. repeat() against the Ready for Committer patch
- 1-byte source, 100 MB: master 173.1 ms, v5 47.2 ms, RfC 47.7 ms
- 1-byte source, 16 MB: master 27.3 ms, v5 7.0 ms, RfC 6.7 ms
- 10-byte source, 100 MB: master 58.6 ms, v5 46.8 ms, RfC 55.3 ms
- 100-byte source, 100 MB: master 46.8 ms, v5 44.9 ms
So on this machine dropping the memset() path costs nothing for a
one-byte source, and v5 also speeds up the 10-byte case, which the RfC
patch does not touch. Jan's M5 Max numbers upthread had chunked
memset() well behind a single one. I have no ARM machine to check
whether that carries over to repeat_bytes().
3. rpad(), 100 MB
- 1-byte pad: master 330.6 ms, v5 45.5 ms
- 10-byte pad: master 342.5 ms, v5 44.6 ms
- 100-byte pad: master 325.9 ms, v5 43.2 ms
4. Results are unchanged
I compared lpad(), rpad() and repeat() output between master and v5
in UTF8, EUC_JP and SQL_ASCII databases: 6075 calls, with lengths
around the 16 kB and 8 MB boundaries, multibyte and mixed pads with
partial tails, and sizes near the 1 GB limit. Lengths, md5 of every
result and error messages are identical, including the same 16
"requested length too large" errors. Invalid pad strings are not in
it, since the encoding.sql tests cover those.
5. Cancellation
On master, lpad() and rpad() have no CHECK_FOR_INTERRUPTS() in the
padding loop. With statement_timeout = 20ms,
rpad('x', 268000000, 'ab') runs for about 900 ms, the whole call,
before the cancel takes effect. With v5 it is cancelled after about
24 ms, as repeat() already is on master (21 ms), because the padding
now goes through repeat_bytes().
Scripts, raw timings and outputs are in the attachment.
Regards,
Manu
Review of v5 (lpad/rpad/repeat): timings, differential check, cancellation
master 2c10c2ce4d7; release builds (default -O2, no asserts)
===== machine =====
Model name: AMD Ryzen AI 9 HX 370 w/ Radeon 890M
L1d cache: 576 KiB (12 instances)
L2 cache: 12 MiB (12 instances)
L3 cache: 24 MiB (2 instances)
cores 0-3 Zen 5, 4-11 Zen 5c; server pinned to CPU 0, pgbench/psql to CPU 2
ldd (GNU libc) 2.43
glibc.cpu.x86_data_cache_size: 0xc000 (min: 0x0, max: 0xffffffffffffffff)
glibc.cpu.x86_memset_non_temporal_threshold: 0x600000 (min: 0x4040, max:
0xffffffffffffffff)
glibc.cpu.x86_non_temporal_threshold: 0x600000 (min: 0x4040, max:
0xfffffffffffffff)
glibc.cpu.x86_rep_movsb_threshold: 0x600000 (min: 0x200, max:
0xffffffffffffffff)
glibc.cpu.x86_shared_cache_size: 0x800000 (min: 0x0, max: 0xffffffffffffffff)
"===== build.sh ====="
#!/bin/bash
# Release builds (no asserts, default -O2) from the same master commit:
# master control
# v5 Sehrope's v5 (0001-0003), REPEAT_BYTES_BLOCK = 16 kB
# v5b8m v5 with REPEAT_BYTES_BLOCK = 8 MB, the only change
# rfc Chenhui's repeat() v2 (Ready for Committer): early return + memset()
set -eu
BASE=${BASE:-2c10c2ce4d7}
SRC=$HOME/Proyectos/postgresql
W=$HOME/pgpad
A=$(cd "$(dirname "$0")" && pwd)
mkdir -p $W
tree() { # name patch...
local name=$1; shift
local t=$W/src-$name
[ -d $t ] && return
git -C $SRC worktree add -q --detach $t $BASE
for p in "$@"; do git -C $t am -q "$p"; done
}
tree master
tree v5 $A/sehrope-v5/v5-000{1,2,3}-*.patch
tree v5b8m $A/sehrope-v5/v5-000{1,2,3}-*.patch
sed -i 's/^#define REPEAT_BYTES_BLOCK\t16384$/#define REPEAT_BYTES_BLOCK\t(8 *
1024 * 1024)/' \
$W/src-v5b8m/src/backend/utils/adt/oracle_compat.c
$W/src-v5b8m/src/backend/utils/adt/varlena.c 2>/dev/null || true
grep -rn 'define REPEAT_BYTES_BLOCK' $W/src-v5b8m/src/backend/utils/adt/
tree rfc $A/chenhui-v2/v2-000{1,2}-*.patch
build() {
local name=$1
mkdir -p $W/b-$name && cd $W/b-$name
$W/src-$name/configure --prefix=$W/i-$name --quiet > configure.log 2>&1
make -j"$(nproc)" -s > build.log 2>&1
make -s install > install.log 2>&1
echo "$name: $(git -C $W/src-$name log -1 --format='%h %s' | cut -c1-60)
warnings=$(grep -c 'warning:' build.log)"
}
build master & build v5 & build v5b8m & build rfc & wait
echo BUILD-DONE
"===== bench.sh ====="
#!/bin/bash
# repeat() and rpad() timings on master and the patched builds, following
# Sehrope's method: pgbench -c 1, one statement per transaction, rounds
# alternating between builds, median of ROUNDS.
#
# Configs (build[:env]):
# master v5 v5b8m rfc
# v5b8m-nont = v5b8m with glibc's non-temporal stores disabled
# (GLIBC_TUNABLES), to see whether a gain comes from them
#
# This CPU mixes Zen 5 and Zen 5c cores, so the server runs pinned to
# SRVCPU and pgbench to CLICPU.
# bench.sh [ROUNDS]
set -u
ROUNDS=${1:-5}
A=$(cd "$(dirname "$0")" && pwd)
W=$HOME/pgpad
SRVCPU=${SRVCPU:-0}; CLICPU=${CLICPU:-2}
CONFIGS=${CONFIGS:-"master v5 v5b8m v5b8m-nont rfc"}
# glibc 2.43 silently keeps the default when the value equals the listed max
# of the memset tunable (0xffffffffffffffff), so both use 0xfffffffffffffff;
# check with: GLIBC_TUNABLES=$NONT /lib64/ld-linux-x86-64.so.2 --list-tunables
# (the first run of this script had the memset one ignored; v5 does not use
# memset(), so that run's v5b8m-nont numbers are unaffected)
NONT='glibc.cpu.x86_non_temporal_threshold=0xfffffffffffffff:glibc.cpu.x86_memset_non_temporal_threshold=0xfffffffffffffff'
PAD=abcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghijabcdefghij
OUT=$A/bench.raw; : > $OUT
port=55480
declare -A PORT DIR
for c in $CONFIGS; do
b=${c%-nont}
D=$(mktemp -d /tmp/claude-1000/pad.XXXX); DIR[$c]=$D; PORT[$c]=$port
$W/i-$b/bin/initdb -D $D -A trust --no-sync -U postgres >/dev/null
printf "port = $port\nunix_socket_directories = '/tmp'\nmax_connections =
5\n" >> $D/postgresql.conf
if [ "$c" != "$b" ]; then
GLIBC_TUNABLES=$NONT taskset -c $SRVCPU $W/i-$b/bin/pg_ctl -D $D -l $D/log
-w start >/dev/null
else
taskset -c $SRVCPU $W/i-$b/bin/pg_ctl -D $D -l $D/log -w start >/dev/null
fi
port=$((port + 1))
done
# case lines: name|sql|transactions per round
cases() {
for len in 1 10 100; do
s=${PAD:0:$len}
for n in 1000000 16000000 100000000; do
t=$(( n >= 100000000 ? 5 : (n >= 16000000 ? 20 : 200) ))
echo "repeat len=$len N=$n|SELECT octet_length(repeat('$s', $((n /
len))))|$t"
echo "rpad len=$len N=$n|SELECT octet_length(rpad('x', $n, '$s'))|$t"
done
done
}
for r in $(seq $ROUNDS); do
# rotate the order of the configs each round
set -- $CONFIGS; for _ in $(seq $(( (r - 1) % $# ))); do set -- "${@:2}"
"$1"; done
order="$*"
cases | while IFS='|' read -r name sql t; do
for c in $order; do
b=${c%-nont}
f=$(mktemp /tmp/claude-1000/q.XXXX.sql); echo "$sql;" > $f
lat=$(taskset -c $CLICPU $W/i-$b/bin/pgbench -n -h /tmp -p ${PORT[$c]} -U
postgres -c 1 -t $t -f $f postgres 2>/dev/null \
| awk '/latency average/ {print $4}')
rm -f $f
echo "$r|$c|$name|$lat" >> $OUT
done
done
echo "round $r done" >&2
done
for c in $CONFIGS; do
b=${c%-nont}
$W/i-$b/bin/pg_ctl -D ${DIR[$c]} -m fast -w stop >/dev/null; rm -rf ${DIR[$c]}
done
# median per (config, case), one line per case: master first, then the rest
python3 - "$OUT" "$CONFIGS" <<'EOF' | tee $A/bench.txt
import sys, statistics, collections
rows = [l.rstrip("\n").split("|") for l in open(sys.argv[1]) if l.strip()]
configs = sys.argv[2].split()
med = collections.defaultdict(list)
order = []
for r, c, name, lat in rows:
if lat:
med[(c, name)].append(float(lat))
if name not in order:
order.append(name)
print("median latency in ms, " + ", ".join(configs))
for name in order:
vals = [statistics.median(med[(c, name)]) if med[(c, name)] else
float("nan") for c in configs]
print(f"{name}: " + ", ".join(f"{v:.3f}" for v in vals))
EOF
===== bench.txt (./bench.sh 5) =====
median latency in ms, master, v5, v5b8m, v5b8m-nont, rfc
repeat len=1 N=1000000: 1.487, 0.211, 0.228, 0.216, 0.219
rpad len=1 N=1000000: 2.949, 0.043, 0.042, 0.045, 2.924
repeat len=1 N=16000000: 27.279, 6.995, 7.623, 7.136, 6.695
rpad len=1 N=16000000: 49.933, 4.108, 4.056, 3.730, 50.064
repeat len=1 N=100000000: 173.132, 47.155, 46.926, 47.027, 47.727
rpad len=1 N=100000000: 330.626, 45.497, 47.149, 48.070, 330.721
repeat len=10 N=1000000: 0.327, 0.220, 0.210, 0.246, 0.327
rpad len=10 N=1000000: 2.999, 0.038, 0.043, 0.043, 2.939
repeat len=10 N=16000000: 8.480, 6.071, 5.930, 6.239, 8.619
rpad len=10 N=16000000: 50.406, 3.777, 4.464, 3.771, 51.329
repeat len=10 N=100000000: 58.642, 46.783, 50.412, 46.664, 55.334
rpad len=10 N=100000000: 342.535, 44.577, 46.989, 46.646, 328.291
repeat len=100 N=1000000: 0.239, 0.229, 0.226, 0.223, 0.262
rpad len=100 N=1000000: 2.929, 0.048, 0.052, 0.060, 2.983
repeat len=100 N=16000000: 6.348, 6.251, 6.729, 6.429, 6.750
rpad len=100 N=16000000: 51.905, 3.849, 4.138, 4.208, 50.296
repeat len=100 N=100000000: 46.811, 44.867, 47.551, 48.362, 46.295
rpad len=100 N=100000000: 325.854, 43.249, 48.334, 50.560, 335.755
===== bench.raw (round|config|case|latency ms) =====
1|master|repeat len=1 N=1000000|1.573
1|v5|repeat len=1 N=1000000|0.211
1|v5b8m|repeat len=1 N=1000000|0.228
1|v5b8m-nont|repeat len=1 N=1000000|0.276
1|rfc|repeat len=1 N=1000000|0.233
1|master|rpad len=1 N=1000000|3.044
1|v5|rpad len=1 N=1000000|0.051
1|v5b8m|rpad len=1 N=1000000|0.044
1|v5b8m-nont|rpad len=1 N=1000000|0.051
1|rfc|rpad len=1 N=1000000|2.921
1|master|repeat len=1 N=16000000|27.815
1|v5|repeat len=1 N=16000000|6.995
1|v5b8m|repeat len=1 N=16000000|7.710
1|v5b8m-nont|repeat len=1 N=16000000|7.377
1|rfc|repeat len=1 N=16000000|7.540
1|master|rpad len=1 N=16000000|52.476
1|v5|rpad len=1 N=16000000|3.600
1|v5b8m|rpad len=1 N=16000000|4.056
1|v5b8m-nont|rpad len=1 N=16000000|3.650
1|rfc|rpad len=1 N=16000000|50.327
1|master|repeat len=1 N=100000000|168.610
1|v5|repeat len=1 N=100000000|46.517
1|v5b8m|repeat len=1 N=100000000|46.926
1|v5b8m-nont|repeat len=1 N=100000000|46.215
1|rfc|repeat len=1 N=100000000|47.727
1|master|rpad len=1 N=100000000|330.626
1|v5|rpad len=1 N=100000000|43.893
1|v5b8m|rpad len=1 N=100000000|45.238
1|v5b8m-nont|rpad len=1 N=100000000|46.845
1|rfc|rpad len=1 N=100000000|327.350
1|master|repeat len=10 N=1000000|0.317
1|v5|repeat len=10 N=1000000|0.220
1|v5b8m|repeat len=10 N=1000000|0.210
1|v5b8m-nont|repeat len=10 N=1000000|0.261
1|rfc|repeat len=10 N=1000000|0.334
1|master|rpad len=10 N=1000000|3.134
1|v5|rpad len=10 N=1000000|0.038
1|v5b8m|rpad len=10 N=1000000|0.042
1|v5b8m-nont|rpad len=10 N=1000000|0.050
1|rfc|rpad len=10 N=1000000|2.939
1|master|repeat len=10 N=16000000|8.018
1|v5|repeat len=10 N=16000000|5.506
1|v5b8m|repeat len=10 N=16000000|5.930
1|v5b8m-nont|repeat len=10 N=16000000|5.743
1|rfc|repeat len=10 N=16000000|8.459
1|master|rpad len=10 N=16000000|52.242
1|v5|rpad len=10 N=16000000|3.798
1|v5b8m|rpad len=10 N=16000000|5.439
1|v5b8m-nont|rpad len=10 N=16000000|5.600
1|rfc|rpad len=10 N=16000000|52.484
1|master|repeat len=10 N=100000000|58.642
1|v5|repeat len=10 N=100000000|46.918
1|v5b8m|repeat len=10 N=100000000|52.342
1|v5b8m-nont|repeat len=10 N=100000000|50.504
1|rfc|repeat len=10 N=100000000|58.684
1|master|rpad len=10 N=100000000|328.584
1|v5|rpad len=10 N=100000000|46.283
1|v5b8m|rpad len=10 N=100000000|46.989
1|v5b8m-nont|rpad len=10 N=100000000|47.474
1|rfc|rpad len=10 N=100000000|332.579
1|master|repeat len=100 N=1000000|0.237
1|v5|repeat len=100 N=1000000|0.206
1|v5b8m|repeat len=100 N=1000000|0.213
1|v5b8m-nont|repeat len=100 N=1000000|0.206
1|rfc|repeat len=100 N=1000000|0.224
1|master|rpad len=100 N=1000000|2.868
1|v5|rpad len=100 N=1000000|0.048
1|v5b8m|rpad len=100 N=1000000|0.062
1|v5b8m-nont|rpad len=100 N=1000000|0.063
1|rfc|rpad len=100 N=1000000|3.028
1|master|repeat len=100 N=16000000|7.206
1|v5|repeat len=100 N=16000000|6.326
1|v5b8m|repeat len=100 N=16000000|6.675
1|v5b8m-nont|repeat len=100 N=16000000|6.117
1|rfc|repeat len=100 N=16000000|6.750
1|master|rpad len=100 N=16000000|49.635
1|v5|rpad len=100 N=16000000|4.055
1|v5b8m|rpad len=100 N=16000000|4.138
1|v5b8m-nont|rpad len=100 N=16000000|4.208
1|rfc|rpad len=100 N=16000000|50.296
1|master|repeat len=100 N=100000000|46.811
1|v5|repeat len=100 N=100000000|38.952
1|v5b8m|repeat len=100 N=100000000|43.863
1|v5b8m-nont|repeat len=100 N=100000000|47.189
1|rfc|repeat len=100 N=100000000|45.777
1|master|rpad len=100 N=100000000|322.829
1|v5|rpad len=100 N=100000000|42.335
1|v5b8m|rpad len=100 N=100000000|45.797
1|v5b8m-nont|rpad len=100 N=100000000|47.126
1|rfc|rpad len=100 N=100000000|336.829
2|v5|repeat len=1 N=1000000|0.210
2|v5b8m|repeat len=1 N=1000000|0.245
2|v5b8m-nont|repeat len=1 N=1000000|0.208
2|rfc|repeat len=1 N=1000000|0.202
2|master|repeat len=1 N=1000000|1.487
2|v5|rpad len=1 N=1000000|0.035
2|v5b8m|rpad len=1 N=1000000|0.041
2|v5b8m-nont|rpad len=1 N=1000000|0.042
2|rfc|rpad len=1 N=1000000|2.886
2|master|rpad len=1 N=1000000|2.930
2|v5|repeat len=1 N=16000000|7.046
2|v5b8m|repeat len=1 N=16000000|6.534
2|v5b8m-nont|repeat len=1 N=16000000|6.739
2|rfc|repeat len=1 N=16000000|6.244
2|master|repeat len=1 N=16000000|25.690
2|v5|rpad len=1 N=16000000|4.209
2|v5b8m|rpad len=1 N=16000000|4.004
2|v5b8m-nont|rpad len=1 N=16000000|3.687
2|rfc|rpad len=1 N=16000000|48.573
2|master|rpad len=1 N=16000000|49.657
2|v5|repeat len=1 N=100000000|47.155
2|v5b8m|repeat len=1 N=100000000|55.758
2|v5b8m-nont|repeat len=1 N=100000000|47.238
2|rfc|repeat len=1 N=100000000|43.216
2|master|repeat len=1 N=100000000|164.346
2|v5|rpad len=1 N=100000000|43.217
2|v5b8m|rpad len=1 N=100000000|47.149
2|v5b8m-nont|rpad len=1 N=100000000|56.707
2|rfc|rpad len=1 N=100000000|340.892
2|master|rpad len=1 N=100000000|325.681
2|v5|repeat len=10 N=1000000|0.210
2|v5b8m|repeat len=10 N=1000000|0.206
2|v5b8m-nont|repeat len=10 N=1000000|0.246
2|rfc|repeat len=10 N=1000000|0.514
2|master|repeat len=10 N=1000000|0.473
2|v5|rpad len=10 N=1000000|0.036
2|v5b8m|rpad len=10 N=1000000|0.045
2|v5b8m-nont|rpad len=10 N=1000000|0.043
2|rfc|rpad len=10 N=1000000|2.893
2|master|rpad len=10 N=1000000|2.857
2|v5|repeat len=10 N=16000000|5.536
2|v5b8m|repeat len=10 N=16000000|5.716
2|v5b8m-nont|repeat len=10 N=16000000|5.801
2|rfc|repeat len=10 N=16000000|7.902
2|master|repeat len=10 N=16000000|7.745
2|v5|rpad len=10 N=16000000|3.293
2|v5b8m|rpad len=10 N=16000000|3.757
2|v5b8m-nont|rpad len=10 N=16000000|3.173
2|rfc|rpad len=10 N=16000000|48.776
2|master|rpad len=10 N=16000000|48.802
2|v5|repeat len=10 N=100000000|40.933
2|v5b8m|repeat len=10 N=100000000|42.228
2|v5b8m-nont|repeat len=10 N=100000000|46.664
2|rfc|repeat len=10 N=100000000|55.334
2|master|repeat len=10 N=100000000|53.506
2|v5|rpad len=10 N=100000000|43.690
2|v5b8m|rpad len=10 N=100000000|50.387
2|v5b8m-nont|rpad len=10 N=100000000|50.047
2|rfc|rpad len=10 N=100000000|328.291
2|master|rpad len=10 N=100000000|333.304
2|v5|repeat len=100 N=1000000|0.229
2|v5b8m|repeat len=100 N=1000000|0.214
2|v5b8m-nont|repeat len=100 N=1000000|0.223
2|rfc|repeat len=100 N=1000000|0.266
2|master|repeat len=100 N=1000000|0.239
2|v5|rpad len=100 N=1000000|0.057
2|v5b8m|rpad len=100 N=1000000|0.052
2|v5b8m-nont|rpad len=100 N=1000000|0.049
2|rfc|rpad len=100 N=1000000|2.983
2|master|rpad len=100 N=1000000|2.924
2|v5|repeat len=100 N=16000000|5.575
2|v5b8m|repeat len=100 N=16000000|6.330
2|v5b8m-nont|repeat len=100 N=16000000|5.573
2|rfc|repeat len=100 N=16000000|6.332
2|master|repeat len=100 N=16000000|6.348
2|v5|rpad len=100 N=16000000|3.478
2|v5b8m|rpad len=100 N=16000000|3.847
2|v5b8m-nont|rpad len=100 N=16000000|4.249
2|rfc|rpad len=100 N=16000000|49.297
2|master|rpad len=100 N=16000000|50.337
2|v5|repeat len=100 N=100000000|47.476
2|v5b8m|repeat len=100 N=100000000|49.504
2|v5b8m-nont|repeat len=100 N=100000000|50.675
2|rfc|repeat len=100 N=100000000|48.258
2|master|repeat len=100 N=100000000|47.584
2|v5|rpad len=100 N=100000000|49.109
2|v5b8m|rpad len=100 N=100000000|50.932
2|v5b8m-nont|rpad len=100 N=100000000|53.914
2|rfc|rpad len=100 N=100000000|335.755
2|master|rpad len=100 N=100000000|325.102
3|v5b8m|repeat len=1 N=1000000|0.225
3|v5b8m-nont|repeat len=1 N=1000000|0.216
3|rfc|repeat len=1 N=1000000|0.204
3|master|repeat len=1 N=1000000|1.444
3|v5|repeat len=1 N=1000000|0.214
3|v5b8m|rpad len=1 N=1000000|0.045
3|v5b8m-nont|rpad len=1 N=1000000|0.045
3|rfc|rpad len=1 N=1000000|2.924
3|master|rpad len=1 N=1000000|2.949
3|v5|rpad len=1 N=1000000|0.039
3|v5b8m|repeat len=1 N=16000000|6.969
3|v5b8m-nont|repeat len=1 N=16000000|7.307
3|rfc|repeat len=1 N=16000000|7.080
3|master|repeat len=1 N=16000000|27.899
3|v5|repeat len=1 N=16000000|7.071
3|v5b8m|rpad len=1 N=16000000|4.553
3|v5b8m-nont|rpad len=1 N=16000000|4.437
3|rfc|rpad len=1 N=16000000|49.731
3|master|rpad len=1 N=16000000|49.445
3|v5|rpad len=1 N=16000000|4.248
3|v5b8m|repeat len=1 N=100000000|46.587
3|v5b8m-nont|repeat len=1 N=100000000|47.027
3|rfc|repeat len=1 N=100000000|49.059
3|master|repeat len=1 N=100000000|173.132
3|v5|repeat len=1 N=100000000|47.468
3|v5b8m|rpad len=1 N=100000000|48.186
3|v5b8m-nont|rpad len=1 N=100000000|48.658
3|rfc|rpad len=1 N=100000000|328.980
3|master|rpad len=1 N=100000000|327.377
3|v5|rpad len=1 N=100000000|46.348
3|v5b8m|repeat len=10 N=1000000|0.204
3|v5b8m-nont|repeat len=10 N=1000000|0.219
3|rfc|repeat len=10 N=1000000|0.320
3|master|repeat len=10 N=1000000|0.327
3|v5|repeat len=10 N=1000000|0.212
3|v5b8m|rpad len=10 N=1000000|0.040
3|v5b8m-nont|rpad len=10 N=1000000|0.041
3|rfc|rpad len=10 N=1000000|2.883
3|master|rpad len=10 N=1000000|2.942
3|v5|rpad len=10 N=1000000|0.052
3|v5b8m|repeat len=10 N=16000000|8.526
3|v5b8m-nont|repeat len=10 N=16000000|6.800
3|rfc|repeat len=10 N=16000000|9.973
3|master|repeat len=10 N=16000000|9.376
3|v5|repeat len=10 N=16000000|6.894
3|v5b8m|rpad len=10 N=16000000|4.485
3|v5b8m-nont|rpad len=10 N=16000000|4.624
3|rfc|rpad len=10 N=16000000|50.511
3|master|rpad len=10 N=16000000|50.804
3|v5|rpad len=10 N=16000000|4.238
3|v5b8m|repeat len=10 N=100000000|57.266
3|v5b8m-nont|repeat len=10 N=100000000|58.048
3|rfc|repeat len=10 N=100000000|65.732
3|master|repeat len=10 N=100000000|65.570
3|v5|repeat len=10 N=100000000|49.838
3|v5b8m|rpad len=10 N=100000000|48.594
3|v5b8m-nont|rpad len=10 N=100000000|46.646
3|rfc|rpad len=10 N=100000000|325.341
3|master|rpad len=10 N=100000000|342.535
3|v5|rpad len=10 N=100000000|45.261
3|v5b8m|repeat len=100 N=1000000|0.226
3|v5b8m-nont|repeat len=100 N=1000000|0.243
3|rfc|repeat len=100 N=1000000|0.248
3|master|repeat len=100 N=1000000|0.227
3|v5|repeat len=100 N=1000000|0.216
3|v5b8m|rpad len=100 N=1000000|0.040
3|v5b8m-nont|rpad len=100 N=1000000|0.045
3|rfc|rpad len=100 N=1000000|2.916
3|master|rpad len=100 N=1000000|2.929
3|v5|rpad len=100 N=1000000|0.041
3|v5b8m|repeat len=100 N=16000000|6.860
3|v5b8m-nont|repeat len=100 N=16000000|6.576
3|rfc|repeat len=100 N=16000000|6.948
3|master|repeat len=100 N=16000000|7.063
3|v5|repeat len=100 N=16000000|6.928
3|v5b8m|rpad len=100 N=16000000|4.148
3|v5b8m-nont|rpad len=100 N=16000000|4.610
3|rfc|rpad len=100 N=16000000|50.180
3|master|rpad len=100 N=16000000|51.905
3|v5|rpad len=100 N=16000000|4.713
3|v5b8m|repeat len=100 N=100000000|45.009
3|v5b8m-nont|repeat len=100 N=100000000|42.563
3|rfc|repeat len=100 N=100000000|40.962
3|master|repeat len=100 N=100000000|39.873
3|v5|repeat len=100 N=100000000|43.671
3|v5b8m|rpad len=100 N=100000000|46.044
3|v5b8m-nont|rpad len=100 N=100000000|50.311
3|rfc|rpad len=100 N=100000000|332.144
3|master|rpad len=100 N=100000000|325.854
3|v5|rpad len=100 N=100000000|41.055
4|v5b8m-nont|repeat len=1 N=1000000|0.208
4|rfc|repeat len=1 N=1000000|0.219
4|master|repeat len=1 N=1000000|1.416
4|v5|repeat len=1 N=1000000|0.209
4|v5b8m|repeat len=1 N=1000000|0.227
4|v5b8m-nont|rpad len=1 N=1000000|0.039
4|rfc|rpad len=1 N=1000000|3.046
4|master|rpad len=1 N=1000000|2.882
4|v5|rpad len=1 N=1000000|0.043
4|v5b8m|rpad len=1 N=1000000|0.042
4|v5b8m-nont|repeat len=1 N=16000000|6.569
4|rfc|repeat len=1 N=16000000|6.695
4|master|repeat len=1 N=16000000|27.279
4|v5|repeat len=1 N=16000000|6.337
4|v5b8m|repeat len=1 N=16000000|7.623
4|v5b8m-nont|rpad len=1 N=16000000|3.730
4|rfc|rpad len=1 N=16000000|52.228
4|master|rpad len=1 N=16000000|51.453
4|v5|rpad len=1 N=16000000|3.720
4|v5b8m|rpad len=1 N=16000000|4.211
4|v5b8m-nont|repeat len=1 N=100000000|45.719
4|rfc|repeat len=1 N=100000000|47.822
4|master|repeat len=1 N=100000000|174.628
4|v5|repeat len=1 N=100000000|40.432
4|v5b8m|repeat len=1 N=100000000|40.503
4|v5b8m-nont|rpad len=1 N=100000000|42.527
4|rfc|rpad len=1 N=100000000|343.643
4|master|rpad len=1 N=100000000|333.860
4|v5|rpad len=1 N=100000000|45.497
4|v5b8m|rpad len=1 N=100000000|42.479
4|v5b8m-nont|repeat len=10 N=1000000|0.202
4|rfc|repeat len=10 N=1000000|0.323
4|master|repeat len=10 N=1000000|0.362
4|v5|repeat len=10 N=1000000|0.240
4|v5b8m|repeat len=10 N=1000000|0.284
4|v5b8m-nont|rpad len=10 N=1000000|0.043
4|rfc|rpad len=10 N=1000000|3.224
4|master|rpad len=10 N=1000000|2.999
4|v5|rpad len=10 N=1000000|0.038
4|v5b8m|rpad len=10 N=1000000|0.043
4|v5b8m-nont|repeat len=10 N=16000000|6.239
4|rfc|repeat len=10 N=16000000|8.619
4|master|repeat len=10 N=16000000|8.480
4|v5|repeat len=10 N=16000000|6.071
4|v5b8m|repeat len=10 N=16000000|5.893
4|v5b8m-nont|rpad len=10 N=16000000|3.291
4|rfc|rpad len=10 N=16000000|51.833
4|master|rpad len=10 N=16000000|48.376
4|v5|rpad len=10 N=16000000|3.425
4|v5b8m|rpad len=10 N=16000000|4.464
4|v5b8m-nont|repeat len=10 N=100000000|43.100
4|rfc|repeat len=10 N=100000000|52.106
4|master|repeat len=10 N=100000000|52.496
4|v5|repeat len=10 N=100000000|43.130
4|v5b8m|repeat len=10 N=100000000|46.421
4|v5b8m-nont|rpad len=10 N=100000000|46.446
4|rfc|rpad len=10 N=100000000|334.879
4|master|rpad len=10 N=100000000|345.571
4|v5|rpad len=10 N=100000000|44.577
4|v5b8m|rpad len=10 N=100000000|46.815
4|v5b8m-nont|repeat len=100 N=1000000|0.216
4|rfc|repeat len=100 N=1000000|0.262
4|master|repeat len=100 N=1000000|0.255
4|v5|repeat len=100 N=1000000|0.254
4|v5b8m|repeat len=100 N=1000000|0.254
4|v5b8m-nont|rpad len=100 N=1000000|0.060
4|rfc|rpad len=100 N=1000000|3.073
4|master|rpad len=100 N=1000000|3.176
4|v5|rpad len=100 N=1000000|0.060
4|v5b8m|rpad len=100 N=1000000|0.057
4|v5b8m-nont|repeat len=100 N=16000000|6.675
4|rfc|repeat len=100 N=16000000|6.211
4|master|repeat len=100 N=16000000|5.921
4|v5|repeat len=100 N=16000000|6.251
4|v5b8m|repeat len=100 N=16000000|6.846
4|v5b8m-nont|rpad len=100 N=16000000|3.940
4|rfc|rpad len=100 N=16000000|51.230
4|master|rpad len=100 N=16000000|52.143
4|v5|rpad len=100 N=16000000|3.849
4|v5b8m|rpad len=100 N=16000000|4.169
4|v5b8m-nont|repeat len=100 N=100000000|48.362
4|rfc|repeat len=100 N=100000000|46.295
4|master|repeat len=100 N=100000000|49.466
4|v5|repeat len=100 N=100000000|44.867
4|v5b8m|repeat len=100 N=100000000|47.551
4|v5b8m-nont|rpad len=100 N=100000000|53.332
4|rfc|rpad len=100 N=100000000|339.208
4|master|rpad len=100 N=100000000|334.218
4|v5|rpad len=100 N=100000000|43.249
4|v5b8m|rpad len=100 N=100000000|51.759
5|rfc|repeat len=1 N=1000000|0.285
5|master|repeat len=1 N=1000000|1.496
5|v5|repeat len=1 N=1000000|0.239
5|v5b8m|repeat len=1 N=1000000|0.241
5|v5b8m-nont|repeat len=1 N=1000000|0.241
5|rfc|rpad len=1 N=1000000|3.045
5|master|rpad len=1 N=1000000|3.028
5|v5|rpad len=1 N=1000000|0.067
5|v5b8m|rpad len=1 N=1000000|0.041
5|v5b8m-nont|rpad len=1 N=1000000|0.050
5|rfc|repeat len=1 N=16000000|6.403
5|master|repeat len=1 N=16000000|27.261
5|v5|repeat len=1 N=16000000|6.948
5|v5b8m|repeat len=1 N=16000000|8.551
5|v5b8m-nont|repeat len=1 N=16000000|7.136
5|rfc|rpad len=1 N=16000000|50.064
5|master|rpad len=1 N=16000000|49.933
5|v5|rpad len=1 N=16000000|4.108
5|v5b8m|rpad len=1 N=16000000|3.908
5|v5b8m-nont|rpad len=1 N=16000000|3.902
5|rfc|repeat len=1 N=100000000|44.838
5|master|repeat len=1 N=100000000|175.551
5|v5|repeat len=1 N=100000000|54.855
5|v5b8m|repeat len=1 N=100000000|51.297
5|v5b8m-nont|repeat len=1 N=100000000|51.311
5|rfc|rpad len=1 N=100000000|330.721
5|master|rpad len=1 N=100000000|333.589
5|v5|rpad len=1 N=100000000|61.238
5|v5b8m|rpad len=1 N=100000000|49.432
5|v5b8m-nont|rpad len=1 N=100000000|48.070
5|rfc|repeat len=10 N=1000000|0.327
5|master|repeat len=10 N=1000000|0.326
5|v5|repeat len=10 N=1000000|0.280
5|v5b8m|repeat len=10 N=1000000|0.263
5|v5b8m-nont|repeat len=10 N=1000000|0.264
5|rfc|rpad len=10 N=1000000|3.169
5|master|rpad len=10 N=1000000|3.388
5|v5|rpad len=10 N=1000000|0.072
5|v5b8m|rpad len=10 N=1000000|0.062
5|v5b8m-nont|rpad len=10 N=1000000|0.058
5|rfc|repeat len=10 N=16000000|9.053
5|master|repeat len=10 N=16000000|9.326
5|v5|repeat len=10 N=16000000|6.838
5|v5b8m|repeat len=10 N=16000000|7.181
5|v5b8m-nont|repeat len=10 N=16000000|6.941
5|rfc|rpad len=10 N=16000000|51.329
5|master|rpad len=10 N=16000000|50.406
5|v5|rpad len=10 N=16000000|3.777
5|v5b8m|rpad len=10 N=16000000|3.994
5|v5b8m-nont|rpad len=10 N=16000000|3.771
5|rfc|repeat len=10 N=100000000|54.950
5|master|repeat len=10 N=100000000|60.661
5|v5|repeat len=10 N=100000000|46.783
5|v5b8m|repeat len=10 N=100000000|50.412
5|v5b8m-nont|repeat len=10 N=100000000|45.181
5|rfc|rpad len=10 N=100000000|325.700
5|master|rpad len=10 N=100000000|346.414
5|v5|rpad len=10 N=100000000|40.888
5|v5b8m|rpad len=10 N=100000000|42.669
5|v5b8m-nont|rpad len=10 N=100000000|42.978
5|rfc|repeat len=100 N=1000000|0.316
5|master|repeat len=100 N=1000000|0.271
5|v5|repeat len=100 N=1000000|0.256
5|v5b8m|repeat len=100 N=1000000|0.286
5|v5b8m-nont|repeat len=100 N=1000000|0.262
5|rfc|rpad len=100 N=1000000|2.935
5|master|rpad len=100 N=1000000|3.027
5|v5|rpad len=100 N=1000000|0.044
5|v5b8m|rpad len=100 N=1000000|0.044
5|v5b8m-nont|rpad len=100 N=1000000|0.068
5|rfc|repeat len=100 N=16000000|7.206
5|master|repeat len=100 N=16000000|6.253
5|v5|repeat len=100 N=16000000|5.819
5|v5b8m|repeat len=100 N=16000000|6.729
5|v5b8m-nont|repeat len=100 N=16000000|6.429
5|rfc|rpad len=100 N=16000000|50.828
5|master|rpad len=100 N=16000000|53.784
5|v5|rpad len=100 N=16000000|3.704
5|v5b8m|rpad len=100 N=16000000|3.630
5|v5b8m-nont|rpad len=100 N=16000000|3.441
5|rfc|repeat len=100 N=100000000|46.388
5|master|repeat len=100 N=100000000|46.266
5|v5|repeat len=100 N=100000000|45.845
5|v5b8m|repeat len=100 N=100000000|48.169
5|v5b8m-nont|repeat len=100 N=100000000|50.347
5|rfc|rpad len=100 N=100000000|332.292
5|master|rpad len=100 N=100000000|333.209
5|v5|rpad len=100 N=100000000|44.833
5|v5b8m|rpad len=100 N=100000000|48.334
5|v5b8m-nont|rpad len=100 N=100000000|50.560
===== diffcheck.sh =====
#!/bin/bash
# Differential check: the same lpad()/rpad()/repeat() calls on master and on
# a patched build, in UTF8, EUC_JP and SQL_ASCII databases. Each call prints
# its length in bytes and characters and the md5 of its result, or the error
# message. Any line that differs between the two builds is printed.
# diffcheck.sh [patched build] (default: v5)
set -u
A=$(cd "$(dirname "$0")" && pwd)
W=$HOME/pgpad
NEW=${1:-v5}
gen() { # the calls, one per line, as SELECT statements
python3 - <<'EOF'
pads = ["", " ", "0", "ab", "abc", "é", "aéb", "æ¥æ¬", "æ¥aæ¬b",
"abcdefghij" * 10, "é" * 7 + "x"]
bases = ["", "x", "café", "æ¥æ¬èªããã¹ã"]
ns = [-1, 0, 1, 2, 3, 5, 7, 99, 100, 101,
16383, 16384, 16385, 16386, 32767, 32768, 32769, 50000,
8388607, 8388608, 8388609, 20000001]
def lit(s): return "'" + s.replace("'", "''") + "'"
for f in ("lpad", "rpad"):
for b in bases:
for p in pads:
for n in ns:
print(f"SELECT $l${f}({lit(b)},{n},{lit(p)[:24]})$l$,
octet_length(r), char_length(r), md5(r) "
f"FROM (SELECT {f}({lit(b)}, {n}, {lit(p)}) r) s;")
print(f"SELECT $l${f} too large$l$, octet_length({f}('x', 1073741823,
'ab'));")
print(f"SELECT $l${f} too large mb$l$, octet_length({f}('x', 400000000,
'æ¥æ¬'));")
for s in ["", "x", "ab", "é", "æ¥æ¬", "abcdefghij" * 10, "aéb"]:
for c in [-1, 0, 1, 2, 3, 16383, 16384, 16385, 100000, 8388608, 8388609,
20000001]:
print(f"SELECT $l$repeat({lit(s)[:24]},{c})$l$, octet_length(r),
char_length(r), md5(r) "
f"FROM (SELECT repeat({lit(s)}, {c}) r) s;")
print("SELECT 'repeat too large', octet_length(repeat('ab', 600000000));")
EOF
}
run() { # build dbname -> output
local b=$1 db=$2
# the script is UTF-8; psql would otherwise read it in the database encoding
PGCLIENTENCODING=UTF8 $W/i-$b/bin/psql -X -qAt -h /tmp -p ${PORT[$b]} -U
postgres -d $db -f $A/diffcheck.sql 2>&1 \
| sed 's/^psql:[^:]*:[0-9]*: //'
}
gen > $A/diffcheck.sql
declare -A PORT DIR
port=55490
for b in master $NEW; do
D=$(mktemp -d /tmp/claude-1000/dc.XXXX); DIR[$b]=$D; PORT[$b]=$port
$W/i-$b/bin/initdb -D $D -A trust --no-sync -U postgres -E UTF8 --locale=C
>/dev/null
printf "port = $port\nunix_socket_directories = '/tmp'\n" >>
$D/postgresql.conf
$W/i-$b/bin/pg_ctl -D $D -l $D/log -w start >/dev/null
$W/i-$b/bin/psql -X -q -h /tmp -p $port -U postgres \
-c "CREATE DATABASE eucjp ENCODING 'EUC_JP' TEMPLATE template0" \
-c "CREATE DATABASE sqlascii ENCODING 'SQL_ASCII' TEMPLATE template0"
port=$((port + 1))
done
total=0; bad=0
for db in postgres eucjp sqlascii; do
run master $db > $A/diffcheck.master.$db.out
run $NEW $db > $A/diffcheck.$NEW.$db.out
n=$(wc -l < $A/diffcheck.master.$db.out)
d=$(diff $A/diffcheck.master.$db.out $A/diffcheck.$NEW.$db.out | grep -c '^<')
echo "$db: $n calls, $d differ"
diff $A/diffcheck.master.$db.out $A/diffcheck.$NEW.$db.out | head -20
total=$((total + n)); bad=$((bad + d))
done
echo "total: $total calls, $bad differ between master and $NEW"
echo "errors seen (same on both unless listed above):"
cat $A/diffcheck.master.*.out | grep -E 'ERROR' | sed 's/^[^E]*ERROR/ERROR/' |
sort | uniq -c
for b in master $NEW; do $W/i-$b/bin/pg_ctl -D ${DIR[$b]} -m fast -w stop
>/dev/null; rm -rf ${DIR[$b]}; done
===== ./diffcheck.sh v5 =====
postgres: 2025 calls, 0 differ
eucjp: 2025 calls, 0 differ
sqlascii: 2025 calls, 0 differ
total: 6075 calls, 0 differ between master and v5
errors seen (same on both unless listed above):
16 ERROR: requested length too large
===== cancel.sh =====
#!/bin/bash
# How long a large lpad()/rpad()/repeat() runs past statement_timeout.
# On master, lpad() and rpad() have no CHECK_FOR_INTERRUPTS() in the padding
# loop (repeat() has one per copy); with v5 all three go through
# repeat_bytes(), which checks once per block.
# cancel.sh [build...] (default: master v5)
set -u
A=$(cd "$(dirname "$0")" && pwd)
W=$HOME/pgpad
for b in ${*:-master v5}; do
D=$(mktemp -d /tmp/claude-1000/cx.XXXX); P=55499
$W/i-$b/bin/initdb -D $D -A trust --no-sync -U postgres >/dev/null
printf "port = $P\nunix_socket_directories = '/tmp'\n" >> $D/postgresql.conf
taskset -c 0 $W/i-$b/bin/pg_ctl -D $D -l $D/log -w start >/dev/null
for sql in "rpad('x', 268000000, 'ab')" "lpad('x', 268000000, 'ab')"
"rpad('x', 268000000, ' ')" "repeat('ab', 134000000)"; do
for i in 1 2 3; do
t=$(taskset -c 2 $W/i-$b/bin/psql -X -qAt -h /tmp -p $P -U postgres \
-c "SET statement_timeout = '20ms'" -c '\timing on' \
-c "SELECT octet_length($sql)" 2>&1 | tr '\n' ' ' \
| sed -E 's/.*(ERROR: +[a-z ]+).*Time: ([0-9.]+) ms.*/\2 ms (\1)/;
s/^([0-9]+) Time: ([0-9.]+) ms.*/\2 ms (completed)/')
echo "$b | $sql | $t"
done
done
$W/i-$b/bin/pg_ctl -D $D -m fast -w stop >/dev/null; rm -rf $D
done | tee $A/cancel.txt
===== ./cancel.sh master v5 =====
master | rpad('x', 268000000, 'ab') | 926.001 ms (ERROR: canceling statement
due to statement timeout )
master | rpad('x', 268000000, 'ab') | 894.644 ms (ERROR: canceling statement
due to statement timeout )
master | rpad('x', 268000000, 'ab') | 904.163 ms (ERROR: canceling statement
due to statement timeout )
master | lpad('x', 268000000, 'ab') | 887.355 ms (ERROR: canceling statement
due to statement timeout )
master | lpad('x', 268000000, 'ab') | 895.831 ms (ERROR: canceling statement
due to statement timeout )
master | lpad('x', 268000000, 'ab') | 894.029 ms (ERROR: canceling statement
due to statement timeout )
master | rpad('x', 268000000, ' ') | 896.186 ms (ERROR: canceling statement
due to statement timeout )
master | rpad('x', 268000000, ' ') | 908.862 ms (ERROR: canceling statement
due to statement timeout )
master | rpad('x', 268000000, ' ') | 905.626 ms (ERROR: canceling statement
due to statement timeout )
master | repeat('ab', 134000000) | 20.651 ms (ERROR: canceling statement due
to statement timeout )
master | repeat('ab', 134000000) | 20.636 ms (ERROR: canceling statement due
to statement timeout )
master | repeat('ab', 134000000) | 21.316 ms (ERROR: canceling statement due
to statement timeout )
v5 | rpad('x', 268000000, 'ab') | 24.980 ms (ERROR: canceling statement due to
statement timeout )
v5 | rpad('x', 268000000, 'ab') | 23.898 ms (ERROR: canceling statement due to
statement timeout )
v5 | rpad('x', 268000000, 'ab') | 23.576 ms (ERROR: canceling statement due to
statement timeout )
v5 | lpad('x', 268000000, 'ab') | 23.567 ms (ERROR: canceling statement due to
statement timeout )
v5 | lpad('x', 268000000, 'ab') | 23.636 ms (ERROR: canceling statement due to
statement timeout )
v5 | lpad('x', 268000000, 'ab') | 23.604 ms (ERROR: canceling statement due to
statement timeout )
v5 | rpad('x', 268000000, ' ') | 23.223 ms (ERROR: canceling statement due to
statement timeout )
v5 | rpad('x', 268000000, ' ') | 23.370 ms (ERROR: canceling statement due to
statement timeout )
v5 | rpad('x', 268000000, ' ') | 23.517 ms (ERROR: canceling statement due to
statement timeout )
v5 | repeat('ab', 134000000) | 23.232 ms (ERROR: canceling statement due to
statement timeout )
v5 | repeat('ab', 134000000) | 24.497 ms (ERROR: canceling statement due to
statement timeout )
v5 | repeat('ab', 134000000) | 24.486 ms (ERROR: canceling statement due to
statement timeout )
