On Wed, Aug 28, 2024 at 12:24 AM Thomas Munro <thomas.mu...@gmail.com> wrote:
> 2. I tested against LLVM 10-18, and found that 10 and 11 lack some
> needed symbols. So I just hid this code from them. Even though our
> stable branches support those and even older versions, I am not sure
> if it's worth trying to do something about that for EOL'd distros that
> no one has ever complained about. I am willing to try harder if
> someone thinks that's important...
I would also assume that people using arm64 are more likely to use
recent versions than not.
I've done some additional tests on different LLVM versions with both
the unpatched version (to make sure the crash was triggered) and the
patched version. I'm joining the test scripts I've used as reference.
They target a kubernetes pod since it was the easiest way for me to
get a test ubuntu Jammy:
- setup_pod.sh: Install necessary packages, get multiple llvm
versions, fetch and compile master and patched version of postgres on
different LLVM version
- run_test.sh: go through all LLVM versions for both unpatched and
patched postgres to run the test_script.sh
- test_script.sh: ran inside the pod to setup the db with the
necessary tables and check if the crash happens
This generated the following output:
Test unpatched version on LLVM 19, : Crash triggered
Test unpatched version on LLVM 18, libLLVM-18.so.18.1: Crash triggered
Test unpatched version on LLVM 17, libLLVM-17.so.1: Crash triggered
Test unpatched version on LLVM 16, libLLVM-16.so.1: Crash triggered
Test unpatched version on LLVM 15, libLLVM-15.so.1: Crash triggered
Test unpatched version on LLVM 14, libLLVM-14.so.1: Crash triggered
Test unpatched version on LLVM 13, libLLVM-13.so.1: Crash triggered
Test patched version on LLVM 19, : Query ran successfully
Test patched version on LLVM 18, libLLVM-18.so.18.1: Query ran successfully
Test patched version on LLVM 17, libLLVM-17.so.1: Query ran successfully
Test patched version on LLVM 16, libLLVM-16.so.1: Query ran successfully
Test patched version on LLVM 15, libLLVM-15.so.1: Query ran successfully
Test patched version on LLVM 14, libLLVM-14.so.1: Query ran successfully
Test patched version on LLVM 13, libLLVM-13.so.1: Query ran successfully
I try to print the libLLVM linked to llvm.jit in the output to double
check whether I test on the correct version. The LLVM 19 package only
provides static libraries (probably because it's still a release
candidate?) so it shows as empty in the output. There was no LLVM 12
available when using the llvm.sh script so I couldn't test it. As for
the result, prepatch PG all crashed as expected while the patched
version was able to run the query successfully.
> Next, I think we should wait to see if the LLVM project commits that
> PR, this so that we can sync with their 19.x stable branch, instead of
> using code from a PR. Our next minor release is in November, so we
> have some time. If they don't commit it, we can consider it anyway: I
> mean, it's crashing all over the place in production, and we see that
> other projects are shipping this code already.
The PR[1] just received an approval and it sounds like they are ok to
eventually merge it.
[1] https://github.com/llvm/llvm-project/pull/71968
#!/bin/bash
# Don't want coredumps
ulimit -c 0
# Stop running PG
/var/lib/postgresql/.local/bin/pg_ctl -D /var/lib/postgresql/db_data stop &>> /dev/null
# Clean db_data
rm -rf /var/lib/postgresql/db_data
# Create new dbdata
/var/lib/postgresql/.local/bin/initdb -D /var/lib/postgresql/db_data -Apeer > /dev/null
echo "port = 5432
shared_buffers='4GB'
" > /var/lib/postgresql/db_data/postgresql.conf
# Start and setup test partitioned table
/var/lib/postgresql/.local/bin/pg_ctl -D /var/lib/postgresql/db_data start -l /tmp/pg.log > /dev/null
/var/lib/postgresql/.local/bin/pgbench -i --partitions=128 2> /dev/null
# Run the query to trigger the issue
/var/lib/postgresql/.local/bin/psql options=-cjit_above_cost=0 -c 'SELECT count(bid) from pgbench_accounts;' &>> /dev/null
exit_code=$?
if (( $exit_code == 2 )); then
echo "Crash triggered"
elif (( $exit_code == 0 )); then
echo "Query ran successfully"
else
echo "Other issue while running the query"
fi
#!/bin/bash
set -eu
TEST_POD=$(kubectl get pod -l app="test_pg" -nsre -o=jsonpath='{.items[*].metadata.name}')
LLVM_VERSIONS=(19 18 17 16 15 14 13)
get_llvm_version()
{
kubectl exec "$pod" -- bash -c "ldd /var/lib/postgresql/.local/lib/llvmjit.so | grep libLLVM | awk '{print \$1}'"
}
run_test_on_pod()
{
local pod="$1"
kubectl cp test_script.sh "$pod":/var/lib/postgresql/
for version in ${LLVM_VERSIONS[@]}; do
build_dir="~/postgres_build/build_${version}"
kubectl exec "$pod" -- bash -c "su postgres -c \" cd $build_dir; make -j19 install &>> /dev/null \" "
echo -n "Test unpatched version on LLVM $version, $(get_llvm_version): "
kubectl exec "$pod" -- bash -c "su postgres -c \" cd; bash test_script.sh \" "
done
for version in ${LLVM_VERSIONS[@]}; do
build_dir="~/postgres_build/build_patched_${version}"
kubectl exec "$pod" -- bash -c "su postgres -c \" cd $build_dir; make -j19 install &>> /dev/null \" "
echo -n "Test patched version on LLVM $version, $(get_llvm_version): "
kubectl exec "$pod" -- bash -c "su postgres -c \" cd; bash test_script.sh \" "
done
}
run_test_on_pod "$TEST_POD"
wait
#!/bin/bash
set -eu
TEST_POD=$(kubectl get pod -l app="test_pg" -nsre -o=jsonpath='{.items[*].metadata.name}')
LLVM_VERSIONS=(13 14 15 16 17 18 19)
setup_test_pod()
{
local pod="$1"
# Install packages
kubectl exec "$pod" -- bash -c 'apt update ; DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y tmux vim make g++ pkg-config coreutils git wget lsb-release wget software-properties-common gnupg bison flex libreadline-dev zlib1g-dev libzstd-dev'
# Create PG user
kubectl exec "$pod" -- bash -c '[ -d /var/lib/postgresql ] || useradd -s /bin/bash -d /var/lib/postgresql postgres'
kubectl exec "$pod" -- bash -c 'mkdir -p /var/lib/postgresql; chown postgres:postgres /var/lib/postgresql'
kubectl exec "$pod" -- bash -c 'mkdir -p /var/run/postgresql/; chown postgres:postgres /var/run/postgresql' &
# Checkout PG
kubectl exec "$pod" -- bash -c 'su postgres -c "cd; [ -d postgres_base ] || git clone --single-branch --branch master https://github.com/postgres/postgres.git postgres_base"'
# Create patched version
kubectl cp "v5-0001-Backport-of-LLVM-code-to-fix-ARM-relocation-bug.patch" "$pod":/var/lib/postgresql/
kubectl exec "$pod" -- bash -c 'su postgres -c "cd; [ -d postgres_patched ] || (git clone postgres_base postgres_patched && cd postgres_patched && git apply ~/v5-0001-Backport-of-LLVM-code-to-fix-ARM-relocation-bug.patch)"'
# Install LLVM packages
kubectl exec "$pod" -- bash -c 'wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh'
for version in ${LLVM_VERSIONS[@]}; do
kubectl exec "$pod" -- bash -c "./llvm.sh $version"
done
wait
# Compile PG against different llvm versions
# Base version
for version in ${LLVM_VERSIONS[@]}; do
build_dir="~/postgres_build/build_${version}"
kubectl exec "$pod" -- bash -c "su postgres -c \" mkdir -p $build_dir; cd $build_dir; [ -f GNUmakefile ] || CXX=clang++-${version} CLANG=clang-${version} LLVM_CONFIG=llvm-config-${version} CFLAGS=-O0 ~/postgres_base/configure --enable-cassert --enable-debug --prefix /var/lib/postgresql/.local --with-llvm > /dev/null \" "
kubectl exec "$pod" -- bash -c "su postgres -c \" cd $build_dir; make -j19 &>> /dev/null \" "
done
# Patched version
for version in ${LLVM_VERSIONS[@]}; do
build_dir="~/postgres_build/build_patched_${version}"
kubectl exec "$pod" -- bash -c "su postgres -c \" mkdir -p $build_dir; cd $build_dir; CXX=clang++-${version} CLANG=clang-${version} LLVM_CONFIG=llvm-config-${version} CFLAGS=-O0 ~/postgres_patched/configure --enable-cassert --enable-debug --prefix /var/lib/postgresql/.local --with-llvm > /dev/null \" "
kubectl exec "$pod" -- bash -c "su postgres -c \" cd $build_dir; make -j19 &>> /dev/null \" "
done
}
setup_test_pod "$TEST_POD"
wait
