Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Furthermore, the queries in this case are 10667 bytes in total, so should fit inside ten-ish MTUs in an AWS VPC with the pg wire protocol overhead. So I'm not too worried about the time spent sending the queries. Still, it would be interesting to measure the impact of storing them in the DB and pre-computing them as far as possible without authorization information. We've planned further profiling and optimization of data access and physical layout. No locks should affect this specific use case. On Thu, Oct 10, 2024 at 11:07 PM Mikael Sand wrote: > One question, > > I wonder if it would be possible to link pgcommon_shlib and pgport_shlib > statically into pq and do LTO already at that point, such that consumers of > libpq could merely link against libpq.a and that would be enough? > > I haven't maintained any c or c++ library intended for public distribution > and haven't needed this kind of functionality, but perhaps someone on the > mailing list would already have the necessary expertise? > > Br Mikael > > On Thu, Oct 10, 2024 at 10:33 PM Mikael Sand wrote: > >> Btw Aleksander >> >> To give some context, this is a component intended to solve our last >> scaling issue / most demanding query, which happens at page load and thus >> need to minimize latency over all else. We've used libpq in pipelined >> binary mode, serialized the results it in a custom binary format, >> compressed that using zstd, sent to the frontend, uncompressed by the >> browser builtin support, and parsed from binary to javascript objects using >> custom parser. This has improved performance more than 50x over the >> existing graphql / postgraphile solution so far. It seems likely that doing >> this work in a read replica would further improve performance >> significantly, so as not to send the large amount of data from the database >> to the backend, but this will be a later project. I can have measurements >> regarding static + lto vs dynamic before that. >> >> Br Mikael >> >> On Thu, Oct 10, 2024 at 10:22 PM Mikael Sand wrote: >> >>> Hi Aleksander >>> >>> Ok. So no actual benefit from using dynamic? >>> Well, it seems postgresql and all dependencies already support it, no? >>> Doesn't go do static linking by default / prefer it? Unless you use some >>> part that uses CGO, in which case many go developers appear to disable CGO >>> anyway and use the plain go implementation instead. >>> Similarly, the rust community seems to have a preference / strong >>> support for static, some with alpine, musl, etc. >>> We're still developing this part of our system, and do intend to measure >>> the difference, for this we need to be able to build the static version as >>> well. I can post results once we have them. Have you measured the >>> difference? >>> >>> Br Mikael >>> >>> On Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev < >>> aleksan...@timescale.com> wrote: >>> >>>> Hi Mikael, >>>> >>>> > We use static linking and link time optimization to squeeze the last >>>> bits of performance out of the code in our most performance-critical >>>> queries, and it simplifies our security audits to have a static binary >>>> running inside chainguard/static as the data we handle is >>>> sensitive/business critical. >>>> >>>> No comments about the security aspect. I'm not a security expert so I >>>> leave this topic for somebody for whom this is an area of expertise. >>>> >>>> I don't buy the optimization part. The performance gain you get is >>>> next to nothing compared to the time spent transferring SQL queries >>>> over the network / UDP sockets, parsing and planning them, accessing >>>> the disk, waiting for the locks, etc. Unless you actually measured it >>>> in order to show the opposite which I seriously doubt you did. A >>>> better time investment would be finding actual bottlenecks in the >>>> given system in order to be able to eliminate them. >>>> >>>> > Aleksander, do you have something against static linking or am I >>>> reading you wrong? I've seen this sentiment in several places but never >>>> understood why anyone would hold this position. Could you elaborate? >>>> >>>> The question is whether it's practical for the PostgreSQL community to >>>> put effort into supporting / testing static linking of libpq. >>>> Personally I'm not convinced that demanding from every open-source >>>> project in existence to support static linking is a sustainable idea >>>> in the long run, given the fact that there are technologies like >>>> Docker and programming languages like Go (with pgx client). >>>> >>>> -- >>>> Best regards, >>>> Aleksander Alekseev >>>> >>>
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Well for getting the potential benefits of link-time optimization into more places, and slightly improved developer ergonomics, and perhaps mostly to eliminate potential confusion if one needs pg_common and pg_ports or the shared library version when compiling statically. Would you happen to know if it is possible? regards, mikael sand On Fri, Oct 11, 2024 at 12:12 AM Tom Lane wrote: > Mikael Sand writes: > > I wonder if it would be possible to link pgcommon_shlib and pgport_shlib > > statically into pq and do LTO already at that point, such that consumers > of > > libpq could merely link against libpq.a and that would be enough? > > What would be the point? A typical build will still have a pile of > outside dependencies that static-link consumers will have to track > manually (openssl and so forth). So I don't see how this'd move > the needle much. > > regards, tom lane >
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Hello Aleksander This is for compiling a c++ application that uses libpq with the -static flag, the server compiles fine. Br Mikael On Thu, Oct 10, 2024 at 12:16 PM Aleksander Alekseev < aleksan...@timescale.com> wrote: > Hi Mikael, > > Please use the "Reply to All" button - send your emails to the mailing > list, not to me directly. > > > The official packages and the original source tar both fail to compile, > the patch enables it to compile. > > I just run: > > ``` > docker run -e POSTGRES_PASSWORD=foo postgres:17.0-alpine3.20 > ``` > > ... on my spare Raspberry Pi and it works just fine. Also PostgreSQL > is tested on Alpine [1]. > > It means that your custom Dockerfile is wrong. I suggest comparing it > with the one published on hub.docker.com [2]. Also I would recommend > reading the documentation section about compiling PostgreSQL from > source code [3]. I have a few scripts [4] that may help you correct > the Dockerfile as well [4]. > > All this of course assumes that you really want your custom Dockerfile > instead of using "official" / well regarded 17.0-alpine3.20 one. > > [1]: https://buildfarm.postgresql.org/cgi-bin/show_status.pl > [2]: > https://github.com/docker-library/postgres/blob/172544062d1031004b241e917f5f3f9dfebc0df5/17/alpine3.20/Dockerfile > [3]: https://www.postgresql.org/docs/current/installation.html > [4]: https://github.com/afiskon/pgscripts/ > > -- > Best regards, > Aleksander Alekseev >
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Sorry for not having a properly minimal reproduction to begin with, seems I
can't reduce this any further at the moment at least:
# syntax=docker/dockerfile:1
> FROM chainguard/git:latest-dev AS builder
> USER root
> WORKDIR /app
> RUN apk update && apk add --no-cache --update-cache \
> clang-19 \
> bison \
> curl \
> flex \
> make \
> perl
>
> ARG PG=17.0
> RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2
> | tar -xj
> COPY < Subject: [PATCH] Fix static build
> ---
> Index: src/bin/initdb/Makefile
> <+>UTF-8
> ===
> diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
> --- a/src/bin/initdb/Makefile (revision
> c4b8a916f8a53379621825028532b038e004f891)
> +++ b/src/bin/initdb/Makefile (date 1728466911177)
> @@ -20,7 +20,8 @@
> # from libpq, else we have risks of version skew if we run with a libpq
> # shared library from a different PG version. Define
> # USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
> -override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir)
> -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
> +#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir)
> -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
> +override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone
> $(ICU_CFLAGS) $(CPPFLAGS)
>
> # We need libpq only because fe_utils does.
> LDFLAGS_INTERNAL += -L$(top_builddir)/src/fe_utils -lpgfeutils
> $(libpq_pgport) $(ICU_LIBS)
> Index: src/common/Makefile
> <+>UTF-8
> ===
> diff --git a/src/common/Makefile b/src/common/Makefile
> --- a/src/common/Makefile (revision c4b8a916f8a53379621825028532b038e004f891)
> +++ b/src/common/Makefile (date 1728466354406)
> @@ -143,7 +143,7 @@
> # Files in libpgcommon.a should use/export the "xxx_private" versions
> # of pg_char_to_encoding() and friends.
> #
> -$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
> +#$(OBJS_FRONTEND): CPPFLAGS += -DUSE_PRIVATE_ENCODING_FUNCS
>
>
> #
> EOF
> RUN cd postgresql-$PG && \
> git apply ../pg.patch && \
> export CC=clang CXX=clang++ && \
> ./configure --without-openssl --without-icu --without-ldap
> --without-readline --without-zlib --without-tcl --without-python
> --prefix=/usr/local && \
> cd src && for lib in include common port interfaces/libpq ; do cd $lib &&
> make && make install && cd - ; done
>
> COPY < #include
> int main(){return PQconnectdb("")==NULL;}
> EOF
>
> RUN clang++ -static -o main main.cpp -lpq -lpgcommon -lpgport
> ENTRYPOINT ["./main"]
> CMD []
>
>
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
E.g. this fails to compile:
FROM chainguard/git:latest-dev AS builder
USER root
RUN apk update && apk add --no-cache --update-cache clang-19
postgresql-17-dev openssl-dev
COPY <
int main(){return PQconnectdb("")==NULL;}
EOF
RUN clang++ -static -o main main.cpp -lpq -lpgcommon -lpgport -lssl -lcrypto
On Thu, Oct 10, 2024 at 12:16 PM Aleksander Alekseev <
aleksan...@timescale.com> wrote:
> Hi Mikael,
>
> Please use the "Reply to All" button - send your emails to the mailing
> list, not to me directly.
>
> > The official packages and the original source tar both fail to compile,
> the patch enables it to compile.
>
> I just run:
>
> ```
> docker run -e POSTGRES_PASSWORD=foo postgres:17.0-alpine3.20
> ```
>
> ... on my spare Raspberry Pi and it works just fine. Also PostgreSQL
> is tested on Alpine [1].
>
> It means that your custom Dockerfile is wrong. I suggest comparing it
> with the one published on hub.docker.com [2]. Also I would recommend
> reading the documentation section about compiling PostgreSQL from
> source code [3]. I have a few scripts [4] that may help you correct
> the Dockerfile as well [4].
>
> All this of course assumes that you really want your custom Dockerfile
> instead of using "official" / well regarded 17.0-alpine3.20 one.
>
> [1]: https://buildfarm.postgresql.org/cgi-bin/show_status.pl
> [2]:
> https://github.com/docker-library/postgres/blob/172544062d1031004b241e917f5f3f9dfebc0df5/17/alpine3.20/Dockerfile
> [3]: https://www.postgresql.org/docs/current/installation.html
> [4]: https://github.com/afiskon/pgscripts/
>
> --
> Best regards,
> Aleksander Alekseev
>
Re: Annoying build warnings from latest Apple toolchain
Good day!
I'm encountering a build issue with postgresql 17, I wonder if this was an
intentional consequence of this commit:
https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545a
Or if this was unintentional. Or is there any way to compile pgcommon with
the correct function names statically into libpq?
https://www.postgresql.org/message-id/CAAwAxZf456NwLKD4ZBpyDmPc5GFmGP%3Db5Vw7pTMY0v9R-%3D%2BDTA%40mail.gmail.com
This works with 16.4 but fails with 17.0:
FROM postgres:16.4-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
libedit-dev \
openssl-dev \
zstd-static \
zlib-static \
lz4-static \
e2fsprogs \
keyutils \
zstd-dev \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <
int main(){return PQconnectdb("")==NULL;}
EOF
ARG KRB5=1.21.3
ARG KRB5MAJMIN=1.21
RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz
| tar xzf -
RUN cd krb5-$KRB5/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
ARG SASL=2.1.28
RUN curl -L
https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz
| tar xzf -
RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make install
RUN clang++ -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon -lpgport \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
Best regards
Mikael Sand
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
We use static linking and link time optimization to squeeze the last bits of performance out of the code in our most performance-critical queries, and it simplifies our security audits to have a static binary running inside chainguard/static as the data we handle is sensitive/business critical. Wonderful, the build works when using -lpgcommon_shlib -lpgport_shlib So this appears to imply that the output of `pkg-config -libs -static libpq`is incorrect? /app # pkg-config -libs -static libpq > -L/usr/local/lib -lpq -L/usr/lib/llvm15/lib -L/usr/local/lib -lpgcommon > -lpgport -lgssapi_krb5 -lm -lldap -lssl -lcrypto -ldl -pthread > On Thu, Oct 10, 2024 at 7:54 PM Aleksander Alekseev < aleksan...@timescale.com> wrote: > Mikael, > > On Thu, Oct 10, 2024 at 8:49 PM Tom Lane wrote: > > > > Mikael Sand writes: > > > RUN clang++ -fno-common -static -o main main.cpp \ > > > -L/usr/local/lib -lpq -lpgcommon -lpgport \ > > > -lldap -lsasl2 -lssl -lcrypto -llber \ > > > -lgssapi_krb5 \ > > > -lkrb5 -lk5crypto -lcom_err -lkrb5support \ > > > -lgdbm > > > > The short answer here is that your link recipe is wrong, and has been > > wrong right along, though you accidentally got away with it before. > > The modules within libpq expect to be linked with libpgcommon_shlib > > and libpgport_shlib, not libpgcommon/libpgport. > > > > Having external code that needs to know explicitly about every one > > of a library's dependencies is one of many reasons why we discourage > > static linking. > > May I ask what problem you are trying to solve with static linking in > the first place? > > -- > Best regards, > Aleksander Alekseev >
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Just for reference in case anyone else who utilizes static linking for any
reason hits upon this issue, here is a working Dockerfile for libpq /
postgresql 17
FROM postgres:17.0-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
libedit-dev \
openssl-dev \
zstd-static \
zlib-static \
lz4-static \
e2fsprogs \
keyutils \
zstd-dev \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <
int main(){return PQconnectdb("")==NULL;}
EOF
ARG KRB5=1.21.3
ARG KRB5MAJMIN=1.21
RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz
| tar xzf -
RUN cd krb5-$KRB5/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
ARG SASL=2.1.28
RUN curl -L
https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz
| tar xzf -
RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make install
RUN clang++ -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon_shlib -lpgport_shlib \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Aleksander, do you have something against static linking or am I reading
you wrong?
I've seen this sentiment in several places but never understood why anyone
would hold this position. Could you elaborate?
At least for executables intended to be run in production inside
containers, they usually run in isolation, what other executables could
they share libraries with?
Should we leave performance on the table, given the energy and resource
requirements it implies?
Certainly, for libraries intended to be distributed in e.g. Linux
distributions and be depended on in end-user executables, it makes sense to
have shared libraries and simplify mitigating security issues.
It appears to me that there are many cases where static linking and link
time optimization would be more appropriate than dynamic.
When the executables never have a chance to share libraries with any other
process and are rebuilt whenever a new version is deployed what benefits
does dynamic linking provide over static + lto?
I expect I'm missing something crucial, but I have spent some time trying
to understand this issue deeper, and still find myself mystified.
Please enlighten me.
Br Mikael
On Thu, Oct 10, 2024 at 8:29 PM Mikael Sand wrote:
> Just for reference in case anyone else who utilizes static linking for any
> reason hits upon this issue, here is a working Dockerfile for libpq /
> postgresql 17
>
> FROM postgres:17.0-alpine3.20 AS builder
> USER root
> WORKDIR /app
> RUN apk update && apk add --no-cache --update-cache \
> openssl-libs-static \
> libevent-static \
> libxml2-static \
> libedit-static \
> libxslt-static \
> sqlite-static \
> openldap-dev \
> libxslt-dev \
> libxml2-dev \
> libedit-dev \
> openssl-dev \
> zstd-static \
> zlib-static \
> lz4-static \
> e2fsprogs \
> keyutils \
> zstd-dev \
> zlib-dev \
> gdbm-dev \
> clang17 \
> lz4-dev \
> libldap \
> bison \
> curl \
> perl \
> make
>
> COPY < #include
> int main(){return PQconnectdb("")==NULL;}
> EOF
>
> ARG KRB5=1.21.3
> ARG KRB5MAJMIN=1.21
> RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz |
> tar xzf -
> RUN cd krb5-$KRB5/src && \
> ./configure && make && make install && \
> ./configure --disable-shared --enable-static && make && make install
>
> ARG SASL=2.1.28
> RUN curl -L
> https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz
> | tar xzf -
> RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make install
>
> RUN clang++ -static -o main main.cpp \
> -L/usr/local/lib -lpq -lpgcommon_shlib -lpgport_shlib \
> -lldap -lsasl2 -lssl -lcrypto -llber \
> -lgssapi_krb5 \
> -lkrb5 -lk5crypto -lcom_err -lkrb5support \
> -lgdbm
>
>
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
// Trace/breakpoint trap
> SIGXCPU, // CPU time limit exceeded (4.2BSD)
> SIGXFSZ, // File size limit exceeded (4.2BSD)
> SIGTERM
> };
>
> for (size_t i = 0; i < signals.size(); ++i) {
> struct sigaction action;
> memset(&action, 0, sizeof action);
> action.sa_flags = static_cast(SA_SIGINFO | SA_ONSTACK | SA_NODEFER |
> SA_RESETHAND);
> sigfillset(&action.sa_mask);
> sigdelset(&action.sa_mask, signals[i]);
> action.sa_sigaction = &sig_handler;
> sigaction(signals[i], &action, nullptr);
> }
> }
>
> int main() {
> registerSignalHandlers();
> std::cout << "start" << std::endl;
> PGconn *conn = PQconnectdb("");
> if (conn == NULL) {
> std::cout << "fail" << std::endl;
> } else {
> if (PQstatus(conn) == CONNECTION_OK) {
> std::cout << "success" << std::endl;
> } else {
> fprintf(stderr, "fail: %s", PQerrorMessage(conn));
> }
> PQfinish(conn);
> }
> while (true) {
> std::this_thread::sleep_for(std::chrono::milliseconds(1000));
> }
> return 0;
> }
> EOF
>
> RUN ln /usr/lib/llvm-19/lib/LLVMgold.so /usr/lib/LLVMgold.so
> RUN CC=clang CXX=clang++ clang++ \
> -std=c++20 \
> -static \
> -flto \
> -O3 \
> -march=native \
> -Wpedantic \
> -Wall \
> -Wextra \
> -Wsign-conversion \
> -Wconversion \
> -o main main.cpp \
> -lpq \
> -lpgcommon \
> -lpgport \
> -lssl \
> -lcrypto \
> -lpthread \
> -ldl
> ENTRYPOINT ["./main"]
> CMD []
>
> FROM cgr.dev/chainguard/static:latest AS runner
> WORKDIR /app
> COPY --from=builder --chown=65532:65532 /app/main .
> USER 65532
> ENTRYPOINT ["./main"]
> CMD []
>
> Hope this can help somebody
Br Mikael
On Wed, Oct 9, 2024 at 11:51 AM Mikael Sand wrote:
> This seems to work:
>
> # syntax=docker/dockerfile:1
> FROM alpine:3.20 AS builder
>
> # Fails with 17.0 succeeds with 16.4
> ARG PG=17.0
>
> USER root
> RUN apk update && apk add --no-cache --update-cache \
> ca-certificates-bundle \
> util-linux-dev \
> execline-dev \
> libedit-dev \
> libxml2-dev \
> clang17-dev \
> llvm18-dev \
> build-base \
> net-tools \
> zlib-dev \
> autoconf \
> automake \
> busybox \
> icu-dev \
> clang17 \
> llvm18 \
> cmake \
> bison \
> meson \
> ninja \
> flex \
> perl \
> curl \
> bash
> WORKDIR /app
>
> RUN curl -L
> https://github.com/openssl/openssl/releases/download/openssl-3.3.2/openssl-3.3.2.tar.gz
> > openssl.tar.gz
> RUN mkdir openssl && tar --extract --file=openssl.tar.gz --strip-components=1
> --directory=openssl && rm openssl.tar.gz
> RUN cd openssl && CC=clang CXX=clang++ perl ./Configure \
> linux-$(uname -m) \
> --prefix=/usr \
> --libdir=lib \
> --openssldir=/etc/ssl \
> enable-ktls \
> shared \
> no-zlib \
> no-async \
> no-comp \
> no-idea \
> no-mdc2 \
> no-rc5 \
> no-ec2m \
> no-sm2 \
> no-sm4 \
> no-ssl3 \
> no-seed \
> no-weak-ssl-ciphers \
> -Wa,--noexecstack && \
> perl configdata.pm --dump && \
> make -j$(nproc) && make install
>
> RUN curl -L https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2
> > postgresql.tar.bz2
> RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2
> --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
> COPY < Subject: [PATCH] Fix static build
> ---
> Index: src/bin/initdb/Makefile
> <+>UTF-8
> ===
> diff --git a/src/bin/initdb/Makefile b/src/bin/initdb/Makefile
> --- a/src/bin/initdb/Makefile (revision
> c4b8a916f8a53379621825028532b038e004f891)
> +++ b/src/bin/initdb/Makefile (date 1728466911177)
> @@ -20,7 +20,8 @@
> # from libpq, else we have risks of version skew if we run with a libpq
> # shared library from a different PG version. Define
> # USE_PRIVATE_ENCODING_FUNCS to ensure that that happens.
> -override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir)
> -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
> +#override CPPFLAGS := -DUSE_PRIVATE_ENCODING_FUNCS -I$(libpq_srcdir)
> -I$(top_srcdir)/src/timezone $(ICU_CFLAGS) $(CPPFLAGS)
> +override CPPFLAGS := -I$(libpq_srcdir) -I$(top_srcdir)/src/timezone
> $(ICU_CFLAGS) $(CPPFLAGS)
>
> # We need libpq only because fe_util
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
(conn == NULL) {
std::cout << "fail" << std::endl;
} else {
if (PQstatus(conn) == CONNECTION_OK) {
std::cout << "success" << std::endl;
} else {
fprintf(stderr, "fail: %s", PQerrorMessage(conn));
}
PQfinish(conn);
}
while (true) {
std::this_thread::sleep_for(std::chrono::milliseconds(1000));
}
return 0;
}
EOF
# Fails with "-static" succeeds without
RUN CC=clang CXX=clang++ clang++ \
-std=c++20 \
-static \
-flto \
-O3 \
-march=native \
-Wpedantic \
-Wall \
-Wextra \
-Wsign-conversion \
-Wconversion \
-o main main.cpp \
-lpq \
-lpgcommon \
-lpgport \
-lssl \
-lcrypto \
-lpthread \
-ldl
ENTRYPOINT ["./main"]
CMD []
On Wed, Oct 9, 2024 at 10:10 AM Mikael Sand wrote:
> Hello dear Hackers
>
> I'm trying to upgrade to v17 and encountering a strange issue.
>
> I've made a minimal reproduction that works with 16.4 but fails with 17.0,
> it also works without the "-static" compile flag:
>
>
> # syntax=docker/dockerfile:1
>> FROM alpine:3.20 AS builder
>>
>> # Fails with 17.0 succeeds with 16.4
>> ARG PG=17.0
>>
>> USER root
>> RUN apk update && apk add --no-cache --update-cache \
>> ca-certificates-bundle \
>> util-linux-dev \
>> clang17-dev \
>> execline-dev \
>> llvm18-dev \
>> libedit-dev \
>> libxml2-dev \
>> build-base \
>> net-tools \
>> clang17 \
>> zlib-dev \
>> autoconf \
>> automake \
>> busybox \
>> llvm18 \
>> icu-dev \
>> cmake \
>> bison \
>> flex \
>> perl \
>> curl \
>> bash \
>> flex
>> WORKDIR /app
>>
>> RUN curl -L
>> https://github.com/openssl/openssl/releases/download/openssl-3.3.2/openssl-3.3.2.tar.gz
>> > openssl.tar.gz
>> RUN mkdir openssl && tar --extract --file=openssl.tar.gz
>> --strip-components=1 --directory=openssl && rm openssl.tar.gz
>> RUN cd openssl && CC=clang CXX=clang++ perl ./Configure \
>> linux-$(uname -m) \
>> --prefix=/usr \
>> --libdir=lib \
>> --openssldir=/etc/ssl \
>> enable-ktls \
>> shared \
>> no-zlib \
>> no-async \
>> no-comp \
>> no-idea \
>> no-mdc2 \
>> no-rc5 \
>> no-ec2m \
>> no-sm2 \
>> no-sm4 \
>> no-ssl3 \
>> no-seed \
>> no-weak-ssl-ciphers \
>> -Wa,--noexecstack && \
>> perl configdata.pm --dump && \
>> make -j$(nproc) && make install
>>
>> RUN curl -L
>> https://ftp.postgresql.org/pub/source/v$PG/postgresql-$PG.tar.bz2 >
>> postgresql.tar.bz2
>> RUN mkdir postgresql && tar --extract --bzip2 --file=postgresql.tar.bz2
>> --strip-components=1 --directory=postgresql && rm postgresql.tar.bz2
>> RUN cd postgresql && CC=clang CXX=clang++ ./configure --with-openssl
>> --with-libedit-preferred --with-uuid=e2fs --with-libxml --prefix=/usr/local
>> RUN cd postgresql/src/include && CC=clang CXX=clang++ make && make install
>> RUN cd postgresql/src/common && CC=clang CXX=clang++ make && make install
>> RUN cd postgresql/src/port && CC=clang CXX=clang++ make && make install
>> RUN cd postgresql/src/interfaces/libpq && CC=clang CXX=clang++ make && make
>> install
>>
>> COPY <> #include
>> #include
>> #include
>> #include
>> #include
>> #include
>> #include
>>
>> #include
>>
>> void sig_handler(int /*_signo*/, siginfo_t * info, void * /*_ctx*/) {
>> raise(info->si_signo);
>> _exit(EXIT_FAILURE);
>> }
>>
>> void registerSignalHandlers() {
>> std::vector signals = {
>> // Signals for which the default action is "Core".
>> SIGABRT, // Abort signal from abort(3)
>> SIGBUS, // Bus error (bad memory access)
>> SIGFPE, // Floating point exception
>> SIGILL, // Illegal Instruction
>> SIGIOT, // IOT trap. A synonym for SIGABRT
>> SIGQUIT, // Quit from keyboard
>> SIGSEGV, // Invalid memory reference
>> SIGSYS, // Bad argument to routine (SVr4)
>> SIGTRAP, // Trace/breakpoint trap
>> SIGXCPU, // CPU time limit exceeded (4.2BSD)
>> SIGXFSZ, // File size limit exceeded (4.2BSD)
>> SIGTERM
>> };
>>
>> for (size_t i = 0; i < signals.size(); ++i) {
>> struct sigaction action;
&g
Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
o main main.cpp \ > -lpq \ > -lpgcommon \ > -lpgport \ > -lssl \ > -lcrypto \ > -lpthread \ > -ldl > ENTRYPOINT ["./main"] > CMD [] > > And here is the output: 15/15 RUN CC=clang CXX=clang++ clang++ -std=c++20 -static -flto -O3 -march=native -Wpedantic -Wall -Wextra -Wsign-conversion -Wconversion -o main main.cpp -lpq -lpgcommon -lpgport -lssl -lcrypto -lpthread -ldl ERROR 1.0s 1 /usr/bin/ld: /usr/local/lib/libpq.a(fe-connect.o): in function `pqConnectOptions2': 2 fe-connect.c:(.text+0x1cb4): undefined reference to `pg_encoding_to_char' 3 /usr/bin/ld: /usr/local/lib/libpq.a(fe-connect.o): in function `PQsetClientEncoding': 4 fe-connect.c:(.text+0x64a8): undefined reference to `pg_encoding_to_char' 5 /usr/bin/ld: /usr/local/lib/libpq.a(fe-exec.o): in function `pqSaveParameterStatus': 6 fe-exec.c:(.text+0x1168): undefined reference to `pg_char_to_encoding' 7 /usr/bin/ld: /usr/local/lib/libpq.a(fe-misc.o): in function `PQenv2encoding': 8 fe-misc.c:(.text+0x1394): undefined reference to `pg_char_to_encoding' 9 clang++: error: linker command failed with exit code 1 (use -v to see invocation) Any ideas? Best regard Mikael Sand Dockerfile Description: Binary data
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Hmm, so is static linking of all applications that include libpq
intentionally broken?
At least googling around, it seems like a relatively common practice.
At least it seems that 2ndquadrant builds and uses libpq statically.
https://www.postgresql.org/message-id/20327.1501536978%40sss.pgh.pa.us
I don't see how your script would fix the naming of these functions, can
you elaborate?
Here is a Dockerfile demonstrating the same issue with
the postgres:17.0-alpine3.20 image:
FROM postgres:17.0-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
cyrus-sasl-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
zstd-static \
zlib-static \
libedit-dev \
openssl-dev \
lz4-static \
e2fsprogs \
zstd-dev \
keyutils \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <
int main(){return PQconnectdb("")==NULL;}
EOF
RUN curl -L https://kerberos.org/dist/krb5/1.21/krb5-1.21.3.tar.gz >
krb5-1.21.3.tar.gz && tar xf krb5-1.21.3.tar.gz
RUN cd krb5-1.21.3/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
RUN curl -L
https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.28/cyrus-sasl-2.1.28.tar.gz
> cyrus-sasl-2.1.28.tar.gz
RUN tar xf cyrus-sasl-2.1.28.tar.gz && cd cyrus-sasl-2.1.28 &&
./configure --enable-static && make && make install
RUN clang++ -fno-common -static -o main main.cpp \
-L/usr/local/lib -lpq -L/usr/lib/llvm15/lib -L/usr/local/lib
-lpgcommon -lpgport -lgssapi_krb5 -lm -lldap -lcrypto -ldl -pthread \
-lldap -levent -lsasl2 -lssl -lcrypto -llber -levent \
-lsqlite3 \
-L/usr/local/lib -lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm \
-lssl \
-lgssapi_krb5
On Thu, Oct 10, 2024 at 1:51 PM Aleksander Alekseev <
aleksan...@timescale.com> wrote:
> Hi Mikael,
>
> > This is for compiling a c++ application that uses libpq with the -static
> flag, the server compiles fine.
>
> OK. I couldn't quite do this because the only Linux machine I have at
> the moment runs Raspbian and there doesn't seem to be a static glibc
> available for it. But here is how to achieve what you want *in
> theory*.
>
> First compile Postgres from the source code, for instance (change the
> flags as needed, you probably want a release build):
>
> ```
> # sudo apt install clang-16
> # git clean -dfx
> CFLAGS="-static" meson setup --buildtype debug -Dicu=disabled
> -Dldap=disabled -Dreadline=disabled -Dzlib=disabled -Dlz4=disabled
> -Dprefix=/home/eax/pginstall build
> ninja -C build
> ```
>
> I recommend using Meson and Ninja. Autotools and Make are still
> supported but are slow and probably will be gone in a few years. The
> "ninja -C build" steps fail for me with various errors about the need
> for static glibc but I think it should work on other distributions and
> architectures.
>
> Next install Postgres. I use a script for this [1]:
>
> ```
> ~/pgscripts/single-install-meson.sh
> ```
>
> Change the script as needed - you probably don't need pg_ctl,
> createdb, etc for your task.
>
> Now in order to compile and execute your C++ program:
>
> ```
> export PRFX=/home/eax/pginstall
> g++ -g -Wall -o test_libpq -I$PRFX/include
> -L$PRFX/lib/aarch64-linux-gnu/ test_libpq.cpp -lpq -static
> LD_LIBRARY_PATH=$PRFX/lib/aarch64-linux-gnu/ ./test_libpq
> ```
>
> This being said I don't recall seeing anything about the support of
> static linking of libpq in the documentation [2]. To my knowledge this
> is not officially supported / tested / maintained which means you and
> your colleagues are on your own with the -static flag. Since you are
> already using Docker, perhaps the easiest thing to do would be to back
> the application and all its dependencies (dynamically linked) in a
> Docker container.
>
> Good luck.
>
> [1]: https://github.com/afiskon/pgscripts/
> [2]: https://www.postgresql.org/docs/current/libpq.html
>
> --
> Best regards,
> Aleksander Alekseev
>
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
I don't mind having this patch in use too much, I have a functional build and nothing to worry about, no luck necessary :) But, I expect a lot of unnecessary churn in the community if this is not fixed properly. Not all applications run inside docker, and not all can use dynamic linking. People can dislike static linking for whatever reason, but that doesn't remove the need for it (or both) in some cases and the desire for it in others. Judging from the commit, it doesn't seem like static linking is intentionally broken. Should a test be added so as not to break this by mistake? I must say I don't fully comprehend the context of that commit. On Thu, Oct 10, 2024 at 5:06 PM Mikael Sand wrote: > This seems to be the commit that caused this: > https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545a > > So it seems the commit is related to static linking. > > Is it somehow possible to statically link in a version of pgcommon with > correct function names into libpq? >
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
E.g. this works with 16.4 but fails in 17.0:
FROM postgres:16.4-alpine3.20 AS builder
USER root
WORKDIR /app
RUN apk update && apk add --no-cache --update-cache \
openssl-libs-static \
cyrus-sasl-static \
libevent-static \
libxml2-static \
libedit-static \
libxslt-static \
sqlite-static \
openldap-dev \
libxslt-dev \
libxml2-dev \
zstd-static \
zlib-static \
libedit-dev \
openssl-dev \
lz4-static \
e2fsprogs \
zstd-dev \
keyutils \
zlib-dev \
gdbm-dev \
clang17 \
lz4-dev \
libldap \
bison \
curl \
perl \
make
COPY <
int main(){return PQconnectdb("")==NULL;}
EOF
RUN curl -L https://kerberos.org/dist/krb5/1.21/krb5-1.21.3.tar.gz >
krb5-1.21.3.tar.gz && tar xf krb5-1.21.3.tar.gz
RUN cd krb5-1.21.3/src && \
./configure && make && make install && \
./configure --disable-shared --enable-static && make && make install
RUN curl -L
https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.28/cyrus-sasl-2.1.28.tar.gz
> cyrus-sasl-2.1.28.tar.gz
RUN tar xf cyrus-sasl-2.1.28.tar.gz && cd cyrus-sasl-2.1.28 &&
./configure --enable-static && make && make install
RUN clang++ -fno-common -static -o main main.cpp \
-L/usr/local/lib -lpq -lpgcommon -lpgport \
-lldap -lsasl2 -lssl -lcrypto -llber \
-lgssapi_krb5 \
-lkrb5 -lk5crypto -lcom_err -lkrb5support \
-lgdbm
On Thu, Oct 10, 2024 at 5:28 PM Mikael Sand wrote:
> I don't mind having this patch in use too much, I have a functional build
> and nothing to worry about, no luck necessary :)
>
> But, I expect a lot of unnecessary churn in the community if this is not
> fixed properly.
> Not all applications run inside docker, and not all can use dynamic
> linking.
> People can dislike static linking for whatever reason, but that doesn't
> remove the need for it (or both) in some cases and the desire for it in
> others.
>
> Judging from the commit, it doesn't seem like static linking is
> intentionally broken.
> Should a test be added so as not to break this by mistake?
> I must say I don't fully comprehend the context of that commit.
>
> On Thu, Oct 10, 2024 at 5:06 PM Mikael Sand wrote:
>
>> This seems to be the commit that caused this:
>> https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545a
>>
>> So it seems the commit is related to static linking.
>>
>> Is it somehow possible to statically link in a version of pgcommon with
>> correct function names into libpq?
>>
>
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
This seems to be the commit that caused this: https://github.com/postgres/postgres/commit/b6c7cfac88c47a9194d76f3d074129da3c46545a So it seems the commit is related to static linking. Is it somehow possible to statically link in a version of pgcommon with correct function names into libpq?
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Tom
Would this be an acceptable patch?
Subject: [PATCH] Fix static build
---
Index: src/Makefile.shlib
<+>UTF-8
===
diff --git a/src/Makefile.shlib b/src/Makefile.shlib
--- a/src/Makefile.shlib (revision fd64ed60b62697984bb69a09a3ae19fbe2905eb6)
+++ b/src/Makefile.shlib (date 1728590127913)
@@ -354,7 +354,7 @@
# those that point inside the build or source tree. Use sort to
# remove duplicates. Also record the -l flags necessary for static
# linking, but not those already covered by Requires.private.
- echo 'Libs.private: $(sort $(filter-out -L.% -L$(top_srcdir)/%,$(filter
-L%,$(LDFLAGS) $(SHLIB_LINK $(filter-out
$(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter
-l%,$(SHLIB_LINK_INTERNAL:%_shlib=%) $(SHLIB_LINK)))' >>$@
+ echo 'Libs.private: $(sort $(filter-out -L.% -L$(top_srcdir)/%,$(filter
-L%,$(LDFLAGS) $(SHLIB_LINK $(filter-out
$(PKG_CONFIG_REQUIRES_PRIVATE:lib%=-l%),$(filter -l%,$(SHLIB_LINK_INTERNAL)
$(SHLIB_LINK)))' >>$@
##
Br Mikael
On Thu, Oct 10, 2024 at 9:08 PM Mikael Sand wrote:
> Aleksander, do you have something against static linking or am I reading
> you wrong?
> I've seen this sentiment in several places but never understood why anyone
> would hold this position. Could you elaborate?
>
> At least for executables intended to be run in production inside
> containers, they usually run in isolation, what other executables could
> they share libraries with?
> Should we leave performance on the table, given the energy and resource
> requirements it implies?
>
> Certainly, for libraries intended to be distributed in e.g. Linux
> distributions and be depended on in end-user executables, it makes sense to
> have shared libraries and simplify mitigating security issues.
>
> It appears to me that there are many cases where static linking and link
> time optimization would be more appropriate than dynamic.
> When the executables never have a chance to share libraries with any other
> process and are rebuilt whenever a new version is deployed what benefits
> does dynamic linking provide over static + lto?
>
> I expect I'm missing something crucial, but I have spent some time trying
> to understand this issue deeper, and still find myself mystified.
> Please enlighten me.
>
> Br Mikael
>
> On Thu, Oct 10, 2024 at 8:29 PM Mikael Sand wrote:
>
>> Just for reference in case anyone else who utilizes static linking for
>> any reason hits upon this issue, here is a working Dockerfile for libpq /
>> postgresql 17
>>
>> FROM postgres:17.0-alpine3.20 AS builder
>> USER root
>> WORKDIR /app
>> RUN apk update && apk add --no-cache --update-cache \
>> openssl-libs-static \
>> libevent-static \
>> libxml2-static \
>> libedit-static \
>> libxslt-static \
>> sqlite-static \
>> openldap-dev \
>> libxslt-dev \
>> libxml2-dev \
>> libedit-dev \
>> openssl-dev \
>> zstd-static \
>> zlib-static \
>> lz4-static \
>> e2fsprogs \
>> keyutils \
>> zstd-dev \
>> zlib-dev \
>> gdbm-dev \
>> clang17 \
>> lz4-dev \
>> libldap \
>> bison \
>> curl \
>> perl \
>> make
>>
>> COPY <> #include
>> int main(){return PQconnectdb("")==NULL;}
>> EOF
>>
>> ARG KRB5=1.21.3
>> ARG KRB5MAJMIN=1.21
>> RUN curl -L https://kerberos.org/dist/krb5/$KRB5MAJMIN/krb5-$KRB5.tar.gz |
>> tar xzf -
>> RUN cd krb5-$KRB5/src && \
>> ./configure && make && make install && \
>> ./configure --disable-shared --enable-static && make && make install
>>
>> ARG SASL=2.1.28
>> RUN curl -L
>> https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-$SASL/cyrus-sasl-$SASL.tar.gz
>> | tar xzf -
>> RUN cd cyrus-sasl-$SASL && ./configure --enable-static && make && make
>> install
>>
>> RUN clang++ -static -o main main.cpp \
>> -L/usr/local/lib -lpq -lpgcommon_shlib -lpgport_shlib \
>> -lldap -lsasl2 -lssl -lcrypto -llber \
>> -lgssapi_krb5 \
>> -lkrb5 -lk5crypto -lcom_err -lkrb5support \
>> -lgdbm
>>
>>
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Hi Aleksander Ok. So no actual benefit from using dynamic? Well, it seems postgresql and all dependencies already support it, no? Doesn't go do static linking by default / prefer it? Unless you use some part that uses CGO, in which case many go developers appear to disable CGO anyway and use the plain go implementation instead. Similarly, the rust community seems to have a preference / strong support for static, some with alpine, musl, etc. We're still developing this part of our system, and do intend to measure the difference, for this we need to be able to build the static version as well. I can post results once we have them. Have you measured the difference? Br Mikael On Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev < aleksan...@timescale.com> wrote: > Hi Mikael, > > > We use static linking and link time optimization to squeeze the last > bits of performance out of the code in our most performance-critical > queries, and it simplifies our security audits to have a static binary > running inside chainguard/static as the data we handle is > sensitive/business critical. > > No comments about the security aspect. I'm not a security expert so I > leave this topic for somebody for whom this is an area of expertise. > > I don't buy the optimization part. The performance gain you get is > next to nothing compared to the time spent transferring SQL queries > over the network / UDP sockets, parsing and planning them, accessing > the disk, waiting for the locks, etc. Unless you actually measured it > in order to show the opposite which I seriously doubt you did. A > better time investment would be finding actual bottlenecks in the > given system in order to be able to eliminate them. > > > Aleksander, do you have something against static linking or am I reading > you wrong? I've seen this sentiment in several places but never understood > why anyone would hold this position. Could you elaborate? > > The question is whether it's practical for the PostgreSQL community to > put effort into supporting / testing static linking of libpq. > Personally I'm not convinced that demanding from every open-source > project in existence to support static linking is a sustainable idea > in the long run, given the fact that there are technologies like > Docker and programming languages like Go (with pgx client). > > -- > Best regards, > Aleksander Alekseev >
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
Btw Aleksander To give some context, this is a component intended to solve our last scaling issue / most demanding query, which happens at page load and thus need to minimize latency over all else. We've used libpq in pipelined binary mode, serialized the results it in a custom binary format, compressed that using zstd, sent to the frontend, uncompressed by the browser builtin support, and parsed from binary to javascript objects using custom parser. This has improved performance more than 50x over the existing graphql / postgraphile solution so far. It seems likely that doing this work in a read replica would further improve performance significantly, so as not to send the large amount of data from the database to the backend, but this will be a later project. I can have measurements regarding static + lto vs dynamic before that. Br Mikael On Thu, Oct 10, 2024 at 10:22 PM Mikael Sand wrote: > Hi Aleksander > > Ok. So no actual benefit from using dynamic? > Well, it seems postgresql and all dependencies already support it, no? > Doesn't go do static linking by default / prefer it? Unless you use some > part that uses CGO, in which case many go developers appear to disable CGO > anyway and use the plain go implementation instead. > Similarly, the rust community seems to have a preference / strong support > for static, some with alpine, musl, etc. > We're still developing this part of our system, and do intend to measure > the difference, for this we need to be able to build the static version as > well. I can post results once we have them. Have you measured the > difference? > > Br Mikael > > On Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev < > aleksan...@timescale.com> wrote: > >> Hi Mikael, >> >> > We use static linking and link time optimization to squeeze the last >> bits of performance out of the code in our most performance-critical >> queries, and it simplifies our security audits to have a static binary >> running inside chainguard/static as the data we handle is >> sensitive/business critical. >> >> No comments about the security aspect. I'm not a security expert so I >> leave this topic for somebody for whom this is an area of expertise. >> >> I don't buy the optimization part. The performance gain you get is >> next to nothing compared to the time spent transferring SQL queries >> over the network / UDP sockets, parsing and planning them, accessing >> the disk, waiting for the locks, etc. Unless you actually measured it >> in order to show the opposite which I seriously doubt you did. A >> better time investment would be finding actual bottlenecks in the >> given system in order to be able to eliminate them. >> >> > Aleksander, do you have something against static linking or am I >> reading you wrong? I've seen this sentiment in several places but never >> understood why anyone would hold this position. Could you elaborate? >> >> The question is whether it's practical for the PostgreSQL community to >> put effort into supporting / testing static linking of libpq. >> Personally I'm not convinced that demanding from every open-source >> project in existence to support static linking is a sustainable idea >> in the long run, given the fact that there are technologies like >> Docker and programming languages like Go (with pgx client). >> >> -- >> Best regards, >> Aleksander Alekseev >> >
Re: Build issue with postgresql 17 undefined reference to `pg_encoding_to_char' and `pg_char_to_encoding'
One question, I wonder if it would be possible to link pgcommon_shlib and pgport_shlib statically into pq and do LTO already at that point, such that consumers of libpq could merely link against libpq.a and that would be enough? I haven't maintained any c or c++ library intended for public distribution and haven't needed this kind of functionality, but perhaps someone on the mailing list would already have the necessary expertise? Br Mikael On Thu, Oct 10, 2024 at 10:33 PM Mikael Sand wrote: > Btw Aleksander > > To give some context, this is a component intended to solve our last > scaling issue / most demanding query, which happens at page load and thus > need to minimize latency over all else. We've used libpq in pipelined > binary mode, serialized the results it in a custom binary format, > compressed that using zstd, sent to the frontend, uncompressed by the > browser builtin support, and parsed from binary to javascript objects using > custom parser. This has improved performance more than 50x over the > existing graphql / postgraphile solution so far. It seems likely that doing > this work in a read replica would further improve performance > significantly, so as not to send the large amount of data from the database > to the backend, but this will be a later project. I can have measurements > regarding static + lto vs dynamic before that. > > Br Mikael > > On Thu, Oct 10, 2024 at 10:22 PM Mikael Sand wrote: > >> Hi Aleksander >> >> Ok. So no actual benefit from using dynamic? >> Well, it seems postgresql and all dependencies already support it, no? >> Doesn't go do static linking by default / prefer it? Unless you use some >> part that uses CGO, in which case many go developers appear to disable CGO >> anyway and use the plain go implementation instead. >> Similarly, the rust community seems to have a preference / strong support >> for static, some with alpine, musl, etc. >> We're still developing this part of our system, and do intend to measure >> the difference, for this we need to be able to build the static version as >> well. I can post results once we have them. Have you measured the >> difference? >> >> Br Mikael >> >> On Thu, Oct 10, 2024 at 10:08 PM Aleksander Alekseev < >> aleksan...@timescale.com> wrote: >> >>> Hi Mikael, >>> >>> > We use static linking and link time optimization to squeeze the last >>> bits of performance out of the code in our most performance-critical >>> queries, and it simplifies our security audits to have a static binary >>> running inside chainguard/static as the data we handle is >>> sensitive/business critical. >>> >>> No comments about the security aspect. I'm not a security expert so I >>> leave this topic for somebody for whom this is an area of expertise. >>> >>> I don't buy the optimization part. The performance gain you get is >>> next to nothing compared to the time spent transferring SQL queries >>> over the network / UDP sockets, parsing and planning them, accessing >>> the disk, waiting for the locks, etc. Unless you actually measured it >>> in order to show the opposite which I seriously doubt you did. A >>> better time investment would be finding actual bottlenecks in the >>> given system in order to be able to eliminate them. >>> >>> > Aleksander, do you have something against static linking or am I >>> reading you wrong? I've seen this sentiment in several places but never >>> understood why anyone would hold this position. Could you elaborate? >>> >>> The question is whether it's practical for the PostgreSQL community to >>> put effort into supporting / testing static linking of libpq. >>> Personally I'm not convinced that demanding from every open-source >>> project in existence to support static linking is a sustainable idea >>> in the long run, given the fact that there are technologies like >>> Docker and programming languages like Go (with pgx client). >>> >>> -- >>> Best regards, >>> Aleksander Alekseev >>> >>
