[PATCH] Support static linking against LLVM
By default, PostgreSQL doesn't explicitly choose whether to link statically or dynamically against LLVM when LLVM JIT is enabled (e.g.: `./configure --with-llvm`). `llvm-config` will choose to dynamically link by default. In order to statically link, one must pass `--link-static` to `llvm-config` when listing linker flags (`--ldflags`) and libraries (`--libs`). This patch enables custom flags to be passed to `llvm-config` linker related invocations through the environment variable `LLVM_CONFIG_LINK_ARGS`. To statically link against LLVM it suffices, then, to call `configure` with environment variable `LLVM_CONFIG_LINK_ARGS=--link-static`. --- config/llvm.m4 | 5 +++-- configure | 6 -- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/config/llvm.m4 b/config/llvm.m4 index 3a75cd8b4d..712bd3de6c 100644 --- a/config/llvm.m4 +++ b/config/llvm.m4 @@ -13,6 +13,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], AC_REQUIRE([AC_PROG_AWK]) AC_ARG_VAR(LLVM_CONFIG, [path to llvm-config command]) + AC_ARG_VAR(LLVM_CONFIG_LINK_ARGS, [extra arguments for llvm-config linker related flags]) PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7 llvm-config-6.0 llvm-config-5.0 llvm-config-4.0 llvm-config-3.9) # no point continuing if llvm wasn't found @@ -52,7 +53,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], esac done - for pgac_option in `$LLVM_CONFIG --ldflags`; do + for pgac_option in `$LLVM_CONFIG --ldflags $LLVM_CONFIG_LINK_ARGS`; do case $pgac_option in -L*) LDFLAGS="$LDFLAGS $pgac_option";; esac @@ -84,7 +85,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], # And then get the libraries that need to be linked in for the # selected components. They're large libraries, we only want to # link them into the LLVM using shared library. - for pgac_option in `$LLVM_CONFIG --libs --system-libs $pgac_components`; do + for pgac_option in `$LLVM_CONFIG --libs --system-libs $LLVM_CONFIG_LINK_ARGS $pgac_components`; do case $pgac_option in -l*) LLVM_LIBS="$LLVM_LIBS $pgac_option";; esac diff --git a/configure b/configure index 86ffccb1ee..974b7f2d4e 100755 --- a/configure +++ b/configure @@ -1595,6 +1595,8 @@ Some influential environment variables: CXX C++ compiler command CXXFLAGSC++ compiler flags LLVM_CONFIG path to llvm-config command + LLVM_CONFIG_LINK_ARGS + extra arguments for llvm-config linker related flags CLANG path to clang compiler to generate bitcode CPP C preprocessor PKG_CONFIG path to pkg-config utility @@ -5200,7 +5202,7 @@ fi esac done - for pgac_option in `$LLVM_CONFIG --ldflags`; do + for pgac_option in `$LLVM_CONFIG --ldflags $LLVM_CONFIG_LINK_ARGS`; do case $pgac_option in -L*) LDFLAGS="$LDFLAGS $pgac_option";; esac @@ -5232,7 +5234,7 @@ fi # And then get the libraries that need to be linked in for the # selected components. They're large libraries, we only want to # link them into the LLVM using shared library. - for pgac_option in `$LLVM_CONFIG --libs --system-libs $pgac_components`; do + for pgac_option in `$LLVM_CONFIG --libs --system-libs $LLVM_CONFIG_LINK_ARGS $pgac_components`; do case $pgac_option in -l*) LLVM_LIBS="$LLVM_LIBS $pgac_option";; esac -- 2.40.1
Re: [PATCH] Support static linking against LLVM
Andres, Tom, I found your names in the git history for JIT and LLVM. Any chance one of you could take a look at the patch? -mj On Thu, Aug 10, 2023 at 2:45 PM Marcelo Juchem wrote: > By default, PostgreSQL doesn't explicitly choose whether to link > statically or dynamically against LLVM when LLVM JIT is enabled (e.g.: > `./configure --with-llvm`). > > `llvm-config` will choose to dynamically link by default. > > In order to statically link, one must pass `--link-static` to > `llvm-config` when listing linker flags (`--ldflags`) and libraries > (`--libs`). > > This patch enables custom flags to be passed to `llvm-config` linker > related invocations through the environment variable > `LLVM_CONFIG_LINK_ARGS`. > > To statically link against LLVM it suffices, then, to call `configure` > with environment variable `LLVM_CONFIG_LINK_ARGS=--link-static`. > --- > config/llvm.m4 | 5 +++-- > configure | 6 -- > 2 files changed, 7 insertions(+), 4 deletions(-) > > diff --git a/config/llvm.m4 b/config/llvm.m4 > index 3a75cd8b4d..712bd3de6c 100644 > --- a/config/llvm.m4 > +++ b/config/llvm.m4 > @@ -13,6 +13,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], >AC_REQUIRE([AC_PROG_AWK]) > >AC_ARG_VAR(LLVM_CONFIG, [path to llvm-config command]) > + AC_ARG_VAR(LLVM_CONFIG_LINK_ARGS, [extra arguments for llvm-config > linker related flags]) >PGAC_PATH_PROGS(LLVM_CONFIG, llvm-config llvm-config-7 llvm-config-6.0 > llvm-config-5.0 llvm-config-4.0 llvm-config-3.9) > ># no point continuing if llvm wasn't found > @@ -52,7 +53,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], > esac >done > > - for pgac_option in `$LLVM_CONFIG --ldflags`; do > + for pgac_option in `$LLVM_CONFIG --ldflags $LLVM_CONFIG_LINK_ARGS`; do > case $pgac_option in >-L*) LDFLAGS="$LDFLAGS $pgac_option";; > esac > @@ -84,7 +85,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], ># And then get the libraries that need to be linked in for the ># selected components. They're large libraries, we only want to ># link them into the LLVM using shared library. > - for pgac_option in `$LLVM_CONFIG --libs --system-libs > $pgac_components`; do > + for pgac_option in `$LLVM_CONFIG --libs --system-libs > $LLVM_CONFIG_LINK_ARGS $pgac_components`; do > case $pgac_option in >-l*) LLVM_LIBS="$LLVM_LIBS $pgac_option";; > esac > diff --git a/configure b/configure > index 86ffccb1ee..974b7f2d4e 100755 > --- a/configure > +++ b/configure > @@ -1595,6 +1595,8 @@ Some influential environment variables: >CXX C++ compiler command >CXXFLAGSC++ compiler flags >LLVM_CONFIG path to llvm-config command > + LLVM_CONFIG_LINK_ARGS > + extra arguments for llvm-config linker related flags >CLANG path to clang compiler to generate bitcode >CPP C preprocessor >PKG_CONFIG path to pkg-config utility > @@ -5200,7 +5202,7 @@ fi > esac >done > > - for pgac_option in `$LLVM_CONFIG --ldflags`; do > + for pgac_option in `$LLVM_CONFIG --ldflags $LLVM_CONFIG_LINK_ARGS`; do > case $pgac_option in >-L*) LDFLAGS="$LDFLAGS $pgac_option";; > esac > @@ -5232,7 +5234,7 @@ fi ># And then get the libraries that need to be linked in for the ># selected components. They're large libraries, we only want to ># link them into the LLVM using shared library. > - for pgac_option in `$LLVM_CONFIG --libs --system-libs > $pgac_components`; do > + for pgac_option in `$LLVM_CONFIG --libs --system-libs > $LLVM_CONFIG_LINK_ARGS $pgac_components`; do > case $pgac_option in >-l*) LLVM_LIBS="$LLVM_LIBS $pgac_option";; > esac > -- > 2.40.1 > >
Re: [PATCH] Support static linking against LLVM
In my case, my product has a very controlled environment. We build all our infrastructure from source and we avoid dynamic linking by design, except where technically not viable (e.g.: pgsql extensions). LLVM is one of the libraries we're specifically required to statically link. Unfortunately I can't share the specifics of why that's the case. Without static link support by PostgreSQL we can't enable LLVM JIT. On a side note, I'd like to take this opportunity to ask you if there's any work being done towards migrating away from deprecated LLVM APIs. If there's no work being done on that front, I might take a stab at it if there's any interest from the PostgreSQL community in that contribution. More specifically, there are some deprecated C APIs that are used in PostgreSQL (more details in https://llvm.org/docs/OpaquePointers.html#frontends). For that reason, PostgreSQL LLVM JIT support will fail to build starting with the next version of LLVM (version 17). Migrating to the new APIs will have PostgreSQL require at the minimum version 8 of LLVM (released in March 2019). Regards, -mj On Fri, Aug 11, 2023 at 12:43 PM Andres Freund wrote: > Hi, > > On 2023-08-10 14:45:47 -0500, Marcelo Juchem wrote: > > By default, PostgreSQL doesn't explicitly choose whether to link > > statically or dynamically against LLVM when LLVM JIT is enabled (e.g.: > > `./configure --with-llvm`). > > > > `llvm-config` will choose to dynamically link by default. > > > > In order to statically link, one must pass `--link-static` to > > `llvm-config` when listing linker flags (`--ldflags`) and libraries > > (`--libs`). > > > > This patch enables custom flags to be passed to `llvm-config` linker > > related invocations through the environment variable > > `LLVM_CONFIG_LINK_ARGS`. > > > > To statically link against LLVM it suffices, then, to call `configure` > > with environment variable `LLVM_CONFIG_LINK_ARGS=--link-static`. > > I'm not opposed to it, but I'm not sure I see much need for it either. Do > you > have a specific use case for this? > > Greetings, > > Andres Freund >
Re: [PATCH] Support static linking against LLVM
On Fri, Aug 11, 2023 at 4:39 PM Andres Freund wrote: > Hi, > > On 2023-08-11 15:06:44 -0500, Marcelo Juchem wrote: > > On Fri, Aug 11, 2023 at 2:53 PM Andres Freund > wrote: > > > > > Hi, > > > > > > On 2023-08-11 13:43:17 -0500, Marcelo Juchem wrote: > > > > I'm not sure I know a good way to cleanly do that. > > > > > > Have you tried the approach I proposed here: > > > > https://postgr.es/m/20230811183154.vlyn5kvteklhym3v%40awork3.anarazel.de > > > ? > > > > > > > I want to check it out but the link is not working for me. > > Oh, I hadn't realized you had dropped the list from CC in the email prior, > that's why it's not archived. My proposal was: > Sorry, that wasn't intentional. I'll add it back. > > On 2023-08-11 11:31:54 -0700, Andres Freund wrote: > > I'd prefer a patch that worked around that oddity, rather than adding a > > separate "argument" that requires everyone encountering it to figure out > the > > argument exists and to specify it. > > > > I don't have a static-only llvm around right now, but I do have a > "dynamic > > only" llvm around. It errors out when using "--link-static --libs" - > assuming > > that's the case with the static-only llvm, we could infer the need to > specify > > --link-static based on --link-dynamic erroring out? > > Does your static only llvm error out if you do llvm-config --link-dynamic > --libs? > Yes, it does not work. I understand the final decision is not up to me, and I respect whatever direction you and the community wants to go with, but I'd like to humbly offer my perspective: The issue I'm facing may as well be a corner case or transitional issue with `llvm-config` or LLVM build system. I've recently submitted a couple LLVM patches for a different build system issue related to static linking (has to do with `iovisor/bcc`). In my experience, static linking support is not as ironed out as shared linking in LLVM. I'm not sure it is in the best interest of PostgreSQL to pick up the slack. Instead of optimizing for my use case, , what about instead simply offering "default" (current behavior), "static" and "shared" (explicit choice)? It seems to me it is easier to implement, and less intrusive towards PostgreSQL build system, as opposed to automatically detecting a possibly odd environment. It also feels more general since in the average case, "default" (--with-llvm) should just work. But if someone is intentional about link mode or, as in my case, needs to work around an issue; then explicitly choosing `--with-llvm=static` or `--with-llvm=shared` should do the job just fine. What do you think? > > Greetings, > > Andres Freund >
