Hello,
I did a full review on the provided patches plus some tests, I was able to
validate that the loading of bitcode modules is working also JIT works for
both backend and contrib modules.
To test JIT on contrib modules I just lowered the costs for all jit
settings and used the intarray extension, using the data/test__int.data:
CREATE EXTENSION intarray;
CREATE TABLE test__int( a int[] );1
\copy test__int from 'data/test__int.data'
For queries any from line 98+ on contrib/intarray/sql/_int.sql will work.
Then I added extra debug messages to llvmjit_inline.cpp
on add_module_to_inline_search_path() function, also
on llvm_build_inline_plan(), I was able to see many functions in this
module being successfully inlined.
I'm attaching a new patch based on your original work which add further
support for generating bitcode from:
- Generated backend sources: processed by flex, bison, etc.
- Generated contrib module sources,
On this patch I just included fmgrtab.c and src/backend/parser for the
backend generated code.
For contrib generated sources I added contrib/cube as an example.
All relevant details about the changes are included in the patch itself.
As you may know already I also created a PR focused on llvm bitcode
emission on meson, it generates bitcode for all backend and contribution
modules, currently under review by some colleagues at Percona:
https://github.com/percona/postgres/pull/103
I'm curious if we should get all or some of the generated backend sources
compiled to bitcode, similar to contrib modules.
Please let me know your thoughts and how we can proceed to get this feature
included, thank you.
Regards,
Diego Fronza
Percona
On Fri, Mar 7, 2025 at 7:52 AM Nazir Bilal Yavuz <byavu...@gmail.com> wrote:
> Hi,
>
> On Thu, 5 Sept 2024 at 12:24, Nazir Bilal Yavuz <byavu...@gmail.com>
> wrote:
> >
> > I found that Andres shared a patch
> > (v17-0021-meson-Add-LLVM-bitcode-emission.patch) a while ago [1].
>
> Andres and I continued to work on that. I think the patches are in
> sharable state now and I wanted to hear opinions before proceeding
> further. After applying the patches, bitcode files should be installed
> into $pkglibdir/bitcode/ directory if the llvm is found.
>
> There are 6 patches attached:
>
> v1-0001-meson-Add-generated-header-stamps:
>
> This patch is trivial. Instead of having targets depending directly on
> the generated headers, have them depend on a stamp file. The benefit
> of using a stamp file is that it makes ninja.build smaller and meson
> setup faster.
> ----------
>
> v1-0002-meson-Add-postgresql-extension.pc-for-building-extension-libraries:
>
> This patch is for generating postgresql-extension.pc file which can be
> used for building extensions libraries.
>
> Normally, there is no need to use this .pc file for generating bitcode
> files. However, since there is no clear way to get all include paths
> for building bitcode files, this .pc file is later used for this
> purpose (by running pkg-config --cflags-only-I
> postgresql-extension-uninstalled.pc) [1].
> ----------
>
> v1-0003-meson-Test-building-extensions-by-using-postgresql-extension.pc:
> [Not needed for generating bitcode files]
>
> This is a patch for testing if extensions can be built by using
> postgresql-extension.pc. I added that commit as an example of using
> postgresql-extension.pc to build extensions.
> ----------
>
> v1-0004-meson-WIP-Add-docs-for-postgresql-extension.pc: [Not needed
> for generating bitcode files]
>
> I added this patch in case we recommend people to use
> postgresql-extension.pc to build extension libraries. I am not sure if
> we want to do that because there are still TODOs about
> postgresql-extension.pc like running test suites. I just wanted to
> show my plan, dividing 'Extension Building Infrastructure' into two,
> 'PGXS' and 'postgresql-extension.pc'.
> ----------
>
> v1-0005-meson-Add-LLVM-bitcode-emission:
>
> This patch adds required infrastructure to generate bitcode files and
> uses postgresql-extension-uninstalled.pc to get include paths for
> generating bitcode files [1].
> ----------
>
> v1-0006-meson-Generate-bitcode-files-of-contrib-extension.patch:
>
> This patch adds manually selected contrib libraries to generate their
> bitcode files. These libraries are selected manually, depending on
> - If they have SQL callable functions
> - If the library functions are short enough (the performance gain from
> bitcode files is too minimal compared to the function's run time, so
> this type of libraries are omitted).
>
> Any kind of feedback would be appreciated.
>
> --
> Regards,
> Nazir Bilal Yavuz
> Microsoft
>
From be09b1f38d107789fc9ffe1cd2b2470552689a20 Mon Sep 17 00:00:00 2001
From: Diego Fronza <diego.fro...@percona.com>
Date: Mon, 10 Mar 2025 18:19:49 -0300
Subject: [PATCH] meson: Add LLVM bitcode emission for generated sources
This commit adds suport for bitcode emission for generated source files
(processed by bison, flex, etc).
Since generated sources are created with custom_target, we must handle
the source files differently when iterating over the list, as fs.parent
expects str or file object. The way to handle it is cast the object as
string `@0@.format(src)` then check if it's a custom target (a string
starting with `<CustomTarget`), if that's the case we can safely call
full_path() on it.
But we also want to generate bitcode files with path separators replaced
by underscore, relative to their source location, for that we strip
meson.build_root() from it.
Since generated backend sources may have their own compilation flags and
must also be included in the postgres.index.bc, the way to make it work
with current code was to create a new variable, called
`bc_generated_backend_sources`, which is a list of dictionaries, each
one having an optional 'additional_flags' and a `srclist` pointing to
the list of custom_target generated sources.
This list then has one dictionary entry for each generated backend
sources which share a separate compilation flag.
An example of a possible structure of bitcode_modules which is processed
by the main meson llvm bitcode emission file
src/backend/jit/llvm/bitcode/meson.build:
```
bitcode_modules = [
{
'name': 'postgres',
'target': postgres_lib,
'src_file': backend_sources,
'gen_srcfiles': [
{
'additional_flags': [
'-I/path/postgresl/src/backend/parser',
'-I/path/postgresl/build/src/backend/parser',
],
'srcfiles': [
<custom_target for scan.c>,
<custom_target for gram.c>
]
}
]
}
]
```
---
contrib/cube/meson.build | 2 ++
src/backend/jit/llvm/bitcode/meson.build | 37 ++++++++++++++++++++----
src/backend/meson.build | 2 ++
src/backend/parser/meson.build | 8 +++++
src/backend/utils/fmgr/meson.build | 1 +
5 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/contrib/cube/meson.build b/contrib/cube/meson.build
index 52bd6a9fb8..bbc03dd319 100644
--- a/contrib/cube/meson.build
+++ b/contrib/cube/meson.build
@@ -12,6 +12,7 @@ cube_scan = custom_target('cubescan',
)
generated_sources += cube_scan
cube_sources += cube_scan
+bc_cube_sources += [cube_scan]
cube_parse = custom_target('cubeparse',
input: 'cubeparse.y',
@@ -19,6 +20,7 @@ cube_parse = custom_target('cubeparse',
)
generated_sources += cube_parse.to_list()
cube_sources += cube_parse
+bc_cube_sources += cube_parse[0]
if host_system == 'windows'
cube_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
diff --git a/src/backend/jit/llvm/bitcode/meson.build b/src/backend/jit/llvm/bitcode/meson.build
index 577b4ddb21..acc13a1d16 100644
--- a/src/backend/jit/llvm/bitcode/meson.build
+++ b/src/backend/jit/llvm/bitcode/meson.build
@@ -9,18 +9,18 @@ foreach bitcode_module : bitcode_modules
bitcode_obj = bitcode_module['target']
bitcode_cflags_local = bitcode_cflags + bitcode_module.get('additional_flags', [])
- if 'name' not in bitcode_module
- bitcode_name = bitcode_obj.name()
- else
- bitcode_name = bitcode_module['name']
- endif
+ bitcode_name = bitcode_module.get('name', bitcode_obj.name())
foreach srcfile : bitcode_module['srcfiles']
- if meson.version().version_compare('>=0.59')
+ srcfilename = '@0@'.format(srcfile)
+ if srcfilename.startswith('<CustomTarget')
+ srcfilename = srcfile.full_path().split(meson.build_root() + '/')[1]
+ elif meson.version().version_compare('>=0.59')
srcfilename = fs.parent(srcfile) / fs.name(srcfile)
else
srcfilename = '@0@'.format(srcfile)
endif
+
targetname = '@0@_@1@.bc'.format(
bitcode_name,
srcfilename.underscorify(),
@@ -36,6 +36,31 @@ foreach bitcode_module : bitcode_modules
)
endforeach
+ # process generated sources, which may include custom compilation flags.
+ foreach gen_srcfiles: bitcode_module.get('gen_srcfiles', [])
+ bitcode_cflags_gen_local = bitcode_cflags_local + gen_srcfiles.get('additional_flags', [])
+
+ foreach srcfile: gen_srcfiles['srcfiles']
+ # generated sources are stored in some folder under meson.build_root()/**
+ # remove the build prefix from the string.
+ srcfilename = srcfile.full_path().split(meson.build_root() + '/')[1]
+
+ targetname = '@0@_@1@.bc'.format(
+ bitcode_name,
+ srcfilename.underscorify(),
+ )
+ bitcode_targets += custom_target(
+ targetname,
+ depends: [bitcode_module['target']],
+ input: [srcfile],
+ output: targetname,
+ command: [llvm_irgen_command, bitcode_cflags_gen_local],
+ install: true,
+ install_dir: dir_bitcode,
+ )
+ endforeach
+ endforeach
+
index_name = '@0...@.index.bc'.format(bitcode_name)
bitcode_index = custom_target('@0@'.format(bitcode_name),
output: index_name,
diff --git a/src/backend/meson.build b/src/backend/meson.build
index 5fb33660d3..76e5939fec 100644
--- a/src/backend/meson.build
+++ b/src/backend/meson.build
@@ -5,6 +5,7 @@ backend_sources = []
backend_link_with = [pgport_srv, common_srv]
generated_backend_sources = []
+bc_generated_backend_sources = []
post_export_backend_sources = []
subdir('access')
@@ -144,6 +145,7 @@ bitcode_modules += {
'name': 'postgres',
'target': postgres_lib,
'srcfiles': backend_sources,
+ 'gen_srcfiles': bc_generated_backend_sources,
}
pg_mod_c_args = cflags_mod
diff --git a/src/backend/parser/meson.build b/src/backend/parser/meson.build
index 874aa749aa..add472a0cd 100644
--- a/src/backend/parser/meson.build
+++ b/src/backend/parser/meson.build
@@ -42,6 +42,14 @@ backend_parser = custom_target('gram',
generated_sources += backend_parser.to_list()
parser_sources += backend_parser
+bc_generated_backend_sources += {
+ 'additional_flags': [
+ '-I@0@'.format(meson.current_build_dir()),
+ '-I@0@'.format(meson.current_source_dir()),
+ ],
+ 'srcfiles': [backend_scanner, backend_parser[0]],
+}
+
parser = static_library('parser',
parser_sources,
dependencies: [backend_code],
diff --git a/src/backend/utils/fmgr/meson.build b/src/backend/utils/fmgr/meson.build
index b1dcab93e7..0119bc40cf 100644
--- a/src/backend/utils/fmgr/meson.build
+++ b/src/backend/utils/fmgr/meson.build
@@ -8,3 +8,4 @@ backend_sources += files(
# fmgrtab.c
generated_backend_sources += fmgrtab_target[2]
+bc_generated_backend_sources += {'srcfiles': [fmgrtab_target[2]]}
\ No newline at end of file
--
2.43.0
