Libpq question related to allocated resources
I've got a fairly sizeable application that runs as a CGI app under Apache which I am attempting to convert to FastCGI. "Once through and done" apps tend not to care much if deallocation is less than perfect, since exit(0) (or otherwise) tends to heal all wounds. Not so much when something's running for hours, days or weeks. There memory leaks are ruinous. I've wrapped all of my internal functionality plus all calls to PQexecParams and the escape/unescape functions, all of which must be deallocated after being used. All counters after a pass through the code are zero (increment on allocate, decrement on PQfremem or PQclear.) But -- I still have a /lot /of memory out on the heap according to jemalloc stats that is not being deallocated, and what's worse is that if I rig the code to call PQfinish and then PQconnect once again I get /even more /imbalanced allocate/free counts (and the memory use in said buckets to go with them.) The application itself is nothing particularly fancy although it typically makes dozens of Postgres calls; single-threaded, no prepared statements or async requests. This is under 14.1; I haven't rolled the code forward, but I see nothing in the notes implying there is a problem in libpq that has been corrected, or that there was one in the past in this regard. Its also possible that the FastCGI wrapper has a problem internally. The app, when run under valgrind to do cron processing, comes back clean -- it does show allocations on exit however, although "still accessibl/e" /and those which do come up are related from OpenSSL's error string initialization in /lib/libcrypto.so (I'm on FreeBSD and openssl was specified as "--with-openssl" when Postgres was built.) The obvious question, given the warnings in the FastCGI library: Does libpq /modify /the process environment? Reading from it (provided you don't modify anything from the pointers you access; if you want to then you must copy them somewhere and make the modification outside of the environment itself) is perfectly fine but writing it, directly or indirectly, is NOT. A quick grep implies that indeed it may in backend/libpq/auth.c at least, but I do not have ENABLE_GSS defined in my configuration so that code /shouldn't /be there. -- Karl Denninger k...@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ smime.p7s Description: S/MIME Cryptographic Signature
Re: Libpq question related to allocated resources
On 6/27/2022 23:22, Tom Lane wrote: Karl Denninger writes: But -- I still have a /lot /of memory out on the heap according to jemalloc stats that is not being deallocated, and what's worse is that if I rig the code to call PQfinish and then PQconnect once again I get /even more /imbalanced allocate/free counts (and the memory use in said buckets to go with them.) Hmmm ... I'm not aware of any memory leaks in libpq, but that doesn't mean there are none. Of course, if you're forgetting to PQclear() some PGresults, that's not libpq's fault ;-). Well, yes, which is why I wrapped those calls to make very sure that's not the case (internal reference count in the code when in "debugging mode", basically) along with all the uses of escape/unescape (e.g. bytea fields.) All come back clean on each "round" through which makes it quite puzzling. I'll do more digging. I've got wrappers around all memory allocation in my development libraries that for internal allocations make quite sure that they're both properly paired and sentinels are on the "bookends" so if the code does smash something it catches it, and there's nothing being flagged; the arena, as my code sees it from what it allocated and the calls it made to libpq, are empty when it comes back as it should be. Obviously there's leakage somewhere but at this point I'm quite certain its not in my code itself (its certainly possible FastCGI's lib has a problem somewhere since it has to construct the environment from the web server's CGI call for each call to the application, each of those is distinct, and if something goes wrong there it will leak like crazy since each of those constructs is unique and then must be properly released when that call is complete.) The obvious question, given the warnings in the FastCGI library: Does libpq /modify /the process environment? No. At least, I see no setenv() calls in it, and I think that it'd be pretty unfriendly for a library to do that to its host application. In this case it would be fatal if that was to happen since the environment is synthetic and different on each call; if any part of the environment gets modified then the release by the caller will either leak or, possibly, result in a SEGV. A quick grep implies that indeed it may in backend/libpq/auth.c at least, backend/libpq is unrelated to interfaces/libpq. (I've seen hints that they arose from a common code base, but if so, that was a few decades and a lot of rewrites ago.) regards, tom lane Gotcha. It wasn't clear that this was or wasn't implicated and I'm digging for potential sources, thus the question. Thanks. -- Karl Denninger k...@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ smime.p7s Description: S/MIME Cryptographic Signature
Re: Libpq question related to allocated resources
On 6/27/2022 23:22, Tom Lane wrote: Karl Denninger writes: But -- I still have a /lot /of memory out on the heap according to jemalloc stats that is not being deallocated, and what's worse is that if I rig the code to call PQfinish and then PQconnect once again I get /even more /imbalanced allocate/free counts (and the memory use in said buckets to go with them.) Hmmm ... I'm not aware of any memory leaks in libpq, but that doesn't mean there are none. Of course, if you're forgetting to PQclear() some PGresults, that's not libpq's fault ;-). To follow up on this a bit my investigation is not yet complete /but /I have constructed some truly-hideous worst-case test code that I can have execute under valgrind but using the same basic codebase for everything else (that is, outside of the FastCGI() wrapper loop) and I've not been able to get libpq to misbehave. Everything it grabs it gives back as it should. I've tentatively concluded that the FastCGI wrapper code is doing this and the fault is likely mine (perhaps due to documentation on using it that is less-than-complete) although I've not yet conclusively determined what it is. Wanted to follow up with what I had found since I did make the request -- Karl Denninger k...@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ smime.p7s Description: S/MIME Cryptographic Signature
FreeBSD 12 and Postgres build failures
Both 10.7 and 11.2 fail on FreeBSD 12-STABLE if --with-openssl is part of the configure string. The issue appears to be that FreeBSD updated OpenSSL to OpenSSL 1.1.1a-freebsd 20 Nov 2018 where the 11.x release of FreeBSD (and previous was on 1.0.x -- e.g. OpenSSL 1.0.2o-freebsd 27 Mar 2018) The errors start here: gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I../../../src/include -c -o be-secure-openssl.o be-secure-openssl.c be-secure-openssl.c: In function 'be_tls_init': be-secure-openssl.c:85:3: warning: implicit declaration of function 'OPENSSL_init_ssl'; did you mean 'OPENSSL_init'? [-Wimplicit-function-declaration] OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL); ^~~~ OPENSSL_init be-secure-openssl.c:85:20: error: 'OPENSSL_INIT_LOAD_CONFIG' undeclared (first use in this function); did you mean 'OPENSSL_NO_PADLOCK'? OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL); ^~~~ OPENSSL_NO_PADLOCK be-secure-openssl.c:85:20: note: each undeclared identifier is reported only once for each function it appears in be-secure-openssl.c: In function 'my_sock_read': be-secure-openssl.c:665:35: warning: implicit declaration of function 'BIO_get_data'; did you mean 'BIO_get_ex_data'? [-Wimplicit-function-declaration] res = secure_raw_read(((Port *) BIO_get_data(h)), buf, size); ^~~~ BIO_get_ex_data be-secure-openssl.c:665:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast] res = secure_raw_read(((Port *) BIO_get_data(h)), buf, size); ^ Has anyone looked into and dealt with this yet or have suggestions? -- Karl Denninger k...@denninger.net <mailto:k...@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ smime.p7s Description: S/MIME Cryptographic Signature
Re: FreeBSD 12 and Postgres build failures
On 3/19/2019 17:12, Tom Lane wrote: > Karl Denninger writes: >> Both 10.7 and 11.2 fail on FreeBSD 12-STABLE if --with-openssl is part >> of the configure string. > Hmmm ... I don't have that exact configuration handy to try, but > we know that PG works against OpenSSL 1.1.1a in principle, and > it does work for me with FreeBSD 12.0-RC1 from mid-November, > which contains > # openssl version > OpenSSL 1.1.1-freebsd 11 Sep 2018 > > The error you quote looks a bit like bad decisions in PG's configure > script --- maybe you need to re-run configure? Be sure to blow > away any autoconf cache file if you're using one. > > regards, tom lane That was on a bare, new grab of the source from your repository and a straight-up run of "./configure --with-openssl --with-perl", then "gmake". It's a new directory tree - there's no autoconf or other cache files laying around. Same grab on FreeBSD 11.x builds fine. Digging around I had the openssl-1.0.2r port on the system, which used to be necessary for some other stuff back before 12.x. I've removed it and it looks like that took care of it; for whatever reason configure was getting confused (in theory this shouldn't get you, but theory and practice sometimes don't play nice. :-) Thanks for the confirm that others do have it working; gave me a place to start looking for the issue. -- Karl Denninger k...@denninger.net <mailto:k...@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ smime.p7s Description: S/MIME Cryptographic Signature
Re: FreeBSD 12 and Postgres build failures
On 3/19/2019 18:15, Andrew Gierth wrote: >>>>>> "Karl" == Karl Denninger writes: > Karl> That was on a bare, new grab of the source from your repository and a > Karl> straight-up run of "./configure --with-openssl --with-perl", then > "gmake". > > That's a configure line that has never worked on FreeBSD: > > % ./configure --with-openssl --with-perl > [...] > configure: error: readline header not found > If you have libedit already installed, see config.log for details on the > failure. It is possible the compiler isn't looking in the proper directory. > Use --without-readline to disable libedit support. > % Uh, it has worked on all my FreeBSD systems for a very long time; I've been running Postgres as the backend for a web forum/blogging system for the last many years. Then again: [karl@NewFS ~]$ pkg info|grep readline readline-7.0.5 Library for editing command lines as they are typed -- Karl Denninger k...@denninger.net <mailto:k...@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ smime.p7s Description: S/MIME Cryptographic Signature
Re: FreeBSD 12 and Postgres build failures
On 3/19/2019 21:12, Andrew Gierth wrote: >>>>>> "Karl" == Karl Denninger writes: > >> That's a configure line that has never worked on FreeBSD: > >> > >> % ./configure --with-openssl --with-perl > >> [...] > >> configure: error: readline header not found > >> If you have libedit already installed, see config.log for details on the > >> failure. It is possible the compiler isn't looking in the proper > directory. > >> Use --without-readline to disable libedit support. > >> % > > Karl> Uh, it has worked on all my FreeBSD systems for a very long time; > > Then you have done something non-standard, because "cc" on FreeBSD does > not look in /usr/local/include or /usr/local/lib without being told to > do so, and the configure checks for readline will fail without that. > Possible explanations include having a "gcc" (without a specific version > suffix) on the $PATH (gcc will be chosen over cc in that case), or > having configured CFLAGS or CPPFLAGS in the environment to ./configure. > > You might post your full output of ./configure rather than leaving us to > guess at it. "cc" on FreeBSD is now (and has been for quite some time) clang: root@NewFS:/home/karl # cc -v FreeBSD clang version 6.0.1 (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1) Target: x86_64-unknown-freebsd12.0 Thread model: posix InstalledDir: /usr/bin If it will compile Postgres now it definitely would not when it was first shifted to; gcc, on the other hand (and gmake) does just fine and yes, they are ports (packages in this case as they're binary installs on this box) and not part of the base system: root@NewFS:/home/karl # which gcc /usr/local/bin/gcc root@NewFS:/home/karl # gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc8/gcc/x86_64-portbld-freebsd12.0/8.3.0/lto-wrapper Target: x86_64-portbld-freebsd12.0 Configured with: /wrkdirs/usr/ports/lang/gcc8/work/gcc-8.3.0/configure --with-build-config=bootstrap-debug --disable-nls --enable-gnu-indirect-function --libdir=/usr/local/lib/gcc8 --libexecdir=/usr/local/libexec/gcc8 --program-suffix=8 --with-as=/usr/local/bin/as --with-gmp=/usr/local --with-gxx-include-dir=/usr/local/lib/gcc8/include/c++/ --with-ld=/usr/local/bin/ld --with-pkgversion='FreeBSD Ports Collection' --with-system-zlib --enable-languages=c,c++,objc,fortran --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/share/info/gcc8 --build=x86_64-portbld-freebsd12.0 Thread model: posix gcc version 8.3.0 (FreeBSD Ports Collection) I'll spin up a test machine on a "virgin" install when I get one free from other things and see exactly what I have to put on there in terms of ports. In any event with the removal of the openssl port it now configures, compiles, links and runs. config.log makes clear it is using gcc (as did the blowup) This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. It was created by PostgreSQL configure 11.2, which was generated by GNU Autoconf 2.69. Invocation command line was $ ./configure --with-openssl --with-perl ## - ## ## Platform. ## ## - ## hostname = NewFS.denninger.net uname -m = amd64 uname -r = 12.0-STABLE uname -s = FreeBSD uname -v = FreeBSD 12.0-STABLE r343809 KSD-SMP /usr/bin/uname -p = amd64 /bin/uname -X = unknown /bin/arch = unknown /usr/bin/arch -k = unknown /usr/convex/getsysinfo = unknown /usr/bin/hostinfo = unknown /bin/machine = unknown /usr/bin/oslevel = unknown /bin/universe = unknown PATH: /sbin PATH: /bin PATH: /usr/sbin PATH: /usr/bin PATH: /usr/local/sbin PATH: /usr/local/bin PATH: /root/bin ## --- ## ## Core tests. ## ## --- ## configure:2840: checking build system type configure:2854: result: x86_64-unknown-freebsd12.0 configure:2874: checking host system type configure:2887: result: x86_64-unknown-freebsd12.0 configure:2909: checking which template to use configure:2977: result: freebsd configure:3102: checking whether NLS is wanted configure:3134: result: no configure:3142: checking for default port number configure:3167: result: 5432 configure:3672: checking for block size configure:3706: result: 8kB configure:3718: checking for segment size configure:3747: result: 1GB configure:3759: checking for WAL block size configure:3794: result: 8kB configure:3891: checking for gcc configure:3907: found /usr/local/bin/gcc configure:3918: result: gcc configure:3949: checking for C compiler version configure:3958: gcc --version >&5 gcc (FreeBSD Ports Collection) 8.3.0 Copyright (C) 2018 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; n
