LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes
Hi all, I have encountered a problem related to LDAP authenticated session with Postgres foreign data wrapper (postgres_fdw). The server crashed with following errors and other active server processes are terminated as well: 2019-02-20 14:53:30.496 SGT [PID=1353 application="" user_name= database= host(port)=] LOG: server process (PID 26306) was terminated by signal 11: Segmentation fault 2019-02-20 14:53:30.496 SGT [PID=1353 application="" user_name= database= host(port)=] LOG: terminating any other active server processes I can reproduce it in a test server with many other sessions connected: 1. login using non-LDAP-authenticated user, query local & foreign tables - OK 2. login using LDAP-authenticated user, query local table - OK 3. login using LDAP-authenticated user, query foreign table - ERROR, server crashes with signal 11: Segmentation fault error when I quit the psql session It seems like the problem only when the LDAP-authenticated session (which queried foreign table) is terminated. In dmesg log, I can see following: [16385512.182231] traps: postmaster[26306] general protection ip:7f1e758b638c sp:7ffef7ed8858 error:0 in libc-2.17.so[7f1e75836000+1b6000] Has anyone encountered similar issue? ## PostgreSQL version: 10.6 Platform: CentOS Linux ## Thank you. Regards, Mike Yeap
Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes
> Are the "postgres" executable and libpq linked with the same version of OpenLDAP? How should I check whether they are linked? My Postgres version is 10.6 and I have this output for "yum list | grep ldap | sort": $ yum list | grep ldap | sort apr-util-ldap.x86_641.5.2-6.el7base bind-dyndb-ldap.x86_64 11.1-4.el7 base compat-openldap.i6861:2.3.43-5.el7 base compat-openldap.x86_64 1:2.3.43-5.el7 base cyrus-sasl-ldap.i6862.1.26-23.el7 base cyrus-sasl-ldap.x86_64 2.1.26-23.el7 base freeradius-ldap.x86_64 3.0.13-9.el7_5 base ipsilon-authldap.noarch 1.0.0-13.el7_3 base krb5-server-ldap.x86_64 1.15.1-37.el7_6 updates ldapjdk-javadoc.noarch 4.19-5.el7 base ldapjdk.noarch 4.19-5.el7 base mod_ldap.x86_64 2.4.6-88.el7.centosbase nss-pam-ldapd.i686 0.8.13-16.el7 base nss-pam-ldapd.x86_640.8.13-16.el7 base openldap-clients.x86_64 2.4.44-21.el7_6 @updates openldap-devel.i686 2.4.44-21.el7_6 updates openldap-devel.x86_64 2.4.44-21.el7_6 updates openldap.i686 2.4.44-21.el7_6 updates openldap-servers-sql.x86_64 2.4.44-21.el7_6 updates openldap-servers.x86_64 2.4.44-21.el7_6 updates openldap.x86_64 2.4.44-21.el7_6 @updates openssh-ldap.x86_64 7.4p1-16.el7 base php-ldap.x86_64 5.4.16-46.el7 base python-ldap2pg-doc.x86_64 4.11-1.rhel7 pgdg10 python-ldap2pg.x86_64 4.11-1.rhel7 pgdg10 python-ldap.x86_64 2.4.15-2.el7 base sssd-ldap.x86_641.16.2-13.el7_6.5 updates And in the database where I encountered this issue I have these extensions installed: repdb=# \dx List of installed extensions Name| Version | Schema | Description +-++ hstore | 1.4 | public | data type for storing sets of (key, value) pairs pg_stat_statements | 1.6 | repdb | track execution statistics of all SQL statements executed plpgsql| 1.0 | pg_catalog | PL/pgSQL procedural language postgres_fdw | 1.0 | repdb | foreign-data wrapper for remote PostgreSQL servers tablefunc | 1.0 | repdb | functions that manipulate whole tables, including crosstab (5 rows) Thank you. Regards, Mike Yeap On Wed, Feb 20, 2019 at 10:17 PM Tom Lane wrote: > Laurenz Albe writes: > > Mike Yeap wrote: > >> I have encountered a problem related to LDAP authenticated session with > Postgres foreign data wrapper (postgres_fdw). > > > Are the "postgres" executable and libpq linked with the same version of > OpenLDAP? > > And which version is that? (And which version of Postgres?) > > Digging around in our git history, I came across this: > > Author: Noah Misch > Branch: master Release: REL9_5_BR [d7cdf6ee3] 2014-07-22 11:01:03 -0400 > > Diagnose incompatible OpenLDAP versions during build and test. > > With OpenLDAP versions 2.4.24 through 2.4.31, inclusive, PostgreSQL > backends can crash at exit. Raise a warning during "configure" based > on > the compile-time OpenLDAP version number, and test the crash scenario > in > the dblink test suite. Back-patch to 9.0 (all supported versions). > > which sounds a fair bit like what you are describing. > > regards, tom lane >
Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes
Hi Tom, when I run "ldd /usr/pgsql-10/bin/postmaster" I got this output: # ldd /usr/pgsql-10/bin/postmaster linux-vdso.so.1 => (0x7ffd4ec65000) libpthread.so.0 => /lib64/libpthread.so.0 (0x7eff8b5d3000) libxml2.so.2 => /lib64/libxml2.so.2 (0x7eff8b268000) libpam.so.0 => /lib64/libpam.so.0 (0x7eff8b059000) libssl.so.10 => /lib64/libssl.so.10 (0x7eff8ade7000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x7eff8a985000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x7eff8a738000) librt.so.1 => /lib64/librt.so.1 (0x7eff8a53) libdl.so.2 => /lib64/libdl.so.2 (0x7eff8a32b000) libm.so.6 => /lib64/libm.so.6 (0x7eff8a029000) libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x7eff89dd4000) libicui18n.so.50 => /lib64/libicui18n.so.50 (0x7eff899d4000) libicuuc.so.50 => /lib64/libicuuc.so.50 (0x7eff8965b000) libsystemd.so.0 => /lib64/libsystemd.so.0 (0x7eff89633000) libc.so.6 => /lib64/libc.so.6 (0x7eff89271000) /lib64/ld-linux-x86-64.so.2 (0x7eff8b7f9000) libz.so.1 => /lib64/libz.so.1 (0x7eff8905b000) liblzma.so.5 => /lib64/liblzma.so.5 (0x7eff88e35000) libaudit.so.1 => /lib64/libaudit.so.1 (0x7eff88c0c000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x7eff88924000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x7eff8872) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x7eff884ec000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x7eff882de000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x7eff880da000) libresolv.so.2 => /lib64/libresolv.so.2 (0x7eff87ebf000) liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x7eff87cb) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x7eff87a93000) libssl3.so => /lib64/libssl3.so (0x7eff8784f000) libsmime3.so => /lib64/libsmime3.so (0x7eff87628000) libnss3.so => /lib64/libnss3.so (0x7eff87302000) libnssutil3.so => /lib64/libnssutil3.so (0x7eff870d5000) libplds4.so => /lib64/libplds4.so (0x7eff86ed1000) libplc4.so => /lib64/libplc4.so (0x7eff86ccc000) libnspr4.so => /lib64/libnspr4.so (0x7eff86a8d000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x7eff86785000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x7eff8656f000) libicudata.so.50 => /lib64/libicudata.so.50 (0x7eff84f9a000) libcap.so.2 => /lib64/libcap.so.2 (0x7eff84d95000) libselinux.so.1 => /lib64/libselinux.so.1 (0x7eff84b6e000) libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x7eff848ec000) libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x7eff846e7000) libdw.so.1 => /lib64/libdw.so.1 (0x7eff844a) libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x7eff84299000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x7eff84062000) libattr.so.1 => /lib64/libattr.so.1 (0x7eff83e5c000) libpcre.so.1 => /lib64/libpcre.so.1 (0x7eff83bfa000) libelf.so.1 => /lib64/libelf.so.1 (0x7eff839e2000) libbz2.so.1 => /lib64/libbz2.so.1 (0x7eff837d1000) libfreebl3.so => /lib64/libfreebl3.so (0x7eff835ce000) On the line that has ldap in it: libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x7eff89dd4000) Sorry but in this case what is my libpq? Regards, Mike Yeap On Thu, Feb 21, 2019 at 10:03 AM Tom Lane wrote: > Mike Yeap writes: > >> Are the "postgres" executable and libpq linked with the same version of > >> OpenLDAP? > > > How should I check whether they are linked? > > "ldd" should show the dependencies of whatever executable or library > you point it at. > > regards, tom lane >
Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes
Hi Thomas, does that mean the bug is still there? Regards, Mike Yeap On Mon, Feb 25, 2019 at 4:06 PM Thomas Munro wrote: > On Thu, Feb 21, 2019 at 2:42 PM Mike Yeap wrote: > > openldap-clients.x86_64 2.4.44-21.el7_6 > @updates > > openldap-devel.i686 2.4.44-21.el7_6 > updates > > openldap-devel.x86_64 2.4.44-21.el7_6 > updates > > openldap.i686 2.4.44-21.el7_6 > updates > > openldap-servers-sql.x86_64 2.4.44-21.el7_6 > updates > > openldap-servers.x86_64 2.4.44-21.el7_6 > updates > > openldap.x86_64 2.4.44-21.el7_6 > @updates > > > On Wed, Feb 20, 2019 at 10:17 PM Tom Lane wrote: > >> With OpenLDAP versions 2.4.24 through 2.4.31, inclusive, PostgreSQL > >> backends can crash at exit. Raise a warning during "configure" > based on > >> the compile-time OpenLDAP version number, and test the crash > scenario in > >> the dblink test suite. Back-patch to 9.0 (all supported versions). > > Clearly 2.4.44 is not in the range 2.4.24 through 2.4.31. Perhaps the > dangerous range is out of date? Hmm, so Noah's analysis[1] says this > is a clash between libldap_r.so (used by libpq) and libldap.so (used > by the server), specifically in destructor/exit code. Curiously, in a > thread about Curl's struggles with this problem, I found a claim[2] > that Debian decided to abandon the non-"_r" variant and just use _r > always. Sure enough, on my Debian buster VM I see a symlink > libldap-2.4.so.2 -> libldap_r-2.4.so.2. So essentially Debian and > friends have already forced Noah's first option on users: > > > 1. Link the backend with libldap_r, so we never face the mismatch. On > some > > platforms, this means also linking in threading libraries. > > FreeBSD and CentOS systems near me have separate libraries still. > > [1] > https://www.postgresql.org/message-id/flat/20140612210219.GA705509%40tornado.leadboat.com > [2] https://www.openldap.org/lists/openldap-technical/201608/msg00094.html > > -- > Thomas Munro > https://enterprisedb.com >
Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes
Hi Thomas, I see. guess I can't use LDAP authentication for now, :-( Hopefully this problem is solved in future version, thank you! Regards, Mike Yeap On Tue, Feb 26, 2019 at 4:12 PM Thomas Munro wrote: > On Tue, Feb 26, 2019 at 8:17 PM Mike Yeap wrote: > > Hi Thomas, does that mean the bug is still there? > > Hi Mike, > > I haven't tried to repro this myself, but it certainly sounds like it. > It also sounds like it would probably go away if you switched to a > Debian-derived distro, instead of a Red Hat-derived distro, but I > doubt that's the kind of advice you were looking for. We need to > figure out a proper solution here, though I'm not sure what. Question > for the list: other stuff in the server needs libpthread (SSL, LLVM, > ...), so why are we insisting on using non-MT LDAP? > > -- > Thomas Munro > https://enterprisedb.com >
Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes
Hi Noah, below is the output from one of the servers having this issue: $ echo "select pg_backend_pid(); load 'dblink'; select pg_sleep(100)" | psql -X & [1] 9731 $ select pg_backend_pid(); load 'dblink'; select pg_sleep(100) pg_backend_pid 9732 (1 row) LOAD $ gdb --batch --pid 9732 -ex 'info sharedlibrary ldap' warning: .dynamic section for "/lib64/libldap-2.4.so.2" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/lib64/liblber-2.4.so.2" is not at the expected address (wrong library or version mismatch?) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x7f1e7592dcf3 in __epoll_wait_nocancel () from /lib64/libc.so.6 >FromTo Syms Read Shared Object Library 0x7f1e7637d0f8 0x7f1e763ae51c Yes (*) /lib64/libldap-2.4.so.2 0x7f1d9f2c16d0 0x7f1d9f2f5ae4 Yes (*) /lib64/libldap_r-2.4.so.2 (*): Shared library is missing debugging information. Regards, Mike Yeap On Thu, Mar 14, 2019 at 1:42 PM Noah Misch wrote: > On Thu, Mar 14, 2019 at 05:18:49PM +1300, Thomas Munro wrote: > > On Thu, Mar 7, 2019 at 4:19 PM Noah Misch wrote: > > > Has anyone else reproduced this? > > > > I tried, but could not reproduce this problem on "CentOS Linux release > > 7.6.1810 (Core)" using OpenLDAP "2.4.44-21.el7_6" (same as Mike > > reported, what yum install is currently serving up). > > > When exiting the session, I was expecting the backend to crash, > > because it had executed libldap.so code during authentication, and > > then it had linked in libldap_r.so via libpq.so while connecting via > > postgres_fdw. But it doesn't crash. I wonder what is different for > > Mike; am I missing something, or is there non-determinism here? > > The test is deterministic. I'm guessing Mike's system is finding ldap > libraries other than the usual system ones. Mike, would you check as > follows? > > $ echo "select pg_backend_pid(); load 'dblink'; select pg_sleep(100)" | > psql -X & > [1] 2530123 > pg_backend_pid > > 2530124 > (1 row) > > LOAD > > $ gdb --batch --pid 2530124 -ex 'info sharedlibrary ldap' > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > 0x76303463 in __epoll_wait_nocancel () from /lib64/libc.so.6 > FromTo Syms Read Shared Object Library > 0x765e1ee0 0x76613304 Yes (*) /lib64/libldap-2.4.so.2 > 0x7fffe998f6d0 0x7fffe99c3ae4 Yes (*) > /lib64/libldap_r-2.4.so.2 > (*): Shared library is missing debugging information. >
Postgres PANIC when it could not open file in pg_logical/snapshots directory
Hi all, I have a Postgres version 11.11 configured with both physical replication slots (for repmgr) as well as some logical replication slots (for AWS Database Migration Service (DMS)). This morning, the server went panic with the following messages found in the log file: 2021-06-22 04:56:35.314 +08 [PID=19457 application="[unknown]" user_name=dms database=** host(port)=**(48360)] PANIC: could not open file "pg_logical/snapshots/969-FD606138.snap": Operation not permitted 2021-06-22 04:56:35.317 +08 [PID=1752 application="" user_name= database= host(port)=] LOG: server process (PID 19457) was terminated by signal 6: Aborted 2021-06-22 04:56:35.317 +08 [PID=1752 application="" user_name= database= host(port)=] LOG: terminating any other active server processes The PG server then terminates all existing PG processes. The process with 19457 is from one of the DMS replication tasks, I have no clue why it suddenly couldn't open a snapshot file. I checked the server load and file systems and didn't find anything unusual at that time. Appreciate if you can give me some guidance on troubleshooting this issue Thanks Regards, Mike Yeap
Virtual patching software for PostgreSQL
Hi all, I'm looking for an alternative software that does virtual patching for PostgreSQL DB servers to the Trellix Database Security (previously named McAfee Database Security) I'm currently using Appreciate if anyone using something like that can share some info Thanks! Regards, Mike Yeap
