Debugging confirms this, tgt uses dlopen to load its optional storage modules and from there into glusterfs the it crashes
(gdb) frame 20 #20 0x0000555555596d3c in bs_init_signalfd () at bs.c:286 (gdb) p soname $1 = 0x555555dbe150 "/usr/lib/tgt/backing-store/bs_glfs.so" (gdb) l This is slightly more tricky in this case, as there are two conditions that can break it: 1. loading tcmalloc late conflicting with existing code (that was LP: #1950777 Debian #999700 2. But on x86 in the cases we had #1 didn't happen, yet there is another problem and that is when multiple libs want to insert tcmalloc (which happens here) Ceph (via tgt-rbd) already can use tcmalloc and tgt is ready to tolerate that. But not ALSO glusterfs uses tcmalloc. So effectively tgt does dlopen on its libs which now both pull in tcmalloc. We can see that in GDB (gdb) b dlopen Breakpoint 1 at 0x14620 (gdb) run -f Starting program: /usr/sbin/tgtd -f [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules? tgtd: work_timer_start(146) use timer_fd based scheduler Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_glfs.so", mode=mode@entry=2) at dlopen.c:77 77 dlopen.c: No such file or directory. (gdb) c Continuing. Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_rbd.so", mode=mode@entry=2) at dlopen.c:77 77 in dlopen.c (gdb) c Continuing. src/tcmalloc.cc:333] Attempt to free invalid pointer 0x5555555ecae0 And since this is name-based ordering we can turn the order around: $ mv /usr/lib/tgt/backing-store/bs_glfs.so /usr/lib/tgt/backing- store/bs_xxglfs.so And now: (gdb) b dlopen Breakpoint 1 at 0x14620 (gdb) run -f Starting program: /usr/sbin/tgtd -f [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules? tgtd: work_timer_start(146) use timer_fd based scheduler Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_xxglfs.so", mode=mode@entry=2) at dlopen.c:77 77 dlopen.c: No such file or directory. (gdb) c Continuing. Breakpoint 1, ___dlopen (file=0x5555555be4a0 "/usr/lib/tgt/backing-store/bs_rbd.so", mode=mode@entry=2) at dlopen.c:77 77 in dlopen.c (gdb) c Continuing. src/tcmalloc.cc:333] Attempt to free invalid pointer 0x5555555ecae0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1951126 Title: 10.0 is breaking tgt autopkgtest To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/glusterfs/+bug/1951126/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
