I don't think it is similar to Xavi fixed in https://review.gluster.org/#/c/glusterfs/+/24099/. Is it possible to share the output "thread apply all bt full" after attaching the core with gdb?
Regards, Mohit Agrawal On Sat, Feb 15, 2020 at 7:25 AM Amar Tumballi <[email protected]> wrote: > Is this crash seen already ? Does > https://review.gluster.org/#/c/glusterfs/+/24099/ fix this? > > On Sat, Feb 15, 2020 at 4:32 AM Joe Julian <[email protected]> wrote: > >> These crashes have been happening almost daily. Any thoughts on how to >> stabilize this? >> >> >> [2020-02-14 19:02:13.932178] I [MSGID: 100030] [glusterfsd.c:2865:main] >> 0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 7.0 >> (args: /usr/bin/glusterfs --process-name fuse --volfile-server=gluster >> --volfile-id=kube /tmp/hostpath_pv) >> [2020-02-14 19:02:13.939208] I [glusterfsd.c:2593:daemonize] >> 0-glusterfs: Pid of current running process is 1828083 >> [2020-02-14 19:02:13.970829] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 0 >> [2020-02-14 19:02:13.970870] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 1 >> [2020-02-14 19:02:14.026097] W [MSGID: 108003] >> [afr.c:98:fix_quorum_options] 0-kube-replicate-0: quorum-type none >> overriding quorum-count 1 >> [2020-02-14 19:02:14.027753] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 2 >> [2020-02-14 19:02:14.027871] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 4 >> [2020-02-14 19:02:14.027939] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 5 >> [2020-02-14 19:02:14.027984] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 3 >> [2020-02-14 19:02:14.028010] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 6 >> [2020-02-14 19:02:14.028410] I [MSGID: 101190] >> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 7 >> [2020-02-14 19:02:14.028826] I [MSGID: 114020] [client.c:2434:notify] >> 0-kube-client-0: parent translators are ready, attempting connect on >> transport >> [2020-02-14 19:02:14.031361] I [MSGID: 114020] [client.c:2434:notify] >> 0-kube-client-1: parent translators are ready, attempting connect on >> transport >> [2020-02-14 19:02:14.031840] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >> 0-kube-client-0: changing port to 49159 (from 0) >> [2020-02-14 19:02:14.031880] I [socket.c:863:__socket_shutdown] >> 0-kube-client-0: intentional socket shutdown(12) >> Final graph: >> >> +------------------------------------------------------------------------------+ >> 1: volume kube-client-0 >> 2: type protocol/client >> 3: option ping-timeout 42 >> 4: option remote-host strabo >> 5: option remote-subvolume /data/gluster/kube >> 6: option transport-type socket >> 7: option transport.address-family inet >> 8: option username 51b95740-bff7-4744-bbfc-dbae17670997 >> 9: option password 021da395-9404-43ac-b358-2f5dfc5acc71 >> 10: option transport.socket.ssl-enabled off >> 11: option filter-O_DIRECT on >> 12: option event-threads 8 >> 13: option transport.tcp-user-timeout 0 >> 14: option transport.socket.keepalive-time 20 >> 15: option transport.socket.keepalive-interval 2 >> 16: option transport.socket.keepalive-count 9 >> 17: option send-gids true >> 18: end-volume >> 19: >> 20: volume kube-client-1 >> 21: type protocol/client >> 22: option ping-timeout 42 >> 23: option remote-host nightshade >> 24: option remote-subvolume /data/gluster/kube >> 25: option transport-type socket >> 26: option transport.address-family inet >> 27: option username 51b95740-bff7-4744-bbfc-dbae17670997 >> 28: option password 021da395-9404-43ac-b358-2f5dfc5acc71 >> 29: option transport.socket.ssl-enabled off >> 30: option filter-O_DIRECT on >> 31: option event-threads 8 >> 32: option transport.tcp-user-timeout 0 >> 33: option transport.socket.keepalive-time 20 >> 34: option transport.socket.keepalive-interval 2 >> 35: option transport.socket.keepalive-count 9 >> 36: option send-gids true >> 37: end-volume >> 38: >> 39: volume kube-replicate-0 >> 40: type cluster/replicate >> 41: option afr-pending-xattr kube-client-0,kube-client-1 >> 42: option quorum-count 1 >> 43: option use-compound-fops off >> 44: subvolumes kube-client-0 kube-client-1 >> 45: end-volume >> 46: >> 47: volume kube-dht >> 48: type cluster/distribute >> 49: option readdir-optimize on >> 50: option lock-migration off >> 51: option force-migration off >> 52: subvolumes kube-replicate-0 >> 53: end-volume >> 54: >> 55: volume kube-io-cache >> 56: type performance/io-cache >> 57: option cache-timeout 30 >> 58: subvolumes kube-dht >> 59: end-volume >> 60: >> 61: volume kube-open-behind >> 62: type performance/open-behind >> 63: subvolumes kube-io-cache >> 64: end-volume >> 65: >> 66: volume kube-md-cache >> 67: type performance/md-cache >> 68: option cache-invalidation on >> 69: subvolumes kube-open-behind >> 70: end-volume >> 71: >> 72: volume kube-io-threads >> 73: type performance/io-threads >> 74: subvolumes kube-md-cache >> 75: end-volume >> 76: >> 77: volume kube >> 78: type debug/io-stats >> 79: option log-level INFO >> 80: option threads 16 >> 81: option latency-measurement off >> 82: option count-fop-hits off >> 83: option global-threading off >> 84: subvolumes kube-io-threads >> 85: end-volume >> 86: >> 87: volume meta-autoload >> 88: type meta >> 89: subvolumes kube >> 90: end-volume >> 91: >> >> +------------------------------------------------------------------------------+ >> [2020-02-14 19:02:14.052685] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >> 0-kube-client-1: changing port to 49159 (from 0) >> [2020-02-14 19:02:14.052792] I [socket.c:863:__socket_shutdown] >> 0-kube-client-1: intentional socket shutdown(13) >> [2020-02-14 19:02:14.056240] I [MSGID: 114057] >> [client-handshake.c:1373:select_server_supported_programs] >> 0-kube-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version >> (400) >> [2020-02-14 19:02:14.076058] I [MSGID: 114046] >> [client-handshake.c:1104:client_setvolume_cbk] 0-kube-client-1: >> Connected to kube-client-1, attached to remote volume >> '/data/gluster/kube'. >> [2020-02-14 19:02:14.076113] I [MSGID: 108005] >> [afr-common.c:5277:__afr_handle_child_up_event] 0-kube-replicate-0: >> Subvolume 'kube-client-1' came back up; going online. >> [2020-02-14 19:02:24.031553] I [fuse-bridge.c:5162:fuse_init] >> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 >> kernel 7.31 >> [2020-02-14 19:02:24.031605] I [fuse-bridge.c:5777:fuse_graph_sync] >> 0-fuse: switched to graph 0 >> [2020-02-14 19:04:27.205776] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >> 0-kube-client-0: changing port to 49159 (from 0) >> [2020-02-14 19:06:40.325688] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >> 0-kube-client-0: changing port to 49159 (from 0) >> [2020-02-14 19:08:53.444994] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >> 0-kube-client-0: changing port to 49159 (from 0) >> [2020-02-14 19:11:06.565527] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >> 0-kube-client-0: changing port to 49159 (from 0) >> pending frames: >> frame : type(0) op(0) >> frame : type(0) op(0) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(FLUSH) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(FLUSH) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> frame : type(1) op(LOOKUP) >> patchset: git://git.gluster.org/glusterfs.git >> signal received: 11 >> time of crash: >> 2020-02-14 19:13:00 >> configuration details: >> argp 1 >> backtrace 1 >> dlfcn 1 >> libpthread 1 >> llistxattr 1 >> setfsid 1 >> spinlock 1 >> epoll.h 1 >> xattr.h 1 >> st_atim.tv_nsec 1 >> package-string: glusterfs 7.0 >> /usr/lib/libglusterfs.so.0(+0x25e12)[0x7fc026b55e12] >> /usr/lib/libglusterfs.so.0(gf_print_trace+0x354)[0x7fc026b60b74] >> /usr/lib/libc.so.6(+0x3bfb0)[0x7fc0268e3fb0] >> /usr/lib/libglusterfs.so.0(mem_get+0xf0)[0x7fc026b81850] >> /usr/lib/libglusterfs.so.0(+0x1a4cd)[0x7fc026b4a4cd] >> /usr/lib/libglusterfs.so.0(dict_setn+0x1f1)[0x7fc026b4a941] >> /usr/lib/libglusterfs.so.0(dict_set_dynptr+0x32)[0x7fc026b4e342] >> /usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x3b318)[0x7fc0210cb318] >> /usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x56d03)[0x7fc0210e6d03] >> /usr/lib/libgfrpc.so.0(+0xe957)[0x7fc026afe957] >> /usr/lib/libgfrpc.so.0(+0xecc9)[0x7fc026afecc9] >> /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fc026afb816] >> /usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0x4629)[0x7fc022294629] >> /usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0xc45c)[0x7fc02229c45c] >> /usr/lib/libglusterfs.so.0(+0x8cedc)[0x7fc026bbcedc] >> /usr/lib/libpthread.so.0(+0x94cf)[0x7fc026a794cf] >> /usr/lib/libc.so.6(clone+0x43)[0x7fc0269a72d3] >> --------- >> >> ________ >> >> Community Meeting Calendar: >> >> APAC Schedule - >> Every 2nd and 4th Tuesday at 11:30 AM IST >> Bridge: https://bluejeans.com/441850968 >> >> NA/EMEA Schedule - >> Every 1st and 3rd Tuesday at 01:00 PM EDT >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> [email protected] >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > > > -- > -- > https://kadalu.io > Container Storage made easy! > >
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
