That's unfortunately not possible in my case as the clients are K8s worker nodes. The gluster client version is tied to the Kuberntes release/deployer - Rancher in my case.
Disabling ctime "gluster volume set $volname ctime off" fixed the mounting issue. On Thu, Jan 30, 2020 at 4:16 AM Mahdi Adnan <[email protected]> wrote: > Hello, > > We had a similar issue when we upgraded one of our clusters to 6.5 and > clients were running 4.1.5 and 4.1.9, both crashed after few seconds of > mounting, we did not dig into the issue instead, we upgraded the clients to > 6.5 and it worked fine. > > On Tue, Jan 28, 2020 at 1:35 AM Laurent Dumont <[email protected]> > wrote: > >> After some more digging, it seems I'm hitting the following bug - >> https://github.com/gluster/glusterfs/issues/658 >> >> On Mon, Jan 27, 2020 at 4:23 PM Laurent Dumont <[email protected]> >> wrote: >> >>> Hi everyone, >>> >>> Small question. I'm trying to mount a Gluster volume (server is at 6.7) >>> and the client is at 4.1.5. I'm seeing the mount start on the client but it >>> looks like the client crashes and is left in a strange state. Is there any >>> inherent compatibility issues between the two versions? Is 4.1.5 too old >>> when talking to a 6.7 server? >>> >>> Client : >>> sh-4.4# glusterfs --version >>> glusterfs 4.1.5 >>> Repository revision: git://git.gluster.org/glusterfs.git >>> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >>> GlusterFS comes with ABSOLUTELY NO WARRANTY. >>> It is licensed to you under your choice of the GNU Lesser >>> General Public License, version 3 or any later version (LGPLv3 >>> or later), or the GNU General Public License, version 2 (GPLv2), >>> in all cases as published by the Free Software Foundation. >>> >>> Server : >>> root@gluster01:~# glusterd --version >>> glusterfs 6.7 >>> Repository revision: git://git.gluster.org/glusterfs.git >>> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> >>> GlusterFS comes with ABSOLUTELY NO WARRANTY. >>> It is licensed to you under your choice of the GNU Lesser >>> General Public License, version 3 or any later version (LGPLv3 >>> or later), or the GNU General Public License, version 2 (GPLv2), >>> in all cases as published by the Free Software Foundation. >>> >>> Steps : >>> sh-4.4# mount -t glusterfs 10.10.99.29:kube_vol /test >>> Mount failed. Please check the log file for more details. >>> >>> >>> sh-4.4# cat /var/log/glusterfs/test.log >>> [2020-01-27 21:03:26.770751] I [MSGID: 100030] [glusterfsd.c:2741:main] >>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.5 >>> (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=10.10.99.29 >>> --volfile-id=kube_vol /test) >>> [2020-01-27 21:03:26.775717] I [MSGID: 101190] >>> [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread >>> with index 1 >>> pending frames: >>> frame : type(0) op(0) >>> patchset: git://git.gluster.org/glusterfs.git >>> signal received: 11 >>> time of crash: >>> 2020-01-27 21:03:26 >>> configuration details: >>> argp 1 >>> backtrace 1 >>> dlfcn 1 >>> libpthread 1 >>> llistxattr 1 >>> setfsid 1 >>> spinlock 1 >>> epoll.h 1 >>> xattr.h 1 >>> st_atim.tv_nsec 1 >>> package-string: glusterfs 4.1.5 >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x237aa)[0x7f2e1061d7aa] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7f2e10627517] >>> /lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7f2e0ec6f060] >>> /lib/x86_64-linux-gnu/libc.so.6(+0xbf944)[0x7f2e0ecfb944] >>> /lib/x86_64-linux-gnu/libc.so.6(fnmatch+0x61)[0x7f2e0ecfce41] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_volume_option_get_list+0x35)[0x7f2e10670905] >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7698e)[0x7f2e1067098e] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach_match+0x87)[0x7f2e10614b97] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach+0x18)[0x7f2e10614d78] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate_list+0x3f)[0x7f2e10670b4f] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate+0x39)[0x7f2e10670bc9] >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5ab99)[0x7f2e10654b99] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_activate+0x24)[0x7f2e106554d4] >>> /usr/sbin/glusterfs(glusterfs_process_volfp+0x100)[0x5643b3bc1f40] >>> /usr/sbin/glusterfs(mgmt_getspec_cbk+0x6bd)[0x5643b3bc889d] >>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe110)[0x7f2e103ec110] >>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe44f)[0x7f2e103ec44f] >>> >>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f2e103e8863] >>> >>> /usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x6cfb)[0x7f2e0b4e9cfb] >>> >>> /usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x9605)[0x7f2e0b4ec605] >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8052e)[0x7f2e1067a52e] >>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f2e0f47c4a4] >>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f2e0ed24d0f] >>> --------- >>> [2020-01-27 21:22:35.074154] I [MSGID: 100030] [glusterfsd.c:2741:main] >>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.5 >>> (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=10.10.99.29 >>> --volfile-id=kube_vol /test) >>> [2020-01-27 21:22:35.078542] I [MSGID: 101190] >>> [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread >>> with index 1 >>> pending frames: >>> frame : type(0) op(0) >>> patchset: git://git.gluster.org/glusterfs.git >>> signal received: 11 >>> time of crash: >>> 2020-01-27 21:22:35 >>> configuration details: >>> argp 1 >>> backtrace 1 >>> dlfcn 1 >>> libpthread 1 >>> llistxattr 1 >>> setfsid 1 >>> spinlock 1 >>> epoll.h 1 >>> xattr.h 1 >>> st_atim.tv_nsec 1 >>> package-string: glusterfs 4.1.5 >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x237aa)[0x7f6ef309a7aa] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7f6ef30a4517] >>> /lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7f6ef16ec060] >>> /lib/x86_64-linux-gnu/libc.so.6(+0xbf944)[0x7f6ef1778944] >>> /lib/x86_64-linux-gnu/libc.so.6(fnmatch+0x61)[0x7f6ef1779e41] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_volume_option_get_list+0x35)[0x7f6ef30ed905] >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7698e)[0x7f6ef30ed98e] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach_match+0x87)[0x7f6ef3091b97] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach+0x18)[0x7f6ef3091d78] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate_list+0x3f)[0x7f6ef30edb4f] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_options_validate+0x39)[0x7f6ef30edbc9] >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5ab99)[0x7f6ef30d1b99] >>> >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_activate+0x24)[0x7f6ef30d24d4] >>> /usr/sbin/glusterfs(glusterfs_process_volfp+0x100)[0x55b5e83a7f40] >>> /usr/sbin/glusterfs(mgmt_getspec_cbk+0x6bd)[0x55b5e83ae89d] >>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe110)[0x7f6ef2e69110] >>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe44f)[0x7f6ef2e6944f] >>> >>> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f6ef2e65863] >>> >>> /usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x6cfb)[0x7f6eedf66cfb] >>> >>> /usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/rpc-transport/socket.so(+0x9605)[0x7f6eedf69605] >>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8052e)[0x7f6ef30f752e] >>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f6ef1ef94a4] >>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f6ef17a1d0f] >>> --------- >>> >> ________ >> >> Community Meeting Calendar: >> >> APAC Schedule - >> Every 2nd and 4th Tuesday at 11:30 AM IST >> Bridge: https://bluejeans.com/441850968 >> >> NA/EMEA Schedule - >> Every 1st and 3rd Tuesday at 01:00 PM EDT >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> [email protected] >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > > > -- > Respectfully > Mahdi >
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
