Adding Krutika to look at gluster logs On Mon, Jun 18, 2018 at 10:39 AM, <[email protected]> wrote:
> from glusterfs/rhev_datacenter > [2018-06-18 12:32:50.854668] W [socket.c:593:__socket_rwv] 0-glusterfs: > readv on 172.16.224.10:24007 failed (No data available) > [2018-06-18 12:33:38.194322] C > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] > 0-engine-client-0: server 172.16.224.10:49152 has not responded in the > last 30 seconds, disconnecting. > [2018-06-18 12:33:38.195797] E [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> > /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) > op(STATFS(14)) called at 2018-06-18 12:33:07.800145 (xid=0x21be) > [2018-06-18 12:33:38.195824] W [MSGID: 114031] > [client-rpc-fops.c:777:client3_3_statfs_cbk] > 0-engine-client-0: remote operation failed [Transport endpoint is not > connected] > [2018-06-18 12:33:38.196036] E [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> > /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GF-DUMP) op(NULL(2)) called > at 2018-06-18 12:33:07.800158 (xid=0x21bf) > [2018-06-18 12:33:38.196057] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] > 0-engine-client-0: socket disconnected > [2018-06-18 12:33:38.196224] E [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> > /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) > called at 2018-06-18 12:33:08.307804 (xid=0x21c0) > [2018-06-18 12:33:38.196243] W [MSGID: 114031] > [client-rpc-fops.c:2922:client3_3_readv_cbk] > 0-engine-client-0: remote operation failed [Transport endpoint is not > connected] > [2018-06-18 12:33:38.196279] W [MSGID: 114061] > [client-common.c:704:client_pre_fstat] > 0-engine-client-0: (6f81ab66-8991-45a2-9576-5c7defdec302) remote_fd is > -1. EBADFD [File descriptor in bad state] > [2018-06-18 12:33:38.196545] E [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> > /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) > called at 2018-06-18 12:33:08.307812 (xid=0x21c4) > > last part get repeated a lot of times. > > 2018-06-18 12:33:38.199480] W [MSGID: 114061] > [client-common.c:704:client_pre_fstat] > 0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is > -1. EBADFD [File descriptor in bad state] > [2018-06-18 12:33:38.199721] E [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> > /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) > called at 2018-06-18 12:33:16.836613 (xid=0x21cb) > > this part also get repeated sometimes > > > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) > called at 2018-06-18 12:33:37.337567 (xid=0x21db) > The message "W [MSGID: 114031] [client-rpc-fops.c:2922:client3_3_readv_cbk] > 0-engine-client-0: remote operation failed [Transport endpoint is not > connected]" repeated 8 times between [2018-06-18 12:33:38.202531] and > [2018-06-18 12:33:38.205070] > [2018-06-18 12:33:38.205097] W [MSGID: 114061] > [client-common.c:704:client_pre_fstat] > 0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is > -1. EBADFD [File descriptor in bad state] > [2018-06-18 12:33:38.205461] E [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb89f2c0efb] (--> > /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb89f085e6e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb89f085f8e] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fb89f087710] > (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fb89f088200] ))))) > 0-engine-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) > called at 2018-06-18 12:33:37.337596 (xid=0x21dc) > > [2018-06-18 12:33:38.208554] W [MSGID: 114031] > [client-rpc-fops.c:2922:client3_3_readv_cbk] > 0-engine-client-0: remote operation failed [Transport endpoint is not > connected] > [2018-06-18 12:33:38.208590] W [MSGID: 114061] > [client-common.c:704:client_pre_fstat] > 0-engine-client-0: (2bddf734-3baa-439f-90e9-93b483260e01) remote_fd is > -1. EBADFD [File descriptor in bad state] > [2018-06-18 12:34:07.234501] E [socket.c:2369:socket_connect_finish] > 0-engine-client-0: connection to 172.16.224.10:24007 failed (No route to > host); disconnecting socket > [2018-06-18 12:39:25.760451] E [MSGID: 114058] > [client-handshake.c:1565:client_query_portmap_cbk] > 0-engine-client-0: failed to get the port number for remote subvolume. > Please run 'gluster volume status' on server to see if brick process is > running. > ---------------------------------------- > that's what's on gluster log on the host that engine was running . (Do you > use some external site to embeed logs ? sorry I'm new on the list) > > agent.log is prety big so I 'll try to find some pastebin like site to > link it > _______________________________________________ > Users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community- > guidelines/ > List Archives: https://lists.ovirt.org/archives/list/[email protected]/ > message/QKNGD7TZT2UVAFG6H6HCPDIJQ6ROAX74/ >
_______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/F5P3ELPTG5Y2BHMPOUGXOHXL5JDADSGN/
