> On 18. Mar 2021, at 13:53, Rodney W. Grimes <freebsd-...@gndrsh.dnsmgr.net> > wrote: > > Note I am NOT a TCP expert, but know enough about it to add a comment... > >> Alan Somers wrote: >> [stuff snipped] >>> Is the 128K limit related to MAXPHYS? If so, it should be greater in 13.0. >> For the client, yes. For the server, no. >> For the server, it is just a compile time constant NFS_SRVMAXIO. >> >> It's mainly related to the fact that I haven't gotten around to testing >> larger >> sizes yet. >> - kern.ipc.maxsockbuf needs to be several times the limit, which means it >> would >> have to increase for 1Mbyte. >> - The session code must negotiate a maximum RPC size > 1 Mbyte. >> (I think the server code does do this, but it needs to be tested.) >> And, yes, the client is limited to MAXPHYS. >> >> Doing this is on my todo list, rick >> >> The client should acquire the attributes that indicate that and set >> rsize/wsize >> to that. "# nfsstat -m" on the client should show you what the client >> is actually using. If it is larger than 128K, set both rsize and wsize to >> 128K. >> >>> Output from the NFS Client when the issue occurs >>> # netstat -an | grep NFS.Server.IP.X >>> tcp 0 0 NFS.Client.IP.X:46896 NFS.Server.IP.X:2049 >>> FIN_WAIT2 >> I'm no TCP guy. Hopefully others might know why the client would be >> stuck in FIN_WAIT2 (I vaguely recall this means it is waiting for a fin/ack, >> but could be wrong?) > > The most common way to get stuck in FIN_WAIT2 is to call > shutdown(2) on a socket, but never following up with a > close(2) after some timeout period. The "client" is still > connected to the socket and can stay in this shutdown state > for ever, the kernel well not reap the socket as it is > associated with a processes, aka not orphaned. I suspect > that the Linux client has a corner condition that is leading > to this socket leak. > > If on the Linux client you can look at the sockets to see > if these are still associated with a process, ala fstat or > what ever Linux tool does this that would be helpfull. > If they are infact connected to a process it is that > process that must call close(2) to clean these up. > > IIRC the server side socket would be gone at this point > and there is nothing the server can do that would allow > a FIN_WAIT2 to close down. Jason reported that the server is in CLOSE-WAIT. This would mean the the server received the FIN, ACKed it, but has not initiated the teardown of the Server->Client direction. So the server side socket is still there and close has not be called yet. > > The real TCP experts can now correct my 30 year old TCP > stack understanding... I wouldn't count myself as a real TCP expert, but the behaviour hasn't changed in the last 30 years, I think...
Best regards Michael > >> >>> # cat /sys/kernel/debug/sunrpc/rpc_xprt/*/info >>> netid: tcp >>> addr: NFS.Server.IP.X >>> port: 2049 >>> state: 0x51 >>> >>> syslog >>> Mar 4 10:29:27 hostname kernel: [437414.131978] -pid- flgs status -client- >>> --rqstp- ->timeout ---ops-- >>> Mar 4 10:29:27 hostname kernel: [437414.133158] 57419 40a1 0 9b723c73 >>> >143cfadf 30000 4ca953b5 nfsv4 OPEN_NOATTR a:call_connect_status >>> [sunrpc] >q:xprt_pending >> I don't know what OPEN_NOATTR means, but I assume it is some variant >> of NFSv4 Open operation. >> [stuff snipped] >>> Mar 4 10:29:30 hostname kernel: [437417.110517] RPC: 57419 >>> xprt_connect_status: >connect attempt timed out >>> Mar 4 10:29:30 hostname kernel: [437417.112172] RPC: 57419 >>> call_connect_status >>> (status -110) >> I have no idea what status -110 means? >>> Mar 4 10:29:30 hostname kernel: [437417.113337] RPC: 57419 call_timeout >>> (major) >>> Mar 4 10:29:30 hostname kernel: [437417.114385] RPC: 57419 call_bind >>> (status 0) >>> Mar 4 10:29:30 hostname kernel: [437417.115402] RPC: 57419 call_connect >>> xprt >00000000e061831b is not connected >>> Mar 4 10:29:30 hostname kernel: [437417.116547] RPC: 57419 xprt_connect >>> xprt >00000000e061831b is not connected >>> Mar 4 10:30:31 hostname kernel: [437478.551090] RPC: 57419 >>> xprt_connect_status: >connect attempt timed out >>> Mar 4 10:30:31 hostname kernel: [437478.552396] RPC: 57419 >>> call_connect_status >(status -110) >>> Mar 4 10:30:31 hostname kernel: [437478.553417] RPC: 57419 call_timeout >>> (minor) >>> Mar 4 10:30:31 hostname kernel: [437478.554327] RPC: 57419 call_bind >>> (status 0) >>> Mar 4 10:30:31 hostname kernel: [437478.555220] RPC: 57419 call_connect >>> xprt >00000000e061831b is not connected >>> Mar 4 10:30:31 hostname kernel: [437478.556254] RPC: 57419 xprt_connect >>> xprt >00000000e061831b is not connected >> Is it possible that the client is trying to (re)connect using the same >> client port#? >> I would normally expect the client to create a new TCP connection using a >> different client port# and then retry the outstanding RPCs. >> --> Capturing packets when this happens would show us what is going on. >> >> If there is a problem on the FreeBSD end, it is most likely a broken >> network device driver. >> --> Try disabling TSO , LRO. >> --> Try a different driver for the net hardware on the server. >> --> Try a different net chip on the server. >> If you can capture packets when (not after) the hang >> occurs, then you can look at them in wireshark and see >> what is actually happening. (Ideally on both client and >> server, to check that your network hasn't dropped anything.) >> --> I know, if the hangs aren't easily reproducible, this isn't >> easily done. >> --> Try a newer Linux kernel and see if the problem persists. >> The Linux folk will get more interested if you can reproduce >> the problem on 5.12. (Recent bakeathon testing of the 5.12 >> kernel against the FreeBSD server did not find any issues.) >> >> Hopefully the network folk have some insight w.r.t. why >> the TCP connection is sitting in FIN_WAIT2. >> >> rick >> >> >> >> Jason Breitman >> >> >> >> >> >> >> _______________________________________________ >> freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscr...@freebsd.org<mailto:freebsd-net-unsubscr...@freebsd.org>" >> >> _______________________________________________ >> freebsd-net@freebsd.org<mailto:freebsd-net@freebsd.org> mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscr...@freebsd.org<mailto:freebsd-net-unsubscr...@freebsd.org>" >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" >> > > -- > Rod Grimes rgri...@freebsd.org > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
