On Tue, Jul 29, 2025 at 07:05:09PM -0400, Greg Troxel wrote: > While running a test to make sure that NetBSD correctly does what I told > Joel it would do :-) I noticed multiple anomalies, but I'm not sure if > they are actually wrong. > > In this case, the topology is: > - lan-client: normal netbsd 10 machine on Ethernet > - router: netbsd 9 Ethernet, and gif0 to he, mtu 1280 (but he configured to > 1480) > (another ethernet to a fiber ONT) > - he endpoint etc. > - vps: netbsd 9 domU at a xen-based hosting service > > On router: I removed the "mtu 1480" dynamic routes. > On lan-client, I ran "wget https:://vps/foo/bar".
Note that our gif(4) fragments IPv6 packets on output. What are the relevant sysctl variables? On what machine was the trace recorded? You never say, but that is important. Also, it's always a good idea to record traces on both ends if that is possible. You might want to decode with -ttt to get relative time stamps. The absolute times rarely matter and it saves a lot of calculating. > 18:29:00.168831 IP6 lan-client.61111 > vps.https: Flags [S], seq 583236747, > win 65535, options [mss 1440,nop,wscale 3,sackOK,TS val 1 ecr 0], length 0 > 18:29:00.168878 IP6 vps.https > lan-client.61111: Flags [S.], seq > 942780486, ack 583236748, win 65535, options [mss 1440,nop,wscale 3,sackOK,TS > val 1 ecr 1], length 0 > 18:29:00.252215 IP6 lan-client.61111 > vps.https: Flags [.], ack 1, win > 16560, options [nop,nop,TS val 1 ecr 1], length 0 > > normal open > > 18:29:00.252229 IP6 lan-client.61111 > vps.https: Flags [P.], seq 1:412, > ack 1, win 16560, options [nop,nop,TS val 1 ecr 1], length 411 > > client part of TLS > > 18:29:00.252332 IP6 vps.https > lan-client.61111: Flags [.], ack 1, win > 16560, options [nop,nop,TS val 2 ecr 1], length 0 > > really good question what this ack is for; doesn't cover 412. Probably > crossing in the mail and sent before the previous client TLS packet is > procesed. But it's sent 83.454 after after the SYNACK from vps and 0.103ms after the last client packet was received. Someone has to meditate over the code. > QUESTION: is this some scheme to send ack pairs to measure link capacity? I don't think we have such code. > 18:29:00.253813 IP6 vps.https > lan-client.61111: Flags [.], seq 1:1429, > ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 1428 > 18:29:00.253815 IP6 vps.https > lan-client.61111: Flags [.], seq 1429:2857, > ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 1428 > 18:29:00.253816 IP6 vps.https > lan-client.61111: Flags [P.], seq > 2857:3149, ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 292 > > vps sends packets #1A, 2A, and 3A, pile of certs. Only 3A fits in the > tunnel. > > 18:29:00.315724 IP6 tserv1.nyc4.he.net > vps: ICMP6, packet too big, mtu > 1480, length 1240 > > he's tunnel endpoint objects to 1A, arriving ~62 ms later. > > 18:29:00.315836 IP6 vps.https > lan-client.61111: Flags [.], seq 1:1409, > ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 1408 > 18:29:00.315838 IP6 vps.https > lan-client.61111: Flags [.], seq 1409:2817, > ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 1408 > 18:29:00.315840 IP6 vps.https > lan-client.61111: Flags [P.], seq > 2817:3149, ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 332 > > vps immediately resends the certs, sliced into 3 packets differently as > 1B, 2B, and 3B. > > 18:29:00.316729 IP6 tserv1.nyc4.he.net > vps: ICMP6, packet too big, mtu > 1480, length 1240 > > he's tunnel endpoint objects to 1B. > > 18:29:00.316760 IP6 vps.https > lan-client.61111: Flags [.], seq 1:1409, > ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 1408 > 18:29:00.316762 IP6 vps.https > lan-client.61111: Flags [.], seq 1409:2817, > ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 1408 > 18:29:00.316763 IP6 vps.https > lan-client.61111: Flags [P.], seq > 2817:3149, ack 412, win 16560, options [nop,nop,TS val 2 ecr 1], length 332 > > vps immediately resends the certs again, with the same split. > > QUESTION: is this correct, or should it have somehow recorded that it > already did this? Seems correct to me: https://nxr.netbsd.org/xref/src/sys/netinet/tcp_subr.c#1585 > 18:29:00.326324 IP6 lan-client.61111 > vps.https: Flags [.], ack 1, win > 16560, options [nop,nop,TS val 2 ecr 2,nop,nop,sack 1 {2857:3149}], length 0 > > ack arrives for packet #3 as SACK, ack point still 1 (the SYN). > > 18:29:00.394163 IP6 lan-client.61111 > vps.https: Flags [.], ack 1409, win > 16384, options [nop,nop,TS val 2 ecr 2,nop,nop,sack 1 {2857:3149}], length 0 > > ack for 1B, and SACKing 3A > > 18:29:00.394165 IP6 lan-client.61111 > vps.https: Flags [.], ack 2817, win > 16208, options [nop,nop,TS val 2 ecr 2,nop,nop,sack 1 {2857:3149}], length 0 > > ack for 2B, and SACKING 3A. Not acking 2817-2857. > > 18:29:00.394166 IP6 lan-client.61111 > vps.https: Flags [.], ack 3149, win > 16166, options [nop,nop,TS val 2 ecr 2], length 0 > > full ack for 3B, not SACK. > > 18:29:00.394166 IP6 lan-client.61111 > vps.https: Flags [.], ack 3149, win > 16515, options [nop,nop,TS val 2 ecr 2], length 0 > > second ack for 3B, not SACK. Yes, the time is the same but tcpdump > really has two lines. > > QUESTION: why? You really have to look at the trace from the client. Maybe this is an ACK for the retransmitted 3B. > 18:29:00.394167 IP6 lan-client.61111 > vps.https: Flags [.], ack 3149, win > 16515, options [nop,nop,TS val 2 ecr 2,nop,nop,sack 1 {1:1409}], length 0 > > dupack for 1B 2nd transmit No, this is an SACK for 1B but an ACK for 1-3149. This doesn't make sense to me. Why send an SACK for a packet that we already have ACKed? The next two are the same. > 18:29:00.394167 IP6 lan-client.61111 > vps.https: Flags [.], ack 3149, win > 16515, options [nop,nop,TS val 2 ecr 2,nop,nop,sack 1 {1409:2817}], length 0 > > dupack for 2B 2nd transmit > > 18:29:00.394168 IP6 lan-client.61111 > vps.https: Flags [.], ack 3149, win > 16515, options [nop,nop,TS val 2 ecr 2,nop,nop,sack 1 {2817:3149}], length 0 > > dupack for 3B 2nd transmit > > from here on, looks mostly normal with full acks, no SACK. > > 18:29:00.394168 IP6 lan-client.61111 > vps.https: Flags [P.], seq 412:492, > ack 3149, win 16560, options [nop,nop,TS val 2 ecr 2], length 80 > 18:29:00.394452 IP6 vps.https > lan-client.61111: Flags [P.], seq > 3149:3436, ack 492, win 16560, options [nop,nop,TS val 2 ecr 2], length 287 > 18:29:00.394509 IP6 vps.https > lan-client.61111: Flags [P.], seq > 3436:3723, ack 492, win 16560, options [nop,nop,TS val 2 ecr 2], length 287 > 18:29:00.468791 IP6 lan-client.61111 > vps.https: Flags [P.], seq 492:650, > ack 3436, win 16524, options [nop,nop,TS val 2 ecr 2], length 158 > 18:29:00.470602 IP6 vps.https > lan-client.61111: Flags [.], seq 3723:5131, > ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.470604 IP6 vps.https > lan-client.61111: Flags [.], seq 5131:6539, > ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.470606 IP6 vps.https > lan-client.61111: Flags [.], seq 6539:7947, > ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.470609 IP6 vps.https > lan-client.61111: Flags [.], seq 7947:9355, > ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.470611 IP6 vps.https > lan-client.61111: Flags [.], seq > 9355:10763, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.550668 IP6 lan-client.61111 > vps.https: Flags [.], ack 5131, win > 16384, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.550756 IP6 lan-client.61111 > vps.https: Flags [.], ack 7947, win > 16032, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.550757 IP6 lan-client.61111 > vps.https: Flags [.], ack 7947, win > 16560, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.550757 IP6 lan-client.61111 > vps.https: Flags [.], ack 10763, win > 16208, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.550758 IP6 lan-client.61111 > vps.https: Flags [.], ack 10763, win > 16560, options [nop,nop,TS val 2 ecr 2], length 0 > > except there are two acks for 7947 and two for 10763. > > QUESTION: why and is this correct? Could there be reordering? Well, you really have to look at the trace taken on the client to see what's going on here. Note that the ACKs are received 80.147ms after the 7947:9355 was sent. > 18:29:00.550760 IP6 vps.https > lan-client.61111: Flags [.], seq > 10763:12171, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.550762 IP6 vps.https > lan-client.61111: Flags [.], seq > 12171:13579, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.550791 IP6 vps.https > lan-client.61111: Flags [.], seq > 13579:14987, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.550793 IP6 vps.https > lan-client.61111: Flags [.], seq > 14987:16395, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.550795 IP6 vps.https > lan-client.61111: Flags [.], seq > 16395:17803, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 1408 > 18:29:00.550796 IP6 vps.https > lan-client.61111: Flags [P.], seq > 17803:18082, ack 650, win 16560, options [nop,nop,TS val 2 ecr 2], length 279 > 18:29:00.626999 IP6 lan-client.61111 > vps.https: Flags [.], ack 13579, win > 16208, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.627001 IP6 lan-client.61111 > vps.https: Flags [.], ack 13579, win > 16560, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.630529 IP6 lan-client.61111 > vps.https: Flags [.], ack 16395, win > 16208, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.630530 IP6 lan-client.61111 > vps.https: Flags [.], ack 18082, win > 15997, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.630532 IP6 lan-client.61111 > vps.https: Flags [.], ack 18082, win > 16560, options [nop,nop,TS val 2 ecr 2], length 0 > > Again extra acks Hmm. These extra acks are send every 2nd received packet. They don't seem to cause retransmissions, though. > 18:29:00.630532 IP6 lan-client.61111 > vps.https: Flags [F.], seq 650, ack > 18082, win 16560, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.630563 IP6 vps.https > lan-client.61111: Flags [.], ack 651, win > 16560, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.630665 IP6 vps.https > lan-client.61111: Flags [F.], seq 18082, > ack 651, win 16560, options [nop,nop,TS val 2 ecr 2], length 0 > 18:29:00.698763 IP6 lan-client.61111 > vps.https: Flags [.], ack 18083, win > 16560, options [nop,nop,TS val 2 ecr 2], length 0 > > normal close (where vps acks the FIN before the server shuts down its > half); seems ok. --chris
