> On 19 Dec 2024, at 08:20, David Gwynne <da...@gwynne.id.au> wrote:
>
>
>
>> On 19 Dec 2024, at 02:17, Maurice Janssen <maur...@z74.net> wrote:
>>
>> On Wed, Dec 18, 2024 at 11:51:26AM +1000, David Gwynne wrote:
>>> Hey Maurice,
>>>
>>> bluhm@ just did a talk at eurobsdcon that might help you understand the
>>> different points to look at. my tldr version is if packets are being
>>> dropped on the firewall it will be somewhere in the receive path. my
>>> experience is that once a packet has made it into the network stack
>>> it tends to get transmitted fine.
>>
>> Thanks for your reply. This has given me some new things to look at.
>>
>>> the places that packets can get dropped are in:
>>>
>>> 1. hardware
>>>
>>> hardware counters from igc should be visible using kstat(1) now. the
>>> igc-stats that you're looking at would be any of the errs/colls, or the
>>> rx no bufs counter.
>>
>> kstat(1) shows me that there are (nearly) no errors on the rx side, but
>> showed about 470k dropped packets (qdrops) on the tx side for igc2.
>
> wait, what?
do you see oactives going up too?
>
> do you have PCIe ASPM enabled in the bios?
>
>>
>>> 2. rxq drops and errors
>>>
>>> the rxq is where packets received by the driver are put before being
>>> handed over to the network stack. again, these are visible as kstats,
>>> but are also aggregated as interface level counters you can see with
>>> netstat -i. you can look at drops and errors individiually with netstat
>>> -id and netstat -ie respectively.
>>
>> netstat -id shows all zeros in de Idrop column, but the same 470k packets
>> in the Odrop column.
>> Not sure if this can also be caused by unplugging the cable. I did that
>> sereral times during testing/swapping switches on the igc2 port.
>> I will reboot to reset the counters and keep an eye on it.
>>
>>> the fact that you're using pppoe on top of a vlan adds some potential
>>> for loss too. there's a queue in between ethernet input processing and
>>> pppoe processing that could be dropping packets. i don't think we have a
>>> way to see drops on that queue though.
>>>
>>> if you're feeling brave you can try the diff i sent to the 'PPPoE
>>> passthrough with "GigaHub" is very slow' thread on misc@ a week or so
>>> ago which skips the queue for pppoe data packets.
>>
>> I think I can give it a try in a week or two.
>> For what it's worth: I see only 138 qdrops for pppoe0:0:txq:0 in kstat
>> output. And no drops on the rxq, but all counters are at zero so I guess
>> that doesn't say much.
>>
>>> 3. pf drops
>>>
>>> i think systat pf and pfctl -si are the best way to see counters here.
>>> however, i feel like there's some things that pf does and doesn't do
>>> sometimes that aren't captured well.
>>>
>>> 4. tx drops
>>>
>>> again, i think this is unlikely, but you should be able to see tx
>>> interface drops using the txq kstats or drops/errors in netstat -i
>>> output. there's tx counters in the igc-stats too.
>>
>> Furthermore, netstat -ss -p udp shows me:
>> udp:
>> 295407643 datagrams received
>> 1 with bad data length field
>> 2100 with bad checksum
>> 592550 with no checksum
>> 289009064 input packets software-checksummed
>> 318512247 output packets software-checksummed
>> 25112 dropped due to no socket
>> 239806 broadcast/multicast datagrams dropped due to no socket
>> 431246 dropped due to full socket buffers
>> 294709378 delivered
>> 348403092 datagrams output
>> 295132792 missed PCB cache
>>
>> But I don't know if there dropped packets are also on igc2.
>>
>> Thanks again and best regards,
>> Maurice
