oops, missed your sysctl -a output (I wasn't expecting to see it, well done ;-)
net.inet.ip.ifq.drops=140720 You would probably benefit from increasing net.inet.ip.ifq.maxlen, maybe double it once or twice and see if net.inet.ip.ifq.drops stops increasing.
