You can configure tunnel bandwidth everywhere, but you can’t configure a given tunnel everywhere, you have to assign it to a particular FPC/PIC/0.
For example, with: set chassis fps 2 pic 3 tunnel-services bandwidth 10g You need to create gr-2/3/0 interfaces for tunnels to use that PFE. You can create multiple tunnel-services bandwidth entries on multiple PICs, but you can only put a given tunnel on one gr-x/y/0 interface. Owen > On Oct 2, 2023, at 23:21, Saku Ytti <s...@ytti.fi> wrote: > > On Mon, 2 Oct 2023 at 20:21, Jeff Behrns via NANOG <nanog@nanog.org> wrote: > >> Encountered an issue with an MX204 using all 4x100G ports and a logical >> tunnel to hairpin a VRF. The tunnel started dropping packets around 8Gbps. >> I bumped up tunnel-services BW from 10G to 100G which made the problem >> worse; the tunnel was now limited to around 1.3Gbps. To my knowledge with >> Trio PFE you shouldn't have to disable a physical port to allocate bandwidth >> for tunnel-services. Any helpful info is appreciated. > > You might have more luck in j-nsp. > > But yes you don't need any physical interface in trio to do tunneling. > I can't explain your problem, and you probably need JTAC help. I would > appreciate it if you'd circle back and tell what the problem was. > > How it works is that when PPE decides it needs to tunnel the packet, > you're going to send the packet back to MQ via SERDES (which will then > send it again to some PPE, not the same). I think what that bandwidth > command does is change the stream allocation, you should see it in > 'show <MQ/XM...> <#> stream'. > > In theory, because PPE can process packet forever (well, until > watchdog kills the PPE for thinking it is stuck) you could very > cheaply do outer+inner at the local PPE, but I think that would mean > that certain features like QoS would not work on the inner interface, > so I think all this expensive recirculation and SERDES consumption is > to satisfy quite limited need, and it should be possible to implement > some 'performance mode' for tunneling, where these MQ/XM provided > features are not available, but performance cost in most cases is > negligible. > > In parallel to opening the JTAC case, you might want to try to > experiment in which FPC/PIC you set the tunneling bandwidth to. I > don't understand how the tunneling would work if the MQ/XM is remote, > like would you then also steal fabric capacity every time you tunnel, > not just MQ>LU>MQ>LU SERDES, but MQ>LU>MQ>FAB>MQ>LU. So intuitively I > would recommend ensuring you have the bandwidth configured at the > local PFE, if you don't know what the local PFE is, just configure it > everywhere? > Also you could consult several counters to see if some stream or > fabric is congested, and these tunneled packets are being sent over > congested fabric every time with lower fabric qos. > > I don't understand why the bandwidth command is a thing, and why you > can configure where it is. To me it seems obvious they should always > handle tunneling strictly locally, never over fabric, because you > always end up stealing more capacity if you send it to remote MQ. That > is, implicitly it should be on for every MQ, and every PPE tunnel via > local MQ. > > -- > ++ytti
