Re: [vpp-dev] classifier for policy based forwarding
Argh, I had written up a response but accidentally sent it to only Sandeep We've also hit a snag with this. We are attempting to send TCP traffic upwards to FRR for the purpose of using BGP but have hit this snag. We need BGP to not only update VPPs FIB but also to receive updates over BGP via VPPs link nets from neighbouring routers. If the group has a better suggestion for this, please do tell. We can't seem to find a clear point when this was changed, or if it's a current bug. This part of the issue seems to fall under vpp-dev and not vppsb-dev. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10754): https://lists.fd.io/g/vpp-dev/message/10754 Mute This Topic: https://lists.fd.io/mt/24602615/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability
Hello, We've got a NAT44 setup configured with separate local and outside linknets, where the internal NAT pool resides on a different subnet and is translated to an outside different subnet. When sending, in this case ICMP packets of size 1500 bytes, to 8.8.8.8 we get no response back from the target. However, if we decrease size to 1472, it works. 1500 is including headers (total packet size is 1500). When sending packets of size 1500 towards 8.8.8.8, the initial returning packet is 1468 and marked as fragmented with more fragments to come. When sending packets of size 1472 towards 8.8.8.8 all the returning packets are of size 1472 The result of sending packets of size 1500 towards 8.8.8.8 is that we get them back fragmented, and for some reason, not received correctly by the internal device, whether not passed through out2in properly or not reassembled properly. Please feel free to attempt reproducing the above and let me know if you need more info. Any thoughts? -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10864): https://lists.fd.io/g/vpp-dev/message/10864 Mute This Topic: https://lists.fd.io/mt/27400627/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability
Hi Matus, Much appreciated, glad you found it! Thanks, John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Friday, October 19, 2018 8:00 AM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, There is bug in NAT code for ICMP fragments. I will fix it as soon as possible. Thanks, Matus From: vpp-dev@lists.fd.io On Behalf Of JB Sent: Thursday, October 18, 2018 12:48 PM To: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Ole, I've added a relevant packet trace. Is that adequate? Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10875): https://lists.fd.io/g/vpp-dev/message/10875 Mute This Topic: https://lists.fd.io/mt/27400627/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability
Hi Matus, Brilliant, thanks! John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Friday, October 19, 2018 2:27 PM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, Fix in master and stable/1810 branch Matus From: vpp-dev@lists.fd.io On Behalf Of JB Sent: Friday, October 19, 2018 9:14 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Much appreciated, glad you found it! Thanks, John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, October 19, 2018 8:00 AM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, There is bug in NAT code for ICMP fragments. I will fix it as soon as possible. Thanks, Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Thursday, October 18, 2018 12:48 PM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Ole, I've added a relevant packet trace. Is that adequate? Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10879): https://lists.fd.io/g/vpp-dev/message/10879 Mute This Topic: https://lists.fd.io/mt/27400627/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] VPP crash after days of heavy traffic, next_index is not associated to an interface #vpp_stability
Hello, I've encountered a crash I've not seen posted anywhere. We have a test setup running NAT and on average around 5 Gbps/500 kpps of traffic. After 24-48 hours, VPP crashes with the following error: vnet[18358]: next_index_to_iface:101: next_index not associated to an interface! vnet[18358]: received signal SIGSEGV, PC 0x7faf14fe8608, faulting address 0x8 vnet[18358]: #0 0x7faf1565718f 0x7faf1565718f vnet[18358]: #1 0x7faf1455e390 0x7faf1455e390 vnet[18358]: #2 0x7faf14fe8608 0x7faf14fe8608 vnet[18358]: #3 0x7faf1561d374 0x7faf1561d374 vnet[18358]: #4 0x7faf1561e1ff vlib_worker_loop + 0x53f vnet[18358]: #5 0x7faf14794838 0x7faf14794838 systemd[1]: vpp.service: Main process exited, code=killed, status=6/ABRT systemd[1]: vpp.service: Unit entered failed state. systemd[1]: vpp.service: Failed with result 'signal'. systemd[1]: vpp.service: Service hold-off time over, scheduling restart. systemd[1]: Stopped vector packet processing engine. systemd[1]: Starting vector packet processing engine... systemd[1]: Started vector packet processing engine. This has now occurred twice over the last four days. Any ideas? Any other information that I can provide? I believe I've seen somewhere that VPP can do a full crash dump (or perhaps just core?) but I haven't seen the location of it. Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10891): https://lists.fd.io/g/vpp-dev/message/10891 Mute This Topic: https://lists.fd.io/mt/27432851/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP crash after days of heavy traffic, next_index is not associated to an interface #vpp_stability
Hi Dave, We have not configured LISP-GPE Could it be that VPP received an encapsulated packet it wasn't expecting causing it to crash? That'd be dangerous I'll have a look at the page for bug reports! Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10893): https://lists.fd.io/g/vpp-dev/message/10893 Mute This Topic: https://lists.fd.io/mt/27432851/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP crash after days of heavy traffic, next_index is not associated to an interface #vpp_stability
Hello Florin, Brilliant, that does seem to address it. We'll patch it tomorrow. Much appreciated. Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10898): https://lists.fd.io/g/vpp-dev/message/10898 Mute This Topic: https://lists.fd.io/mt/27432851/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability
?Hi Matus! I've picked this up again for testing and verifying. I am running it in deterministic mode now, and it's still displaying the same issue. Do you by any chance have the commit at hand? Is the fix only applied to dynamic NAT and not deterministic? Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net From: vpp-dev@lists.fd.io on behalf of JB Sent: Friday, October 19, 2018 3:19 PM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco); vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Brilliant, thanks! John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Friday, October 19, 2018 2:27 PM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, Fix in master and stable/1810 branch Matus From: vpp-dev@lists.fd.io On Behalf Of JB Sent: Friday, October 19, 2018 9:14 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Much appreciated, glad you found it! Thanks, John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, October 19, 2018 8:00 AM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, There is bug in NAT code for ICMP fragments. I will fix it as soon as possible. Thanks, Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Thursday, October 18, 2018 12:48 PM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Ole, I've added a relevant packet trace. Is that adequate? Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11536): https://lists.fd.io/g/vpp-dev/message/11536 Mute This Topic: https://lists.fd.io/mt/27400627/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability
Hi Matus, Is there some functional reasoning behind this, or is it a yet-to-be-implemented feature? ?Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Monday, December 10, 2018 10:46 AM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi, Deterministic mode doesn't support fragments. Matus From: John Biscevic Sent: Monday, December 10, 2018 10:36 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability ?Hi Matus! I've picked this up again for testing and verifying. I am running it in deterministic mode now, and it's still displaying the same issue. Do you by any chance have the commit at hand? Is the fix only applied to dynamic NAT and not deterministic? Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net<mailto:john.bisce...@bahnhof.net> From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of JB mailto:john.bisce...@bahnhof.net>> Sent: Friday, October 19, 2018 3:19 PM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco); vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Brilliant, thanks! John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, October 19, 2018 2:27 PM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, Fix in master and stable/1810 branch Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Friday, October 19, 2018 9:14 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Much appreciated, glad you found it! Thanks, John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, October 19, 2018 8:00 AM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, There is bug in NAT code for ICMP fragments. I will fix it as soon as possible. Thanks, Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Thursday, October 18, 2018 12:48 PM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Ole, I've added a relevant packet trace. Is that adequate? Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11540): https://lists.fd.io/g/vpp-dev/message/11540 Mute This Topic: https://lists.fd.io/mt/27400627/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability
?Hi Matus, Yeah, I've seen that. Gotcha, thanks! I'll spend some time looking at it and see what I can do. Thanks! Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Monday, December 10, 2018 12:45 PM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi, It is yet-to-be-implemented feature, deterministic mode is PoC code, it has only some basic functionality Matus From: John Biscevic Sent: Monday, December 10, 2018 12:40 PM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Is there some functional reasoning behind this, or is it a yet-to-be-implemented feature? ?Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net<mailto:john.bisce...@bahnhof.net> From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Monday, December 10, 2018 10:46 AM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi, Deterministic mode doesn't support fragments. Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Monday, December 10, 2018 10:36 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability ?Hi Matus! I've picked this up again for testing and verifying. I am running it in deterministic mode now, and it's still displaying the same issue. Do you by any chance have the commit at hand? Is the fix only applied to dynamic NAT and not deterministic? Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net<mailto:john.bisce...@bahnhof.net> From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of JB mailto:john.bisce...@bahnhof.net>> Sent: Friday, October 19, 2018 3:19 PM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco); vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Brilliant, thanks! John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, October 19, 2018 2:27 PM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, Fix in master and stable/1810 branch Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Friday, October 19, 2018 9:14 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Matus, Much appreciated, glad you found it! Thanks, John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, October 19, 2018 8:00 AM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi John, There is bug in NAT code for ICMP fragments. I will fix it as soon as possible. Thanks, Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Thursday, October 18, 2018 12:48 PM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Packet error-drop with fragmented packets through NAT #vpp_stability Hi Ole, I've added a relevant packet trace. Is that adequate? Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11542): https://lists.fd.io/g/vpp-dev/message/11542 Mute This Topic: https://lists.fd.io/mt/27400627/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp router plugin threads? (vpp + router + netlink + FRRouting)
Hi Brian, I've successfully built the router plugin on 18.10 and "19.01" What errors are you encountering when you attempt to build it? Kind regards, John Biscevic Systems Architect, Bahnhof AB Mobile: +46 76 111 01 24 E-mail: john.bisce...@bahnhof.net From: vpp-dev@lists.fd.io on behalf of Brian Dickson Sent: Wednesday, December 12, 2018 4:17 AM To: vpp-dev@lists.fd.io Cc: vppsb-...@lists.fd.io Subject: [vpp-dev] vpp router plugin threads? (vpp + router + netlink + FRRouting) Greetings, VPP folks. I am continuing to work on my vpp + router-plugin (+FRRouting) set-up. I have things mostly working with very large routing tables (source from multiple BGP peers), but am having some challenges when trying to use multi-threaded (additional worker threads) for increasing overall VPP forwarding performance. When using just a single thread, the BGP peers take a long time to sync up but it is relatively stable. Forwarding performance on a 10G NIC (i40e driver and vfio-pci selected), is pretty decent, but I am interested in finding ways to improve performance (and getting things to the point where I can use a 40G card also in the system). The limit seems to be packets per second, and maxes out at about 11Mpps. The problem is, when I try to use worker threads, I start running into issues with rtnetlink buffers, and BGP, ICMP, ARP, etc, all become "flaky". My suspicion is that it has something to do with which thread(s) handle the netlink traffic, and which thread(s) handle the TCP port 179 (BGP) traffic, which needs to go via the tap-inject path to the kernel, and then to the BGP speaking application (FRR sub-unit "bgpd"). Is there anyone who can provide information or advice on this issue? NB: the flakiness is in a COMPLETELY unloaded environment - no other traffic is being handled, nothing else is consuming CPU cycles. It is just the BGP traffic itself plus related stuff (ARP) and any diagnostic traffic I use (ping). Is this a case where I need to adjust the RSS to direct incoming packets to the right subset of cores, and do I also need to direct particular traffic (TCP 179) to the main core? Do I need to ensure anything else, like using a separate core (and set the core afinity with taskset -c ) for my BGP speaker? Any suggestions or advice would be greatly appreciated. Also, any updates on bringing netlink and router plugins into the main vppsb tree? Building them on anything other than 18.07 just doesn't work for me, and even on 18.07 is rather brittle, and I'm not 100% sure about the build steps, which actually involve passing CFLAGS in to make, which suggests something isn't quite right... Thanks in advance, Brian -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11578): https://lists.fd.io/g/vpp-dev/message/11578 Mute This Topic: https://lists.fd.io/mt/28729084/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Up-to-date documentation for classifiers?
Hello everyone, I've been looking for some documentation surrounding the classify feature of VPP. Any documentation or any decent information I've stumbled upon seems to be outdated with syntax that's no longer applicable. Do we have anything that I've missed? -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11580): https://lists.fd.io/g/vpp-dev/message/11580 Mute This Topic: https://lists.fd.io/mt/28732603/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Sanity check re: NAT for same-service mapping
Hello group, This might be best answered by Matus since it regards NAT, but I'll throw it out there for the whole group. The endpoint-dependent feature of the NAT plugin – Endpoint address AND port dependent I presume from the 6-tuple description of it – allows us to map the same internal source IP and port to the same external IP when targeting a certain past destination IP AND port, correct? My concern is more of the situations where services initially create a connection to one endpoint address, and then create another session to another endpoint address, expecting the same source address to match the client. Client opens a connection to endpoint X using external IP A, which proceeds to instruct client to open a session to endpoint Y, both endpoints share the same backend and expect the client to have IP A but since it's a new session and we're doing dynamic NAT, the client ends up with external IP B, breaking the chain. Many services depend on this. The idea is that when a new NAT source IP is seen, that we reserve a certain number of internal ports for that IP to the same number of external ports on a single IP, so all connections originating from that NAT source IP will always have the same external IP, thus allowing for endpoint services to not lose track of client due to IP mismatch, which breaks service. This differs from deterministic NAT in that we don't preallocate entire subnets and match them to external addresses from the start, but rather only individual adresses when needed. Is this feature reasoning sound, or is there a better solution suggested? -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11637): https://lists.fd.io/g/vpp-dev/message/11637 Mute This Topic: https://lists.fd.io/mt/28785710/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hi Ole, Absolutely, Endpoint independent mapping is the safest bet, which is why it is recommended. It is unfortunate that we cannot rely on services being IP source agnostic or that STUN will be used. Thus, even though Endpoint independent mapping can be considered non-efficient in its address and port allocation, we should perhaps consider it being the default implementation as the other implementation would be circumstantial and would benefit specific NAT deployments rather than require them, or else we run the risk of seeing broken services during NAT, especially on larger deployments. That way, exactly as you say, we could do endpoint dependent address and port mapping for the applications we know are safe, but would come second in priority. Interestingly enough, there's a scenario where we run into the danger of being unable to enact a proper remapping according to Endpoint independent mapping if all we do is track source IP and source port to the same external IP and external port from a pool, which is that unless those are reserved for the source IP and source port, in any larger deployment, another source IP might occupy it before we get a chance to. RFC4787 Section 4, REQ-2 and Section 5 describe it rather well: https://tools.ietf.org/html/rfc4787 Also, RFC6888 does reflect its implementation for CGN, in which Endpoint independent mapping is almost necessary, unless deterministic NAT (potentially highly wasteful though) is used, as in the case where we use deterministic NAT it requires the provider to be strict on their internal allocation of RFC6598 (CGNAT) address space otherwise they'll have to allocate large networks that would be mostly unused, and very wasteful computationally when traversing the NAT allocations. Kind regards, John From: Ole Troan Sent: Monday, December 17, 2018 10:26 PM To: John Biscevic Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping > This might be best answered by Matus since it regards NAT, but I'll throw it > out there for the whole group. > > The endpoint-dependent feature of the NAT plugin – Endpoint address AND port > dependent I presume from the 6-tuple description of it – allows us to map the > same internal source IP and port to the same external IP when targeting a > certain past destination IP AND port, correct? > My concern is more of the situations where services initially create a > connection to one endpoint address, and then create another session to > another endpoint address, expecting the same source address to match the > client. > > Client opens a connection to endpoint X using external IP A, which proceeds > to instruct client to open a session to endpoint Y, both endpoints share the > same backend and expect the client to have IP A but since it's a new session > and we're doing dynamic NAT, the client ends up with external IP B, breaking > the chain. Many services depend on this. > > The idea is that when a new NAT source IP is seen, that we reserve a certain > number of internal ports for that IP to the same number of external ports on > a single IP, so all connections originating from that NAT source IP will > always have the same external IP, thus allowing for endpoint services to not > lose track of client due to IP mismatch, which breaks service. Yes, and this is the reason why the IETF recommendeds against it, and recommends the use of endpoint independent mapping. In one sense, us at your peril, in the other sense, given that we’ve run out of Ipv4 addresses, that will have consequences for applications too. Address and port dependent mapping obviously has the great benefit of using the address/port space efficiently. One option we discussed, which I’m happy to encourage patches for, is to make the NAT mode, NAT traversal aware. E.g. it detects a STUN packet and makes that connection have endpoint independent mapping (for some time period). Inverse we could only do address and port dependent mappings for “applications” (aka UDP/TCP ports) we knew were safe. Opinions welcome! Cheers, Ole -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11639): https://lists.fd.io/g/vpp-dev/message/11639 Mute This Topic: https://lists.fd.io/mt/28785710/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Build/rebuild problems (adding dpdk support)
Hey Brian, I believe it fetches it again because the file doesn't match the hash string present in one of the files inside the dpdk directory. At least that was the issue I faced building for Mellanox cards ages ago. You could also specify that it uses an external DPDK source. Sent from ny phone On Tue, Dec 18, 2018 at 5:20 AM +0100, "Brian Dickson" mailto:brian.peter.dick...@gmail.com>> wrote: Hi, VPP folks, I am trying to add in support for a new card/driver. The info on the DPDK website says the drivers I have should work fine. Thus, the only issue is adding in the correct manufacturer/device IDs into the init.c file: src/plugins/dpdk/device/init.c I edit the file (one-line change), put it back into the .tar.xz file, put it where it should be being picked up (in build-root or so I believe), and it gets blown away. It always seems to want to pull everything from git regardless of the presence of the right tar.xz file in build-root, pointed to by the vpp-latest symbolic link. I am using "make pkg-rpm" as the command to rebuild things from the main vpp source root (parent of build-root). What am I doing wrong? Where should I be rebuilding this? What command should I use? Or what make target? In what directory? Thanks in advance. Brian P.S. I already spent a lot of time chasing a red herring. Does someone want to explain to me the purpose of the "sed -e s/-/_/" in extras/rpm/Makefile, full line: RELEASE=$(shell echo $(BASENAME) | cut -f3- -d- | sed -e s/-/_/g) It seems really out of place, and leads to lots of difficulty in tracking down the place where something that looks like "foo-bar-blech" becomes "foo-bar_blech", for no obvious reason. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11642): https://lists.fd.io/g/vpp-dev/message/11642 Mute This Topic: https://lists.fd.io/mt/28789545/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hi Matus, There must be some misunderstanding! We're discussing endpoint INdependent mapping which as far as I know isn't currently present in VPP. The issue being that the current NAT implementations break certain services as clients end up with different external IPs in various situations where an endpoint service might use a different IP and port but still expect the same IP, or where the desired external IP and port can't be reused since another session has occupied them because they're not reserved for the client. Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 8:57 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Hi, Endpoint-dependent NAT is not default behaviour, when you want to use endpoint-dependent NAT you need to adjust startup config https://wiki.fd.io/view/VPP/NAT#NAT44 Matus -Original Message- From: vpp-dev@lists.fd.io On Behalf Of JB Sent: Tuesday, December 18, 2018 12:02 AM To: Ole Troan Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Ole, Absolutely, Endpoint independent mapping is the safest bet, which is why it is recommended. It is unfortunate that we cannot rely on services being IP source agnostic or that STUN will be used. Thus, even though Endpoint independent mapping can be considered non-efficient in its address and port allocation, we should perhaps consider it being the default implementation as the other implementation would be circumstantial and would benefit specific NAT deployments rather than require them, or else we run the risk of seeing broken services during NAT, especially on larger deployments. That way, exactly as you say, we could do endpoint dependent address and port mapping for the applications we know are safe, but would come second in priority. Interestingly enough, there's a scenario where we run into the danger of being unable to enact a proper remapping according to Endpoint independent mapping if all we do is track source IP and source port to the same external IP and external port from a pool, which is that unless those are reserved for the source IP and source port, in any larger deployment, another source IP might occupy it before we get a chance to. RFC4787 Section 4, REQ-2 and Section 5 describe it rather well: https://tools.ietf.org/html/rfc4787 Also, RFC6888 does reflect its implementation for CGN, in which Endpoint independent mapping is almost necessary, unless deterministic NAT (potentially highly wasteful though) is used, as in the case where we use deterministic NAT it requires the provider to be strict on their internal allocation of RFC6598 (CGNAT) address space otherwise they'll have to allocate large networks that would be mostly unused, and very wasteful computationally when traversing the NAT allocations. Kind regards, John From: Ole Troan Sent: Monday, December 17, 2018 10:26 PM To: John Biscevic Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping > This might be best answered by Matus since it regards NAT, but I'll throw it > out there for the whole group. > > The endpoint-dependent feature of the NAT plugin - Endpoint address AND port > dependent I presume from the 6-tuple description of it - allows us to map the > same internal source IP and port to the same external IP when targeting a > certain past destination IP AND port, correct? > My concern is more of the situations where services initially create a > connection to one endpoint address, and then create another session to > another endpoint address, expecting the same source address to match the > client. > > Client opens a connection to endpoint X using external IP A, which proceeds > to instruct client to open a session to endpoint Y, both endpoints share the > same backend and expect the client to have IP A but since it's a new session > and we're doing dynamic NAT, the client ends up with external IP B, breaking > the chain. Many services depend on this. > > The idea is that when a new NAT source IP is seen, that we reserve a certain > number of internal ports for that IP to the same number of external ports on > a single IP, so all connections originating from that NAT source IP will > always have the same external IP, thus allowing for endpoint services to not > lose track of client due to IP mismatch, which breaks service. Yes, and this is the reason why the IETF recommendeds against it, and recommends the use of endpoint independent mapping. In one sense, us at your peril, in the other sense, given that we've run out of Ipv4 addresses, that will have consequences for applications too. Address and port dependent mapping obviously has the great benefit of usin
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hi Matus, That is... Interesting. Is the behaviour dependent on the presence of STUN packets? Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 10:08 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Endpoint independent mapping is default behaviour Matus From: John Biscevic Sent: Tuesday, December 18, 2018 10:03 AM To: Ole Troan ; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, There must be some misunderstanding! We're discussing endpoint INdependent mapping which as far as I know isn't currently present in VPP. The issue being that the current NAT implementations break certain services as clients end up with different external IPs in various situations where an endpoint service might use a different IP and port but still expect the same IP, or where the desired external IP and port can't be reused since another session has occupied them because they're not reserved for the client. Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 8:57 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Hi, Endpoint-dependent NAT is not default behaviour, when you want to use endpoint-dependent NAT you need to adjust startup config https://wiki.fd.io/view/VPP/NAT#NAT44 Matus -Original Message- From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> On Behalf Of JB Sent: Tuesday, December 18, 2018 12:02 AM To: Ole Troan Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Ole, Absolutely, Endpoint independent mapping is the safest bet, which is why it is recommended. It is unfortunate that we cannot rely on services being IP source agnostic or that STUN will be used. Thus, even though Endpoint independent mapping can be considered non-efficient in its address and port allocation, we should perhaps consider it being the default implementation as the other implementation would be circumstantial and would benefit specific NAT deployments rather than require them, or else we run the risk of seeing broken services during NAT, especially on larger deployments. That way, exactly as you say, we could do endpoint dependent address and port mapping for the applications we know are safe, but would come second in priority. Interestingly enough, there's a scenario where we run into the danger of being unable to enact a proper remapping according to Endpoint independent mapping if all we do is track source IP and source port to the same external IP and external port from a pool, which is that unless those are reserved for the source IP and source port, in any larger deployment, another source IP might occupy it before we get a chance to. RFC4787 Section 4, REQ-2 and Section 5 describe it rather well: https://tools.ietf.org/html/rfc4787 Also, RFC6888 does reflect its implementation for CGN, in which Endpoint independent mapping is almost necessary, unless deterministic NAT (potentially highly wasteful though) is used, as in the case where we use deterministic NAT it requires the provider to be strict on their internal allocation of RFC6598 (CGNAT) address space otherwise they'll have to allocate large networks that would be mostly unused, and very wasteful computationally when traversing the NAT allocations. Kind regards, John From: Ole Troan Sent: Monday, December 17, 2018 10:26 PM To: John Biscevic Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping > This might be best answered by Matus since it regards NAT, but I'll throw it > out there for the whole group. > > The endpoint-dependent feature of the NAT plugin - Endpoint address AND port > dependent I presume from the 6-tuple description of it - allows us to map the > same internal source IP and port to the same external IP when targeting a > certain past destination IP AND port, correct? > My concern is more of the situations where services initially create a > connection to one endpoint address, and then create another session to > another endpoint address, expecting the same source address to match the > client. > > Client opens a connection to endpoint X using external IP A, which proceeds > to instruct client to open a session to endpoint Y, both endpoints share the > same backend and expect the client to have IP A but since it's a new session > and we're doing dynamic NAT, the client ends up with external IP B, breaking > the chain. Many services depend on this. > > The idea is that when a
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hi Matus, Brilliant, thanks! However, then isn't it possible for a client to end up exposing two different external IPs to an endpoint if the client opens two separate sessions over two different ports? John Sent from my phone On Tue, Dec 18, 2018 at 10:28 AM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES@Cisco) via Lists.Fd.Io" mailto:matfabia=cisco@lists.fd.io>> wrote: Session/mapping key is 4-tuple (client address, port, fib index and protocol), internal address and port is mapped always to same external address and port no matter what is the endpoint https://gerrit.fd.io/r/gitweb?p=vpp.git;a=blob;f=src/plugins/nat/nat.h;h=3ce83ea26022fac43045fc88bfb37466c78c98dd;hb=HEAD#l58 Matus From: John Biscevic Sent: Tuesday, December 18, 2018 10:53 AM To: Ole Troan ; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, That is... Interesting. Is the behaviour dependent on the presence of STUN packets? Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 10:08 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Endpoint independent mapping is default behaviour Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 10:03 AM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, There must be some misunderstanding! We're discussing endpoint INdependent mapping which as far as I know isn't currently present in VPP. The issue being that the current NAT implementations break certain services as clients end up with different external IPs in various situations where an endpoint service might use a different IP and port but still expect the same IP, or where the desired external IP and port can't be reused since another session has occupied them because they're not reserved for the client. Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 8:57 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Hi, Endpoint-dependent NAT is not default behaviour, when you want to use endpoint-dependent NAT you need to adjust startup config https://wiki.fd.io/view/VPP/NAT#NAT44 Matus -Original Message- From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> On Behalf Of JB Sent: Tuesday, December 18, 2018 12:02 AM To: Ole Troan Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Ole, Absolutely, Endpoint independent mapping is the safest bet, which is why it is recommended. It is unfortunate that we cannot rely on services being IP source agnostic or that STUN will be used. Thus, even though Endpoint independent mapping can be considered non-efficient in its address and port allocation, we should perhaps consider it being the default implementation as the other implementation would be circumstantial and would benefit specific NAT deployments rather than require them, or else we run the risk of seeing broken services during NAT, especially on larger deployments. That way, exactly as you say, we could do endpoint dependent address and port mapping for the applications we know are safe, but would come second in priority. Interestingly enough, there's a scenario where we run into the danger of being unable to enact a proper remapping according to Endpoint independent mapping if all we do is track source IP and source port to the same external IP and external port from a pool, which is that unless those are reserved for the source IP and source port, in any larger deployment, another source IP might occupy it before we get a chance to. RFC4787 Section 4, REQ-2 and Section 5 describe it rather well: https://tools.ietf.org/html/rfc4787 Also, RFC6888 does reflect its implementation for CGN, in which Endpoint independent mapping is almost necessary, unless deterministic NAT (potentially highly wasteful though) is used, as in the case where we use deterministic NAT it requires the provider to be strict on their internal allocation of RFC6598 (CGNAT) address space otherwise they'll have to allocate large networks that would be mostly unused, and very wasteful computationally when traversing the NAT allocations. Kind regards, John From: Ole Troan Sent: Monday, December 17, 2018 10:26 PM To: John Biscevic Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-servi
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hi Matus, Thanks, that's how I figured that it works, and was the root of my concern and the idea of reserving ports for a client equivalent to max translations per user to avoid such scenarios. It's potentially wasteful but a fail-safe against such scenarios. Much appreciated that you've taken your time so far! John Sent from my phone On Tue, Dec 18, 2018 at 1:29 PM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: When external address used in first session has allocated all available ports second session use another external address if it is available (when you have 2 external addresses nat start using first and when all ports are allocated it moves to second one) Matus From: John Biscevic Sent: Tuesday, December 18, 2018 2:23 PM To: Ole Troan ; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, Brilliant, thanks! However, then isn't it possible for a client to end up exposing two different external IPs to an endpoint if the client opens two separate sessions over two different ports? John Sent from my phone On Tue, Dec 18, 2018 at 10:28 AM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES@Cisco) via Lists.Fd.Io" mailto:matfabia=cisco@lists.fd.io>> wrote: Session/mapping key is 4-tuple (client address, port, fib index and protocol), internal address and port is mapped always to same external address and port no matter what is the endpoint https://gerrit.fd.io/r/gitweb?p=vpp.git;a=blob;f=src/plugins/nat/nat.h;h=3ce83ea26022fac43045fc88bfb37466c78c98dd;hb=HEAD#l58 Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 10:53 AM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, That is... Interesting. Is the behaviour dependent on the presence of STUN packets? Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 10:08 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Endpoint independent mapping is default behaviour Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 10:03 AM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, There must be some misunderstanding! We're discussing endpoint INdependent mapping which as far as I know isn't currently present in VPP. The issue being that the current NAT implementations break certain services as clients end up with different external IPs in various situations where an endpoint service might use a different IP and port but still expect the same IP, or where the desired external IP and port can't be reused since another session has occupied them because they're not reserved for the client. Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 8:57 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Hi, Endpoint-dependent NAT is not default behaviour, when you want to use endpoint-dependent NAT you need to adjust startup config https://wiki.fd.io/view/VPP/NAT#NAT44 Matus -Original Message- From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> On Behalf Of JB Sent: Tuesday, December 18, 2018 12:02 AM To: Ole Troan Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Ole, Absolutely, Endpoint independent mapping is the safest bet, which is why it is recommended. It is unfortunate that we cannot rely on services being IP source agnostic or that STUN will be used. Thus, even though Endpoint independent mapping can be considered non-efficient in its address and port allocation, we should perhaps consider it being the default implementation as the other implementation would be circumstantial and would benefit specific NAT deployments rather than require them, or else we run the risk of seeing broken services during NAT, especially on larger deployments. That way, exactly as you say, we could do endpoint dependent address and port mapping for the applications we know are safe, but would come second in priority. Interestingly enough, there's a scenario where we run into the danger of being unable to enact a proper remapping accordin
[vpp-dev] NAT workers above 4 completely tanks performance
Hello everyone, This is on "19.01", I've yet to test with 18.10. I've setup dynamic NAT. I've tested this with a clean setup without specifying anything special in startup.conf. The results are irrelevant of other NAT settings except NAT workers. I've tried specifying various networks, single IP as external network, or a larger subnet of /24 or /23, no difference. If I set VPP workers to no more than 4, I see no issues in throughput or latency, since then NAT workers are also limited to 4. However, if I specify workers at 5 or above, latency is high on every single packet processed, with averages of around 18 ms. Throughput is also severely affected, hitting around 100 Mbps from an external server to a NAT:ed client. If NAT workers are set to 4 or less, the issues go away. Server has a single 12 core Xeon Gold CPU, 96GB DDR4 RAM in optimal placement using all memory channels, and interfaces are two Mellanox ConnectX-5 100GbE. I'm stumped, not seen this behaviour in the past or in anyone's experience. Suggestions? -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11698): https://lists.fd.io/g/vpp-dev/message/11698 Mute This Topic: https://lists.fd.io/mt/28802889/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
I've noted a potential modification as to how we do dynamic NAT that would alleviate issues seen by for example banks (they depend on this a lot and both current dynamic NAT implementations of VPP break this). Today, we use a 4-tuple session key for endpoint independent NAT, but, if the source IP has before communicated with the same destination IP, regardless of source or destination port, the new session should use the same external IP. I realise that if we hash this, we'd have to hash source and destination IP straight off. Otherwise we do end up with quite a few scenarios where NAT is broken. This is also the reason why the Linux iptables implementation for their "persistent" argument (replaces the old "SAME") is that a source IP communicating with a destination IP will always attempt to use the same external IP, regardless of ports, saving such scenarios. This in a sense bridges endpoint dependent with independent since for the former we'd use the same external IP when a source IP and source port is matched, and the latter where both source ip, source IP, destination IP, and destination port are matched. The situation we do not account for with the current implementation is when a source IP uses a different source port but intends to communicate with the same destination IP as a previous session. John From: vpp-dev@lists.fd.io on behalf of JB Sent: Tuesday, December 18, 2018 2:54 PM To: Ole Troan; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, Thanks, that's how I figured that it works, and was the root of my concern and the idea of reserving ports for a client equivalent to max translations per user to avoid such scenarios. It's potentially wasteful but a fail-safe against such scenarios. Much appreciated that you've taken your time so far! John Sent from my phone On Tue, Dec 18, 2018 at 1:29 PM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: When external address used in first session has allocated all available ports second session use another external address if it is available (when you have 2 external addresses nat start using first and when all ports are allocated it moves to second one) Matus From: John Biscevic Sent: Tuesday, December 18, 2018 2:23 PM To: Ole Troan ; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, Brilliant, thanks! However, then isn't it possible for a client to end up exposing two different external IPs to an endpoint if the client opens two separate sessions over two different ports? John Sent from my phone On Tue, Dec 18, 2018 at 10:28 AM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES@Cisco) via Lists.Fd.Io" mailto:matfabia=cisco@lists.fd.io>> wrote: Session/mapping key is 4-tuple (client address, port, fib index and protocol), internal address and port is mapped always to same external address and port no matter what is the endpoint https://gerrit.fd.io/r/gitweb?p=vpp.git;a=blob;f=src/plugins/nat/nat.h;h=3ce83ea26022fac43045fc88bfb37466c78c98dd;hb=HEAD#l58 Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 10:53 AM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, That is... Interesting. Is the behaviour dependent on the presence of STUN packets? Sincerely, John Sent from my phone On Tue, Dec 18, 2018 at 10:08 AM +0100, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: Endpoint independent mapping is default behaviour Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 10:03 AM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, There must be some misunderstanding! We're discussing endpoint INdependent mapping which as far as I know isn't currently present in VPP. The issue being that the current NAT implementations break certain services as clients end up with different external IPs in various situations where an endpoint service might use a different IP and port but still expect the same IP, or where the desired external IP and port can't be reus
Re: [vpp-dev] NAT workers above 4 completely tanks performance
Hi Damjan, Absolutely. I raw one case with the default number of NAT workers (10) which has poor performance, and another case with a fewer number of NAT workers (4) showing great performance. They're separated by two different files, both are attached. John vpp# sh run Thread 0 vpp_main (lcore 1) Time 20.7, average vectors/node 0.00, last 128 main loops 0.00 per node 0.00 vector rates in 0.e0, out 0.e0, drop 0.e0, punt 0.e0 Name State Calls Vectors Suspends Clocks Vectors/Call Perf Ticks api-rx-from-ringany wait 0 0 1 1.13e40.00 cdp-process any wait 0 0 2 6.32e30.00 dpdk-processany wait 0 0 7 6.07e60.00 fib-walkany wait 0 0 10 5.81e30.00 ikev2-manager-process any wait 0 0 21 4.66e30.00 ip4-reassembly-expire-walk any wait 0 0 2 6.61e30.00 ip6-icmp-neighbor-discovery-ev any wait 0 0 21 4.11e30.00 ip6-reassembly-expire-walk any wait 0 0 2 6.16e30.00 lisp-retry-service any wait 0 0 10 6.29e30.00 statseg-collector-process time wait0 0 2 5.39e30.00 unix-cli-local:1 active 0 0 7 3.20e60.00 unix-epoll-input polling 46896 0 0 1.64e60.00 vpe-oam-process any wait 0 0 10 5.05e30.00 --- Thread 1 vpp_wk_0 (lcore 2) Time 20.7, average vectors/node 1.01, last 128 main loops 0.00 per node 0.00 vector rates in 8.1284e4, out 1.1746e5, drop 4.3071e1, punt 0.e0 Name State Calls Vectors Suspends Clocks Vectors/Call Perf Ticks HundredGigabitEthernet65/0/1-o active 747464 747591 0 1.43e21.00 HundredGigabitEthernet65/0/1-t active 747464 747591 0 4.00e21.00 HundredGigabitEthernetb3/0/1-o active1675943 1679595 0 1.47e21.00 HundredGigabitEthernetb3/0/1-t active1675943 1679595 0 2.24e21.00 dpdk-input polling 116041927 1679595 0 2.46e4 .01 error-drop active890 890 0 4.26e21.00 ethernet-input active1675943 1679595 0 2.78e21.00 ip4-icmp-echo-requestactive 1 1 0 3.23e31.00 ip4-icmp-input active 1 1 0 9.34e21.00 ip4-input-no-checksumactive1675942 1679594 0 2.44e21.00 ip4-load-balance active2394167 2427187 0 1.24e21.01 ip4-localactive 1 1 0 3.31e31.00 ip4-lookup active2394166 2427186 0 2.08e21.01 ip4-rewrite active2394166 2427186 0 1.90e21.01 lldp-input active 1 1 0 3.08e31.00 nat44-in2out active1675942 1679594 0 3.09e21.00 nat44-in2out-worker-handoff active1675942 1679594 0 2.56e21.00 nat44-out2in active 748354 748481 0 6.37e21.00 --- Thread 2 vpp_wk_1 (lcore 3) Time 20.7, average vectors/node 1.00, last 128 main loops 0.00 per node 0.00 vector rates in 3.6237e4, out 0.e0, drop 5.7589e0, punt 0.e0 Name Sta
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hi Matus, Thanks, that's what I figured. I'll see about appending the code in case more people find use for it. John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Thursday, December 20, 2018 7:20 AM To: John Biscevic; Ole Troan Cc: vpp-dev@lists.fd.io Subject: RE: [vpp-dev] Sanity check re: NAT for same-service mapping Hi, NAT address and port allocation is pluggable, you can write your own algorithm and use it instead of default (currently we support two additional port restricted algorithms map-e and port range) Matus From: John Biscevic Sent: Wednesday, December 19, 2018 6:12 PM To: Ole Troan ; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping I've noted a potential modification as to how we do dynamic NAT that would alleviate issues seen by for example banks (they depend on this a lot and both current dynamic NAT implementations of VPP break this). Today, we use a 4-tuple session key for endpoint independent NAT, but, if the source IP has before communicated with the same destination IP, regardless of source or destination port, the new session should use the same external IP. I realise that if we hash this, we'd have to hash source and destination IP straight off. Otherwise we do end up with quite a few scenarios where NAT is broken. This is also the reason why the Linux iptables implementation for their "persistent" argument (replaces the old "SAME") is that a source IP communicating with a destination IP will always attempt to use the same external IP, regardless of ports, saving such scenarios. This in a sense bridges endpoint dependent with independent since for the former we'd use the same external IP when a source IP and source port is matched, and the latter where both source ip, source IP, destination IP, and destination port are matched. The situation we do not account for with the current implementation is when a source IP uses a different source port but intends to communicate with the same destination IP as a previous session. John From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of JB mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 2:54 PM To: Ole Troan; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, Thanks, that's how I figured that it works, and was the root of my concern and the idea of reserving ports for a client equivalent to max translations per user to avoid such scenarios. It's potentially wasteful but a fail-safe against such scenarios. Much appreciated that you've taken your time so far! John Sent from my phone On Tue, Dec 18, 2018 at 1:29 PM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco)" mailto:matfa...@cisco.com>> wrote: When external address used in first session has allocated all available ports second session use another external address if it is available (when you have 2 external addresses nat start using first and when all ports are allocated it moves to second one) Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 2:23 PM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, Brilliant, thanks! However, then isn't it possible for a client to end up exposing two different external IPs to an endpoint if the client opens two separate sessions over two different ports? John Sent from my phone On Tue, Dec 18, 2018 at 10:28 AM +, "Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES@Cisco) via Lists.Fd.Io" mailto:matfabia=cisco@lists.fd.io>> wrote: Session/mapping key is 4-tuple (client address, port, fib index and protocol), internal address and port is mapped always to same external address and port no matter what is the endpoint https://gerrit.fd.io/r/gitweb?p=vpp.git;a=blob;f=src/plugins/nat/nat.h;h=3ce83ea26022fac43045fc88bfb37466c78c98dd;hb=HEAD#l58 Matus From: John Biscevic mailto:john.bisce...@bahnhof.net>> Sent: Tuesday, December 18, 2018 10:53 AM To: Ole Troan mailto:otr...@employees.org>>; Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Sanity check re: NAT for same-service mapping Hi Matus, That is... Interesting. Is the behaviour depe
Re: [vpp-dev] NAT workers above 4 completely tanks performance
Hi Matus, Thanks! Any suggestions on what that can be done to alleviate the issues? The above test was done with a single client, but the same symptoms are shown when throwing far more flows at it, around 5.5 million sessions, from thousands of L3 sources.? John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Friday, December 21, 2018 6:21 AM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] NAT workers above 4 completely tanks performance Hi, in your case most of NAT translations are done in one core. With 4 cores you are lucky and flows arrive at same core where translations are processing (no worker handoff) and with 10 cores there is worker handoff between two workers and it is reason of performance drop. Basically your flows are not symmetrically distributed between cores. Matus From: vpp-dev@lists.fd.io On Behalf Of JB Sent: Friday, December 21, 2018 4:25 AM To: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] NAT workers above 4 completely tanks performance Hi Damjan, Absolutely. I raw one case with the default number of NAT workers (10) which has poor performance, and another case with a fewer number of NAT workers (4) showing great performance. They're separated by two different files, both are attached. John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11748): https://lists.fd.io/g/vpp-dev/message/11748 Mute This Topic: https://lists.fd.io/mt/28802889/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] NAT workers above 4 completely tanks performance
Hi Matus, I've not yet had the chance to check what that looks like with multiple clients. I did check the rest of what you wrote, and that made everything a bit clearer! I have it setup so one physical interface handles all internal traffic, and another physical interface handles all external traffic. I've setup each interface with 4 RX queues each. The internal interface queues are on the first four workers (0-3), while the external interface queues are on the last four workers (4-7). However, setting NAT workers on the last four workers showed not as many issues. I confirmed with "show run" that they're on the latter workers. We see some performance issues if the NAT workers are on different workers than the interfaces handling inside traffic. We see extreme issues if the NAT workers are on workers that have no queues on them, for example worker 8+. We see the best performance if the NAT workers are using the same workers as those handling the inside interface RX queues, which would make sense why the issues go away if I limit NAT workers to 0-3. John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) Sent: Friday, December 21, 2018 7:02 AM To: John Biscevic; vpp-dev@lists.fd.io Subject: RE: [vpp-dev] NAT workers above 4 completely tanks performance Is worker distribution same in case of multiple clients (you ca see this with same "show run" exercise, take a look at number of interface and nat44-in2out calls for each core)? Maybe you should try to play with interface rx queue placement (you can see it in "show interface rx-placement" output) or RSS options of the card/interface. Matus From: John Biscevic Sent: Friday, December 21, 2018 6:34 AM To: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] NAT workers above 4 completely tanks performance Hi Matus, Thanks! Any suggestions on what that can be done to alleviate the issues? The above test was done with a single client, but the same symptoms are shown when throwing far more flows at it, around 5.5 million sessions, from thousands of L3 sources.? John From: Matus Fabian -X (matfabia - PANTHEON TECHNOLOGIES at Cisco) mailto:matfa...@cisco.com>> Sent: Friday, December 21, 2018 6:21 AM To: John Biscevic; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] NAT workers above 4 completely tanks performance Hi, in your case most of NAT translations are done in one core. With 4 cores you are lucky and flows arrive at same core where translations are processing (no worker handoff) and with 10 cores there is worker handoff between two workers and it is reason of performance drop. Basically your flows are not symmetrically distributed between cores. Matus From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of JB Sent: Friday, December 21, 2018 4:25 AM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] NAT workers above 4 completely tanks performance Hi Damjan, Absolutely. I raw one case with the default number of NAT workers (10) which has poor performance, and another case with a fewer number of NAT workers (4) showing great performance. They're separated by two different files, both are attached. John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#11750): https://lists.fd.io/g/vpp-dev/message/11750 Mute This Topic: https://lists.fd.io/mt/28802889/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Sanity check re: NAT for same-service mapping
Hello everyone, I've just come back from vacation and picked this up again. I've not yet found a pretty solution to the issues that are present when using NAT on a large scale and communicating with sources that require clients to communicate from the same external IP. I had hoped we wouldn't need to reserve ports as per the last idea as it is wasteful so I've looked at it again. I turned towards the source code of netfilter used in Linux to see how they implemented their SAME target (deprecated), and later (current) "persistent" flag as this works very well in a crowded NAT (lots of clients easily and quickly filling up port slots of several external IPs at once). They previously hashed both source and destination IP but this caused the aforementioned issues (essentially endpoint dependent). The current implementation, and has been current for a few years now, is that they only care about the client source IP and match that always to the same external IP. It is extremely important that ports and destinations are not taken into consideration as it breaks many services. The idea is therefor that if we have seen a client before, and we have assigned an external IP to that client, we should aim to keep assigning that same external IP to that client for all future connections, regardless of ports. My understanding per the previous comments and the construction of the 4-tuple is that the user will get the same external IP if both the source IP and source port match a previous connection. This unfortunately breaks functionality (tested to verify) with most services that open several connections from a client IP, from several source ports, expecting to see all connections coming from the same IP. The above can also occur if the client IP is assigned to a typical residential CPE, where multiple connections originate from the same client IP. In which case users attempting to use something like IP-verified two-factor authentication (banks) behind their CPE will fail. An addition to the above but not a functional requirement, in order to alleviate the number of occupied ports for a given external IP so we increase the chances of having an available port for a client who needs to reuse it is that new users added to the first least-used external IP in the pool. However, this part falls within NAT address and port allocation and could be added as a separate algorithm. I'm struggling to see a good implementation other than messing with the actual session keys and I'll happily accept suggestions -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12060): https://lists.fd.io/g/vpp-dev/message/12060 Mute This Topic: https://lists.fd.io/mt/28785710/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] [NAT] Assign same external IP
Hello, Breaking this out into its own thread. Currently when creating a new dynamic NAT session, the source IP and source port are considered. If I've understood this right, the next time the user (source IP) sends traffic matching the previous traffic (source IP + source port), the same external IP should be assigned. However, I'd want to make sure the source IP always gets the same external IP, regardless of port used. I'm looking for suggestions and ideas as to how this can be achieved. Is the 4-tuple hash key the only deciding factor here? Thanks! -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12142): https://lists.fd.io/g/vpp-dev/message/12142 Mute This Topic: https://lists.fd.io/mt/29639823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [NAT] Assign same external IP
Hi Ole!
Thanks for the response!
I've followed the code you mentioned which has lead me to the default address
and port assignment algorithm. I can see how we can easily plug our own, but
I'm trying to first break down the code for the default one in order to
understand how exactly the algorithm works right now. Is the default algorithm
also responsible for the endpoint independent logic?
It's a bit lost on me as to why the default one checks against the fib_index
(the other ones do not).
The second time we check if it's a bitwise all 1 (~0), before then running the
exact same code. Is this a hack for a previous issue?
>
>
>
> else if (a-> fib_index == ~ 0 ) { ga = a; }
>
>
Thanks,
John
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#12183): https://lists.fd.io/g/vpp-dev/message/12183
Mute This Topic: https://lists.fd.io/mt/29639823/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [NAT] Assign same external IP
Hi Matus, Thanks for the response! Ah, I see, that makes more sense as to why we check against the FIB. However, if we just pick a random port (per protocol) from the "first address with some available ports" (dictated by the "busy ports" I presume), how does this ensure that a user ever gets the same external IP? I can see it happening if we only have one external IP, or by sheer luck. Looking at the algorithm, it's just as you say, we take it from the first available one, but there's no logic in place to assign a user a previously-assigned external IP, or am I missing some logic in the code here? I'm referring to an older discussion you and I had where you mentioned > > Session/mapping key is 4-tuple (client address, port, fib index and > protocol), internal address and port is mapped always to same external > address and port no matter what is the endpoint > https://gerrit.fd.io/r/gitweb?p=vpp.git;a=blob;f=src/plugins/ > nat / nat.h;h=3ce83ea26022fac43045fc88bfb37466c78c98dd;hb=HEAD#l58 The link refers to the definition of the snat session key (4-tuple). Trying to find the above logic in the code. Is that due to the nature of the 4-tuple and has nothing to do with the assignment algorithm? Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12185): https://lists.fd.io/g/vpp-dev/message/12185 Mute This Topic: https://lists.fd.io/mt/29639823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [NAT] Assign same external IP
Hi Matus, Thanks once again. That seems to be it. After reading RFC4787 section 4.1 REQ 1, their mention of PAP (Paired-Address-Pooling) seems to be on-point. As they mention: > > NATs that use an "IP address pooling" behavior of "Arbitrary" can cause > issues for applications that use multiple ports from the same endpoint, > but that do not negotiate IP addresses individually (e.g., some > applications using RTP and RTCP). and the REQ 2 mentions the standard recommendation: > > It is RECOMMENDED that a NAT have an "IP address pooling" behavior of > "Paired". Note that this requirement is not applicable to NATs that do not > support IP address pooling. That's the issues we've faced and are attempting to avoid, since it would seem that arbitrary address pooling causes issues for a lot of services when we use multiple external addresses. Are there any plans to implemented PAP? I'm unsure how a clean and efficient implementation would look since we don't want to reserve the entire public IP for a single internal IP, but still attempt to keep traffic over the same external IP. Would a feasible implementation perhaps reserve a number of slots for the internal IP (wasteful)? Would it perhaps make it so that for each new internal user, we bump them to the least-used external address in the vector, so that we lessen the likelihood of running out of ports? It's sadly extremely frustrating that a lot of services depend so much on all connections being from the same IP in order to maintain user authentication, as I'd imagine more service providers migrating to NAT-based solutions (such as for CGN), and PAP doesn't seem to be as efficient as AAP. Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12188): https://lists.fd.io/g/vpp-dev/message/12188 Mute This Topic: https://lists.fd.io/mt/29639823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [NAT] Assign same external IP
After reading Cisco's implementation for PAP for IOS XE, it seems they limit the number of local addresses per global address. The default is 120 local addresses per global address. That way we can make sure that there are never more than a certain number of local users per global IP, but can still impose limits such as maximum translations per user https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/xe-3s/nat-xe-3s-book/iadnat-addr-pool.html -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12190): https://lists.fd.io/g/vpp-dev/message/12190 Mute This Topic: https://lists.fd.io/mt/29639823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] [NAT] Assign same external IP
Hi Matus, That's unfortunate. That would work as an immediate solution. I've considered a solution like that, but I'm worried it might be wasteful. I considered that very setup when I was contemplating a sort of hybrid NAT between dynamic NAT and CGN. In CGN, just as we allocate a number of ports per IP by dividing all external IPs and ports with the number of internal IPs, we'd allocate a block of ports for each new user created, just as you say. In reality, you have some users who only occupy a handful of ports, and others who occupy hundreds. I'd imagine a potential sane compromise might be to have both a limit for the max number of local users per global IP, and max translations per user. That way we can avoid having, say, ten thousand local clients on a global IP with just a few ports each, and at the same time we can ensure that no single client takes up too many ports. It would be a compromise that can scale, and won't necessarily punish the system if most users only use a few ports, rather than allocating the ports in advance. The downside is that if we have, say, a limit of 120 clients, and each client only uses a handful of ports, we'll have thousands of unused ports on that global IP. For the sake of such an implementation, it might be good if the limit can be changed during runtime instead of only at startup. As I mentioned, PAP seems less efficient than AAP, but it might be a necessary loss of efficiency in order to maintain functionality with all the services that break without it, as per the RFC4787 requirements and recommendations. Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12192): https://lists.fd.io/g/vpp-dev/message/12192 Mute This Topic: https://lists.fd.io/mt/29639823/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Submitting code for NAT PAP
Hi, I've added support to the NAT plugin for Paired-Address-Pooling (PAP) and wanted to see if there is interest for me to submit it as a patch for review? The changes modify the behaviour of user creation, address allocation, and address management. Fundamentally it pairs a NAT user with an external IP when the user is created. The plugin will then only hand out ports within that external IP to that NAT user. The ceiling for max translations is overridden by (ports per IP / max_users_per_IP), but one can manually set a lower number of max translations. The max number of users per external IP is also configurable. When a new user is seen, the system will pick the external IP with the lowest number of paired addresses. This ensures that if we have a lot of external addresses, we spread usage across them. I've so far tested this in a lab with a few thousand simulated clients and it has worked as intended. This fixes issues for services that require all user connections to originate from the same source IP otherwise authentication breaks, such as banks. Sincerely, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12446): https://lists.fd.io/g/vpp-dev/message/12446 Mute This Topic: https://lists.fd.io/mt/30286653/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Submitting code for NAT PAP
Hi Ole, I'll see to submitting it then! Sincerely, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12456): https://lists.fd.io/g/vpp-dev/message/12456 Mute This Topic: https://lists.fd.io/mt/30286653/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Fresh install shows large latency increase when using workers #vpp_stability
Hi everyone, I noticed something today using v19.04-rc0, running Ubuntu 18.04 Fresh install, nothing modified, cloned straight from https://gerrit.fd.io/r/vpp Either built manually or via vpp/build-root/vagrant/build.sh, then installed non-dev and non-debug debs When NOT using workers, latency is fine, but when enabling workers from the startup configuration, even just a single worker, immediately increases the latency by 20-40 fold Not using NAT, simply adding an IP to one of the interfaces, and pinging a host on the same subnet. I've attempted to play around with rx-placement on the interfaces without any success. Placing them all on the same worker increases the latency to 25-27 ms I've tested this on three different machines, both physical and virtual, some running Skylake-X, others running Ivy-EP all displaying the same issue. No workers: 0.25 - 0.38 ms latency Any other number of workers: 10-28 ms latency depending on rx-placement -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12457): https://lists.fd.io/g/vpp-dev/message/12457 Mute This Topic: https://lists.fd.io/mt/30295958/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Fresh install shows large latency increase when using workers #vpp_stability
I can confirm that this does NOT happen in 19.01 stable -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12458): https://lists.fd.io/g/vpp-dev/message/12458 Mute This Topic: https://lists.fd.io/mt/30295958/21656 Mute #vpp_stability: https://lists.fd.io/mk?hashtag=vpp_stability&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Dynamic NAT breaks TCP punting
Hi, If we enable dynamic NAT, then TCP packets are no longer punted or even seen on tap interfaces, as they are simply dropped if there's no matching translation. This isn't ideal since we still want to be able to process packets that are meant for us, for example if our interface address is not used in the NAT pool. We could enable "nat44 forwarding", but that essentially breaks dynamic NAT as it only works with endpoint dependent NAT and static mappings. I suspect we should fiddle with the order of the graph nodes but I'd love a pointer. John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12714): https://lists.fd.io/g/vpp-dev/message/12714 Mute This Topic: https://lists.fd.io/mt/30923501/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Dynamic NAT breaks TCP punting
For anyone reading this, a dirty way around it is to simply add a static mapping using the external IP and port as both the local and external IP and port. This forwards the packet internally, punts it, and goes straight to the TAP interface. Not ideal, not pretty, but it works. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12722): https://lists.fd.io/g/vpp-dev/message/12722 Mute This Topic: https://lists.fd.io/mt/30923501/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
