Public bug reported:
As of this moment, Neutron uses random-fully[1] PAT when performing NAT on L3
agent, meaning that source port is randomized for every outgoing connection.
This breaks some workflows for software that performs UDP hole punching, such
as Nebula[2] for example. There're other use cases where knowing post-NAT
source port value for an outgoing connection is desirable.
Neutron currently provides a `use_random_fully` setting[3] that controls use of
random-fully PAT, but it's global and affects cloud as a whole.
My proposal is to implement control over random-fully setting on a per-
Floating-IP basis.
I've already implemented this in a Devstack environment. My change required
updates in neutron(L3 agent code, database schema update(1 additional column of
a boolean type for floatingip table)), neutron-lib(API support, introduction of
a new validator type, etc), and openstackclient(CLI support).
In this implementation a new 'random_fully' setting belonging to a FIP can take
3 values: True, False, or None.
If True(API JSON: {"floatingip": {"random_fully": true}}), random-fully is
always enabled on a FIP, disregarding global `use_random_fully` setting.
If False(API JSON: {"floatingip": {"random_fully": false}}), random-fully is
always disabled on a FIP, disregarding global `use_random_fully` setting.
If None(API JSON: {"floatingip": {"random_fully": null}}), random-fully mode is
inherited from the global `use_random_fully` setting.
It works pretty much as expected, L3 agent updates iptables rules after API
call.
I'll be glad to share that code to expedite this feature implementation.
Short example output from a Devstack environment:
```
stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating
IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c
'Random Fully'
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| ID | Floating IP Address | Fixed IP Address
| Port | Router |
Status | Description | Random Fully |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11 | 10.0.0.21
| b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 |
ACTIVE | My FIP description 4 | None |
| 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207 | 10.0.0.39
| 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2 |
ACTIVE | | None |
| b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231 | 10.0.0.19
| 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 |
ACTIVE | | False |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
ubuntu@vlab007:~$ sudo ip netns exec
qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep
"neutron-l3-agent-float-snat -s"
-A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source 172.24.4.11
--random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source 172.24.4.207
--random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source 172.24.4.231
stack@vlab007:~/neutron$ openstack floating ip set --disable-random-fully
387fdc61-d386-4917-bd82-23055ebca273
stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating
IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c
'Random Fully'
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| ID | Floating IP Address | Fixed IP Address
| Port | Router |
Status | Description | Random Fully |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11 | 10.0.0.21
| b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 |
ACTIVE | My FIP description 4 | None |
| 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207 | 10.0.0.39
| 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2 |
ACTIVE | | False |
| b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231 | 10.0.0.19
| 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2 |
ACTIVE | | False |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
ubuntu@vlab007:~$ sudo ip netns exec
qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep
"neutron-l3-agent-float-snat -s"
-A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source 172.24.4.11
--random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source 172.24.4.207
-A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source 172.24.4.231
```
[1] https://ipset.netfilter.org/iptables-extensions.man.html
[2] https://github.com/slackhq/nebula
[3]
https://opendev.org/openstack/neutron/commit/553777c52d0fbfa18a1830d41ac018d2e1dc8282
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2083214
Title:
[RFE] control random-fully behavior on a per-FIP base
Status in neutron:
New
Bug description:
As of this moment, Neutron uses random-fully[1] PAT when performing NAT on L3
agent, meaning that source port is randomized for every outgoing connection.
This breaks some workflows for software that performs UDP hole punching, such
as Nebula[2] for example. There're other use cases where knowing post-NAT
source port value for an outgoing connection is desirable.
Neutron currently provides a `use_random_fully` setting[3] that controls use
of random-fully PAT, but it's global and affects cloud as a whole.
My proposal is to implement control over random-fully setting on a
per-Floating-IP basis.
I've already implemented this in a Devstack environment. My change required
updates in neutron(L3 agent code, database schema update(1 additional column of
a boolean type for floatingip table)), neutron-lib(API support, introduction of
a new validator type, etc), and openstackclient(CLI support).
In this implementation a new 'random_fully' setting belonging to a FIP can
take 3 values: True, False, or None.
If True(API JSON: {"floatingip": {"random_fully": true}}), random-fully is
always enabled on a FIP, disregarding global `use_random_fully` setting.
If False(API JSON: {"floatingip": {"random_fully": false}}), random-fully is
always disabled on a FIP, disregarding global `use_random_fully` setting.
If None(API JSON: {"floatingip": {"random_fully": null}}), random-fully mode
is inherited from the global `use_random_fully` setting.
It works pretty much as expected, L3 agent updates iptables rules after API
call.
I'll be glad to share that code to expedite this feature implementation.
Short example output from a Devstack environment:
```
stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating
IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c
'Random Fully'
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| ID | Floating IP Address | Fixed IP
Address | Port | Router
| Status | Description | Random Fully |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11 | 10.0.0.21
| b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2
| ACTIVE | My FIP description 4 | None |
| 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207 | 10.0.0.39
| 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2
| ACTIVE | | None |
| b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231 | 10.0.0.19
| 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2
| ACTIVE | | False |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
ubuntu@vlab007:~$ sudo ip netns exec
qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep
"neutron-l3-agent-float-snat -s"
-A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source
172.24.4.11 --random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source
172.24.4.207 --random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source
172.24.4.231
stack@vlab007:~/neutron$ openstack floating ip set --disable-random-fully
387fdc61-d386-4917-bd82-23055ebca273
stack@vlab007:~/neutron$ openstack floating ip list --long -c ID -c 'Floating
IP Address' -c 'Fixed IP Address' -c Port -c Router -c Status -c Description -c
'Random Fully'
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| ID | Floating IP Address | Fixed IP
Address | Port | Router
| Status | Description | Random Fully |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
| 0d97ed4c-15ae-4d01-a69c-ffd14e46ead0 | 172.24.4.11 | 10.0.0.21
| b5b29b90-350c-4d4e-8e27-35e76e9b8204 | 90364e18-a104-49b0-bbb5-41a516ea9bd2
| ACTIVE | My FIP description 4 | None |
| 387fdc61-d386-4917-bd82-23055ebca273 | 172.24.4.207 | 10.0.0.39
| 64413e38-d611-461d-b1e5-20e38d3795dd | 90364e18-a104-49b0-bbb5-41a516ea9bd2
| ACTIVE | | False |
| b47db56a-f944-43c2-ab16-271d3d809e20 | 172.24.4.231 | 10.0.0.19
| 97acadf3-7ed2-4dee-8e9c-db3b359c2319 | 90364e18-a104-49b0-bbb5-41a516ea9bd2
| ACTIVE | | False |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+--------+----------------------+--------------+
ubuntu@vlab007:~$ sudo ip netns exec
qrouter-90364e18-a104-49b0-bbb5-41a516ea9bd2 iptables-legacy-save -t nat|grep
"neutron-l3-agent-float-snat -s"
-A neutron-l3-agent-float-snat -s 10.0.0.21/32 -j SNAT --to-source
172.24.4.11 --random-fully
-A neutron-l3-agent-float-snat -s 10.0.0.39/32 -j SNAT --to-source
172.24.4.207
-A neutron-l3-agent-float-snat -s 10.0.0.19/32 -j SNAT --to-source
172.24.4.231
```
[1] https://ipset.netfilter.org/iptables-extensions.man.html
[2] https://github.com/slackhq/nebula
[3]
https://opendev.org/openstack/neutron/commit/553777c52d0fbfa18a1830d41ac018d2e1dc8282
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2083214/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
