Hi!
As mentioned in a previous post
(https://sourceforge.net/p/shorewall/mailman/message/36599490/) we are
experimenting with Docker 18.09 in swarm mode and Shorewall 5.2.3, and the
current integration doesn't work.
We decided to try to quickly implement a patch, which can be found here
https://sourceforge.net/u/nikm/shorewall/ci/master/tree/. We tried to
extend the existing integration to reflect the current chain and
interface configuration utilized in Docker in swarm mode. In general it
would be nice if we could get some feedback on that sometime to be able
to prepare a merge request if wanted, but there is one thing that would
be especially interesting. Docker 18.09 in swarm mode applies some sort
of firewall strategy and it would be interesting where you'd see the
ideal Shorewall integration.
Docker iptables usage:
1) In the nat table Docker adds the two chains DOCKER and
DOCKER-INGRESS. Jump rules added to the PREROUTING chain direct to those
chains on dst-type LOCAL.
PREROUTING chain:
DOCKER-INGRESS all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
DOCKER all -- * * 0.0.0.0/0
0.0.0.0/0 ADDRTYPE match dst-type LOCAL
In those chains Docker dynamically adds DNAT rules to direct the
registered task ports to the concerned interface. Which chain it uses
depends on the utilization of the routing-mesh.
Sample DOCKER chain:
RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
RETURN all -- docker_gwbridge * 0.0.0.0/0 0.0.0.0/0
DNAT tcp -- !docker0 * 0.0.0.0/0
0.0.0.0/0 tcp dpt:10001 to:172.17.0.2:443
Sample DOCKER-INGRESS chain:
DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:10002 to:172.18.0.2:10002
RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
2) The DNAT rules above don't direct to local interfaces directly
(docker0: inet 172.17.0.1/16, docker_gwbridge: inet 172.18.0.1/16) so
that takes us to the FORWARD chain in the filter table.
In the filter table Docker adds five chains DOCKER-USER,
DOCKER-ISOLATION-STAGE-1, DOCKER-ISOLATION-STAGE-2, DOCKER,
DOCKER-INGRESS. Jump rules in the FORWARD chain distribute
traffic to those.
FORWARD chain:
DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0
0.0.0.0/0
ACCEPT all -- * docker0
0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- * docker0
0.0.0.0/0 0.0.0.0/0
ACCEPT all -- docker0 !docker0
0.0.0.0/0 0.0.0.0/0
ACCEPT all -- docker0 docker0
0.0.0.0/0 0.0.0.0/0
ACCEPT all -- * docker_gwbridge 0.0.0.0/0
0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- * docker_gwbridge 0.0.0.0/0
0.0.0.0/0
ACCEPT all -- docker_gwbridge !docker_gwbridge
0.0.0.0/0 0.0.0.0/0
DROP all -- docker_gwbridge docker_gwbridge
0.0.0.0/0 0.0.0.0/0
We can see here that all traffic passes the DOCKER-USER chain to
allow custom rules. Afterwards all traffic passes the DOCKER-INGRESS
chain which applies ACCEPT rules for NEW connections
as well as RELATED,ESTABLISHED connections for back traffic to
registered task ports utilizing the routing-mesh.
Sample DOCKER-INGRESS chain:
ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
tcp dpt:10002
ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED tcp spt:10002
RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 are there for
docker network isolation. And finally the DOCKER chain applies ACCEPT
rules for registered task ports that are running locally.
Sample DOCKER chain:
ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2
tcp dpt:10001
Possible solutions:
So for now we are wondering where the best place for integrating
Shorewall could be. One thing to keep in mind is that rules in
DOCKER/DOCKER-INGRESS chain get applied dynamically on scheduling of tasks.
1) As a first approach we decided to simply remove the jump rules to
DOCKER/DOCKER-INGRESS chains from the FORWARD chain, as those rules
pretty much seem to be what we do want to set explicitly, and doing so
allows package flow to directly enter the Shorewall logic. As those
rules are created explicitly (see
https://sourceforge.net/u/nikm/shorewall/ci/master/tree/Shorewall/Perl/Shorewall/Misc.pm[681-ff])
in the Shorewall integration for Docker, this is also easy to do. (We
saw that in
https://sourceforge.net/u/nikm/shorewall/ci/master/tree/Shorewall/Perl/Shorewall/Chains.pm[8720-ff]
there is some sort of export logic for forward rules which isn't used in
the end. Is there something missing?)
But there might also be a scenario where we add a custom rule to the
DOCKER-USER chain that would allow hooking in with the Shorewall logic.
Any thoughts are greatly appreciated.
Thanks and best regards,
Niko
_______________________________________________
Shorewall-users mailing list
Shorewall-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/shorewall-users
