Hello! I'm having a trouble with multiple routing tables (FreeBSD 7.2 release). Either I'm missing something in my setup or packets for daemons started with setfib are being sent out via the wrong interface.
What I'd like to implement: em0 - internal management network with ip address 10.2.5.2/24 and default route 10.2.5.1 em1 - public interface to be used in jail with ip address 195.208.245.229/27 and default route 195.208.245.225 Here are my routing tables: r61net-fbsdhost-1, / # setfib -0 netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 10.2.5.1 UGS 0 350 em0 10.2.5.0/24 link#1 UC 0 0 em0 10.2.5.1 00:1e:4a:b4:ea:c0 UHLW 2 0 em0 1182 127.0.0.1 127.0.0.1 UH 0 30 lo0 r61net-fbsdhost-1, / # setfib -1 netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 195.208.245.225 UGS 0 0 em1 195.208.245.224/27 link#2 UC 0 0 em1 195.208.245.225 link#2 UHLW 2 0 em1 Firewall: r61net-fbsdhost-1, / # ipfw show 00001 0 0 setfib 1 ip from any to any in recv em1 00010 0 0 count ip from any to any dst-port 2222 fib 0 00011 0 0 count ip from any 2222 to any fib 0 00012 0 0 count ip from any to any dst-port 2222 fib 1 00013 0 0 count ip from any 2222 to any fib 1 00100 0 0 allow ip from any to any via lo0 00200 0 0 deny ip from any to 127.0.0.0/8 00300 0 0 deny ip from 127.0.0.0/8 to any 65000 30 2648 allow ip from any to any 65535 0 0 deny ip from any to any With this setup almost everything works as I expect. For example ICMP echo requests and responses are being received and sent via em1. Both when ping runs on this host as "setfib 1 ping other_host" and when other host pings ip address of em1. Connection attempts (setfib 1 telnet other_host) are also being sent out of the right interface. But when it comes to the daemons I run into troubles. I use sshd for tests (have also tried other daemons with no luck): r61net-fbsdhost-1, / # setfib 1 /usr/sbin/sshd -o 'ListenAddress 195.208.245.229:2222' -D sshd is bound only to ip address of em1: r61net-fbsdhost-1, / # sockstat | grep 2222 root sshd 839 3 tcp4 195.208.245.229:2222 *:* While doing telnet 195.208.249.229 2222 from another host I got following packet traces: r61net-fbsdhost-1, / # tcpdump -i em0 port 2222 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes 17:39:34.872475 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: S 2590499299:2590499299(0) ack 3939022576 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3248254533 147282318> 17:39:34.902622 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 1:41(40) ack 1 win 8326 <nop,nop,timestamp 3248254533 147282318> 17:39:37.572271 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 41:60(19) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585> 17:39:37.572293 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: F 60:60(0) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585> 17:39:37.572986 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: . ack 8 win 8325 <nop,nop,timestamp 3248254593 147282585> r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes 17:39:34.872370 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: S 3939022575:3939022575(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147282318 0> 17:39:34.872803 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 2590499300 win 8326 <nop,nop,timestamp 147282318 3248254533> 17:39:35.002882 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 41 win 8326 <nop,nop,timestamp 147282331 3248254533> 17:39:37.571659 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: P 0:6(6) ack 41 win 8326 <nop,nop,timestamp 147282585 3248254533> 17:39:37.572923 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 61 win 8323 <nop,nop,timestamp 147282585 3248254593> 17:39:37.572945 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: F 6:6(0) ack 61 win 8326 <nop,nop,timestamp 147282585 3248254593> And firewall counters: r61net-fbsdhost-1, / # ipfw show 00001 6 326 setfib 1 ip from any to any in recv em1 00010 0 0 count ip from any to any dst-port 2222 fib 0 00011 5 327 count ip from any 2222 to any fib 0 00012 6 326 count ip from any to any dst-port 2222 fib 1 00013 0 0 count ip from any 2222 to any fib 1 00100 0 0 allow ip from any to any via lo0 00200 0 0 deny ip from any to 127.0.0.0/8 00300 0 0 deny ip from 127.0.0.0/8 to any 65000 60 5057 allow ip from any to any 65535 0 0 deny ip from any to any So the packets, generated by sshd are being sent out via em0 instead of em1. With ipfw add 2 setfib 1 ip from 195.208.245.229 to any outgoing packets are being tagged with correct fib, but still sent via em0. With ipfw add 60003 fwd 195.208.245.225 src-ip me src-ip 195.208.245.224/27 not dst-ip 195.208.245.224/27 first SYN packet from 195.208.245.229 is being sent correctly via em1, but I cannot see any further packets at all: r61net-fbsdhost-1, / # tcpdump -i em1 port 2222 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes 18:01:56.665341 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: S 2484180116:2484180116(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147415433 0> 18:01:56.665463 IP stat.r61.net.2222 > brain.cc.rsu.ru.50435: S 3905497961:3905497961(0) ack 2484180117 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3376909218 147415433> 18:01:56.665798 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: . ack 1 win 8326 <nop,nop,timestamp 147415433 3376909218> and no packets at em0. TCP connection establishes but no data packets come from daemon which is rather weird. I would appreciate any help with this issue. -- Oleg Sharoyko. Software and Network Engineer Computer Center of Rostov State University. _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
