Additionally I have now built and installed Linux-4.6.4 and run the debug
script.
Setup is unchanged from the previous one, except that Node 1 is now running
Linux-4.6.4.
Node 1: fd01:1b10:1000::1 is running 4.6.4
Debug log from Node 1, Receiver running 4.6.4:
==================Begin==========================
Linux node-0-001 4.6.4-node-test #2 Thu Jul 21 13:22:34 CEST 2016 i686 i686
i686 GNU/Linux
tcpdump -lni lan1 -c 5
14:21:49.737030 IP6 fd01:1b10:1000::3 > ff0e::1: ESP(spi=0x00000001,seq=0xff),
length 136
14:21:50.737092 IP6 fd01:1b10:1000::3 > ff0e::1: ESP(spi=0x00000001,seq=0x100),
length 136
14:21:51.737155 IP6 fd01:1b10:1000::3 > ff0e::1: ESP(spi=0x00000001,seq=0x101),
length 136
14:21:52.737217 IP6 fd01:1b10:1000::3 > ff0e::1: ESP(spi=0x00000001,seq=0x102),
length 136
14:21:53.737280 IP6 fd01:1b10:1000::3 > ff0e::1: ESP(spi=0x00000001,seq=0x103),
length 136
cat /proc/net/xfrm_stat
XfrmInError 0
XfrmInBufferError 0
XfrmInHdrError 0
XfrmInNoStates 0
XfrmInStateProtoError 0
XfrmInStateModeError 0
XfrmInStateSeqError 0
XfrmInStateExpired 0
XfrmInStateMismatch 0
XfrmInStateInvalid 0
XfrmInTmplMismatch 0
XfrmInNoPols 0
XfrmInPolBlock 0
XfrmInPolError 0
XfrmOutError 0
XfrmOutBundleGenError 0
XfrmOutBundleCheckError 0
XfrmOutNoStates 0
XfrmOutStateProtoError 0
XfrmOutStateModeError 0
XfrmOutStateSeqError 0
XfrmOutStateExpired 0
XfrmOutPolBlock 0
XfrmOutPolDead 0
XfrmOutPolError 0
XfrmFwdHdrError 0
XfrmOutStateInvalid 0
XfrmAcquireError 0
ip -s xfrm state
src fd01:1b10:1000::1 dst ff0e::1
proto esp spi 0x00000001(1) reqid 0(0x00000000) mode tunnel
replay-window 0 seq 0x00000000 flag (0x00000000)
enc cbc(aes) 0x7bef6ecaf06d29ef55b24aca6e19964b332e02e75be676a3 (192
bits)
sel src ::/0 dst ::/0 uid 0
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2016-07-21 14:18:08 use -
stats:
replay-window 0 replay 0 failed 0
ip -s xfrm policy
src ::/0 dst ff0e::1/128 uid 0
dir out action allow index 17 priority 2147483648 share any flag
(0x00000000)
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2016-07-21 14:18:08 use -
tmpl src fd01:1b10:1000::1 dst ff0e::1
proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
level required share any
enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
src ::/0 dst ff0e::1/128 uid 0
dir fwd action allow index 10 priority 2147483648 share any flag
(0x00000000)
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2016-07-21 14:18:08 use -
tmpl src fd01:1b10:1000::1 dst ff0e::1
proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
level required share any
enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
src ::/0 dst ff0e::1/128 uid 0
dir in action allow index 8 priority 2147483648 share any flag
(0x00000000)
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2016-07-21 14:18:08 use -
tmpl src fd01:1b10:1000::1 dst ff0e::1
proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
level required share any
enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
===================End===========================
Kind regards
Joerg
> -----Ursprüngliche Nachricht-----
> Von: Pommnitz Jörg
> Gesendet: Donnerstag, 21. Juli 2016 11:58
> An: 'Ilan Tayari'; 'Shanker Wang'
> Cc: netdev@vger.kernel.org
> Betreff: AW: IPv6 IPSec incompatibilities between 2.6.23 and 3.6.18 (and
> probably later)
>
> Ilan and Shanker,
> I did as you asked.
> Setup: Three nodes all connected to the same Ethernet hub.
> Node 1: fd01:1b10:1000::1 is running 3.18.36
> Node 2: fd01:1b10:1000::2 is running 2.6.23.12
> Node 3: fd01:1b10:1000::3 is running 3.18.36 and generates the traffic with
> the command "ping6 -I lan1 ff0e::1"
>
> All three nodes are configured with the following script (IP6ADDR adjusted
> for every node):
> ==================Begin==========================
> #!/bin/bash
>
> IP6ANYADDR=::/0
> IP6BCAST=ff0e::1
> KEY="0x7bef6ecaf06d29ef55b24aca6e19964b332e02e75be676a3"
> #IFNAME=radio
> IFNAME=lan1
> IP6ADDR=fd01:1b10:1000::X
> PREFIX6=64
> SPI=0x1
>
> ip link set dev ${IFNAME} up
> ip addr add ${IP6ADDR}/${PREFIX6} dev ${IFNAME}
> echo "flush; spdflush;" | setkey -c
> echo "add ${IP6ADDR} ${IP6BCAST} esp ${SPI} -m tunnel -E aes-cbc ${KEY};" |
> setkey -c
> echo "spdadd ${IP6ANYADDR} ${IP6BCAST} any -P in ipsec
> esp/tunnel/${IP6ADDR}-${IP6BCAST}/require;" | setkey -c
> echo "spdadd ${IP6ANYADDR} ${IP6BCAST} any -P out ipsec
> esp/tunnel/${IP6ADDR}-${IP6BCAST}/require;" | setkey -c
> ===================End===========================
>
> To capture the debug information I used the following script:
>
> ==================Begin==========================
> uname -a
> echo "tcpdump -lni lan1 -c 5"
> tcpdump -lni lan1 -c 5
> echo "cat /proc/net/xfrm_stat"
> cat /proc/net/xfrm_stat
> echo "ip -s xfrm state"
> ip -s xfrm state
> echo "ip -s xfrm policy"
> ip -s xfrm policy
> ===================End===========================
>
> Debug log from Node 1, Receiver running 3.6.18:
> ==================Begin==========================
> Linux node-0-001 3.18.36-node-5875 #1 Tue Jul 12 14:00:52 CEST 2016 i686
> i686 i686 GNU/Linux
> tcpdump -lni lan1 -c 5
> 12:03:09.580943 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x69), length 136
> 12:03:10.581006 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x6a), length 136
> 12:03:11.581068 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x6b), length 136
> 12:03:12.581131 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x6c), length 136
> 12:03:13.581193 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x6d), length 136
> cat /proc/net/xfrm_stat
> XfrmInError 0
> XfrmInBufferError 0
> XfrmInHdrError 0
> XfrmInNoStates 0
> XfrmInStateProtoError 0
> XfrmInStateModeError 0
> XfrmInStateSeqError 0
> XfrmInStateExpired 0
> XfrmInStateMismatch 0
> XfrmInStateInvalid 0
> XfrmInTmplMismatch 0
> XfrmInNoPols 0
> XfrmInPolBlock 0
> XfrmInPolError 0
> XfrmOutError 0
> XfrmOutBundleGenError 0
> XfrmOutBundleCheckError 0
> XfrmOutNoStates 0
> XfrmOutStateProtoError 0
> XfrmOutStateModeError 0
> XfrmOutStateSeqError 0
> XfrmOutStateExpired 0
> XfrmOutPolBlock 0
> XfrmOutPolDead 0
> XfrmOutPolError 0
> XfrmFwdHdrError 0
> XfrmOutStateInvalid 0
> XfrmAcquireError 0
> ip -s xfrm state
> src fd01:1b10:1000::1 dst ff0e::1
> proto esp spi 0x00000001(1) reqid 0(0x00000000) mode tunnel
> replay-window 0 seq 0x00000000 flag (0x00000000)
> enc cbc(aes) 0x7bef6ecaf06d29ef55b24aca6e19964b332e02e75be676a3 (192
> bits)
> sel src ::/0 dst ::/0 uid 0
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 12:01:44 use -
> stats:
> replay-window 0 replay 0 failed 0
> ip -s xfrm policy
> src ::/0 dst ff0e::1/128 uid 0
> dir out action allow index 41 priority 2147483648 share any flag
> (0x00000000)
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 12:01:44 use -
> tmpl src fd01:1b10:1000::1 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
> src ::/0 dst ff0e::1/128 uid 0
> dir fwd action allow index 34 priority 2147483648 share any flag
> (0x00000000)
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 12:01:44 use -
> tmpl src fd01:1b10:1000::1 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
> src ::/0 dst ff0e::1/128 uid 0
> dir in action allow index 24 priority 2147483648 share any flag
> (0x00000000)
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 12:01:44 use -
> tmpl src fd01:1b10:1000::1 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
> ===================End===========================
>
> Debug log from Node 2, Receiver running 2.6.23.12 (Note: 2.6.23 did not have
> /proc/net/xfrm_stat):
> ==================Begin==========================
> Linux node-0-001 2.6.23.12-node-4908 #3 Mon Dec 15 17:28:03 CET 2014 i686
> GNU/Linux
> tcpdump -lni lan1 -c 5
> 11:54:06.506723 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x75), length 136
> 11:54:06.506723 IP6 fd01:1b10:1000::3 > ff0e::1: ICMP6, echo request, seq
> 117, length 64
> 11:54:07.506696 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x76), length 136
> 11:54:07.506696 IP6 fd01:1b10:1000::3 > ff0e::1: ICMP6, echo request, seq
> 118, length 64
> 11:54:08.506731 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x77), length 136
> cat /proc/net/xfrm_stat
> ip -s xfrm state
> src fd01:1b10:1000::2 dst ff0e::1
> proto esp spi 0x00000001(1) reqid 0(0x00000000) mode tunnel
> replay-window 0 seq 0x00000000 flag (0x00000000)
> enc cbc(aes) 0x7bef6ecaf06d29ef55b24aca6e19964b332e02e75be676a3 (192
> bits)
> sel src ::/0 dst ::/0 uid 0
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 8320(bytes), 80(packets)
> add 2016-07-21 11:52:48 use 2016-07-21 11:52:49
> stats:
> replay-window 0 replay 0 failed 0
> ip -s xfrm policy
> src ::/0 dst ff0e::1/128 uid 0
> dir in action allow index 24 priority 2147483648 share any flag
> 0x00000000
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 11:52:48 use -
> tmpl src fd01:1b10:1000::2 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
> src ::/0 dst ff0e::1/128 uid 0
> dir out action allow index 41 priority 2147483648 share any flag
> 0x00000000
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 11:52:48 use -
> tmpl src fd01:1b10:1000::2 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
> src ::/0 dst ff0e::1/128 uid 0
> dir fwd action allow index 34 priority 2147483648 share any flag
> 0x00000000
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 11:52:48 use -
> tmpl src fd01:1b10:1000::2 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
> ===================End===========================
>
> Debug log from Node 3, Sender running 3.6.18:
> ==================Begin==========================
> Linux node-0-005 3.18.36-node-5875 #1 Tue Jul 12 14:00:52 CEST 2016 i686
> i686 i686 GNU/Linux
> tcpdump -lni lan1 -c 5
> 10:27:07.369198 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x5b), length 136
> 10:27:08.369261 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x5c), length 136
> 10:27:09.369323 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x5d), length 136
> 10:27:10.369386 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x5e), length 136
> 10:27:11.369448 IP6 fd01:1b10:1000::3 > ff0e::1:
> ESP(spi=0x00000001,seq=0x5f), length 136
> cat /proc/net/xfrm_stat
> XfrmInError 0
> XfrmInBufferError 0
> XfrmInHdrError 0
> XfrmInNoStates 0
> XfrmInStateProtoError 0
> XfrmInStateModeError 0
> XfrmInStateSeqError 0
> XfrmInStateExpired 0
> XfrmInStateMismatch 0
> XfrmInStateInvalid 0
> XfrmInTmplMismatch 0
> XfrmInNoPols 0
> XfrmInPolBlock 0
> XfrmInPolError 0
> XfrmOutError 0
> XfrmOutBundleGenError 0
> XfrmOutBundleCheckError 0
> XfrmOutNoStates 0
> XfrmOutStateProtoError 0
> XfrmOutStateModeError 0
> XfrmOutStateSeqError 0
> XfrmOutStateExpired 0
> XfrmOutPolBlock 0
> XfrmOutPolDead 0
> XfrmOutPolError 0
> XfrmFwdHdrError 0
> XfrmOutStateInvalid 0
> XfrmAcquireError 0
> ip -s xfrm state
> src fd01:1b10:1000::3 dst ff0e::1
> proto esp spi 0x00000001(1) reqid 0(0x00000000) mode tunnel
> replay-window 0 seq 0x00000000 flag (0x00000000)
> enc cbc(aes) 0x7bef6ecaf06d29ef55b24aca6e19964b332e02e75be676a3 (192
> bits)
> sel src ::/0 dst ::/0 uid 0
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 9880(bytes), 95(packets)
> add 2016-07-21 10:25:32 use 2016-07-21 10:25:37
> stats:
> replay-window 0 replay 0 failed 0
> ip -s xfrm policy
> src ::/0 dst ff0e::1/128 uid 0
> dir out action allow index 41 priority 2147483648 share any flag
> (0x00000000)
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 10:25:32 use 2016-07-21 10:27:11
> tmpl src fd01:1b10:1000::3 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
> src ::/0 dst ff0e::1/128 uid 0
> dir fwd action allow index 34 priority 2147483648 share any flag
> (0x00000000)
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 10:25:32 use -
> tmpl src fd01:1b10:1000::3 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
> src ::/0 dst ff0e::1/128 uid 0
> dir in action allow index 24 priority 2147483648 share any flag
> (0x00000000)
> lifetime config:
> limit: soft (INF)(bytes), hard (INF)(bytes)
> limit: soft (INF)(packets), hard (INF)(packets)
> expire add: soft 0(sec), hard 0(sec)
> expire use: soft 0(sec), hard 0(sec)
> lifetime current:
> 0(bytes), 0(packets)
> add 2016-07-21 10:25:32 use -
> tmpl src fd01:1b10:1000::3 dst ff0e::1
> proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
> level required share any
> enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
> ===================End===========================
>
> A difference between 2.6.23 and 3.6.18 is the value of the Masks in the
> policies:
>
> 2.6.23: enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
> 3.6.18: enc-mask 00000000 auth-mask 00000000 comp-mask 00000000
>
> I have no idea, what these masks mean. Is this important?
>
> Regards
> Joerg
>
> > -----Ursprüngliche Nachricht-----
> > Von: Ilan Tayari [mailto:il...@mellanox.com]
> > Gesendet: Dienstag, 19. Juli 2016 17:04
> > An: Pommnitz Jörg
> > Cc: netdev@vger.kernel.org
> > Betreff: RE: IPv6 IPSec incompatibilities between 2.6.23 and 3.6.18 (and
> > probably later)
> >
> > > On the receiving side (e.g. fd01:1b10:1000::1) I see the decrypted
> > packets with
> > > the 2.6.23 kernel:
> > > but NOT with the newer kernel:
> >
> > Hi Joerg,
> >
> > First steps to debug this would be:
> > cat /proc/net/xfrm_stat
> > ip -s xfrm state
> > ip -s xfrm policy
> >
> > First command will show some error accounting, which can point to the
> > culprit code.
> > Second and third command will show existing objects, and some statistics
> > like when the last packet was used with them.
> >
> > Last thing - for your safety you should keep those session keys private.
> >
> > Ilan.
________________________________
Industrieanlagen-Betriebsgesellschaft mbH
Sitz der Gesellschaft: Ottobrunn, Registergericht: Amtsgericht München, HRB 5499
Geschäftsführung: Prof. Dr.-Ing. Rudolf F. Schwarz
Vorsitzender des Aufsichtsrats: RA Engelbert Kupka MdL a.D.
