Public bug reported: After upgrading from 6.8.0-1023.25~22.04.1 to 6.8.0-1025.27~22.04.1 in some Kubernetes worker nodes, our kube-proxy started failing with these logs: ``` 2025-03-06T00:55:59.784251404Z stderr F E0306 00:55:59.784126 1 proxier.go:1432] "Failed to execute iptables-restore" err=< 2025-03-06T00:55:59.784266755Z stderr F exit status 2: Warning: Extension MARK revision 0 not supported, missing kernel module? 2025-03-06T00:55:59.784269955Z stderr F ip6tables-restore v1.8.9 (nf_tables): unknown option "--xor-mark" 2025-03-06T00:55:59.784272495Z stderr F Error occurred at line: 11 2025-03-06T00:55:59.784274584Z stderr F Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information. 2025-03-06T00:55:59.784277344Z stderr F > rules="*nat\n:KUBE-SERVICES - [0:0]\n:KUBE-POSTROUTING - [0:0]\n:KUBE-NODE-PORT - [0:0]\n:KUBE-LOAD-BALANCER - [0:0]\n:KUBE-MARK-MASQ - [0:0]\n-A KUBE-SERVICES -s ::1/128 -j RETURN\n-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT\n-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ\n-A KUBE-POSTROUTING -m mark ! --mark 0x00004000/0x00004000 -j RETURN\n-A KUBE-POSTROUTING -j MARK --xor-mark 0x00004000\n-A KUBE-POSTROUTING -m comment --comment \"kubernetes service traffic requiring SNAT\" -j MASQUERADE --random-fully\n-A KUBE-MARK-MASQ -j MARK --or-mark 0x00004000\nCOMMIT\n*filter\n:KUBE-FORWARD - [0:0]\n:KUBE-NODE-PORT - [0:0]\n:KUBE-PROXY-FIREWALL - [0:0]\n:KUBE-SOURCE-RANGES-FIREWALL - [0:0]\n:KUBE-IPVS-FILTER - [0:0]\n:KUBE-IPVS-OUT-FILTER - [0:0]\n-A KUBE-SOURCE-RANGES-FIREWALL -j DROP\n-A KUBE-FORWARD -m comment --comment \"kubernetes forwarding rules\" -m mark --mark 0x00004000/0x00004000 -j ACCEPT\n-A KUBE-FORWARD -m comment --comment \"kubernetes forwarding conntrack rule\" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT\n-A KUBE-NODE-PORT -m comment --comment \"Kubernetes health check node port\" -m set --match-set KUBE-6-HEALTH-CHECK-NODE-PORT dst -j ACCEPT\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-LOAD-BALANCER dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-CLUSTER-IP dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-EXTERNAL-IP dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-EXTERNAL-IP-LOCAL dst,dst -j RETURN\n-A KUBE-IPVS-FILTER -m set --match-set KUBE-6-HEALTH-CHECK-NODE-PORT dst -j RETURN\n-A KUBE-IPVS-FILTER -m conntrack --ctstate NEW -m set --match-set KUBE-6-IPVS-IPS dst -j REJECT\nCOMMIT\n" ```
This error about "--xor-mark" being unknown looks very similar to what was reported in https://github.com/bottlerocket- os/bottlerocket/issues/4295. That issue mentioned that https://web.git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.15.170&id=90baa455aa7e099152898cfa5eb3928d6152da12 should fix it. I verified that ip6tables-restore works fine on 6.8.0-1024.26~22.04.1 (without changing any Kubernetes-related package), so I think this issue is specific to 1025. That fix commit says that it: Fixes: 0bfcb7b71e73 ("netfilter: xtables: avoid NFPROTO_UNSPEC where needed"). It looks like the buggy commit is the latest commit to touch xt_mark.c in the jammy aws-6.8-next branch: https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux- aws/+git/jammy/log/net/netfilter/xt_mark.c?h=aws-6.8-next Is there any way to fast-track the fix commit into linux-aws? Will 1025 (without the fix) get promoted from jammy-proposed to jammy? I'm not familiar with the process by which commits are merged into linux-aws and published, so I apologize if this is not the right place for this, and would appreciate pointers to the right place to ask. ** Affects: linux-aws (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2101914 Title: nt_tables reporting unknown option "--xor-mark" in 6.8.0-1025.27 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2101914/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
