[Fwd: Re: PR 4230 -- kernel panic on pf]

[Paolo Perrucci]

Hi all,

I need help to solve the problem described in the attached thread.
I didn't received any reply from Brad to my last mail (12 days ago).
Anyone on the list can help me.

Thank you
Paolo



-------- Messaggio Originale --------
Oggetto:        Re: PR 4230 -- kernel panic on pf
Data:   Wed, 08 Jun 2005 12:28:32 +0200
Da:     Paolo Perrucci <[EMAIL PROTECTED]>
A:      Brad <[EMAIL PROTECTED]>
Referenze:      <[EMAIL PROTECTED]>
<[EMAIL PROTECTED]>
<[EMAIL PROTECTED]>
<[EMAIL PROTECTED]>
<[EMAIL PROTECTED]>



Hi Brad,

while my primary firewall was running yesterday I re-installed and
reconfigured the secondary using the 3.7 release. I recompiled the
-stable kernel cvs source tree.

This morning I activated the pfsync device on the secondary firewall
without switching on the carp interfaces in order to try to sinchronize
the connection state table. After 1 minute the secondary firewall crashed:

panic: kernel diagnostic assertion "state->timeout < PFTM_MAX" failed:
file "/usr/src/sys/net/pf.c", line 887
ddb> trace
Debugger(d5dc9b74,d06d2000,d06d3df4,e38cf9c4,e38cf9c4) at Debugger+0x4
panic(d04dea80,d04affb7,d04d5c83,d04d5c9d,377) at panic+0x63
tablefull(d04affb7,d04d5c9d,377,d04d5c83,d05ab760) at tablefull
pf_purge_expired_src_nodes(e38cf9c4,ffffffff,d0563170,0,20) at
pf_purge_expired_src_nodes
pf_purge_expired_states(30,d01feb16,d0b68a80,d06d3e54,d01021b1) at
pf_purge_expired_states+0x33
pf_purge_timeout(d05ab72c,5305,10,d06d0058,0) at pf_purge_timeout+0x15
softclock(58,d05b0010,d06d0010,d0460010,d06d2000) at softclock+0x22c
Bad frame pointer: 0xd06d3e74

I hope you can help me.
Thanks
Paolo


Brad ha scritto:

At this point I don't see anything I could reasonably backport to
3.6. I would recommend looking at 3.7 and give me any feedback on
that.

On Fri, Jun 03, 2005 at 12:55:05PM +0200, Paolo Perrucci wrote:

Hi Brad,

bad news...

with the secondary firewall patched the primary fails with the same error.

So I patched the primary too and after a day of work it fails at the same assert pf.c:747.

What can I do to help you debug the issue ?
Paolo

Brad ha scritto:

This is a diff taken from CVS that was commited 8 months ago.
I will look into having this commited to -stable, though no
guarentee. But it is an "official" patch so to speak.

On Tue, May 31, 2005 at 04:06:54PM +0200, Paolo Perrucci wrote:

Hi Brad,

I applied your patches and now the backup firewall seems to be stable.

Can I consider your patches as definitives (can I apply the patches to the primary firewall too ?) ?

Thank you very much
Paolo

PS I: I rebuilt the binaries too

PS II: for Abraham, see PR numer 4230 here http://cvs.openbsd.org/query-pr.html to understand what's is happening... ;-)

Brad ha scritto:

Hi Paolo and Abraham,

Can you guys try this diff out against a 3.6-stable src tree?

Index: if_pfsync.c
===================================================================
RCS file: /cvs/src/sys/net/if_pfsync.c,v
retrieving revision 1.37
retrieving revision 1.38
diff -u -p -r1.37 -r1.38
--- if_pfsync.c 30 Aug 2004 07:44:28 -0000      1.37
+++ if_pfsync.c 17 Sep 2004 21:49:15 -0000      1.38
@@ -199,6 +199,7 @@ pfsync_insert_net_state(struct pfsync_st
        st->rule.ptr = r;
        /* XXX get pointers to nat_rule and anchor */

+       /* XXX when we have nat_rule/anchors, use STATE_INC_COUNTERS */
        r->states++;

        /* fill in the rest of the state entry */
@@ -227,6 +228,8 @@ pfsync_insert_net_state(struct pfsync_st

        if (pf_insert_state(kif, st)) {
                pfi_maybe_destroy(kif);

+ /* XXX when we have nat_rule/anchors, use STATE_DEC_COUNTERS */

+               r->states--;
                pool_put(&pf_state_pl, st);
                return (EINVAL);
        }
Index: pf.c
===================================================================
RCS file: /cvs/src/sys/net/pf.c,v
retrieving revision 1.457
retrieving revision 1.458
diff -u -p -r1.457 -r1.458
--- pf.c        11 Jul 2004 15:54:21 -0000      1.457
+++ pf.c        17 Sep 2004 21:49:15 -0000      1.458
@@ -250,6 +250,24 @@ struct pf_pool_limit pf_pool_limits[PF_L
        ((r)->rule_flag & PFRULE_GRBOUND) ? (k)->pfik_parent :               \
        (k)->pfik_parent->pfik_parent)

+#define STATE_INC_COUNTERS(s)                          \
+       do {                                            \
+               s->rule.ptr->states++;                    \
+               if (s->anchor.ptr != NULL)           \
+                       s->anchor.ptr->states++;  \
+               if (s->nat_rule.ptr != NULL)         \
+                       s->nat_rule.ptr->states++;        \
+       } while (0)
+
+#define STATE_DEC_COUNTERS(s)                          \
+       do {                                            \
+               if (s->nat_rule.ptr != NULL)         \
+                       s->nat_rule.ptr->states--;        \
+               if (s->anchor.ptr != NULL)           \
+                       s->anchor.ptr->states--;  \
+               s->rule.ptr->states--;                    \
+       } while (0)
+

static __inline int pf_src_compare(struct pf_src_node *, struct pf_src_node *);

static __inline int pf_state_compare_lan_ext(struct pf_state *,
        struct pf_state *);
@@ -2713,14 +2731,10 @@ cleanup:
                        return (PF_DROP);
                }
                bzero(s, sizeof(*s));
-               r->states++;
-               if (a != NULL)
-                       a->states++;
                s->rule.ptr = r;
                s->nat_rule.ptr = nr;
-               if (s->nat_rule.ptr != NULL)
-                       s->nat_rule.ptr->states++;
                s->anchor.ptr = a;
+               STATE_INC_COUNTERS(s);
                s->allow_opts = r->allow_opts;
                s->log = r->log & 2;
                s->proto = IPPROTO_TCP;
@@ -2799,6 +2813,7 @@ cleanup:
                    off, pd, th, &s->src, &s->dst)) {
                        REASON_SET(&reason, PFRES_MEMORY);
                        pf_src_tree_remove_state(s);
+                       STATE_DEC_COUNTERS(s);
                        pool_put(&pf_state_pl, s);
                        return (PF_DROP);
                }
@@ -2810,6 +2825,7 @@ cleanup:

("pf_normalize_tcp_stateful failed on first pkt"));

                        pf_normalize_tcp_cleanup(s);
                        pf_src_tree_remove_state(s);
+                       STATE_DEC_COUNTERS(s);
                        pool_put(&pf_state_pl, s);
                        return (PF_DROP);
                }
@@ -2817,6 +2833,7 @@ cleanup:
                        pf_normalize_tcp_cleanup(s);
                        REASON_SET(&reason, PFRES_MEMORY);
                        pf_src_tree_remove_state(s);
+                       STATE_DEC_COUNTERS(s);
                        pool_put(&pf_state_pl, s);
                        return (PF_DROP);
                } else
@@ -3054,14 +3071,10 @@ cleanup:
                        return (PF_DROP);
                }
                bzero(s, sizeof(*s));
-               r->states++;
-               if (a != NULL)
-                       a->states++;
                s->rule.ptr = r;
                s->nat_rule.ptr = nr;
-               if (s->nat_rule.ptr != NULL)
-                       s->nat_rule.ptr->states++;
                s->anchor.ptr = a;
+               STATE_INC_COUNTERS(s);
                s->allow_opts = r->allow_opts;
                s->log = r->log & 2;
                s->proto = IPPROTO_UDP;
@@ -3110,6 +3123,7 @@ cleanup:
                if (pf_insert_state(BOUND_IFACE(r, kif), s)) {
                        REASON_SET(&reason, PFRES_MEMORY);
                        pf_src_tree_remove_state(s);
+                       STATE_DEC_COUNTERS(s);
                        pool_put(&pf_state_pl, s);
                        return (PF_DROP);
                } else
@@ -3340,14 +3354,10 @@ cleanup:
                        return (PF_DROP);
                }
                bzero(s, sizeof(*s));
-               r->states++;
-               if (a != NULL)
-                       a->states++;
                s->rule.ptr = r;
                s->nat_rule.ptr = nr;
-               if (s->nat_rule.ptr != NULL)
-                       s->nat_rule.ptr->states++;
                s->anchor.ptr = a;
+               STATE_INC_COUNTERS(s);
                s->allow_opts = r->allow_opts;
                s->log = r->log & 2;
                s->proto = pd->proto;
@@ -3390,6 +3400,7 @@ cleanup:
                if (pf_insert_state(BOUND_IFACE(r, kif), s)) {
                        REASON_SET(&reason, PFRES_MEMORY);
                        pf_src_tree_remove_state(s);
+                       STATE_DEC_COUNTERS(s);
                        pool_put(&pf_state_pl, s);
                        return (PF_DROP);
                } else
@@ -3606,14 +3617,10 @@ cleanup:
                        return (PF_DROP);
                }
                bzero(s, sizeof(*s));
-               r->states++;
-               if (a != NULL)
-                       a->states++;
                s->rule.ptr = r;
                s->nat_rule.ptr = nr;
-               if (s->nat_rule.ptr != NULL)
-                       s->nat_rule.ptr->states++;
                s->anchor.ptr = a;
+               STATE_INC_COUNTERS(s);
                s->allow_opts = r->allow_opts;
                s->log = r->log & 2;
                s->proto = pd->proto;
@@ -3652,6 +3659,7 @@ cleanup:
                if (pf_insert_state(BOUND_IFACE(r, kif), s)) {
                        REASON_SET(&reason, PFRES_MEMORY);
                        pf_src_tree_remove_state(s);
+                       STATE_DEC_COUNTERS(s);
                        pool_put(&pf_state_pl, s);
                        return (PF_DROP);
                } else