Frode, Thanks for the help . We will try to apply these patches .
Out of interest why would OVS crash exactly on an hourly interval ? Is there some process that runs on an hourly interval ? Gav On Sat, Oct 28, 2023 at 7:56 AM Frode Nordahl <frode.nord...@canonical.com> wrote: > Hello, Gavin, > > This looks familiar and I wonder if it is fixed by [0]? It is also > available in 2.17.7 [1]. > > 0: > https://github.com/openvswitch/ovs/commit/106ef21860c935e5e0017a88bf42b94025c4e511 > 1: > https://github.com/openvswitch/ovs/commit/111c7be3193e15e2acf8af8ceb74a1177a95806d > > -- > Frode Nordahl > > lør. 28. okt. 2023, 00:50 skrev Gavin McKee via discuss < > ovs-discuss@openvswitch.org>: > >> Hi, >> >> Every hour we are seeing OVS crash with the following message >> >> ovs-vsctl --version >> ovs-vsctl (Open vSwitch) 2.17.6 >> DB Schema 8.3.0 >> >> 2023-10-27T00:33:47.163Z|00006|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T01:33:47.277Z|00003|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T02:33:47.391Z|00003|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T03:33:47.503Z|00004|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T04:33:47.613Z|00006|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T05:33:47.729Z|00003|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T06:33:47.841Z|00006|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T07:33:47.885Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T08:33:47.981Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T09:33:48.040Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T10:33:48.112Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T11:33:48.228Z|00001|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T12:33:48.249Z|00003|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T13:33:48.273Z|00001|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T14:33:48.315Z|00005|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T15:33:48.433Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T16:33:48.488Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T17:33:48.514Z|00002|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T18:33:48.630Z|00003|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T19:33:48.744Z|00001|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T20:33:48.857Z|00001|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T21:33:48.973Z|00001|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> 2023-10-27T22:33:49.093Z|00956|util(handler17)|EMER|../include/openvswitch/ofpbuf.h:194: >> assertion offset + size <= b->size failed in ofpbuf_at_assert() >> >> >> gdb shows the following >> >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock >> -vconsole:emer -vsyslog:err -vfi'. >> Program terminated with signal SIGABRT, Aborted. >> #0 0x00007f1b7e54854c in __pthread_kill_implementation () from >> /lib64/libc.so.6 >> [Current thread is 1 (Thread 0x7f1b6fbfd640 (LWP 1617989))] >> (gdb) bt >> #0 0x00007f1b7e54854c in __pthread_kill_implementation () from >> /lib64/libc.so.6 >> #1 0x00007f1b7e4fbce6 in raise () from /lib64/libc.so.6 >> #2 0x00007f1b7e4cf7f3 in abort () from /lib64/libc.so.6 >> #3 0x000055eb8de11d74 in ovs_abort_valist (err_no=err_no@entry=0, >> format=format@entry=0x55eb8e01ca18 "%s: assertion %s failed in %s()", >> args=args@entry=0x7f1b6fbad030) at ../lib/util.c:444 >> #4 0x000055eb8de1c335 in vlog_abort_valist (module_=<optimized out>, >> message=0x55eb8e01ca18 "%s: assertion %s failed in %s()", >> args=args@entry=0x7f1b6fbad030) at ../lib/vlog.c:1249 >> #5 0x000055eb8de1c3da in vlog_abort ( >> module=module@entry=0x55eb8e36a4a0 <this_module.lto_priv>, >> message=message@entry=0x55eb8e01ca18 "%s: assertion %s failed in %s()") >> at ../lib/vlog.c:1263 >> #6 0x000055eb8de1c41b in ovs_assert_failure (where=<optimized out>, >> function=<optimized out>, condition=<optimized out>) at ../lib/util.c:86 >> #7 0x000055eb8de555b8 in ofpbuf_at_assert (b=0x7f1b3001cb70, >> b=0x7f1b3001cb70, >> offset=16, size=20) at ../include/openvswitch/ofpbuf.h:194 >> #8 tc_replace_flower (id=<optimized out>, flower=<optimized out>) >> --Type <RET> for more, q to quit, c to continue without paging-- >> at ../lib/tc.c:3285 >> #9 0x000055eb8de4c5c8 in netdev_tc_flow_put (netdev=0x55eb968e5e50, >> match=<optimized out>, actions=<optimized out>, actions_len=<optimized >> out>, >> ufid=<optimized out>, info=<optimized out>, stats=<optimized out>) >> at ../lib/netdev-offload-tc.c:2181 >> #10 0x000055eb8de31078 in netdev_flow_put (stats=<optimized out>, >> info=0x7f1b6fbb5710, ufid=<optimized out>, act_len=<optimized out>, >> actions=<optimized out>, match=0x7f1b6fbb5910, netdev=0x55eb968e5e50) >> at ../lib/netdev-offload.c:259 >> #11 parse_flow_put (put=0x7f1b6fbb7c30, dpif=<optimized out>) >> at ../lib/dpif-netlink.c:2297 >> #12 try_send_to_netdev (op=0x7f1b6fbb7c28, dpif=<optimized out>) >> at ../lib/dpif-netlink.c:2384 >> #13 dpif_netlink_operate (dpif_=<optimized out>, ops=0x7f1b6fbb6800, >> n_ops=<optimized out>, offload_type=DPIF_OFFLOAD_AUTO) >> at ../lib/dpif-netlink.c:2455 >> #14 0x000055eb8dd3adda in dpif_operate (dpif=0x55eb968cab10, >> ops=0x7f1b6fbb6800, >> n_ops=3, offload_type=<optimized out>) at ../lib/dpif.c:1372 >> --Type <RET> for more, q to quit, c to continue without paging-- >> #15 0x000055eb8dce6a6b in handle_upcalls (n_upcalls=<optimized out>, >> upcalls=0x7f1b6fbdb430, udpif=0x55eb968983d0) >> at ../ofproto/ofproto-dpif-upcall.c:1677 >> #16 recv_upcalls (handler=handler@entry=0x55eb96931558) >> at ../ofproto/ofproto-dpif-upcall.c:915 >> #17 0x000055eb8dce6c44 in udpif_upcall_handler (arg=0x55eb96931558) >> at ../ofproto/ofproto-dpif-upcall.c:815 >> #18 0x000055eb8dddab13 in ovsthread_wrapper (aux_=<optimized out>) >> at ../lib/ovs-thread.c:422 >> #19 0x00007f1b7e546802 in start_thread () from /lib64/libc.so.6 >> #20 0x00007f1b7e4e6450 in clone3 () from /lib64/libc.so.6 >> (gdb) q >> >> Attached a core dump file >> >> >> Can anyone help here? >> >> core.ovs-vswitchd.984.e5ad44fdff9d4094a14ea0434... >> <https://drive.google.com/file/d/1G5OfoJygSHK_tfhLaFTn4kG_OjSBE1h5/view?usp=drive_web> >> >> >> _______________________________________________ >> discuss mailing list >> disc...@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
