Ok, killing openvpn with -9 leaves the routes around, and particularly interesting are the following routes :
78.90.222.xx 10.255.255.0 UGHS 0 5841 epair0 => 78.90.222.xx/32 10.255.255.0 UGS 0 0 epair0 Now, if I do : route delete 78.90.222.xx 10.255.255.0 The route, with the H flag is deleted. If I repeat the command the second route is deleted as well, even if the second command specifies a netmask no panic. However the first delete command specifies the /32 mask like this : route delete 78.90.222.xx 10.255.255.0 255.255.255.255 Then I get "rtfree 2" kernel panic immediately. This seems to be happening as I'm manually installing static routes in the vnet jail for the VPN remote endpoints , however OpenVPN adds such routes too however differently, which results in two routing entries. For example : route add $IP $GW and route add $IP $GW 255.255.255.255 add to different route entries, one is /32 network, the other is a host route. --Nikolay On Wed, Jan 1, 2014 at 1:21 PM, Nikolay Denev <nde...@gmail.com> wrote: > On Wed, Jan 1, 2014 at 1:10 PM, Nikolay Denev <nde...@gmail.com> wrote: >> >> On Sun, Dec 22, 2013 at 1:10 PM, <freebsd-gnats-sub...@freebsd.org> wrote: >>> >>> Thank you very much for your problem report. >>> It has the internal identification `misc/185092'. >>> The individual assigned to look at your >>> report is: freebsd-bugs. >>> >>> You can access the state of your problem report at any time >>> via this link: >>> >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=185092 >>> >>> >Category: misc >>> >Responsible: freebsd-bugs >>> >Synopsis: panic: rtfree 2 (using RADIX_MPATH in a VNET jail) >>> >Arrival-Date: Sun Dec 22 13:10:00 UTC 2013 >> >> >> I'm trying to understand exactly what is happening here, and examining a >> core dump with kgdb I'm getting some output that confuses me : >> >> (kgdb) bt >> #0 doadump (textdump=-1011569920) at pcpu.h:233 >> #1 0xc06069b2 in kern_reboot (howto=260) at >> /usr/src/sys/kern/kern_shutdown.c:447 >> #2 0xc0606d0e in panic (fmt=<value optimized out>) at >> /usr/src/sys/kern/kern_shutdown.c:754 >> #3 0xc06de639 in rtfree (rt=<value optimized out>) at >> /usr/src/sys/net/route.c:464 >> #4 0xc06e188d in route_output (m=<value optimized out>) at >> /usr/src/sys/net/rtsock.c:951 >> #5 0xc06de18f in raw_usend (so=<value optimized out>, flags=0, m=<value >> optimized out>, nam=0x0, control=<value optimized out>, >> td=0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 >> #6 0xc066eca9 in sosend_generic (so=0xc3e9c1a8, uio=<value optimized >> out>, top=<value optimized out>, control=0x0, >> flags=<value optimized out>, td=<value optimized out>) at >> /usr/src/sys/kern/uipc_socket.c:1271 >> #7 0xc066efc7 in sosend (so=0xc3e9c1a8, addr=0x0, uio=0xd9b9cc10, >> top=0x0, control=0x0, flags=0, td=0xc3bd2000) >> at /usr/src/sys/kern/uipc_socket.c:1315 >> #8 0xc0654af4 in soo_write (fp=0xc3c0c818, uio=0xd9b9cc10, >> active_cred=0xc3f1dd00, flags=0, td=0xc3bd2000) >> at /usr/src/sys/kern/sys_socket.c:103 >> #9 0xc064c866 in dofilewrite (td=0xc3bd2000, fd=3, fp=0xc3c0c818, >> auio=0xd9b9cc10, offset=-1, flags=0) at file.h:303 >> #10 0xc064c566 in kern_writev (td=0xc3bd2000, fd=3, auio=<value optimized >> out>) at /usr/src/sys/kern/sys_generic.c:467 >> #11 0xc064c4bc in sys_write (td=<value optimized out>, uap=<value >> optimized out>) at /usr/src/sys/kern/sys_generic.c:382 >> #12 0xc08614d3 in syscall (frame=<value optimized out>) at >> subr_syscall.c:134 >> #13 0xc084cca1 in Xint0x80_syscall () at >> /usr/src/sys/i386/i386/exception.s:270 >> #14 0x281975b7 in ?? () >> Previous frame inner to this frame (corrupt stack?) >> Current language: auto; currently minimal >> (kgdb) fr 3 >> #3 0xc06de639 in rtfree (rt=<value optimized out>) at >> /usr/src/sys/net/route.c:464 >> 464 panic("rtfree 2"); >> (kgdb) print *rt >> $1 = {rt_nodes = {{rn_mklist = 0xc3b4ab30, rn_parent = 0x1, rn_bit = 0, >> rn_bmask = 0 '\0', rn_flags = 0 '\0', rn_u = {rn_leaf = { >> rn_Key = 0xc0882687 "shutdown_post_sync", rn_Mask = 0x1030000 >> <Address 0x1030000 out of bounds>, rn_Dupedkey = 0x0}, rn_node = { >> rn_Off = -1064819065, rn_L = 0x1030000, rn_R = 0x0}}}, >> {rn_mklist = 0x0, rn_parent = 0x4, rn_bit = -18048, rn_bmask = -94 '?', >> rn_flags = 195 '?', rn_u = {rn_leaf = {rn_Key = 0xc3a545e0 "", >> rn_Mask = 0xc3a4e440 " ??(???\020'", rn_Dupedkey = 0xc3a4e880}, >> rn_node = {rn_Off = -1012578848, rn_L = 0xc3a4e440, rn_R = >> 0xc3a4e880}}}}, rt_gateway = 0x74756873, rt_flags = 1853321060, >> rt_refcnt = 1936683103, rt_ifp = 0x79735f74, rt_ifa = 0x636e, rt_rmx = >> {rmx_mtu = 0, rmx_expire = 0, rmx_pksent = 0, rmx_weight = 0}, >> rt_fibnum = 0, rt_mtx = {lock_object = {lo_name = 0x0, lo_flags = 0, >> lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}} >> >> >> >> rn_Key with value of “shutdown_post_sync” ? >> >> It’s visible also in the raw_usend() frame: >> >> (kgdb) fr 5 >> #5 0xc06de18f in raw_usend (so=<value optimized out>, flags=0, m=<value >> optimized out>, nam=0x0, control=<value optimized out>, >> td=0xc3bd2000) at /usr/src/sys/net/raw_usrreq.c:238 >> 238 return ((*so->so_proto->pr_output)(m, so)); >> (kgdb) print *m >> $2 = {m_hdr = {mh_next = 0xc3b4ab30, mh_nextpkt = 0x1, mh_data = 0x0, >> mh_len = -1064819065, mh_type = 0, mh_flags = 66304, mh_pad = 0}, >> M_dat = {MH = {MH_pkthdr = {rcvif = 0x0, tags = {slh_first = 0x4}, len = >> -1012745856, flowid = 3282388448, >> csum_flags = 14097648373312316480, fibnum = 26739, cosqos = 117 >> 'u', rsstype = 116 't', l2hlen = 100 'd', l3hlen = 111 'o', >> l4hlen = 119 'w', l5hlen = 110 'n', PH_per = {eigth = "_post_sy", >> sixteen = {28767, 29551, 24436, 31091}, thirtytwo = {1936683103, >> 2037604212}, sixtyfour = {8751443454668533855}, unintptr = >> {1936683103}, ptr = 0x736f705f}, PH_loc = { >> eigth = "nc\000\000\000\000\000", sixteen = {25454, 0, 0, 0}, >> thirtytwo = {25454, 0}, sixtyfour = {25454}, unintptr = {25454}, >> ptr = 0x636e}}, MH_dat = {MH_ext = {ref_cnt = 0x0, ext_buf = >> 0x0, ext_size = 0, ext_type = 0, ext_flags = 0, ext_free = 0, >> ext_arg1 = 0x0, ext_arg2 = 0x0}, >> MH_databuf = '\0' <repeats 56 times>, "file", '\0' <repeats 20 >> times>, >> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\00000Y?", >> '\0' <repeats 12 times>, >> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000\000\000???", >> '\0' <repeats 23 times>}}, >> M_databuf = >> "\000\000\000\000\004\000\000\000\200????E??@??\200??shutdown_post_sync", >> '\0' <repeats 62 times>, "file", '\0' <repeats 20 times>, >> "\006\000\000\000\020\000\000\000??\215?\000\000C\001\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\00000Y?", >> '\0' <repeats 12 times>, >> "`2Y?\000\000\000\000\000\000\000\000T\211\223?\022\000\000\000\000\203??\000\000\000\000\000???", >> '\0' <repeats 23 times>}} >> >> >> This is 10.0-PRERELEASE r259547M (with applied the recent nd6_nbr.c rtfree >> patch, which I thought earlier might be the cause of the panics I'm >> seeing). >> >> The machine is Soekris Net5501-70 with this kernel config : >> >> cpu I586_CPU >> cpu I686_CPU >> ident MARS >> options CPU_GEODE >> options CPU_SOEKRIS >> >> options HZ=2000 >> options DEVICE_POLLING >> options BPF_JITTER >> >> makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols >> >> options SCHED_ULE # ULE scheduler >> options PREEMPTION # Enable kernel thread preemption >> options INET # InterNETworking >> options INET6 # IPv6 communications protocols >> options TCP_OFFLOAD # TCP offload >> options FFS # Berkeley Fast Filesystem >> options SOFTUPDATES # Enable FFS soft updates support >> options UFS_DIRHASH # Improve performance on big directories >> options PROCFS # Process filesystem (requires PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_PART_GPT # GUID Partition Tables. >> options GEOM_LABEL # Provides labelization >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >> options SCSI_DELAY=500 # Delay (in ms) before probing SCSI >> options KTRACE # ktrace(1) support >> options STACK # stack(9) support >> options SYSVSHM # SYSV-style shared memory >> options SYSVMSG # SYSV-style message queues >> options SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions >> options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) >> options CAPABILITY_MODE # Capsicum capability mode >> options CAPABILITIES # Capsicum capabilities >> options PROCDESC # Support for process descriptors >> options INCLUDE_CONFIG_FILE # Include this file in kernel >> >> # Debugging support. Always need this: >> options KDB # Enable kernel debugger support. >> options KDB_TRACE # Print a stack trace for a panic. >> options KDB_UNATTENDED >> >> options TEXTDUMP_PREFERRED >> options TEXTDUMP_VERBOSE >> >> device pci >> device ata # Legacy ATA/SATA controllers >> options ATA_STATIC_ID # Static device numbering >> >> # ATA/SCSI peripherals >> device scbus # SCSI bus (required for ATA/SCSI) >> device da # Direct Access (disks) >> device pass # Passthrough device (direct ATA/SCSI access) >> >> # Add suspend/resume support for the i8254. >> device pmtimer >> >> # Serial (COM) ports >> device uart # Generic UART driver >> >> device miibus # MII bus support >> device vr # VIA Rhine, Rhine II >> >> # Wireless NIC cards >> device wlan # 802.11 support >> options IEEE80211_DEBUG # enable debug msgs >> options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's >> options IEEE80211_SUPPORT_MESH # enable 802.11s draft support >> device wlan_wep # 802.11 WEP support >> device wlan_ccmp # 802.11 CCMP support >> device wlan_tkip # 802.11 TKIP support >> device wlan_amrr # AMRR transmit rate control algorithm >> device ath # Atheros NICs >> device ath_pci # Atheros pci/cardbus glue >> device ath_hal # pci/cardbus chip support >> options AH_SUPPORT_AR5416 # enable AR5416 tx/rx descriptors >> options AH_AR5416_INTERRUPT_MITIGATION # AR5416 interrupt mitigation >> options ATH_ENABLE_11N # Enable 802.11n support for AR5416 and later >> device ath_rate_sample # SampleRate tx rate control for ath >> >> # Pseudo devices. >> device loop # Network loopback >> device random # Entropy device >> device ether # Ethernet support >> device vlan # 802.1Q VLAN support >> device tun # Packet tunnel. >> device md # Memory "disks" >> device gif # IPv6 and IPv4 tunneling >> device gre >> device faith # IPv6-to-IPv4 relaying (translation) >> device firmware # firmware assist module >> device if_bridge >> >> options VIMAGE >> options ROUTETABLES=8 >> options RADIX_MPATH >> >> options SW_WATCHDOG >> >> device crypto >> device cryptodev >> device glxsb >> >> options BOOTVERBOSE=1 >> >> #device pf >> #device pflog >> #device pfsync >> device carp >> device enc >> device lagg >> device epair >> >> #options ALTQ >> #options ALTQ_CBQ >> #options ALTQ_RED >> #options ALTQ_RIO >> #options ALTQ_HFSC >> #options ALTQ_PRIQ >> >> options IPFIREWALL >> options IPFIREWALL_DEFAULT_TO_ACCEPT >> options IPFIREWALL_NAT >> options LIBALIAS >> options IPDIVERT >> options DUMMYNET >> >> device bpf # Berkeley packet filter >> >> # USB support >> options USB_DEBUG # enable debug msgs >> device uhci # UHCI PCI->USB interface >> device ohci # OHCI PCI->USB interface >> device ehci # EHCI PCI->USB interface (USB 2.0) >> device usb # USB Bus (required) >> device umass # Disks/Mass storage - Requires scbus and da >> >> >> Also src.conf and make.conf : >> >> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/src.conf >> WITHOUT_ACCT=yes >> WITHOUT_ACPI=yes >> WITHOUT_AMD=yes >> WITHOUT_APM=yes >> WITHOUT_ASSERT_DEBUG=yes >> WITHOUT_AT=yes >> WITHOUT_ATF=yes >> WITHOUT_ATM=yes >> WITHOUT_AUDIT=yes >> WITHOUT_BLUETOOTH=yes >> WITHOUT_CALENDAR=yes >> WITHOUT_CDDL=yes >> WITHOUT_CTM=yes >> WITHOUT_DICT=yes >> WITHOUT_FLOPPY=yes >> WITHOUT_GAMES=yes >> WITHOUT_HTML=yes >> WITHOUT_INFO=yes >> WITHOUT_IPFILTER=yes >> WITHOUT_IPX=yes >> #WITHOUT_KERNEL_SYMBOLS=yes >> WITHOUT_LEGACY_CONSOLE=yes >> WITHOUT_LOCALES=yes >> WITHOUT_LPR=yes >> WITHOUT_MAIL=yes >> WITHOUT_NDIS=yes >> WITHOUT_QUOTAS=yes >> WITHOUT_ROUTED=yes >> WITHOUT_SENDMAIL=yes >> WITH_SVN=yes >> WITHOUT_ZFS=yes >> >> root@vpn_vrf:[VNET(x)]:/usr/src/sys # cat /etc/make.conf >> CFLAGS=-O2 >> COPTFLAGS= -O -pipe >> CPUTYPE=geode >> KERNCONF=MARS >> NO_MODULES=yes >> BOOTWAIT=0 >> DOC_LANG=en_US.ISO8859-1 >> >> >> >> --Nikolay >> > > Also, originally I thought that the panic is when a multi path route is > being deleted, however again from the coredump it seems that the panic > happens when openvpn deletes the host route it installs for the remote > openvpn server pointed to the default gw (before openvpn installs the new > default gw pointing to the vpn tunnel) : > > (kgdb) fr 12 > (kgdb) x/12sb td->td_proc->p_args > 0xc4269780: "\001" > 0xc4269782: "" > 0xc4269783: "" > 0xc4269784: "B" > 0xc4269786: "" > 0xc4269787: "" > 0xc4269788: "/sbin/route" > 0xc4269794: "delete" > 0xc426979b: "-net" > 0xc42697a0: "78.90.222.xxx" > 0xc42697ad: "10.255.255.0" > 0xc42697ba: "255.255.255.255" > (kgdb) > > I'm trying to reproduce this on a VirtualBox instance now, however so far no > luck (no OpenVPN running, just adding and removing routes). > > > --Nikolay _______________________________________________ freebsd-bugs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"