On 2022-12-18 00:01, Zhenlei Huang wrote:
I'm currently working on route nexthop caching feature for tunneling
interfaces such as
if_gif, if_gre, if_vxlan, and potentially if_wg. I encounter a nasty
bug related to VNET lifecycle.
More preciously I'd like to call `rib_unsubscribe()` to unsubscribe
route event when the interface
tunnel is deleted (gif_delete_tunnel).
While on VNET shutting down, VNET SYSUNINIT was called and the routing
vnet subsystem
is destroyed before the interface going down and hence cause
pagefault. I do not want to check
`vnet.vnet_shutdown` state as it looks messed up.
I'm recently reviewing the life cycles of prison and get some
inspirations.
When the jail / prison is submitted to destroy ( by jail_remove
syscall ) then SIGKILL is sent to
the prison's processes. I think it is correct order to destroy jail /
prison. To summarize, the life cycle
of jail / prison is:
on jail create: PRISON_STATE_INVALID -> create VNET ->
PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses,
routing, etc. -> create / attach (network) processes
on jail destroy: jexec kill processes (1) by user -> mark it as
PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) ->
destroy VNET (if prison pr_ref go to the last one) -> DYED
The (2) is a cleanup by kernel as (1) is possible not done by user.
So it comes the idea about the life cycle of VNET.
While on jail destroy, the network resources are cleaned up by
vnet_destroy ( SYSUNINIT ). Then the
order of SYSUNINIT of network components is hacking as circular
network resource dependency is possible.
For example the routing table entries (nhop) have reference of ifnet,
and ifnet have reference to route nhop (cache), as
I encountered.
Just like the cleanup processes by kernel, we can introduce a new
stage `vnet_shutdown` that clean up network resources.
When jail / prison is going to dye, after kernel has cleaned up
processes it call `vnet_shutdown` to cleanup network resources,
then vnet_destroy will go smoothly as there's no circular network
resource dependency right now.
The life cycle of prison becomes:
on jail create: PRISON_STATE_INVALID -> create VNET ->
PRISON_STATE_ALIVE -> setup network resources, ifnet, if addresses,
routing, etc. -> create / attach (network) processes
on jail destroy: jexec kill processes (1) by user -> mark it as
PRISON_STATE_DYING -> send SIGKILL to processes by kernel (2) ->
vnet_shutdown cleanup network resources -> destroy VNET (if prison
pr_ref go to the last one) -> DYED
This idea is still unmature and I hope to hear more voices about it.
This is absolutely the direction things need to go. Vnet isn't the
only thing that can have these problems, though it's been the biggest
offender. There could also be cycles that involve more than one
subsystem, which could be helped by broad application of this idea.
There's a function in kern_jail.c ready for this: prison_cleanup.
It's called in "mark PRISON_STATE_DYING" stage of things. That's
before the "send SIGKILL" part of your sequence, but otherwise fits.
- Jamie
