Re: VNET jails not going away
Hi, I have the same/similar problem, it's quite annoying, with R10.1 and R10.2: jail -r shuts down the (vnet-)jail, jls does not list them anymore, but with jls -d, they are still there - apparently in a 'dying state' ? I cannot restart the jail as long as the old jail still appears in 'jls -d'. Really annoying. But they go away, eventually, sometimes after 5 minutes, sometimes after half an hour or so. Kaj. >Monday, February 22, 2016 1:41 PM UTC from "Bjoern A. Zeeb" >: > >Hi, > >has anyone else experienced VNET jails to not fully go away anymore on a >recent HEAD kernel (or possibly an older kernel)? > >I have test cases with which I can have them in DYING state (see jls -av) for >ever or at least more than half a day. I am in the process of trying to find >the cause but would be good to know if anyone else is experiencing this? > > >Thanks, >Bjoern > >Example (after more than 12 hours of jail -r ..): > ># jls -av > JID Hostname Path >Name State >CPUSetID >IP Address(es) > 1 left.example.net / >lef827DYING >18 > 2 center.example.net/ >mid827DYING >19 > 3 right.example.net / >right827 DYING >20 > 6 right.example.net / >right923 DYING >23 >___ >freebsd-jail@freebsd.org mailing list >https://lists.freebsd.org/mailman/listinfo/freebsd-jail >To unsubscribe, send any mail to " freebsd-jail-unsubscr...@freebsd.org " Best regards, Mail Lists mli...@mail.ru ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: VNET jails not going away
> On 24 Feb 2016, at 19:20 , Mail Lists wrote: > > > > Hi, > > I have the same/similar problem, it's quite annoying, with R10.1 and R10.2: > jail -r shuts down the (vnet-)jail, jls does not list them anymore, but with > jls -d, they are still there - > apparently in a 'dying state' ? > > I cannot restart the jail as long as the old jail still appears in 'jls -d'. > Really annoying. > > But they go away, eventually, sometimes after 5 minutes, sometimes after half > an hour or so. Yeah, if they eventually go away that’s fine. Hanging around for a bit can be expected. That’s, e.g., TCP timeouts from sockets, or similar. The problem is if they never go away. Bjoern ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Jail management
On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote: > I've been using FreeBSD jails (with ezjail) for many years and they work > very well. However I'm now reaching a critical mass (30+ jails) where I > want to be able to manage them in bulk more easily. > > In this environment, each jail runs just a single application, installed > from a package built using poudriere from a custom port. That package > depends on Java, so lots of other packages also get pulled in. That > application gets new versions roughly once every 4 weeks. The problems I > have right now are: > > * FreeBSD's packaging system doesn't understand the concept of installing > a particular package version, so all my scripts will by default upgrade > the application to the current version even if I don't want to. I can't > easily install a new jail at an old version. > > * It is hard to reproduce the environment exactly, matching the > application to the same version of Java that was available at the time of > deployment. Again I'm fighting against the pkg system which always wants > the latest version. The package system *could* handle this, but it doesn't fit our design. We aren't like RedHat/Debian where we "freeze" packages at a certain version at the OS release and then backport only changes. With that method different versions of packages will just work with everything else in the system. With FreeBSD's ports system it's really a rolling release as the entire ports tree moves together. Mixing packages build from different checkouts of the ports tree is dangerous and not guaranteed to work. You may be better served with the quarterly branch of the ports tree where things are mostly static for 4 months at a time. Only security and major bugfixes trickle in. Software will get upgraded to fix security issues -- the fixes are not "backported" as that overhead is unmaintainable and is even being criticized in the Linux world these days. The entire software ecosystem needs to stay nimble if we are to stay secure. Upstream projects need to either learn to not break functionality or to provide Long Term Support releases that people can rely on. We definitely have growing pains. > > * For failover I want each jail reproduced exactly on another host, or at > least a snapshot which could be sent to another host within a few > seconds. The jails are quite small (< 500Mb). Most of that is just the > openjdk pkg. > > > As I understand, ezjail doesn't support multiple base jails. If it did, > then I could simply install the application (and packages) to the base > jail and have versions of the base. Then by shutting down a jail, > switching the base to the new version and starting up, everything would > upgrade easily. Even better would be some concept of hierarchy with > customer_jail sitting on top of base_version_1.0 which in turn sits on > top of base_jail. > > Would I need to abandon ezjail and be able to build all the above myself > with a combination of nullfs (basejail) and unionfs (intermediate > versioned jail)? Does unionfs now work with ZFS? > > > Alternatively I could simply use zfs clones to deploy a new version of > the application by destroying the whole jail and replacing it with a new > one. I'd need to then script (I use saltstack) deploying the 2-3 config > files which are different in each jail. > > > > Thoughts? What seems like a more robust long term approach to jail > management? > > I don't use ezjail. It doesn't upgrade well, and changes to the base jail require you stop all your jails. FreeBSD fat jails are so small (300MB?) it's not worth it in my opinion. I simply wrote a shell script to create fat jails and another script to handle updating them all. They're all treated like full servers/VMs, and configs/roles are managed with Ansible/Salt/etc. -- Mark Felder ports-secteam member f...@freebsd.org ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Jail management
On 25/02/2016 7:52am, Mark Felder wrote: > > > On Sun, Feb 21, 2016, at 19:13, Aristedes Maniatis wrote: >> * It is hard to reproduce the environment exactly, matching the >> application to the same version of Java that was available at the time of >> deployment. Again I'm fighting against the pkg system which always wants >> the latest version. > > The package system *could* handle this, but it doesn't fit our design. > We aren't like RedHat/Debian where we "freeze" packages at a certain > version at the OS release and then backport only changes. With that > method different versions of packages will just work with everything > else in the system. With FreeBSD's ports system it's really a rolling > release as the entire ports tree moves together. Mixing packages build > from different checkouts of the ports tree is dangerous and not > guaranteed to work. Hi Mark Yes, that makes sense and I've frequently struggled in Linux systems to get all the bits to match each other properly. So the FreeBSD solution here works. But for me, where I use poudriere and roll my own custom packages, versioning becomes complicated. I'd need to either create a new poudriere jail for every release of my software, or go down the path of snap-shotting the entire jail (I'm choosing the second). > I don't use ezjail. It doesn't upgrade well, and changes to the base > jail require you stop all your jails. FreeBSD fat jails are so small > (300MB?) it's not worth it in my opinion. I simply wrote a shell script > to create fat jails and another script to handle updating them all. > They're all treated like full servers/VMs, and configs/roles are managed > with Ansible/Salt/etc. That's a good point. And after all the excellent advice here, this is probably what I'll do: 1. Discard salt-minions inside every jail. That's become more trouble to look after as the number of jails grows. That's also really tricky to handle in salt when I want to fail over jails to another host. Since then every jail is running in two places and that confuses salt. 2. Create a single master/template jail. That might include the basejail or perhaps I'll keep the nullfs thing which works fine. 3. With every release of my software, upgrade that jail using 'pkg upgrade', test and 'zfs snapshot pool/template@v8.10'. I'll keep accumulating snapshots for every release, bundling up the changes to my software plus all the dependency packages and config. 4. When I want to upgrade a customer jail, I stop the jail and: # zfs destroy pool/customerJail # zfs clone pool/template@v8.10 pool/customerJail I'll have salt help with automating this of course. By using the zfs clone command, even the 300Mb of FreeBSD userland takes zero bytes to clone. These commands should really take less than a second to execute, so the upgrade speed is great. 5. Then salt will write out the appropriate customer specific configuration into the newly cloned jail (for us that's just 2-3 files) 6. Start jail Where having no basejail falls down is when the host system goes from FreeBSD 10 to 11, then I'll need to upgrade every jail but I'll have no way to go backwards to an older version once this is done since the old snapshot will include the old userland. That's probably not a problem for something that is rare. Also, all the above works only because we use logstash and therefore don't care about losing all the logs inside each jail with every upgrade. I guess syslog would do the same thing for other people. I hope this little summary helps other people facing similar challenges. Ari -- --> Aristedes Maniatis CEO, ish https://www.ish.com.au GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A signature.asc Description: OpenPGP digital signature
