Jail management
I've been using FreeBSD jails (with ezjail) for many years and they work very well. However I'm now reaching a critical mass (30+ jails) where I want to be able to manage them in bulk more easily. In this environment, each jail runs just a single application, installed from a package built using poudriere from a custom port. That package depends on Java, so lots of other packages also get pulled in. That application gets new versions roughly once every 4 weeks. The problems I have right now are: * FreeBSD's packaging system doesn't understand the concept of installing a particular package version, so all my scripts will by default upgrade the application to the current version even if I don't want to. I can't easily install a new jail at an old version. * It is hard to reproduce the environment exactly, matching the application to the same version of Java that was available at the time of deployment. Again I'm fighting against the pkg system which always wants the latest version. * For failover I want each jail reproduced exactly on another host, or at least a snapshot which could be sent to another host within a few seconds. The jails are quite small (< 500Mb). Most of that is just the openjdk pkg. As I understand, ezjail doesn't support multiple base jails. If it did, then I could simply install the application (and packages) to the base jail and have versions of the base. Then by shutting down a jail, switching the base to the new version and starting up, everything would upgrade easily. Even better would be some concept of hierarchy with customer_jail sitting on top of base_version_1.0 which in turn sits on top of base_jail. Would I need to abandon ezjail and be able to build all the above myself with a combination of nullfs (basejail) and unionfs (intermediate versioned jail)? Does unionfs now work with ZFS? Alternatively I could simply use zfs clones to deploy a new version of the application by destroying the whole jail and replacing it with a new one. I'd need to then script (I use saltstack) deploying the 2-3 config files which are different in each jail. Thoughts? What seems like a more robust long term approach to jail management? Thanks Ari -- --> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A signature.asc Description: OpenPGP digital signature
Re: Jail management
Checkout qjail from your description I think it will do what you want. On 22/02/2016 01:13, Aristedes Maniatis wrote: I've been using FreeBSD jails (with ezjail) for many years and they work very well. However I'm now reaching a critical mass (30+ jails) where I want to be able to manage them in bulk more easily. In this environment, each jail runs just a single application, installed from a package built using poudriere from a custom port. That package depends on Java, so lots of other packages also get pulled in. That application gets new versions roughly once every 4 weeks. The problems I have right now are: * FreeBSD's packaging system doesn't understand the concept of installing a particular package version, so all my scripts will by default upgrade the application to the current version even if I don't want to. I can't easily install a new jail at an old version. * It is hard to reproduce the environment exactly, matching the application to the same version of Java that was available at the time of deployment. Again I'm fighting against the pkg system which always wants the latest version. * For failover I want each jail reproduced exactly on another host, or at least a snapshot which could be sent to another host within a few seconds. The jails are quite small (< 500Mb). Most of that is just the openjdk pkg. As I understand, ezjail doesn't support multiple base jails. If it did, then I could simply install the application (and packages) to the base jail and have versions of the base. Then by shutting down a jail, switching the base to the new version and starting up, everything would upgrade easily. Even better would be some concept of hierarchy with customer_jail sitting on top of base_version_1.0 which in turn sits on top of base_jail. Would I need to abandon ezjail and be able to build all the above myself with a combination of nullfs (basejail) and unionfs (intermediate versioned jail)? Does unionfs now work with ZFS? Alternatively I could simply use zfs clones to deploy a new version of the application by destroying the whole jail and replacing it with a new one. I'd need to then script (I use saltstack) deploying the 2-3 config files which are different in each jail. Thoughts? What seems like a more robust long term approach to jail management? Thanks Ari ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Jail management
> On 22 Feb 2016, at 14:13, Aristedes Maniatis wrote: > > Thoughts? What seems like a more robust long term approach to jail management? Take a look at bsdploy https://github.com/ployground/bsdploy or just come and ask ezjails author. ;) Also unionfs does not work very stable. erdgeist signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Jail management
On 22/02/2016 12:57pm, erdgeist wrote: > >> On 22 Feb 2016, at 14:13, Aristedes Maniatis wrote: >> >> Thoughts? What seems like a more robust long term approach to jail >> management? > > Take a look at bsdploy https://github.com/ployground/bsdploy or just come and > ask ezjails author. ;) Hello there! Thanks for ezjail: a very useful tool for avoiding the pain of setting up nullfs and friends. However I think that bsdploy is orthogonal to my problem. I'm already embedded in saltstack, so moving to ansible doesn't solve any problems for me. And I can't see how it solves the pkg versioning problem any better. That's why I was thinking to move to a snapshot clone/restore approach to jail management. But that idea butts up against ezjail's assumptions. > Also unionfs does not work very stable. OK, I'll cross that option off my list. That then leaves just ZFS clone as the way to create a reproducible and deployable jail environment with the correct (old) package versions. I did have another idea: create a poudriere environment for each version of the app and switch /usr/local/etc/pkg/repos/my.conf each time. But that seems awkward and still very hard to go back in time and apply small fixes to an old deployed version. Have I just now outgrown ezjail and should set off on my own? I'm afraid of how I'd go about upgrading the basejail for new FreeBSD host versions without your tool :-) Thanks Ari -- --> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A signature.asc Description: OpenPGP digital signature
Re: Jail management
One of the solutions I have found to the version issue is to build my
own package repo. I build the packages the way I want, and then upload
them to my own package repo (which is just another jail running
thttpd). I also keep a jail running with the ports tree frozen at the
versions I am using for production.
Add the following to /usr/local/etc/pkg.conf
repos_dir: [
"/usr/local/etc/pkg/repos",
"/etc/pkg",
]
The tells pkg to look in your private repo first
Then, create /usr/local/etc/pkg/repos/private.conf
private: {
url: "pkg+http://pkg.ssimicro.com/${ABI}/latest";,
enabled: true,
signature_type: "PUBKEY",
PUBKEY: "/usr/local/etc/pkg/repos/ssi.pub",
mirror_type: "srv"
}
Note: you also need to create a public/private key pair for this using
openssl. I don't recall the specifics though, but it looks like a
pretty standard self-signed key/cert pair.
The private key is stored on the repo and used to sign the packages when
you add initialize the repo:
pkg repo /home/pkg/repo/freebsd:10:x86:64/latest /home/pkg/repo.key
Best,
-Markham
On 2016-02-21 6:13 PM, Aristedes Maniatis wrote:
> I've been using FreeBSD jails (with ezjail) for many years and they work very
> well. However I'm now reaching a critical mass (30+ jails) where I want to be
> able to manage them in bulk more easily.
>
> In this environment, each jail runs just a single application, installed from
> a package built using poudriere from a custom port. That package depends on
> Java, so lots of other packages also get pulled in. That application gets new
> versions roughly once every 4 weeks. The problems I have right now are:
>
> * FreeBSD's packaging system doesn't understand the concept of installing a
> particular package version, so all my scripts will by default upgrade the
> application to the current version even if I don't want to. I can't easily
> install a new jail at an old version.
>
> * It is hard to reproduce the environment exactly, matching the application
> to the same version of Java that was available at the time of deployment.
> Again I'm fighting against the pkg system which always wants the latest
> version.
>
> * For failover I want each jail reproduced exactly on another host, or at
> least a snapshot which could be sent to another host within a few seconds.
> The jails are quite small (< 500Mb). Most of that is just the openjdk pkg.
>
>
> As I understand, ezjail doesn't support multiple base jails. If it did, then
> I could simply install the application (and packages) to the base jail and
> have versions of the base. Then by shutting down a jail, switching the base
> to the new version and starting up, everything would upgrade easily. Even
> better would be some concept of hierarchy with customer_jail sitting on top
> of base_version_1.0 which in turn sits on top of base_jail.
>
> Would I need to abandon ezjail and be able to build all the above myself with
> a combination of nullfs (basejail) and unionfs (intermediate versioned jail)?
> Does unionfs now work with ZFS?
>
>
> Alternatively I could simply use zfs clones to deploy a new version of the
> application by destroying the whole jail and replacing it with a new one. I'd
> need to then script (I use saltstack) deploying the 2-3 config files which
> are different in each jail.
>
>
>
> Thoughts? What seems like a more robust long term approach to jail management?
>
>
> Thanks
> Ari
>
>
>
signature.asc
Description: OpenPGP digital signature
