Corosync and Pacemaker with Jails. Is it worth the effort?
Hello,
Are these cluster tools actually usable on FBSD and in Jails?
I got a 3 way PostgreSQL cluster with floating IPs to _mostly_ work across
3 Bastille-based jails but the whole process has been very painful, so I'm
wondering if this stuff is actually usable on FBSD or am I just wasting my
time, and this port is just a curiosity ?
I have noticed that several of the resource agents are very Linux oriented,
for example, requiring the ip command, and some of the very basic
clustering logic seem to depend on things like systemd.
So far I have discovered that you have to use VNET jail and I have had to
add:
allow.mlock = 1;
sysvmsg = new;
sysvsem = new;
sysvshm = new;
As well as this to corosync:
system {
qb_ipc_type: shm
sched_rr: no
stonith-enabled: false
}
Not sure if there are other things I haven't noticed because even though I
finally got the cluster to run, the promotion and demotion seems very
unstable still.
Also, I cannot get rid of these warnings in every command:
WARNING: could not get the pacemaker version, bad installation?
WARNING: list index out of range
I am happy to continue the calvary if it's worthwhile and there are other
people using and/or interested in this, but the lack of information seems
to suggest that maybe this is just a waste of time, and these tools are
nowhere near a usable state without heavy lifting.
If the latter is the case, are there any other stable HA/Clustering
solutions that are mature for FBSD+Jails, and that include resource
managers such as databases (namely Pg and MariaDB) and floating IPs ?
This is the environment:
FreeBSD 14.0-RELEASE
Bastille: 0.10.20231125
Pacemaker 2.1.6
Corosync 3.1.7
Many thanks in advance, for any pointers or comments.
--
Alex
Dockerfile to Bastille interpreter/translator
I understand there's an ongoing effort to support Linux Docker images "natively" through amd64 translation layer to the FBSD kernel. I think that is a great endeavour but it got me thinking that perhaps there's an alternative way to leverage the full power of FBSD and Bastille. If you look at Dockerfile, they mostly follow the same pattern: 1 - package manager installs 2 - system commands 3 - package, inheritance 4 - repeat 5 - launch entrypoint The package manager AFAICT are only 3: alpine, debian, red hat The commands are almost translatable 1:1 to Bastille So my question is, has anybody thought or have started work on an Dockerfile interpreter that can build native Bastille/ZFS "images" from a Dockerfile ? I think most of the work would be creating and maintaining the package names and their equivs which for the most part will be very similar to FBSD ports. Any thoughts? Best, -- Alex
Re: Dockerfile to Bastille interpreter/translator
On Thu, Jan 11, 2024 at 10:34 AM Zhenlei Huang wrote: .. > > So my question is, has anybody thought or have started work on an > Dockerfile interpreter that can build native Bastille/ZFS "images" from a > Dockerfile ? > > If I read you right, you may look at buildah / podman, and the runtime > runj [1] . > > Good luck :) > > 1. https://github.com/samuelkarp/runj > > Thank you! I was already looking at runj. But what I had in mind was more like a translator from Dockerfile to Bastillefile and/or CBSDFile, something like that. Although honestly, I don't think that it would be too hard for ops people to translate Dockerfiles to native FBSD ones like Bastillefile or CBSDFile Best, -- Alex
If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
I made the title provocative for a flame but in good faith, so hear me out ;-) Being a FreeBSD user for many years I perfectly understand why supporting Docker is useless outside of virtualization or just for fun. I only put it in the subject to get your attention. But can the same be said of Kubernetes? In the understanding that k8s relies on some Linux specific things like namespaces and control groups etc. but we have similar utilities that could be used, so not sure why no serious effort to port k8s to BSD so far... Moreover, why is there not a serious alternative to Kubernetes-like, or a FreeBSd native Kubernetes port, at this point in 2024 ? I've read comments about Nomad but does it actually work with Bastille for example ? I mean we have all the building blocks, and we invented "container" so wtf is there no similar orchestration and auto scaling jail technology at our disposal on FBSD so far ? What am I missing?
Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
On Thu, Feb 1, 2024 at 1:40 PM Odhiambo Washington wrote: .. > You're missing the fact that you could virtualize a Linux host and run all > those Docker/k8s inside it and just proxy access to them. > For example, I am running https://ushahidi.kictanet.or.ke/views/map > inside an Ubuntu VM under bhyve. Apache on the FreeBSD host is doing the > proxy. > > PS: I am lucky to be the 1st one to pour more fuel into your flame :-) > > Hi Odhiambo, thanks for opening the discussion! I am aware that hypervisor and full virtualization is an option, no doubt. But IMHO ideally it should be native jails with Bastille, CBSD or whatever. Best, -- Alex >
Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
On Thu, Feb 1, 2024 at 1:44 PM Mario Marietto wrote: > Would be cool to investigate if the Linuxulator supports Kubernetes > > I understand that Linux native container can already run on BSD with Linuxulator layer. But I would rather traduce Dockerfile to Bastillefile of CBSDFile and go with native code. IMO, the issue is not so much the container tech, we invented that in the first place. The issue is not having a decent orchestrator / autoscaling service. Although I think that runj is the correct starting point: https://github.com/samuelkarp/runj Although I am not a k8s expert not sure if runj counts as an alternative to k8s or more like container runtime for k8s. Would be nice that actual k8s experts could chime in here ... > On Thu, Feb 1, 2024 at 1:40 PM Odhiambo Washington > wrote: > >> >> >> On Thu, Feb 1, 2024 at 3:13 PM Alejandro Imass >> wrote: >> >>> >>> .. > You're missing the fact that you could virtualize a Linux host and run all >> those Docker/k8s inside it and just proxy access to them. >> For example, I am running https://ushahidi.kictanet.or.ke/views/map >> inside an Ubuntu VM under bhyve. Apache on the FreeBSD host is doing the >> proxy. >> >> PS: I am lucky to be the 1st one to pour more fuel into your flame :-) >> >> >> -- >> Best regards, >> Odhiambo WASHINGTON, >> Nairobi,KE >> +254 7 3200 0004/+254 7 2274 3223 >> In an Internet failure case, the #1 suspect is a constant: DNS. >> "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) >> [How to ask smart questions: >> http://www.catb.org/~esr/faqs/smart-questions.html] >> > > > -- > Mario. >
Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
On Thu, Feb 1, 2024 at 3:56 PM Patrick M. Hausen wrote: > Hi all, > > > Am 01.02.2024 um 15:47 schrieb Alejandro Imass : > > IMO, the issue is not so much the container tech, we invented that in > the first place. The issue is not having a decent orchestrator / > autoscaling service. > > There's Nomad (https://www.nomadproject.io/) integration > with pot: https://github.com/bsdpot/pot > > The right person to contact is probably Luca (https://github.com/pizzamig > ). > I'll send him a hint at this thread instead of Cc'ing him without asking. > > Thank you! I saw Nomad but I am worried about the move to BUSL. I didn't know it had integration with pot. Shouldn't be that hard to support Bastille, CBSD and IOCage if the latter is still alive. May forking Nomad is an option, they did it for Terraform and pretty quick: https://opentofu.org Best, -- Alex > Kind regards, > Patrick > -- > >
Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
On Thu, Feb 1, 2024 at 4:05 PM Mario Marietto wrote: > ---> I understand that Linux native containers can already run on BSD with > the Linuxulator layer. > > Docker can't run on the linuxulator. Curious to see if kubernetes and or > runj do it. Me too I know that native code is better,but I know that no one > on FreeBSD wants,so instead of using bhyve,if I can,I prefer the lighter > Linuxulator. > Well, it has run in the past: https://wiki.freebsd.org/Docker But I don't think anyone really wants Docker and there's the hypervisor for that. The intent of the thread is to deliberate on native freebsd orchestration and autoscaling. I don't think anyone would mind re-writing Dockerfile to Bastillefile or whatever. What's missing is the other part, the k8s equivalent.
Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
On Fri, Feb 2, 2024 at 12:40 AM Mario Marietto wrote: > Excuse me. I'm not very experienced,but isn't the L4 or any other > microkernel a valid alternative to containers ? If I have understood > correctly how it works,it allows multiple instances of the various services > implemented within the microkernel OS. How many instances can we have ? For > example in the L4 Linux kernel webpage it is explained that it can boot > FreeBSD in cooperation with Linux. It works like xen. WIth xen we can have > multiple virtual machines. But xen today has been preferred to kvm. And > anyway,we always talk about monolithic kernels. So,I want to ask : is a > microkernel OS a valid alternative to the containers ? If it allows to run > only some services of the "virtualized" os,why not use it ? Why not invest > effort and time to implement this solution as an alternative to the > containers that FreeBSD already has ? Personally I like the idea of seeing > Linux and FreeBSD work together. > > IMHO, way beyond the scope of this thread. AFAIK neither Linux nor FreeBSD are microkernels. The original question is more like what would it take to run k8s natively using different jail managers like CBSD and BastilleBSD as drivers? Does it already exist? maybe: https://github.com/tnorlin/kubernetes What is the state? What are the limitations? etc. etc.
Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?
On Fri, Feb 2, 2024 at 2:29 PM Mario Marietto wrote: > Alejandro,I'm not sure if you are aware that the discussion at a certain > point has broadened and deepened. I don't understand why I can't propose > an alternative to the actual jail and jail managers that you are talking > about. And no. On the L4 website there is a userland based on new versions > of Ubuntu and Debian that can be used with the microkernel offered. They > also say that FreeBSD can be compiled,too,using the same technique used to > build the Linux userland proposed. If you want to talk about containers,why > not also add the microkernel topic,that's not a container,but its working > can be compared to the microservices offered by Linux and FreeBSD if they > run with a microkernel. > > Never said you can't propose it, it just seems off-topic to this thread. Maybe start a new thread on microkernel versus X. This thread is about containers and container configuration, auto-scaling, discovery, updating and monitoring, etc. What I usually do in this case is start a new thread with a new subject and (was: xxx). That way, people interested in that branch of the discussion can continue on its own topic, and the mail archives can do a better job for later research. BTW, I think what you are saying already exists and macOS is exactly that: the FreeBSD core and the mach microkernel. Best, -- Alex
