Re: Best way to have a FreeBSD VM for automated testing?

[Paul Vixie]

<>


Not new. Been in production for years now.


https://github.com/firecracker-microvm/firecracker


p vixie

Re: bhyve/nmdm/cu console paste in problem

[Paul Vixie]

i've never been able to get tip or cu to accept a paste. this was not 
the only reason but it was one reason why i wrote rtty many years ago.


/usr/ports/sysutils/rtty

hard to configure, sorry, it didn't have to scale.

Re: RFC: rc(8) script for bhyve(8) on FreeBSD

[Paul Vixie]

pkg install vm-bhyve

-- 
P Vixie

Re: bhyve guests: what clock source to use?

[Paul Vixie]

in the old days i used to have to hard wire it. now it picks the same 
value i used to hard code. ntpd encountered chaos with other values.


kern.timecounter.hardware: TSC-low

i don't know if this has become an urban legend since i last fought it.

re:

Sean McBride wrote on 2023-10-11 11:28:

Hi all,

As part of debugging a problem I'm having, I'd like to understand:

What clock source should bhyve guests use?  tsc? hpet? acpi_pm?  What are 
pros/cons of the choices?  Does it depend on the guest OS? (for me FreeBSD and 
Ubuntu)  Should it match the host?

Thanks,

Sean




--
P Vixie

Re: If we are so opposed to Docker and Kubernetes, what is the real alternative on BSD?

[Paul Vixie]

Alejandro Imass wrote on 2024-02-01 07:18:

...

But I don't think anyone really wants Docker and there's the hypervisor 
for that. The intent of the thread is to deliberate on native freebsd 
orchestration and autoscaling.
I don't think anyone would mind re-writing Dockerfile to Bastillefile or 
whatever. What's missing is the other part, the k8s equivalent.


I think Docker and K8S and other successful / dominant forms of 
containers in the OSS world are platform-specific simply because that's 
what their creators and early adopters cared about. Adding more forms of 
platform-specific container technology (for example, taking explicit 
advantage of Bastille or other FreeBSD features) would not be a value 
add since its adopters would likely still have to support other 
platforms. To be worth doing, the outcome should be platform-agnostic, 
allowing a container creator to not-have-to-care what the underlying 
operating system was. "Write Once Run Anywhere."


 seems to be an example 
of putting the container-maker first and insulating them from details 
they won't care about such as what the underlying platform is running. 
Good abstraction boundaries make good neighbors, as they say.


"We" should not be opposed to Docker per se nor K8S. Linux became 
dominant by focusing on what its users wanted to be able to do. Docker 
and K8S likewise. If we have value to add to that mix, it won't be in 
the form of bespoke or BSD-lockin alternatives. It might be in inclusive 
and platform-agnostic alternatives.


--
P Vixie

Re: bhyve vm and virtio-console

[Paul Vixie]

if you want to use cu, you have to use -l to map guest com1 to some host nmdm 
device. if you want to use virtio-console, it will create unix domain sockets 
not pseudo tty endpoints. from bhyve(8) we see:

>  Virtio console device backends:
>  
>  •  
>  port1=/path/to/port1.sock[,portN=/path/to/port2.sock ...]
>  
>  A maximum of 16 ports per device can be created.  Every
>  port
>  is named and corresponds to a Unix domain socket created by
>  bhyve.  bhyve accepts at most one connection per port at a
>  time.
>  
>  Limitations:
>  
>  •   Due to lack of destructors in bhyve, sockets on the
>  filesystem must be cleaned up manually after bhyve
>  exits.
>  
>  •   There is no way to use the “console port” feature, nor
>  the console port resize at present.
>  
>  •   Emergency write is advertised, but no-op at present.

i know of no host-side client which will connect to these unix domain sockets 
and offer a cu-like user interface. "unidirectional" may be the experience 
you'll get from "socat".

if what you're trying to accomplish is a graphical console, virtio-console is 
not the way to do it. rather, you'll want "fbuf" to create a guest VGA/similar 
that gateways to a host VNC.

re:

On Friday, February 14, 2025 9:55:23 PM UTC Petru Garstea wrote:
> I tried with the following command, and got the same result.
> 
> */usr/sbin/bhyve -H -P -A -c 1 -m 1500M \
> -s 0,hostbridge \
> -s 6,virtio-console,0=/tmp/vtcon.0 \
> -s 31,lpc \
> -l bootrom,/tmp/BHYVE_UEFI.fd \
> -s 1,virtio-blk,/tmp/fbsd.img \
> -s 2,virtio-net,tap0 \
> tmp*
> 
> 
> I also able to connect but the bidirectional connection is not happening
> 
> *socat - UNIX-CONNECT:/tmp/vtcon.0
> ls
> *
> PS: if I use null modem emulation instead virtio-console, then
> connecting with cu is not an issue.
> 
> any ideas ?
> 
> Thanks
> 
> On 2/14/25 14:57, Michael Gmelin wrote:
> > On Fri, 14 Feb 2025 14:21:07 -0500
> > 
> > Petru Garstea wrote:
> >> Hi,
> >> I am trying to configure bhyve vm with support of virtio-console, the
> >> following option is added to bhyve command line
> >> 
> >> *-s 3,virtio-console,port1=/test/console.sock
> >> 
> >> *However when trying to connect to socket with socat trere are no
> >> input/ouptut is observed.
> >> 
> >> Has virtio-console ever worked ?
> > 
> > Yes it works, I just tried it using sysutils/vm-bhyve (just add
> > virt_console0="Yes" to the configuration file).
> > 
> > Based on the logs, these device settings are used by it:
> >  bhyve devices:
> >-s 0,hostbridge \
> >-s 31,lpc \
> >-s 4:0,virtio-blk,/dev/zvol/zroot/vms/freebsd14-test/disk0 \
> >-s 5:0,virtio-net,tap0,mac=58:9c:fc:aa:aa:aa \
> >-s 6:0,virtio-console,0=/vms/freebsd14-test/vtcon.0
> > 
> > I could connect to /vms/freebsd14-test/vtcon.0 using socat:
> >  socat - UNIX-CONNECT:/vms/freebsd14-test/vtcon.0
> > 
> > Inside the vm it's /dev/vtcon/0.
> > 
> > -m


-- 
Paul Vixie

Re: bhyve vCPU limit

[Paul Vixie via freebsd-virtualization]

Miroslav Lachman wrote on 2021-12-01 08:52:

On 01/12/2021 17:17, John Doherty via freebsd-virtualization wrote:

...


I am sorry for hijacking this thread but your information is very 
interesting. I was playing with VMs in VirtualBox and Bhyve and compared 
performance with increasing vCPU count. The more cores VM get the slower 
was even a simple single threaded task like loading PF rules from 
/etc/pf.conf. It was tested on FreeBSD 11.4 and 12.2, I tested ULE and 
4BSD schedulers. Maybe it was somewhat HW related but it always shows 
VMs with more than 2 v CPUs significantly slower. VMs with 6+ vCPU was 
almost unusable (loading of PF ruleset takes about 8 seconds instead of 
fraction on single vCPU VM).


...


loading a PF ruleset requires a fair bit of locking and unlocking of 
kernel data structures for each system call, per rule. while pfctl is 
single threaded, the acquisition process of those kernel locks probably 
requires a memory buffer flush to guaranty atomicity, and the lock's 
domain may overlap with other non-PF kernel activities that different 
hypervisors virtualize differently.


this makes loading a PF ruleset a poor benchmark for hypervisors, unless 
that activity is so common that the unusable slowness is interfering 
with other work. it could be debugged or optimized in that case, but how 
often do you really need to add a PF ruleset?


--
vixie