Re: are emails getting through to the list?
On Wed, 9 Jun 2021 16:26:47 +0100 tech-lists wrote: > Is virtualization@freebsd.org working? Yes, "J." aka "tech-lists". -- Michael Gmelin
Should we include ttyu* to devfs_ruleset 3 (devfsrules_unhide_login)?
Hi, There are many TTY devices in devfsrules_unhide_login=3, but ttyu* (serial lines) are not part of it. As a result, certain things won't work as expected when connecting over a serial console, one example being connecting to a local bhyve vm over serial console (e.g., `vm console myvm' when using vm-bhyve). The example that brought this to my attention is using ssh within a jail that's running inside of a VM, while being connected to that VM over serial console. So the setup is: - FreeBSD 13 host - bhyve vm running FreeBSD 13 on top - Jail using mount.devfs running within the bhyve vm, using the default devfs_ruleset inside of the bhyve vm (which in turn loads devfsrules_jail=4, which includes devfsrules_unhide_login=3). Now, ssh within that jail won't work, as /dev/tty can't be accessed. Example (while being connected to the vm over a serial line): # jail -c path=/ mount.devfs ip4=inherit command=ssh localhost Host key verification failed. jail: ssh localhost: failed Now, adding in an extra rule to ruleset 3: # devfs rule -s 3 add 3250 path "ttyu*" unhide Things work as expected: # jail -c path=/ mount.devfs ip4=inherit command=ssh localhost The authenticity of host 'localhost (127.0.0.1)' can't be established... Are you sure you want to continue connecting (yes/no)? Now the question is, would it make sense to add ttyu* (or at least ttyu0) to [devfsrules_unhide_login=3] in /etc/defaults/devfs.rules, or are there any (security) reasons why this might be a bad idea? Best, Michael -- Michael Gmelin
Re: bastille : poudriere not working in jail: jail: jail:_set: Operation not permitted!
On Mon, 28 Feb 2022 16:15:45 +0100
FreeBSD User wrote:
> Hello folks,
>
> we run at least two poudriere build systems on recent CURRENT boxes
> and one of these poudriere build systems is working within a jail -
> setup via FreeBSD's /etc/jail.conf and by misusing the port ezjail
> for copying/deploying our self-compiled jail binary. The poudriere
> jail uses ZFS and is, to make it short, working like a charme.
>
> Now we try to setup another poudriere, but this time the base is
> XigmaNAS 12.3.0.4/9009, which is based upon 12.X-RELENG, utilizing
> "bastille". Bastille is up to date (in terms od the XigmaNAS plugin).
>
> Following the setup we used on the native CURRENT "jailed poudriere"
> builder and also following this reference (for those who want to
> check on this)
>
> https://www.mimar.rs/blog/host-your-own-services-with-freebsd-jails-part-3-poudriere
>
> which seems quite recent and with the exception, that we use "vnet"
> on all of our systems for jails and so does XigmaNAS.
>
> Starting a building process via poudriere ends up with
>
>
> # poudriere bulk -p head -z default -j 123-amd64 -f
> /usr/local/etc/poudriere.d/zeit4-default.pkglist [00:00:00] Creating
> the reference jail... done [00:00:01] Mounting system devices for
> 123-amd64-head-default [00:00:01] Warning: Using packages from
> previously failed, or uncommitted, build:
> /mnt/poudriere/data/packages/123-amd64-head-default/.building
> [00:00:01] Mounting ports from: /mnt/poudriere/ports/head [00:00:01]
> Mounting packages from:
> /mnt/poudriere/data/packages/123-amd64-head-default [00:00:01]
> Mounting distfiles from: /mnt/poudriere/ports/distfiles [00:00:01]
> Copying /var/db/ports from:
> /usr/local/etc/poudriere.d/head-amd64-head-default-options [00:00:02]
> Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
> /etc/resolv.conf ->
> /mnt/poudriere/data/.m/123-amd64-head-default/ref/etc/resolv.conf
> [00:00:02] Starting jail 123-amd64-head-default jail: jail_set:
> Operation not permitted [00:00:02] Cleaning up [00:00:02] Unmounting
> file systems
>
> poudriere jail -l:
>
> # poudriere jail -l
> JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
> 123-amd64 12.3-RELEASE amd64
> url=https://download.freebsd.org/releases/a ... 3-RELEASE/ 2022-02-24
> 14:14:25 /mnt/poudriere/jails/123-amd64 130-amd64 13.0-RELEASE amd64
> url=https://download.freebsd.org/releases/a ... 0-RELEASE/ 2022-02-24
> 14:11:32 /mnt/poudriere/jails/130-amd64
>
> The jail.conf for this specific jail is as follows:
>
> [...]
> pulverfass-001 {
> devfs_ruleset = 13;
> enforce_statfs = 1;
> exec.clean;
> exec.consolelog =
> /mnt/extensions/bastille/logs/pulverfass-001_console.log; exec.start
> = '/bin/sh /etc/rc'; exec.stop = '/bin/sh /etc/rc.shutdown';
> host.hostname = X;
> mount.devfs;
> mount.fstab = /mnt/extensions/bastille/jails/pulverfass-001/fstab;
> path = /mnt/extensions/bastille/jails/pulverfass-001/root;
> securelevel = 0;
>
> vnet;
> vnet.interface = e0b_bastille4;
> exec.prestart += "jib addm bastille4 igb0";
> exec.prestart += "ifconfig e0a_bastille4 description \"vnet host
> interface for Bastille jail pulverfass-001\""; exec.poststop += "jib
> destroy bastille4";
>
> allow.mount;
> allow.mount.fdescfs;
> allow.mount.devfs;
> allow.mount.tmpfs;
> allow.mount.nullfs;
> allow.mount.procfs;
> allow.mount.linsysfs;
> allow.mount.linprocfs;
> allow.mount.zfs;
>
> allow.chflags;
> allow.raw_sockets;
> allow.socket_af;
> allow.sysvipc;
>
> linux = new;
>
> exec.created += "/sbin/zfs jail ${name} BUNKER00/poudriere";
> exec.start += "/sbin/zfs mount -a";
> exec.poststop += "/sbin/zfs unjail BUNKER00/poudriere";
>
> }
> [...]
>
> Tracking the execution of the build process by issuing
>
> poudriere -x bulk ...
>
> and examin the resulting trace doesn' tgive me any hint, the error
> reported above immediately occurs when the jail is about to be
> started:
>
> + set -u +x
> + jail -c persist 'name=123-amd64-head-default'
> 'path=/mnt/poudriere/data/.m/ \ 123-amd64-head-default/ref'
> 'host.hostname=basehost.local.domain' \ 'ip4.addr=127.0.0.1'
> 'ip6.addr=::1' allow.chflags allow.sysvipc jail: jail_set: Operation
> not permitted
> + exit_handler
> [...]
>
> Searching the net revealed some issues with setting IP4 and IP6 in
> poudriere, but those findings are dated back to 2017 and 2014 and I
> guess this is solved right now.
>
> The difference between our manually jail.conf drive
Re: (vm-)bhyve stopped working recently on 13.2
> On 12. Jan 2024, at 21:50, Tomek CEDRO wrote: > > Hello world :-) > > Aftere recent 13.2-RELEASE freebsd-update and pkg bhyve stopped > working. It starts a machine that stops right after start. I am using > vm-bhyve (1.5.0) frontend and enabling debug="yes" does not really > reveal anything more except error code and its cause. > > Does anyone know what can be the problem and solution here? :-) > > Any hints appreciated :-) > Tomek > What are you running inside bhyve (OS and version)? Could you maybe share vm-bhyve‘s config file? Best Michael > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >
Re: (vm-)bhyve stopped working recently on 13.2
> On 12. Jan 2024, at 23:08, Tomek CEDRO wrote: > > On Fri, Jan 12, 2024 at 10:54 PM Michael Gmelin wrote: >> What are you running inside bhyve (OS and version)? Could you maybe share >> vm-bhyve‘s config file? > > I was running Debian 11 and since it stopped working I tried Debian 12. > > debug="yes" > #loader="bhyveload" > loader="uefi" > uefi_vars="yes" > graphics="yes" > graphics_res="1920x1080" > graphics_listen="127.0.0.1" > graphics_wait="no" > graphics_vga="no" > cpu="4" > memory="20G" > network0_type="virtio-net" > network0_switch="public" > #disk0_type="virtio-blk" > disk0_type="nvme" > disk0_name="disk0.img" > sound="yes" > sound_play="/dev/dsp" > passthru0="6/0/0" > xhci_mouse="yes" > uuid="XXX" > network0_mac="XXX" > Hi, Does it work when you disable passthrough? > … > My last reply went to the questions list only, so as this one goes > also to the virtualisation I would like to humbly express an idea of > moving vh-bhyve under FreeeBSD wings and make it a default frontend > for bhyve as it is amazingly simple and does the job :-) Not sure if the author would like that. It’s a great tool for sure. Best > > Thanks Michael :-) > Tomek > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Re: (vm-)bhyve stopped working recently on 13.2
> On 12. Jan 2024, at 23:49, Tomek CEDRO wrote: > > On Fri, Jan 12, 2024 at 11:45 PM Michael Gmelin wrote: >> Does it work when you disable passthrough? > > Tried that no change :-( > Probably best to strip down the config to see if it starts and then - assuming it does - bring back various settings (including sound, graphics etc.). Cheers >> Not sure if the author would like that. It’s a great tool for sure. > > Indeed! It deserves more attention / support :-) > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info >
Re: (vm-)bhyve stopped working recently on 13.2
> On 13. Jan 2024, at 01:59, Michael Gmelin wrote: > > > >>> On 12. Jan 2024, at 23:49, Tomek CEDRO wrote: >>> >>> On Fri, Jan 12, 2024 at 11:45 PM Michael Gmelin wrote: >>> Does it work when you disable passthrough? >> >> Tried that no change :-( >> > > Probably best to strip down the config to see if it starts and then - > assuming it does - bring back various settings (including sound, graphics > etc.). > > Cheers Actually, this looks like it might be related: https://github.com/churchers/vm-bhyve/pull/525 Found the link in here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273560 Cheers
Re: bhyve vm and virtio-console
> On 14. Feb 2025, at 23:46, Petru Garstea > wrote: > > Spinning up the vm with /dev/nmdm* and connecting with cu -l is not an issue > on my end. > I am trying to create a vm using freebsd cloud image and connect to guest OS > terminal with virtio-console and socat(no X11). > When spawning the vm the unix socket indeed is created, however when I > connect to it with socat nothing is being shown(not actions on any terminal > commands as well). > Also, I noticed there is a kernel module(virtio_console.ko) and apparently I > believe it has to be loaded it, not sure here. > > Please advise, > You can connect to the device inside the vm like I wrote and pass data to the socat connected socket on the outside (this is what I tested, please see my previous email). Are you trying to get some sort of shell over this connection? -m >> On 2/14/25 17:23, Paul Vixie wrote: >> if you want to use cu, you have to use -l to map guest com1 to some host nmdm >> device. if you want to use virtio-console, it will create unix domain sockets >> not pseudo tty endpoints. from bhyve(8) we see: >> >>> Virtio console device backends: >>>• >>> port1=/path/to/port1.sock[,portN=/path/to/port2.sock ...] >>>A maximum of 16 ports per device can be >>> created. Every >>> port >>> is named and corresponds to a Unix domain socket created by >>> bhyve. bhyve accepts at most one connection per port at a >>> time. >>>Limitations: >>>• Due to lack of destructors in bhyve, >>> sockets on the >>> filesystem must be cleaned up manually after bhyve >>> exits. >>>• There is no way to use the “console >>> port” feature, nor >>> the console port resize at present. >>>• Emergency write is advertised, but >>> no-op at present. >> i know of no host-side client which will connect to these unix domain sockets >> and offer a cu-like user interface. "unidirectional" may be the experience >> you'll get from "socat". >> >> if what you're trying to accomplish is a graphical console, virtio-console is >> not the way to do it. rather, you'll want "fbuf" to create a guest >> VGA/similar >> that gateways to a host VNC. >> >> re: >> >>> On Friday, February 14, 2025 9:55:23 PM UTC Petru Garstea wrote: >>> I tried with the following command, and got the same result. >>> >>> */usr/sbin/bhyve -H -P -A -c 1 -m 1500M \ >>> -s 0,hostbridge \ >>> -s 6,virtio-console,0=/tmp/vtcon.0 \ >>> -s 31,lpc \ >>> -l bootrom,/tmp/BHYVE_UEFI.fd \ >>> -s 1,virtio-blk,/tmp/fbsd.img \ >>> -s 2,virtio-net,tap0 \ >>> tmp* >>> >>> >>> I also able to connect but the bidirectional connection is not happening >>> >>> *socat - UNIX-CONNECT:/tmp/vtcon.0 >>> ls >>> * >>> PS: if I use null modem emulation instead virtio-console, then >>> connecting with cu is not an issue. >>> >>> any ideas ? >>> >>> Thanks >>> >>> On 2/14/25 14:57, Michael Gmelin wrote: >>>> On Fri, 14 Feb 2025 14:21:07 -0500 >>>> >>>> Petru Garstea wrote: >>>>> Hi, >>>>> I am trying to configure bhyve vm with support of virtio-console, the >>>>> following option is added to bhyve command line >>>>> >>>>> *-s 3,virtio-console,port1=/test/console.sock >>>>> >>>>> *However when trying to connect to socket with socat trere are no >>>>> input/ouptut is observed. >>>>> >>>>> Has virtio-console ever worked ? >>>> Yes it works, I just tried it using sysutils/vm-bhyve (just add >>>> virt_console0="Yes" to the configuration file). >>>> >>>> Based on the logs, these device settings are used by it: >>>> bhyve devices: >>>>-s 0,hostbridge \ >>>>-s 31,lpc \ >>>>-s 4:0,virtio-blk,/dev/zvol/zroot/vms/freebsd14-test/disk0 \ >>>>-s 5:0,virtio-net,tap0,mac=58:9c:fc:aa:aa:aa \ >>>>-s 6:0,virtio-console,0=/vms/freebsd14-test/vtcon.0 >>>> >>>> I could connect to /vms/freebsd14-test/vtcon.0 using socat: >>>> socat - UNIX-CONNECT:/vms/freebsd14-test/vtcon.0 >>>> >>>> Inside the vm it's /dev/vtcon/0. >>>> >>>> -m >> >
Re: bhyve vm and virtio-console
On Fri, 14 Feb 2025 14:21:07 -0500 Petru Garstea wrote: > Hi, > I am trying to configure bhyve vm with support of virtio-console, the > following option is added to bhyve command line > > *-s 3,virtio-console,port1=/test/console.sock > > *However when trying to connect to socket with socat trere are no > input/ouptut is observed. > > Has virtio-console ever worked ? > Yes it works, I just tried it using sysutils/vm-bhyve (just add virt_console0="Yes" to the configuration file). Based on the logs, these device settings are used by it: bhyve devices: -s 0,hostbridge \ -s 31,lpc \ -s 4:0,virtio-blk,/dev/zvol/zroot/vms/freebsd14-test/disk0 \ -s 5:0,virtio-net,tap0,mac=58:9c:fc:aa:aa:aa \ -s 6:0,virtio-console,0=/vms/freebsd14-test/vtcon.0 I could connect to /vms/freebsd14-test/vtcon.0 using socat: socat - UNIX-CONNECT:/vms/freebsd14-test/vtcon.0 Inside the vm it's /dev/vtcon/0. -m -- Michael Gmelin
Re: bhyve vm and virtio-console
On Fri, 14 Feb 2025 19:36:21 -0500 peter garshtja wrote: > Alright, I was missing the point, in my case if I want to get shell > access via virtio-console then I should have attached getty to that > console inside the guest os, and eventually I expected to see the > same behavior as I have experienced with null modem emulator. > Yes, e.g., adding this in the vm's /etc/ttys will give you access to a console without login/password prompt: ttyV0.0 "/usr/libexec/getty autologin" xterm on secure -m p.s. To make the life of those reading threads later, and if your email client allows, please avoid top posting on the mailing lists: https://docs.freebsd.org/en/articles/mailing-list-faq/#_what_are_the_special_etiquette_consideration_when_replying_to_an_existing_posting_on_the_mailing_lists Thank you! -- Michael Gmelin
