Re: Building a Linuxulator userland from source

[Felix Palmen]

* Alexander Leidinger  [20230822 01:25]:
> Am 2023-08-18 11:26, schrieb Felix Palmen:
> > 1.) Of course, Uses/linux.mk would need quite some switching to handle
> > c7 as well as something new that works completely differently (maybe
> > call it src). All still open issues.
> 
> I suggest to write a new Uses/xxx.mk for this. Much more easy for you to do
> what you want, and less error prone and less QA to do for the existing
> linux_base stuff.

Thanks! Actually, I had this thought already but was unsure about it.
So, someone else suggesting exactly the same is quite convincing.

My goal is to replace -c7 with my project, but of course, even if that
succeeds, it means both userlands will need to coexist for quite some
time, and that's probably indeed easier with completely separate Uses.

> > 2.) Could you please elaborate how e.g. some config file "visible" to
> > the Linux processes could "pollute" a Linux build? Besides, this could
> > only affect files from base /etc I think...
> 
> Well... the config part was more to highlight what the linux_base ports use
> the fallthrough for. In case of building I worry more that some includes
> from /usr/local are used than anything else. Also some other stuff
> configure-runs might pick-up from the installed FreeBSD ports.

I assume/hope that's a minor risk. /usr/local is not in the standard
search paths of the toolchain, so, must be added explicitly. A build
system doing that without being requested to do so would be pretty much
broken. Furthermore, the toolchain is built --with-sysroot=/compat/linux
so prepends that to all the system search paths.

Configure scripts finding *tools* in FreeBSD's /usr/local *might* be a
risk. Not an issue building with poudriere (the build jail will only
have what we want), but maybe an issue when someone builds the ports in
a live system.

Well, we will see :) At least, I already have the first ports building
fine using shell and make from the Linux userland, e.g. here:
https://github.com/Zirias/zfbsd-ports/blob/linux/sysutils/linux-man-db/Makefile

Cheers, Felix

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer -- {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature

Re: Building a Linuxulator userland from source

[Trenton Schulz]

Felix Palmen  writes:



I assume/hope that's a minor risk. /usr/local is not in the 
standard
search paths of the toolchain, so, must be added explicitly. A 
build
system doing that without being requested to do so would be 
pretty much
broken. Furthermore, the toolchain is built 
--with-sysroot=/compat/linux

so prepends that to all the system search paths.

Configure scripts finding *tools* in FreeBSD's /usr/local 
*might* be a
risk. Not an issue building with poudriere (the build jail will 
only
have what we want), but maybe an issue when someone builds the 
ports in

a live system.

Well, we will see :) At least, I already have the first ports 
building

fine using shell and make from the Linux userland, e.g. here:
https://github.com/Zirias/zfbsd-ports/blob/linux/sysutils/linux-man-db/Makefile


This is really fascinating work, and I see value in this even if 
some other way of doing things eventually replaces the Centos-7 items.


Some of this has a bit of overlap with Gentoo prefix 
(https://wiki.gentoo.org/wiki/Project:Prefix), where one puts the 
bare bones of a Gentoo distro under a "prefix" (for example, 
/compat/linux), but then you can use Gentoo's portage 
infrastructure to build the other parts of the system.


I imagine, you are maybe thinking of your own set of linux-* in 
the ports tree, but this might also be useful area to borrow from?


Anyway, I'll lurk back into the shadows to see how this develops.

Best regards,

--
Trenton

Re: Building a Linuxulator userland from source

[Mario Marietto]

CentOS has been replaced already successfully with Ubuntu and Devuan. On
the FreeBSD forums there are a couple of nice tutorials. BTW if we can use
even different user lands,we will be even happier.

On Tue, Aug 22, 2023 at 1:04 PM Trenton Schulz 
wrote:

>
> Felix Palmen  writes:
>
> >
> > I assume/hope that's a minor risk. /usr/local is not in the
> > standard
> > search paths of the toolchain, so, must be added explicitly. A
> > build
> > system doing that without being requested to do so would be
> > pretty much
> > broken. Furthermore, the toolchain is built
> > --with-sysroot=/compat/linux
> > so prepends that to all the system search paths.
> >
> > Configure scripts finding *tools* in FreeBSD's /usr/local
> > *might* be a
> > risk. Not an issue building with poudriere (the build jail will
> > only
> > have what we want), but maybe an issue when someone builds the
> > ports in
> > a live system.
> >
> > Well, we will see :) At least, I already have the first ports
> > building
> > fine using shell and make from the Linux userland, e.g. here:
> >
> https://github.com/Zirias/zfbsd-ports/blob/linux/sysutils/linux-man-db/Makefile
>
> This is really fascinating work, and I see value in this even if
> some other way of doing things eventually replaces the Centos-7 items.
>
> Some of this has a bit of overlap with Gentoo prefix
> (https://wiki.gentoo.org/wiki/Project:Prefix), where one puts the
> bare bones of a Gentoo distro under a "prefix" (for example,
> /compat/linux), but then you can use Gentoo's portage
> infrastructure to build the other parts of the system.
>
> I imagine, you are maybe thinking of your own set of linux-* in
> the ports tree, but this might also be useful area to borrow from?
>
> Anyway, I'll lurk back into the shadows to see how this develops.
>
> Best regards,
>
> --
> Trenton
>
>

-- 
Mario.

Re: Building a Linuxulator userland from source

[Felix Palmen]

* Mario Marietto  [20230822 13:59]:
> CentOS has been replaced already successfully with Ubuntu and Devuan.

No. You can install whatever you like in some Linux jail, you could even
use it as an alternate compat.linux.emul_path if you want, in both cases
it will partially work.

You *won't* get e.g. full unhindered access to the whole / filesystem
tree, you won't be able to use FreeBSD ports/packages of Linux software
with it, and so on.

FreeBSD's official Linuxulator userland is -c7, nothing has been
replaced. Please stop spreading such unfounded claims.

Bye, Felix

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer -- {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature

Re: Building a Linuxulator userland from source

[Mario Marietto]

---> you won't be able to use FreeBSD ports/packages of Linux software with
it...

I'm interested to understand what you mean better herecan you elaborate
using different words ? thanks.

On Tue, Aug 22, 2023 at 2:13 PM Felix Palmen  wrote:

> * Mario Marietto  [20230822 13:59]:
> > CentOS has been replaced already successfully with Ubuntu and Devuan.
>
> No. You can install whatever you like in some Linux jail, you could even
> use it as an alternate compat.linux.emul_path if you want, in both cases
> it will partially work.
>
> You *won't* get e.g. full unhindered access to the whole / filesystem
> tree, you won't be able to use FreeBSD ports/packages of Linux software
> with it, and so on.
>
> FreeBSD's official Linuxulator userland is -c7, nothing has been
> replaced. Please stop spreading such unfounded claims.
>
> Bye, Felix
>
> --
>  Felix Palmen  {private}   fe...@palmen-it.de
>  -- ports committer -- {web}  http://palmen-it.de
>  {pgp public key}  http://palmen-it.de/pub.txt
>  {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231
>


-- 
Mario.

Re: Building a Linuxulator userland from source

[Felix Palmen]

* Trenton Schulz  [20230822 12:55]:
> This is really fascinating work, and I see value in this even if some other
> way of doing things eventually replaces the Centos-7 items.

Thanks a lot!

> Some of this has a bit of overlap with Gentoo prefix
> (https://wiki.gentoo.org/wiki/Project:Prefix), where one puts the bare bones
> of a Gentoo distro under a "prefix" (for example, /compat/linux), but then
> you can use Gentoo's portage infrastructure to build the other parts of the
> system.

Hm, kind of interesting project ;) well sure, might be another source to
look at when hitting some weird issues. But using "portage" IMHO
wouldn't make much sense, we already have our ports system ;)

> I imagine, you are maybe thinking of your own set of linux-* in the ports
> tree, but this might also be useful area to borrow from?
> 
> Anyway, I'll lurk back into the shadows to see how this develops.

Well, I guess it'll take me a few days to stabilize stuff and a few
*more* days to create some helpful USING for it ... but then, I'll try
to build some additional libs and find some proof of concept of some
Linux binary (closed-source?) working on it. That's the rough plan ... I
have some hopes ;)

Cheers, Felix

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer -- {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature

Re: Building a Linuxulator userland from source

[Trenton Schulz]

Felix Palmen  writes:

* Trenton Schulz  [20230822 
12:55]:
This is really fascinating work, and I see value in this even 
if some other

way of doing things eventually replaces the Centos-7 items.


Thanks a lot!


No problem. Having been struggling with some cross-compiling 
issues at work, this hits home. :-)



Some of this has a bit of overlap with Gentoo prefix
(https://wiki.gentoo.org/wiki/Project:Prefix), where one puts 
the bare bones
of a Gentoo distro under a "prefix" (for example, 
/compat/linux), but then
you can use Gentoo's portage infrastructure to build the other 
parts of the

system.


Hm, kind of interesting project ;) well sure, might be another 
source to

look at when hitting some weird issues. But using "portage" IMHO
wouldn't make much sense, we already have our ports system ;)



Yes, yes. I guess this was more for hitting weird issues or 
another way for getting the Linux userland. When I have to do 
anything some GUI app in the Linuxulator, it's a struggle to track 
down all those extra libraries. Then, I start to think, I should 
make a "port for this", but by then, I have forgot to keep track 
off all the RPMs, plus I have it working on the one system I need 
it on, and then I move onto something else. (Master PDF Editor has 
been a victim of this a couple of times *sigh*).


In theory, with Gentoo prefix one "emerges" them out of the 
prefix. Of course, the ports tree could easily do this too, but I 
just thought you might be able to borrow some things out of 
portage.


Regardless, I'm really interested in how this turns out, so I 
think I'll just get back to watching and wishing you luck.



I imagine, you are maybe thinking of your own set of linux-* in 
the ports

tree, but this might also be useful area to borrow from?

Anyway, I'll lurk back into the shadows to see how this 
develops.


Well, I guess it'll take me a few days to stabilize stuff and a 
few
*more* days to create some helpful USING for it ... but then, 
I'll try
to build some additional libs and find some proof of concept of 
some
Linux binary (closed-source?) working on it. That's the rough 
plan ... I

have some hopes ;)


Again, good luck! I guess Master PDF Editor could be a candidate 
for a closed source binary (I have made it work before in the 
regular Linuxulator, but don't use it now).


Best regards,

--
Trenton



Cheers, Felix

Re: Building a Linuxulator userland from source

[Cy Schubert]

In message 
, F
elix Palmen writes:
> Hi all,
>
> for the last two weeks, I've been working on a spike in ports which now
> reached a state where I want to show it to and discuss it with fellow
> ports hackers.
>
> First, a link to my feature branch (warning, will be rebased every now
> and then):
> 
>
> The goal is to create a replacement for the now antiquated linux-c7
> userland. While the classic approach would be to find another Linux
> distribution that's not too much of a moving target and start
> "repackaging" that, I want to try something different: Build the
> required packages from source.
>
> ** Why
>
> It will be quite some work to do this, I'm not really sure about it yet
> (and how it would compare to the repackaging approach), so feasibility
> is yet to be decided. But I hope to get at least these two advantages:
>
> - Provide the newest GNU libs (glibc, libstdc++, ...) built against
>   exactly the Linux version emulated by the FreeBSD version this will
>   run on. This should make it possible to run a lot more Linux binaries
>   without relying on e.g. Linux jails.
> - When binaries don't work for missing Linux libraries, make it somewhat
>   easy to add them, maybe based on already existing FreeBSD ports.
>
> ** State
>
> I just reached a state where I can build a working Linux-native GNU
> toolchain (binutils, glibc, gcc) for C and C++ on aarch64, amd64 and
> i386. From here on, it should be simpler, there are already two ports in
> my branch (archivers/linux-bzip2 and archivers/linux-xz) using that
> native toolchain for building.
>
> ** How
>
> The native toolchain is built by a cross toolchain, the packages for
> this cross-toolchain are prefixed "lxcross-". For building this cross
> toolchain, bootstrapping versions of binutils and gcc are needed to
> build the initial glibc, these versions are suffixed "-bootstrap".
>
> lxcross ports set PREFIX to ${LXCROSSBASE}, which defaults to
> ${LOCALBASE}/linux-cross. lxcross-*-bootstrap ports set PREFIX to
> ${LXBOOTSTRAP}, this one defaults to ${LXCROSSBASE}/bootstrap.
>
> ** Open issues
>
> This is an unordered list off my head, so most likely incomplete.
>
> - Some trickery with PREFIX is currently needed. The ports framework
>   expects PREFIX to be used as is by the upstream build system. This
>   won't hold for building Linux packages, PREFIX must be /compat/linux
>   for that, but passed to the upstream build system in DESTDIR.
> - LIB_DEPENDS don't work, which could probably be solved in the
>   framework. Right now, I'm using a hacky workaround to define
>   LINLIB_DEPENDS and add it to both RUN_ and BUILD_DEPENDS.
> - A lot of smaller things that *should* be provided by the framework,
>   some of them probably by USES=3Dlinux, are currently copy&pasted to
>   every port needing them. I wanted to keep it simple while first trying
>   to get it to work, so the framework isn't touched yet at all.
> - Some stage-qa checks get confused, some (e.g. checking that everything
>   is stripped) don't work.
> - In my tests, "poudriere testport" failed at least on i386, because it
>   mounts linprocfs on /compat/linux/proc and then tries to remove
>   /compat/linux (remove pre-existing PREFIX). To test the ports, I had
>   to slightly modify the testport script for now.
> - For the Linux headers, there should be a metaport picking the Linux
>   version based on ${OSVERSION}. This doesn't exist yet, Linux 4.4.x is
>   always used.
> - Building the final linux-gcc ports, I get weird error messages
>   directly to poudriere's terminal (they do NOT appear in the build
>   log!) like this:
> ELF interpreter /usr/lib/ld-linux.so.2 not found, error 2
>   I have no idea where this comes from, so far I couldn't identify any
>   negative effect though.
>
> Acknowledgement: I found quite some useful info for doing this in the
> "Linux from Scratch" book. Of course you can't just follow the book
> (very different scenario, it assumes building on Linux and not doing any
> staging/packaging), but it *does* have some helpful hints.
>
> Cheers, Felix

Basically this would become another Linux distro, albeit a virtual one that 
runs under our Linuxulator.

Avoiding discussion about packaging -- we can package this any way we wish -- 
how will this support software written for distro A, B, or C. For example, Red 
Hat software doesn't neccesarily run on SuSE or Ubuntu because shared library 
dependencies may be different.

Building our own "distro" to run under the Linuxulator may require a complete 
set of packages and end-user applications because existing Linux software may 
require a Fedora, Debian or Red Hat library. Wouldn't this negate the need for 
a Linuxulator because a person can build most Linux software to run on native 
FreeBSD.

I think a better path might be to support multiple Linux userlands in parallel. 
Thus a user could simply copy or install vendor software for a Red Hat in one 
envir

Unmaintained FreeBSD ports which are out of date

[portscout]

Dear port maintainers,

The portscout new distfile checker has detected that one or more
unmaintained ports appears to be out of date. Please take the opportunity
to check each of the ports listed below, and if possible and appropriate,
submit/commit an update. Please consider also adopting this port.
If any ports have already been updated, you can safely ignore the entry.

An e-mail will not be sent again for any of the port/version combinations
below.

Full details can be found at the following URL:
http://portscout.freebsd.org/po...@freebsd.org.html


Port| Current version | New version
+-+
devel/py-archinfo   | 9.0.5405| v9.2.65
+-+
devel/py-cle| 9.0.5405| v9.2.65
+-+
finance/R-cran-quantmod | 0.4.23  | 0.4.25
+-+
math/py-claripy | 9.0.5405| v9.2.65
+-+
security/py-ailment | 9.0.5405| v9.2.65
+-+
security/py-angr| 9.0.5405| v9.2.65
+-+
security/py-pyvex   | 9.0.5405| v9.2.65
+-+
sysutils/google-compute-engine-oslogin  | 20191018.00 | 20230823.00
+-+


If any of the above results are invalid, please check the following page
for details on how to improve portscout's detection and selection of
distfiles on a per-port basis:

http://portscout.freebsd.org/info/portscout-portconfig.txt

Reported by:portscout!

Re: Building a Linuxulator userland from source

[Felix Palmen]

* Cy Schubert  [20230822 10:34]:
> Basically this would become another Linux distro, albeit a virtual one
> that runs under our Linuxulator.

And also a pretty minimal one. Right now, I'm just building a truly
minimal userland (the GNU toolchain, openssl, GNU make/grep/sed/awk, GNU
coreutils and man-db) and working on putting together some sane USES for
that.

> Avoiding discussion about packaging -- we can package this any way we
> wish -- how will this support software written for distro A, B, or C.
> For example, Red Hat software doesn't neccesarily run on SuSE or
> Ubuntu because shared library dependencies may be different.
> 
> Building our own "distro" to run under the Linuxulator may require a
> complete set of packages and end-user applications because existing
> Linux software may require a Fedora, Debian or Red Hat library.
> Wouldn't this negate the need for a Linuxulator because a person can
> build most Linux software to run on native FreeBSD.

Well first, when I ask why "Linuxulator" is needed, the answer in my
head is: Mostly for closed-source Linux software. Because exactly as you
say, anything else should better be ported and built to run natively on
FreeBSD, if possible.

Now, maybe I'm looking at the wrong software? In my experience with
closed-source Linux Software, sure, it *might* offer
distribution-specific packages, but almost always offers a plain binary
tarball as well. The latter could easily be used to create a port (like
was done in the past as well in our tree), and then it's just a question
of adding ports for the (hopefully few) shared libraries needed by this
software.

> I think a better path might be to support multiple Linux userlands in
> parallel. Thus a user could simply copy or install vendor software for
> a Red Hat in one environment and a SuSE vendor software in another.

This would be the consequence if you really want to support
distribution-specific software packages. I don't think it's feasible in
practice, at least it would make it very hard to still have ports of
Linux software (like my makemkv port), these would need to build and run
with any of these userlands.

To challenge my source-based approach, I'm looking for "proof of
concept" closed-source software to try get running with it, I'll
probably start with makemkv because I already maintain that port. Open
to suggestions what else to test there. In the end, getting to run e.g.
Google Chrome would be perfect, but I imagine this requires creating a
lot of ports for shared libs first.

Cheers, Felix

-- 
 Felix Palmen  {private}   fe...@palmen-it.de
 -- ports committer -- {web}  http://palmen-it.de
 {pgp public key}  http://palmen-it.de/pub.txt
 {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231


signature.asc
Description: PGP signature

Re: Building a Linuxulator userland from source

[Mario Marietto]

It would be nice to try that tool that can hack / convert ./ add another
layer (another linux distro inside the first one. I dont remember the name
now.

Il mer 23 ago 2023, 08:21 Felix Palmen  ha scritto:

> * Cy Schubert  [20230822 10:34]:
> > Basically this would become another Linux distro, albeit a virtual one
> > that runs under our Linuxulator.
>
> And also a pretty minimal one. Right now, I'm just building a truly
> minimal userland (the GNU toolchain, openssl, GNU make/grep/sed/awk, GNU
> coreutils and man-db) and working on putting together some sane USES for
> that.
>
> > Avoiding discussion about packaging -- we can package this any way we
> > wish -- how will this support software written for distro A, B, or C.
> > For example, Red Hat software doesn't neccesarily run on SuSE or
> > Ubuntu because shared library dependencies may be different.
> >
> > Building our own "distro" to run under the Linuxulator may require a
> > complete set of packages and end-user applications because existing
> > Linux software may require a Fedora, Debian or Red Hat library.
> > Wouldn't this negate the need for a Linuxulator because a person can
> > build most Linux software to run on native FreeBSD.
>
> Well first, when I ask why "Linuxulator" is needed, the answer in my
> head is: Mostly for closed-source Linux software. Because exactly as you
> say, anything else should better be ported and built to run natively on
> FreeBSD, if possible.
>
> Now, maybe I'm looking at the wrong software? In my experience with
> closed-source Linux Software, sure, it *might* offer
> distribution-specific packages, but almost always offers a plain binary
> tarball as well. The latter could easily be used to create a port (like
> was done in the past as well in our tree), and then it's just a question
> of adding ports for the (hopefully few) shared libraries needed by this
> software.
>
> > I think a better path might be to support multiple Linux userlands in
> > parallel. Thus a user could simply copy or install vendor software for
> > a Red Hat in one environment and a SuSE vendor software in another.
>
> This would be the consequence if you really want to support
> distribution-specific software packages. I don't think it's feasible in
> practice, at least it would make it very hard to still have ports of
> Linux software (like my makemkv port), these would need to build and run
> with any of these userlands.
>
> To challenge my source-based approach, I'm looking for "proof of
> concept" closed-source software to try get running with it, I'll
> probably start with makemkv because I already maintain that port. Open
> to suggestions what else to test there. In the end, getting to run e.g.
> Google Chrome would be perfect, but I imagine this requires creating a
> lot of ports for shared libs first.
>
> Cheers, Felix
>
> --
>  Felix Palmen  {private}   fe...@palmen-it.de
>  -- ports committer -- {web}  http://palmen-it.de
>  {pgp public key}  http://palmen-it.de/pub.txt
>  {pgp fingerprint} 6936 13D5 5BBF 4837 B212  3ACC 54AD E006 9879 F231
>