Re: CFT: FreeBSD Package Base

[Emmanuel Vadot]

 Hi Kris,

On Sun, 28 Apr 2019 15:52:21 -0400
 wrote:

> FreeBSD Community,
> 
>  
> 
> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and 13-current
> using "TrueOS-inspired" packaged base. These are stock FreeBSD images which
> will allow users to perform all updating via the 'pkg' command directly.
> Rather than trying to answer all questions in this announcement, we've
> created a FAQ page with more details. Please refer to this page, and let us
> know if you have additional questions that we can include on that page going
> forward.
> 

 While I appreciate the effort I have some doubt about your
"re-implementation" of pkgbase. I don't see any improvement compared to
what is in base currently, I even see downside of your implementation.

 - How do you plan with the need of updating kernel first, reboot and
updating the rest of the userland after ? (Needed for major and minor
upgrade, 12.0 to 12.1 for example, and simple update in -STABLE and
-HEAD branch). This is still a problem with the base pkgbase.
 - This is even worse because you are using the same repository for
base and pkg so if a user pkg update and both kernel and pkg(8) needs
to be updated and pkg use a new syscall or capsicum thing it will be
updated first and couldn't proceed with the rest of the update (this is
a supposition, I haven't personally tested).
 - It seems that multiple kernels isn't supported in your
implementation, this is already supported in pkgbase but still need
some love. This is an important point as it will allow user to choose
easily the kernel that they want to use and will also allow us
developper to push kernels with new features to help testing.
 - Since you reduced the granularity on the userland bits it would mean
that if we use your implementation for -p updates we would download the
whole userland packages instead of just updating the package that was
patched. For example with pkgbase, updating from 12.0 to 12.0p1 will
only update the FreeBSD-runtime package. Yes this package is still big
to download when you compare to what have changed but until pkg(8) have
delta pkg supports (and if it will have support, I don't know if
this is a wish or not) this is the best way to go.
 - I see that you are sorting the plist for kernel and userland based
on the line length [1], why is that ?

 I think that the only advantage that your solution offers is that if
we remove a componant of base (rcmds for example in 12-CURRENT) those
files would be removed as they are in the userland-base package while
for pkgbase the FreeBSD-rcmd package will be deleted in the repo and
will not be deleted in the user computer.

> 
> Additionally, I will be hosting a Package Base working group at BSDCan 2019,
> and welcome user and developer attendance to discuss this and other ongoing
> package work:
> 
>  
> 
> https://wiki.freebsd.org/DevSummit/201905/PackageBase
> 

 I will be present and looking forward to work with you on this.

 Cheers,

 P.S. :  FYI I'm working on pkgbase currently and I will have some
patches to commit soon (bsdinstall support, memstick creation that
install a pkgbase aware installaton etc ...).

[1] :
https://github.com/trueos/trueos-ports/blob/trueos-master/os/userland-base/Makefile#L35

-- 
Emmanuel Vadot  
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Konstantin Belousov]

Cc: list trimmed to relevant.  Very long essey below, be warned.

On Sun, Apr 28, 2019 at 03:52:21PM -0400, k...@ixsystems.com wrote:
> FreeBSD Community,
> 
>  
> 
> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and 13-current
> using "TrueOS-inspired" packaged base. These are stock FreeBSD images which
> will allow users to perform all updating via the 'pkg' command directly.
> Rather than trying to answer all questions in this announcement, we've
> created a FAQ page with more details. Please refer to this page, and let us
> know if you have additional questions that we can include on that page going
> forward.
> 
>  
> 
> Additionally, I will be hosting a Package Base working group at BSDCan 2019,
> and welcome user and developer attendance to discuss this and other ongoing
> package work:
> 
>  
> 
> https://wiki.freebsd.org/DevSummit/201905/PackageBase
> 
>  
> 
>  
> 
> FAQ
> 
> -
> 
> https://trueos.github.io/pkgbase-docs/
> 

I do not know what are design decisions for trueos pkgbase are, but I
do know something about in-tree split and why some packaging decisions
where made. I cannot attend your WG, but I believe the reasoning used
for the in-tree is important enough to represent it intact from the
source.  I have to start with some explanatory long text to put it into
the proper perspective.

There are two knots of interdependinces which are critical for correctness
of any upgrade where the target system cannot be simply discarded on failure:
1. C runtime
2. Minimal boot path to prompt.
Let me elaborate both, starting from point 1, which is typically very obscure
despite having the fundamental nature for anything related to upgrades.

The basic execution environment for any program executed by the FreeBSD
kernel is formed by combination of kernel' syscall interface and some
system userspace code which makes the expected environment over the
bare-bone image state after execve. The environment is typically named
C runtime environment since C language ABI is directly tied into it,
and normal C programs only get whatever is provided by the C runtime
unless additional libraries are linked in. Trully, it is not just C
runtime, any other execution environment on top of the OS is based on
this one, but since almost every 'advanced' language runtime is backed
by C language and its runtime, the name stuck.

FreeBSD C runtime, arguably, is provided by the following four objects:
/libexec/ld-elf.so.1
/lib/libc.so.7
/lib/libthr.so.3
/lib/libm.so.5
There, we do *guarantee* that the external ABI of the whole pack of
these four objects is backward compatible, i.e. if the binary was
compiled against set if base libraries at earlier date (may be also
on earlier branch), then the binary behaviour would be same when
executed on newer C runtime pack. This is not trivial to achieve,
besides technical measures that helps there, like backward-compatible
syscall interface, symbol versioning, providing fall-back code for
older interface, a lot of overhead in the development is enforced, like
carefull reviews of the changes, the policy and related discipline of
versioning, following published ABI standards, and so on.

But, internal ABI of the C runtime pack, i.e. interfaces which make rtld
work with libc and libthr, or way by which libthr, when loaded, makes
libc thread-aware, are not stable, and more, they are often changed
in backward-incompatible way. Requiring backward-compatibility there
would stop our ability to evolve the system. Answering some questions in
advance, yes, rtld delves into libc, libthr patches libc on load, libc
has hooks to control some libthr behaviour.

The only provision that we make is that ld-elf.so.1 is required to work
with older libc/libthr combination, but even then libc and libthr must
be built from the same sources with the same options set.

Now, returning to pkgbase, if you look at what libs are packed into clibs,
you see:
ld-elf.so.1
libc.so.7 (and modules like iconv tables or nss, if any)
libthr.so.3
libdl.so.1
libgcc{, _eh, _s}.so.1
libm.so.5
libedit.so.7
libncurses{, w}.so.8
libc++.so.1
It adds very popular libs like libncurses/libedit, and C++ runtime. The
basic reasoning is that this package is small and chances of something
going wrong while installing it are small as result. Put it other way,
the small clibs package organization makes it highly probable that
system is left in the consistent state (either all new libs, or all old
libs) after the upgrade, whetever the outcome is.

If the C runtime pack is not split from the whole 700MB+ update blob, libthr
update has almost certain chance to occur long after or before libc update,
so failures do tend to leave inconsistent rtld/libc/libthr set.  At best,
it gives you strange glitches, at worst you get unusable system that cannot
be repaired without exter

Re: CFT: FreeBSD Package Base

[Ken Moore]

On Monday, April 29, 2019 8:08:08 AM EDT, Konstantin Belousov wrote:

Cc: list trimmed to relevant.  Very long essey below, be warned.

On Sun, Apr 28, 2019 at 03:52:21PM -0400, k...@ixsystems.com wrote:

FreeBSD Community,

 

I'm pleased to announce a CFT for builds of FreeBSD 12-stable 
and 13-current
using "TrueOS-inspired" packaged base. These are stock FreeBSD 
images which
will allow users to perform all updating via the 'pkg' command 
directly. ...


I do not know what are design decisions for trueos pkgbase are, but I
do know something about in-tree split and why some packaging decisions
where made. I cannot attend your WG, but I believe the reasoning used
for the in-tree is important enough to represent it intact from the
source.  I have to start with some explanatory long text to put it into
the proper perspective.

There are two knots of interdependinces which are critical for correctness
of any upgrade where the target system cannot be simply 
discarded on failure:

1. C runtime
2. Minimal boot path to prompt.
Let me elaborate both, starting from point 1, which is 
typically very obscure

despite having the fundamental nature for anything related to upgrades.

The basic execution environment for any program executed by the FreeBSD
kernel is formed by combination of kernel' syscall interface and some
system userspace code which makes the expected environment over the
bare-bone image state after execve. The environment is typically named
C runtime environment since C language ABI is directly tied into it,
and normal C programs only get whatever is provided by the C runtime
unless additional libraries are linked in. Trully, it is not just C
runtime, any other execution environment on top of the OS is based on
this one, but since almost every 'advanced' language runtime is backed
by C language and its runtime, the name stuck.

FreeBSD C runtime, arguably, is provided by the following four objects:
/libexec/ld-elf.so.1
/lib/libc.so.7
/lib/libthr.so.3
/lib/libm.so.5
There, we do *guarantee* that the external ABI of the whole pack of
these four objects is backward compatible, i.e. if the binary was
compiled against set if base libraries at earlier date (may be also
on earlier branch), then the binary behaviour would be same when
executed on newer C runtime pack. This is not trivial to achieve,
besides technical measures that helps there, like backward-compatible
syscall interface, symbol versioning, providing fall-back code for
older interface, a lot of overhead in the development is enforced, like
carefull reviews of the changes, the policy and related discipline of
versioning, following published ABI standards, and so on.

But, internal ABI of the C runtime pack, i.e. interfaces which make rtld
work with libc and libthr, or way by which libthr, when loaded, makes
libc thread-aware, are not stable, and more, they are often changed
in backward-incompatible way. Requiring backward-compatibility there
would stop our ability to evolve the system. Answering some questions in
advance, yes, rtld delves into libc, libthr patches libc on load, libc
has hooks to control some libthr behaviour.

The only provision that we make is that ld-elf.so.1 is required to work
with older libc/libthr combination, but even then libc and libthr must
be built from the same sources with the same options set.

Now, returning to pkgbase, if you look at what libs are packed into clibs,
you see:
ld-elf.so.1
libc.so.7 (and modules like iconv tables or nss, if any)
libthr.so.3
libdl.so.1
libgcc{, _eh, _s}.so.1
libm.so.5
libedit.so.7
libncurses{, w}.so.8
libc++.so.1
It adds very popular libs like libncurses/libedit, and C++ runtime. The
basic reasoning is that this package is small and chances of something
going wrong while installing it are small as result. Put it other way,
the small clibs package organization makes it highly probable that
system is left in the consistent state (either all new libs, or all old
libs) after the upgrade, whetever the outcome is.

If the C runtime pack is not split from the whole 700MB+ update blob, libthr
update has almost certain chance to occur long after or before libc update,
so failures do tend to leave inconsistent rtld/libc/libthr set.  At best,
it gives you strange glitches, at worst you get unusable system that cannot
be repaired without external media.

Now, the second item, the minimal boot path. By definition, it consists
of everything that is required to get bare-bone shell prompt in single
user mode, and where user can repair failed upgrade. Arguably, it should
also include the tools to configure the network and fix filesystems. So
it should consists of
loader (including forth/lua scripts)
kernel
C runtime
/sbin/init
/bin/sh
newfs/fsck/tunefs for UFS
zfs/zfspool and libs for ZFS
ifconfig/route/ping
In this set, zfs and ne

Re: CFT: FreeBSD Package Base

[Lev Serebryakov]

On 28.04.2019 22:52, k...@ixsystems.com wrote:

> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and 13-current
> using "TrueOS-inspired" packaged base. These are stock FreeBSD images which
> will allow users to perform all updating via the 'pkg' command directly.
> Rather than trying to answer all questions in this announcement, we've
> created a FAQ page with more details. Please refer to this page, and let us
> know if you have additional questions that we can include on that page going
> forward.
 Is it too coarse, isn't it?

 I'm not very interested in packetized base for "big servers" which
contains full FreeBSd installation, but I have several NanoBSD
installations, which have more than 100 "WITHOUT_XXX" options in
src.conf. I want to have packetized base to create such images via `pkg'
Not all these options could be converted to packages, options like
WITHOUT_KERBEROS is more build option, but about 2/3 of these options
turn off some file-based features, like sendmail, PPP, toolchain or bzip2.

 IMHO, to be really useful packets in base should be based on these
src.conf options to have ability to skip unneeded "optional" base
components (including, for example, man pages!).

 And one more, not covered with src.conf WITHOUT_XXX: static libraries
and header files, of course!

-- 
// Lev Serebryakov



signature.asc
Description: OpenPGP digital signature

Re: CFT: FreeBSD Package Base

[Kris Moore]

On Mon, Apr 29, 2019 at 8:12 AM Emmanuel Vadot 
wrote:

>
>  Hi Kris,
>
> On Sun, 28 Apr 2019 15:52:21 -0400
>  wrote:
>
> > FreeBSD Community,
> >
> >
> >
> > I'm pleased to announce a CFT for builds of FreeBSD 12-stable and
> 13-current
> > using "TrueOS-inspired" packaged base. These are stock FreeBSD images
> which
> > will allow users to perform all updating via the 'pkg' command directly.
> > Rather than trying to answer all questions in this announcement, we've
> > created a FAQ page with more details. Please refer to this page, and let
> us
> > know if you have additional questions that we can include on that page
> going
> > forward.
> >
>
>  While I appreciate the effort I have some doubt about your
> "re-implementation" of pkgbase. I don't see any improvement compared to
> what is in base currently, I even see downside of your implementation.
>
>  - How do you plan with the need of updating kernel first, reboot and
> updating the rest of the userland after ? (Needed for major and minor
> upgrade, 12.0 to 12.1 for example, and simple update in -STABLE and
> -HEAD branch). This is still a problem with the base pkgbase.
>

We've written our own tool "sysutils/sysup" in GO which handles this. It
performs updates using Boot-Environments to ensure that kernel/world are
updated at same time.




>  - This is even worse because you are using the same repository for
> base and pkg so if a user pkg update and both kernel and pkg(8) needs
> to be updated and pkg use a new syscall or capsicum thing it will be
> updated first and couldn't proceed with the rest of the update (this is
> a supposition, I haven't personally tested).
>

See above.


>  - It seems that multiple kernels isn't supported in your
> implementation, this is already supported in pkgbase but still need
> some love. This is an important point as it will allow user to choose
> easily the kernel that they want to use and will also allow us
> developper to push kernels with new features to help testing.
>

Incorrect, on the 13-CURRENT build if you install kernel-debug, you'll get
the Witness-enabled kernel installed alongside non-debugging one.


>  - Since you reduced the granularity on the userland bits it would mean
> that if we use your implementation for -p updates we would download the
> whole userland packages instead of just updating the package that was
> patched. For example with pkgbase, updating from 12.0 to 12.0p1 will
> only update the FreeBSD-runtime package. Yes this package is still big
> to download when you compare to what have changed but until pkg(8) have
> delta pkg supports (and if it will have support, I don't know if
> this is a wish or not) this is the best way to go.
>

Correct, this is by design. We used the in-tree pkg base for nearly a year,
and found that the granularity didn't really offer any savings from a
download or time perspective. Updating 100+ packages took far longer than a
single one, due to all the meta operations. Additionally in real-world
usage, we found that base packages tended to all get updated at the same
time, which took far longer via pkg, since it had to go and perform 100+
fetch operations just to download the base system bits.



>  - I see that you are sorting the plist for kernel and userland based
> on the line length [1], why is that ?


Whoops! I'll fix :)


>
>  I think that the only advantage that your solution offers is that if
> we remove a componant of base (rcmds for example in 12-CURRENT) those
> files would be removed as they are in the userland-base package while
> for pkgbase the FreeBSD-rcmd package will be deleted in the repo and
> will not be deleted in the user computer.
>


Correct, this is one of the things which prompted us to go this direction.
Being able to handle crazy mixed WITH/WITHOUT flags was important to us,
current pkg base did not handle that so gracefully. Additionally we've
added some additional features, such as being able to 'pkg install os/src'
to get system sources used in exact build, as well as being able to rebuild
your local world / kernel packages using ports "make config" framework is
super handy.


>
> >
> > Additionally, I will be hosting a Package Base working group at BSDCan
> 2019,
> > and welcome user and developer attendance to discuss this and other
> ongoing
> > package work:
> >
> >
> >
> > https://wiki.freebsd.org/DevSummit/201905/PackageBase
> >
>
>  I will be present and looking forward to work with you on this.
>
>  Cheers,
>
>  P.S. :  FYI I'm working on pkgbase currently and I will have some
> patches to commit soon (bsdinstall support, memstick creation that
> install a pkgbase aware installaton etc ...).
>

Great! Looking forward to discussion then!


>
> [1] :
>
> https://github.com/trueos/trueos-ports/blob/trueos-master/os/userland-base/Makefile#L35
>
> --
> Emmanuel Vadot  
>
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
T

RE: CFT: FreeBSD Package Base

[kris]

Just echo'ing what Ken has stated here. This is part of the reason we
implemented this style with the less granular pkgs. The entire
'userland-base' is one single archive, minus docs/tests/debug files. This
means a single 'pkg upgrade' of userland-base will be able to finish
extraction in one pass, ensuring that libc/libthr/libelf and friends all are
splatted on disk in the same pass. 


-- 
Kris Moore
Vice President of Engineering
iXsystems, Inc
Ph: (408) 943-4100
Ph: (408) 943-4101
The Groundbreaking TrueNAS M-Series -
Enterprise Storage & Servers Driven By Open Source

-Original Message-
From: owner-freebsd-pkgb...@freebsd.org 
On Behalf Of Ken Moore
Sent: Monday, April 29, 2019 8:55 AM
To: freebsd-pkgbase@freebsd.org
Subject: Re: CFT: FreeBSD Package Base

On Monday, April 29, 2019 8:08:08 AM EDT, Konstantin Belousov wrote:
> Cc: list trimmed to relevant.  Very long essey below, be warned.
>
> On Sun, Apr 28, 2019 at 03:52:21PM -0400, k...@ixsystems.com wrote:
>> FreeBSD Community,
>> 
>>  
>> 
>> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and 
>> 13-current using "TrueOS-inspired" packaged base. These are stock 
>> FreeBSD images which will allow users to perform all updating via the 
>> 'pkg' command directly. ...
>
> I do not know what are design decisions for trueos pkgbase are, but I 
> do know something about in-tree split and why some packaging decisions 
> where made. I cannot attend your WG, but I believe the reasoning used 
> for the in-tree is important enough to represent it intact from the 
> source.  I have to start with some explanatory long text to put it 
> into the proper perspective.
>
> There are two knots of interdependinces which are critical for 
> correctness of any upgrade where the target system cannot be simply 
> discarded on failure:
> 1. C runtime
> 2. Minimal boot path to prompt.
> Let me elaborate both, starting from point 1, which is typically very 
> obscure despite having the fundamental nature for anything related to 
> upgrades.
>
> The basic execution environment for any program executed by the 
> FreeBSD kernel is formed by combination of kernel' syscall interface 
> and some system userspace code which makes the expected environment 
> over the bare-bone image state after execve. The environment is 
> typically named C runtime environment since C language ABI is directly 
> tied into it, and normal C programs only get whatever is provided by 
> the C runtime unless additional libraries are linked in. Trully, it is 
> not just C runtime, any other execution environment on top of the OS 
> is based on this one, but since almost every 'advanced' language 
> runtime is backed by C language and its runtime, the name stuck.
>
> FreeBSD C runtime, arguably, is provided by the following four objects:
>   /libexec/ld-elf.so.1
>   /lib/libc.so.7
>   /lib/libthr.so.3
>   /lib/libm.so.5
> There, we do *guarantee* that the external ABI of the whole pack of 
> these four objects is backward compatible, i.e. if the binary was 
> compiled against set if base libraries at earlier date (may be also on 
> earlier branch), then the binary behaviour would be same when executed 
> on newer C runtime pack. This is not trivial to achieve, besides 
> technical measures that helps there, like backward-compatible syscall 
> interface, symbol versioning, providing fall-back code for older 
> interface, a lot of overhead in the development is enforced, like 
> carefull reviews of the changes, the policy and related discipline of 
> versioning, following published ABI standards, and so on.
>
> But, internal ABI of the C runtime pack, i.e. interfaces which make 
> rtld work with libc and libthr, or way by which libthr, when loaded, 
> makes libc thread-aware, are not stable, and more, they are often 
> changed in backward-incompatible way. Requiring backward-compatibility 
> there would stop our ability to evolve the system. Answering some 
> questions in advance, yes, rtld delves into libc, libthr patches libc 
> on load, libc has hooks to control some libthr behaviour.
>
> The only provision that we make is that ld-elf.so.1 is required to 
> work with older libc/libthr combination, but even then libc and libthr 
> must be built from the same sources with the same options set.
>
> Now, returning to pkgbase, if you look at what libs are packed into 
> clibs, you see:
>   ld-elf.so.1
>   libc.so.7 (and modules like iconv tables or nss, if any)
>   libthr.so.3
>   libdl.so.1
>   libgcc{, _eh, _s}.so.1
>   libm.so.5
>   libedit.so.7
>   libncurses{, w}.so.8
>   libc++.so.1
> It adds very popular libs like libncurses/libedit, and C++ runtime. 
> The basic reasoning is that this package is small and chances of 
> something going wrong while installing it are small as result. Put it 
> other way, the small clibs package organization makes it highly 
> probable that system is left in the consistent state (either all new

RE: CFT: FreeBSD Package Base

[kris]

This should be very doable with this package base. We use it for FreeNAS in a 
similar manner, where we disable a couple dozen things from base, resulting in 
a much more stripped down userland-base package. By default we also break out 
the doc/tests/debug bits into their own userland-* packages, for same reasons, 
to keep image nice and small.

-- 
Kris Moore
Vice President of Engineering
iXsystems, Inc
Ph: (408) 943-4100
Ph: (408) 943-4101
The Groundbreaking TrueNAS M-Series -
Enterprise Storage & Servers Driven By Open Source

-Original Message-
From: Lev Serebryakov  
Sent: Monday, April 29, 2019 9:20 AM
To: k...@ixsystems.com; freebsd-curr...@freebsd.org; 
freebsd-pkgbase@freebsd.org; freebsd-hack...@freebsd.org
Subject: Re: CFT: FreeBSD Package Base

On 28.04.2019 22:52, k...@ixsystems.com wrote:

> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and 
> 13-current using "TrueOS-inspired" packaged base. These are stock 
> FreeBSD images which will allow users to perform all updating via the 'pkg' 
> command directly.
> Rather than trying to answer all questions in this announcement, we've 
> created a FAQ page with more details. Please refer to this page, and 
> let us know if you have additional questions that we can include on 
> that page going forward.
 Is it too coarse, isn't it?

 I'm not very interested in packetized base for "big servers" which contains 
full FreeBSd installation, but I have several NanoBSD installations, which have 
more than 100 "WITHOUT_XXX" options in src.conf. I want to have packetized base 
to create such images via `pkg'
Not all these options could be converted to packages, options like 
WITHOUT_KERBEROS is more build option, but about 2/3 of these options turn off 
some file-based features, like sendmail, PPP, toolchain or bzip2.

 IMHO, to be really useful packets in base should be based on these src.conf 
options to have ability to skip unneeded "optional" base components (including, 
for example, man pages!).

 And one more, not covered with src.conf WITHOUT_XXX: static libraries and 
header files, of course!

--
// Lev Serebryakov


___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Lev Serebryakov]

On 29.04.2019 16:39, k...@ixsystems.com wrote:

> 
> This should be very doable with this package base. We use it for FreeNAS in a 
> similar manner, where we disable a couple dozen things from base, resulting 
> in a much more stripped down userland-base package. By default we also break 
> out the doc/tests/debug bits into their own userland-* packages, for same 
> reasons, to keep image nice and small.
> 
 Ok, after


# tar tf FreeBSD-HEAD-pkgbase-x64-20190426.iso | grep All
dist/FreeBSD:13:amd64/latest/All
dist/FreeBSD:13:amd64/latest/All/ca_root_nss-3.43_1.txz
dist/FreeBSD:13:amd64/latest/All/jq-1.6.txz
dist/FreeBSD:13:amd64/latest/All/kernel-20190420203550_1.txz
dist/FreeBSD:13:amd64/latest/All/oniguruma-6.9.1.txz
dist/FreeBSD:13:amd64/latest/All/pkg-1.10.5_5.txz
dist/FreeBSD:13:amd64/latest/All/userland-20190420203550.txz
dist/FreeBSD:13:amd64/latest/All/userland-base-20190420203550_7.txz
dist/FreeBSD:13:amd64/latest/All/userland-docs-20190420203550.txz
dist/FreeBSD:13:amd64/latest/All/userland-lib32-20190420203550.txz
#

 I was under impression, that there is only 3 userland packages, not
100+ :-)

-- 
// Lev Serebryakov



signature.asc
Description: OpenPGP digital signature

Re: CFT: FreeBSD Package Base

[Goran Mekić]

On Mon, Apr 29, 2019 at 09:25:05AM -0400, Kris Moore wrote:
> We've written our own tool "sysutils/sysup" in GO which handles this. It
> performs updates using Boot-Environments to ensure that kernel/world are
> updated at same time.

If I'm right, UFS doesn't support boot environments, so how would it
work for UFS based installs?

I personally feel GO is a bit ackward choice of language for something
that practically should be part of base. At least I would expect OS
update/upgrade not to require any external package.

Regards,
meka


signature.asc
Description: PGP signature

Re: CFT: FreeBSD Package Base

[Warner Losh]

On Mon, Apr 29, 2019 at 7:50 AM Lev Serebryakov  wrote:

> On 29.04.2019 16:39, k...@ixsystems.com wrote:
>
> >
> > This should be very doable with this package base. We use it for FreeNAS
> in a similar manner, where we disable a couple dozen things from base,
> resulting in a much more stripped down userland-base package. By default we
> also break out the doc/tests/debug bits into their own userland-* packages,
> for same reasons, to keep image nice and small.
> >
>  Ok, after
>
>
> # tar tf FreeBSD-HEAD-pkgbase-x64-20190426.iso | grep All
> dist/FreeBSD:13:amd64/latest/All
> dist/FreeBSD:13:amd64/latest/All/ca_root_nss-3.43_1.txz
> dist/FreeBSD:13:amd64/latest/All/jq-1.6.txz
> dist/FreeBSD:13:amd64/latest/All/kernel-20190420203550_1.txz
> dist/FreeBSD:13:amd64/latest/All/oniguruma-6.9.1.txz
> dist/FreeBSD:13:amd64/latest/All/pkg-1.10.5_5.txz
> dist/FreeBSD:13:amd64/latest/All/userland-20190420203550.txz
> dist/FreeBSD:13:amd64/latest/All/userland-base-20190420203550_7.txz
> dist/FreeBSD:13:amd64/latest/All/userland-docs-20190420203550.txz
> dist/FreeBSD:13:amd64/latest/All/userland-lib32-20190420203550.txz
> #
>
>  I was under impression, that there is only 3 userland packages, not
> 100+ :-)
>

It's a tradeoff... 100+ packages is super granular, but also a PITA to
manage. 3 package installs quite a bit faster than the 100+ packages due to
a large fixed cost in pkg per package, but isn't granular enough to tailor
for NanoBSD systems.

Warner


> --
> // Lev Serebryakov
>
>
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Kris Moore]

On Mon, Apr 29, 2019 at 9:53 AM Warner Losh  wrote:

>
>
> On Mon, Apr 29, 2019 at 7:50 AM Lev Serebryakov  wrote:
>
>> On 29.04.2019 16:39, k...@ixsystems.com wrote:
>>
>> >
>> > This should be very doable with this package base. We use it for
>> FreeNAS in a similar manner, where we disable a couple dozen things from
>> base, resulting in a much more stripped down userland-base package. By
>> default we also break out the doc/tests/debug bits into their own
>> userland-* packages, for same reasons, to keep image nice and small.
>> >
>>  Ok, after
>>
>>
>> # tar tf FreeBSD-HEAD-pkgbase-x64-20190426.iso | grep All
>> dist/FreeBSD:13:amd64/latest/All
>> dist/FreeBSD:13:amd64/latest/All/ca_root_nss-3.43_1.txz
>> dist/FreeBSD:13:amd64/latest/All/jq-1.6.txz
>> dist/FreeBSD:13:amd64/latest/All/kernel-20190420203550_1.txz
>> dist/FreeBSD:13:amd64/latest/All/oniguruma-6.9.1.txz
>> dist/FreeBSD:13:amd64/latest/All/pkg-1.10.5_5.txz
>> dist/FreeBSD:13:amd64/latest/All/userland-20190420203550.txz
>> dist/FreeBSD:13:amd64/latest/All/userland-base-20190420203550_7.txz
>> dist/FreeBSD:13:amd64/latest/All/userland-docs-20190420203550.txz
>> dist/FreeBSD:13:amd64/latest/All/userland-lib32-20190420203550.txz
>> #
>>
>>  I was under impression, that there is only 3 userland packages, not
>> 100+ :-)
>>
>
> It's a tradeoff... 100+ packages is super granular, but also a PITA to
> manage. 3 package installs quite a bit faster than the 100+ packages due to
> a large fixed cost in pkg per package, but isn't granular enough to tailor
> for NanoBSD systems.
>
> Warner
>
>

Correct. Its all in the FAQ, there are about a dozen available packages:

https://trueos.github.io/pkgbase-docs/#which-base-packages-are-available

Really you only need a few of them on typical system,
userland(meta-pkg)/userland-base/userland-docs/kernel


> --
>> // Lev Serebryakov
>>
>>
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

RE: CFT: FreeBSD Package Base

[kris]

Correct, this is ZFS only. And it's something we're using specific to FreeNAS / 
TrueOS, which is why I didn't originally mention it as apart of our CFT. 

For UFS, there will need to be additional care taken when doing updates. 

-- 
Kris Moore
Vice President of Engineering
iXsystems, Inc
Ph: (408) 943-4100
Ph: (408) 943-4101
The Groundbreaking TrueNAS M-Series -
Enterprise Storage & Servers Driven By Open Source

-Original Message-
From: Goran Mekić  
Sent: Monday, April 29, 2019 9:43 AM
To: Kris Moore 
Cc: Emmanuel Vadot ; FreeBSD Stable 
; FreeBSD Current ; 
freebsd-pkgbase@freebsd.org; freebsd-...@freebsd.org; 
freebsd-hack...@freebsd.org; freebsd-po...@freebsd.org
Subject: Re: CFT: FreeBSD Package Base

On Mon, Apr 29, 2019 at 09:25:05AM -0400, Kris Moore wrote:
> We've written our own tool "sysutils/sysup" in GO which handles this. 
> It performs updates using Boot-Environments to ensure that 
> kernel/world are updated at same time.

If I'm right, UFS doesn't support boot environments, so how would it work for 
UFS based installs?

I personally feel GO is a bit ackward choice of language for something that 
practically should be part of base. At least I would expect OS update/upgrade 
not to require any external package.

Regards,
meka

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Emmanuel Vadot]

On Mon, 29 Apr 2019 09:25:05 -0400
Kris Moore  wrote:

> On Mon, Apr 29, 2019 at 8:12 AM Emmanuel Vadot 
> wrote:
> 
> >
> >  Hi Kris,
> >
> > On Sun, 28 Apr 2019 15:52:21 -0400
> >  wrote:
> >
> > > FreeBSD Community,
> > >
> > >
> > >
> > > I'm pleased to announce a CFT for builds of FreeBSD 12-stable and
> > 13-current
> > > using "TrueOS-inspired" packaged base. These are stock FreeBSD images
> > which
> > > will allow users to perform all updating via the 'pkg' command directly.
> > > Rather than trying to answer all questions in this announcement, we've
> > > created a FAQ page with more details. Please refer to this page, and let
> > us
> > > know if you have additional questions that we can include on that page
> > going
> > > forward.
> > >
> >
> >  While I appreciate the effort I have some doubt about your
> > "re-implementation" of pkgbase. I don't see any improvement compared to
> > what is in base currently, I even see downside of your implementation.
> >
> >  - How do you plan with the need of updating kernel first, reboot and
> > updating the rest of the userland after ? (Needed for major and minor
> > upgrade, 12.0 to 12.1 for example, and simple update in -STABLE and
> > -HEAD branch). This is still a problem with the base pkgbase.
> >
> 
> We've written our own tool "sysutils/sysup" in GO which handles this. It
> performs updates using Boot-Environments to ensure that kernel/world are
> updated at same time.
> 

 Which could never be imported into FreeBSD.

> 
> 
> >  - This is even worse because you are using the same repository for
> > base and pkg so if a user pkg update and both kernel and pkg(8) needs
> > to be updated and pkg use a new syscall or capsicum thing it will be
> > updated first and couldn't proceed with the rest of the update (this is
> > a supposition, I haven't personally tested).
> >
> 
> See above.
> 
> 
> >  - It seems that multiple kernels isn't supported in your
> > implementation, this is already supported in pkgbase but still need
> > some love. This is an important point as it will allow user to choose
> > easily the kernel that they want to use and will also allow us
> > developper to push kernels with new features to help testing.
> >
> 
> Incorrect, on the 13-CURRENT build if you install kernel-debug, you'll get
> the Witness-enabled kernel installed alongside non-debugging one.

 Mhm no, the kernel-debug packages only add the debug file
in /usr/lib/debug/boot/
 I'm talking about installing multiple kernels in //
(i.e. /boot/kernel.GENERIC /boot/kernel.MYFEATUREIWANTTOTEST) like
describe here :
https://wiki.freebsd.org/PkgBase#Project_goals_and_additional_unresolved_issues
in the "How to handle /boot/kernel and /boot/kernel.$KERNCONF" point.

> 
> >  - Since you reduced the granularity on the userland bits it would mean
> > that if we use your implementation for -p updates we would download the
> > whole userland packages instead of just updating the package that was
> > patched. For example with pkgbase, updating from 12.0 to 12.0p1 will
> > only update the FreeBSD-runtime package. Yes this package is still big
> > to download when you compare to what have changed but until pkg(8) have
> > delta pkg supports (and if it will have support, I don't know if
> > this is a wish or not) this is the best way to go.
> >
> 
> Correct, this is by design. We used the in-tree pkg base for nearly a year,
> and found that the granularity didn't really offer any savings from a
> download or time perspective. Updating 100+ packages took far longer than a
> single one, due to all the meta operations. Additionally in real-world
> usage, we found that base packages tended to all get updated at the same
> time, which took far longer via pkg, since it had to go and perform 100+
> fetch operations just to download the base system bits.
> 

 But you never need to update 100+ packages on a proper pkgbase setup
for -p updates.
 Again on a 12.0 to 12.0-p1 update only one package will be updated.

> 
> >  - I see that you are sorting the plist for kernel and userland based
> > on the line length [1], why is that ?
> 
> 
> Whoops! I'll fix :)
> 
> 
> >
> >  I think that the only advantage that your solution offers is that if
> > we remove a componant of base (rcmds for example in 12-CURRENT) those
> > files would be removed as they are in the userland-base package while
> > for pkgbase the FreeBSD-rcmd package will be deleted in the repo and
> > will not be deleted in the user computer.
> >
> 
> 
> Correct, this is one of the things which prompted us to go this direction.
> Being able to handle crazy mixed WITH/WITHOUT flags was important to us,
> current pkg base did not handle that so gracefully. 

 Can you give me more info on this ? What where the WITH/WITHOUT flags
that causes problems ?

> Additionally we've
> added some additional features, such as being able to 'pkg install os/src'
> to get system sources used in exact build, as well as being able to rebuild
> your local world 

Re: CFT: FreeBSD Package Base

[Rodney W. Grimes]

> 
> Correct, this is ZFS only. And it's something we're using specific to FreeNAS 
> / TrueOS, which is why I didn't originally mention it as apart of our CFT. 

Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
calling this FreeBSD pkg base when it is not was wrong,
and miss leading.

> For UFS, there will need to be additional care taken when doing updates. 
> 
> -- 
> Kris Moore
> Vice President of Engineering
> iXsystems, Inc
> Ph: (408) 943-4100
> Ph: (408) 943-4101
> The Groundbreaking TrueNAS M-Series -
> Enterprise Storage & Servers Driven By Open Source
> 
> -Original Message-
> From: Goran Meki?  
> Sent: Monday, April 29, 2019 9:43 AM
> To: Kris Moore 
> Cc: Emmanuel Vadot ; FreeBSD Stable 
> ; FreeBSD Current ; 
> freebsd-pkgbase@freebsd.org; freebsd-...@freebsd.org; 
> freebsd-hack...@freebsd.org; freebsd-po...@freebsd.org
> Subject: Re: CFT: FreeBSD Package Base
> 
> On Mon, Apr 29, 2019 at 09:25:05AM -0400, Kris Moore wrote:
> > We've written our own tool "sysutils/sysup" in GO which handles this. 
> > It performs updates using Boot-Environments to ensure that 
> > kernel/world are updated at same time.
> 
> If I'm right, UFS doesn't support boot environments, so how would it work for 
> UFS based installs?
> 
> I personally feel GO is a bit ackward choice of language for something that 
> practically should be part of base. At least I would expect OS update/upgrade 
> not to require any external package.
> 
> Regards,
> meka
> 
> ___
> freebsd-curr...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 
> 

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Kris Moore]

On Mon, Apr 29, 2019 at 9:55 AM Emmanuel Vadot 
wrote:

> On Mon, 29 Apr 2019 09:25:05 -0400
> Kris Moore  wrote:
>
> > On Mon, Apr 29, 2019 at 8:12 AM Emmanuel Vadot 
> > wrote:
> >
> > >
> > >  Hi Kris,
> > >
> > > On Sun, 28 Apr 2019 15:52:21 -0400
> > >  wrote:
> > >
> > > > FreeBSD Community,
> > > >
> > > >
> > > >
> > > > I'm pleased to announce a CFT for builds of FreeBSD 12-stable and
> > > 13-current
> > > > using "TrueOS-inspired" packaged base. These are stock FreeBSD images
> > > which
> > > > will allow users to perform all updating via the 'pkg' command
> directly.
> > > > Rather than trying to answer all questions in this announcement,
> we've
> > > > created a FAQ page with more details. Please refer to this page, and
> let
> > > us
> > > > know if you have additional questions that we can include on that
> page
> > > going
> > > > forward.
> > > >
> > >
> > >  While I appreciate the effort I have some doubt about your
> > > "re-implementation" of pkgbase. I don't see any improvement compared to
> > > what is in base currently, I even see downside of your implementation.
> > >
> > >  - How do you plan with the need of updating kernel first, reboot and
> > > updating the rest of the userland after ? (Needed for major and minor
> > > upgrade, 12.0 to 12.1 for example, and simple update in -STABLE and
> > > -HEAD branch). This is still a problem with the base pkgbase.
> > >
> >
> > We've written our own tool "sysutils/sysup" in GO which handles this. It
> > performs updates using Boot-Environments to ensure that kernel/world are
> > updated at same time.
> >
>
>  Which could never be imported into FreeBSD.
>

Not suggesting it should be. Just information on how we solved that problem
in our own appliance / platforms. For FreeBSD it would need some tooling
still to handle this style of updating, regardless of which pkg base is
used.

And for what it's worth, FreeBSD is all the poorer for not being able to
bring modern language based tools into the base. Personally I'm hoping the
shift to base-packages makes this a moot point since the idea of 'what is
base' can be diluted to just a manifest of what gets installed out of box.
Just my 2C on the matter though :)


>
> >
> >
> > >  - This is even worse because you are using the same repository for
> > > base and pkg so if a user pkg update and both kernel and pkg(8) needs
> > > to be updated and pkg use a new syscall or capsicum thing it will be
> > > updated first and couldn't proceed with the rest of the update (this is
> > > a supposition, I haven't personally tested).
> > >
> >
> > See above.
>

You can selectively update os/kernel and reboot before doing rest.


> >
> >
> > >  - It seems that multiple kernels isn't supported in your
> > > implementation, this is already supported in pkgbase but still need
> > > some love. This is an important point as it will allow user to choose
> > > easily the kernel that they want to use and will also allow us
> > > developper to push kernels with new features to help testing.
> > >
> >
> > Incorrect, on the 13-CURRENT build if you install kernel-debug, you'll
> get
> > the Witness-enabled kernel installed alongside non-debugging one.
>
>  Mhm no, the kernel-debug packages only add the debug file
> in /usr/lib/debug/boot/
>  I'm talking about installing multiple kernels in //
> (i.e. /boot/kernel.GENERIC /boot/kernel.MYFEATUREIWANTTOTEST) like
> describe here :
>
> https://wiki.freebsd.org/PkgBase#Project_goals_and_additional_unresolved_issues
> in the "How to handle /boot/kernel and /boot/kernel.$KERNCONF" point.
>
>
Incorrect, os/kernel-debug installs /boot/kernel-debug which is (on
13-CURRENT) the Witness enabled kernel. os/kernel-debug-symbols are the
/usr/lib/debug bits.


>
> > >
> > >  I think that the only advantage that your solution offers is that if
> > > we remove a componant of base (rcmds for example in 12-CURRENT) those
> > > files would be removed as they are in the userland-base package while
> > > for pkgbase the FreeBSD-rcmd package will be deleted in the repo and
> > > will not be deleted in the user computer.
> > >
> >
> >
> > Correct, this is one of the things which prompted us to go this
> direction.
> > Being able to handle crazy mixed WITH/WITHOUT flags was important to us,
> > current pkg base did not handle that so gracefully.
>
>  Can you give me more info on this ? What where the WITH/WITHOUT flags
> that causes problems ?
>


I may have to pick Miwi's brain on this, but I believe some of the issues
we saw were when introducing flags such as WITHOUT_RADIUS. Additionally
there is a runtime problem to solve. I.E. if you change flags mid-stream,
and user updates, there was no clean way on pkg-side to remove those
already installed granular packages. Not without external tooling anyway.



>
> --
> Emmanuel Vadot  
>
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, se

Re: CFT: FreeBSD Package Base

[Kris Moore]

On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
freebsd-...@gndrsh.dnsmgr.net> wrote:

> >
> > Correct, this is ZFS only. And it's something we're using specific to
> FreeNAS / TrueOS, which is why I didn't originally mention it as apart of
> our CFT.
>
> Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
> calling this FreeBSD pkg base when it is not was wrong,
> and miss leading.
>

Sorry, I disagree. This pkg base is independent of the ZFS tool we're using
to wrangle boot-environments. Hence why it wasn't mentioned in the CFT.
These base packages work the same as existing in-tree pkg base on UFS, no
difference. If anything are probably safer due to being able to update all
of userland in single extract operation, so you don't have out of order
extraction of libc or some such.


>
> > For UFS, there will need to be additional care taken when doing updates.
> >
> > --
> > Kris Moore
> > Vice President of Engineering
> > iXsystems, Inc
> > Ph: (408) 943-4100
> > Ph: (408) 943-4101
> > The Groundbreaking TrueNAS M-Series -
> > Enterprise Storage & Servers Driven By Open Source
> >
> > -Original Message-
> > From: Goran Meki? 
> > Sent: Monday, April 29, 2019 9:43 AM
> > To: Kris Moore 
> > Cc: Emmanuel Vadot ; FreeBSD Stable <
> freebsd-sta...@freebsd.org>; FreeBSD Current ;
> freebsd-pkgbase@freebsd.org; freebsd-...@freebsd.org;
> freebsd-hack...@freebsd.org; freebsd-po...@freebsd.org
> > Subject: Re: CFT: FreeBSD Package Base
> >
> > On Mon, Apr 29, 2019 at 09:25:05AM -0400, Kris Moore wrote:
> > > We've written our own tool "sysutils/sysup" in GO which handles this.
> > > It performs updates using Boot-Environments to ensure that
> > > kernel/world are updated at same time.
> >
> > If I'm right, UFS doesn't support boot environments, so how would it
> work for UFS based installs?
> >
> > I personally feel GO is a bit ackward choice of language for something
> that practically should be part of base. At least I would expect OS
> update/upgrade not to require any external package.
> >
> > Regards,
> > meka
> >
> > ___
> > freebsd-curr...@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> freebsd-current-unsubscr...@freebsd.org"
> >
> >
>
> --
> Rod Grimes
> rgri...@freebsd.org
>
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[David Chisnall]

On 29/04/2019 14:19, Lev Serebryakov wrote:

  I'm not very interested in packetized base for "big servers" which
contains full FreeBSd installation


'Big servers' may have a full FreeBSD installation in the base system, 
but they may also have hundreds of jails that want the absolute minimum 
required for the service that they're exporting.


FreeBSD is currently suffering quite a lot from the lack of any solid 
story here.  The vast majority of cloud deployments are now using some 
combination of Docker and Kubernetes or equivalents to spin up a large 
number of VMs and an even larger number of microservice containers 
within them.  This should be something that FreeBSD is ideal for - jails 
preform better and provide a more coherent interface than the mess of 
cgroups and seccomp-bpf that Linux containers use.


It *ought* to be trivial to create a jail that has basically nothing 
other than the core libraries (and maybe a shell) and is managed from 
the outside.  Even the few FreeBSD core utilities that support jails 
don't really work like this (for example, I can use pkg to install 
something in a jail, but doing so implicitly installs a copy of the pkg 
tool inside the jail and invokes that).


David
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

RE: CFT: FreeBSD Package Base

[kris]

> > >
> > Incorrect, os/kernel-debug installs /boot/kernel-debug which is (on
> > 13-CURRENT) the Witness enabled kernel. os/kernel-debug-symbols are
> > the /usr/lib/debug bits.
> 
>  I only see kernel-20190420203550_1.txz and kernel-debug-
> 20190420203550.txz in https://pkg.trueos.org/pkg/freebsd-
> pkgbase/FreeBSD%3A13%3Aamd64/latest/All/
> and kernel-debug only contain the debug files.
>  If I'm not looking in the right directory please correct me.
> 
> 
> --
> Emmanuel Vadot  


Ahh, you are correct. I checked and those packages haven't pushed to the
mirrors yet, Jenkins is still chewing on a build of them here. I was using
the 12-stable packages yesterday which has these changes. They should be
synced up to the mirrors in the next 24-48 hours. Sorry about the confusion.


-- 
Kris Moore
Vice President of Engineering
iXsystems, Inc
Ph: (408) 943-4100
Ph: (408) 943-4101
The Groundbreaking TrueNAS M-Series -
Enterprise Storage & Servers Driven By Open Source

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Emmanuel Vadot]

On Mon, 29 Apr 2019 10:05:59 -0400
Kris Moore  wrote:

> On Mon, Apr 29, 2019 at 9:55 AM Emmanuel Vadot 
> wrote:
> 
> > On Mon, 29 Apr 2019 09:25:05 -0400
> > Kris Moore  wrote:
> >
> > > On Mon, Apr 29, 2019 at 8:12 AM Emmanuel Vadot 
> > > wrote:
> > >
> > > >
> > > >  Hi Kris,
> > > >
> > > > On Sun, 28 Apr 2019 15:52:21 -0400
> > > >  wrote:
> > > >
> > > > > FreeBSD Community,
> > > > >
> > > > >
> > > > >
> > > > > I'm pleased to announce a CFT for builds of FreeBSD 12-stable and
> > > > 13-current
> > > > > using "TrueOS-inspired" packaged base. These are stock FreeBSD images
> > > > which
> > > > > will allow users to perform all updating via the 'pkg' command
> > directly.
> > > > > Rather than trying to answer all questions in this announcement,
> > we've
> > > > > created a FAQ page with more details. Please refer to this page, and
> > let
> > > > us
> > > > > know if you have additional questions that we can include on that
> > page
> > > > going
> > > > > forward.
> > > > >
> > > >
> > > >  While I appreciate the effort I have some doubt about your
> > > > "re-implementation" of pkgbase. I don't see any improvement compared to
> > > > what is in base currently, I even see downside of your implementation.
> > > >
> > > >  - How do you plan with the need of updating kernel first, reboot and
> > > > updating the rest of the userland after ? (Needed for major and minor
> > > > upgrade, 12.0 to 12.1 for example, and simple update in -STABLE and
> > > > -HEAD branch). This is still a problem with the base pkgbase.
> > > >
> > >
> > > We've written our own tool "sysutils/sysup" in GO which handles this. It
> > > performs updates using Boot-Environments to ensure that kernel/world are
> > > updated at same time.
> > >
> >
> >  Which could never be imported into FreeBSD.
> >
> 
> Not suggesting it should be. Just information on how we solved that problem
> in our own appliance / platforms. For FreeBSD it would need some tooling
> still to handle this style of updating, regardless of which pkg base is
> used.
> 
> And for what it's worth, FreeBSD is all the poorer for not being able to
> bring modern language based tools into the base. Personally I'm hoping the
> shift to base-packages makes this a moot point since the idea of 'what is
> base' can be diluted to just a manifest of what gets installed out of box.
> Just my 2C on the matter though :)
> 
> 
> >
> > >
> > >
> > > >  - This is even worse because you are using the same repository for
> > > > base and pkg so if a user pkg update and both kernel and pkg(8) needs
> > > > to be updated and pkg use a new syscall or capsicum thing it will be
> > > > updated first and couldn't proceed with the rest of the update (this is
> > > > a supposition, I haven't personally tested).
> > > >
> > >
> > > See above.
> >
> 
> You can selectively update os/kernel and reboot before doing rest.
> 
> 
> > >
> > >
> > > >  - It seems that multiple kernels isn't supported in your
> > > > implementation, this is already supported in pkgbase but still need
> > > > some love. This is an important point as it will allow user to choose
> > > > easily the kernel that they want to use and will also allow us
> > > > developper to push kernels with new features to help testing.
> > > >
> > >
> > > Incorrect, on the 13-CURRENT build if you install kernel-debug, you'll
> > get
> > > the Witness-enabled kernel installed alongside non-debugging one.
> >
> >  Mhm no, the kernel-debug packages only add the debug file
> > in /usr/lib/debug/boot/
> >  I'm talking about installing multiple kernels in //
> > (i.e. /boot/kernel.GENERIC /boot/kernel.MYFEATUREIWANTTOTEST) like
> > describe here :
> >
> > https://wiki.freebsd.org/PkgBase#Project_goals_and_additional_unresolved_issues
> > in the "How to handle /boot/kernel and /boot/kernel.$KERNCONF" point.
> >
> >
> Incorrect, os/kernel-debug installs /boot/kernel-debug which is (on
> 13-CURRENT) the Witness enabled kernel. os/kernel-debug-symbols are the
> /usr/lib/debug bits.

 I only see kernel-20190420203550_1.txz and
kernel-debug-20190420203550.txz in
https://pkg.trueos.org/pkg/freebsd-pkgbase/FreeBSD%3A13%3Aamd64/latest/All/
and kernel-debug only contain the debug files.
 If I'm not looking in the right directory please correct me.

> 
> >
> > > >
> > > >  I think that the only advantage that your solution offers is that if
> > > > we remove a componant of base (rcmds for example in 12-CURRENT) those
> > > > files would be removed as they are in the userland-base package while
> > > > for pkgbase the FreeBSD-rcmd package will be deleted in the repo and
> > > > will not be deleted in the user computer.
> > > >
> > >
> > >
> > > Correct, this is one of the things which prompted us to go this
> > direction.
> > > Being able to handle crazy mixed WITH/WITHOUT flags was important to us,
> > > current pkg base did not handle that so gracefully.
> >
> >  Can you give me more info on this ? What where the WITH/WITHOUT flags
> > that causes pro

Re: CFT: FreeBSD Package Base

[Matthias Apitz]

Why this thread has to go to all these lists? I receive any mail 5
times!

matthias
-- 
Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub
N € I N zur EU!
"Gegen das EU-Europa der Banken, Konzerne und Kriegstreiber.
Für ein soziales und friedliches Europa der Völker." DKP


signature.asc
Description: PGP signature

RE: CFT: FreeBSD Package Base

[kris]

> -Original Message-
> From: Matthias Apitz 
> Sent: Monday, April 29, 2019 10:50 AM
> To: Emmanuel Vadot 
> Cc: Kris Moore ; FreeBSD Stable  sta...@freebsd.org>; freebsd-...@freebsd.org; freebsd-
> hack...@freebsd.org; FreeBSD Current ;
> freebsd-pkgbase@freebsd.org; freebsd-po...@freebsd.org
> Subject: Re: CFT: FreeBSD Package Base
> 
> 
> Why this thread has to go to all these lists? I receive any mail 5 times!
> 
>   Matthias

Fair point. I'll restrict my replies to the -pkgbase list from here on out, 
suggest others do the same. Sorry about the noise 😊

> --
> Matthias Apitz, ✉ g...@unixarea.de, http://www.unixarea.de/ +49-176-
> 38902045 Public GnuPG key: http://www.unixarea.de/key.pub N € I N zur EU!
> "Gegen das EU-Europa der Banken, Konzerne und Kriegstreiber.
> Für ein soziales und friedliches Europa der Völker." DKP

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

RE: CFT: FreeBSD Package Base

[kris]

> -Original Message-
> From: Rodney W. Grimes 
> Sent: Monday, April 29, 2019 10:41 AM
> To: Kris Moore 
> Cc: Rodney W. Grimes ; Goran Mekić
> ; Emmanuel Vadot ; FreeBSD
> Stable ; FreeBSD Current  curr...@freebsd.org>; freebsd-pkgbase@freebsd.org; freebsd-
> p...@freebsd.org; freebsd-hack...@freebsd.org; freebsd-po...@freebsd.org
> Subject: Re: CFT: FreeBSD Package Base
> 
> > On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
> > freebsd-...@gndrsh.dnsmgr.net> wrote:
> >
> > > >
> > > > Correct, this is ZFS only. And it's something we're using specific
> > > > to
> > > FreeNAS / TrueOS, which is why I didn't originally mention it as
> > > apart of our CFT.
> > >
> > > Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only", calling
> > > this FreeBSD pkg base when it is not was wrong, and miss leading.
> > >
> >
> > Sorry, I disagree.
> Which is fine.
> 
> > This pkg base is independent of the ZFS tool we're using
> > to wrangle boot-environments. Hence why it wasn't mentioned in the CFT.
> > These base packages work the same as existing in-tree pkg base on UFS,
no
> > difference. If anything are probably safer due to being able to update
all
> > of userland in single extract operation, so you don't have out of order
> > extraction of libc or some such.
> 
> You missed the major string change and focused on the edge,
> No comment on calling iXsystems :stuff: FreeBSD instead of
> FreeNAS/TrueOS?
> 
> That was the major point of my statement, your miss leading the user
> community, you yourself said this would never be imported into FreeBSD
> base, so I see no reason that it should be called "FreeBSD package Base",
> as it is not, that is a different project.
> 
> --
> Rod Grimes
rgri...@freebsd.org

I think somehow you've missed the entire point here. This is being brought
forth as a FreeBSD CFT in the hopes of upstream adoption. No misleading here
whatsoever. The only thing that I wouldn't expect to be imported into base
was this external tool we use on FreeNAS/TrueOS to handle our specific
use-case of ZFS only. Total strawman here.

Seriously, suggest you bother looking at it and reading further to get the
full context. If anything this is far less invasive since it doesn't require
lots of hacking on base, and can even be used to package old versions of
FreeBSD if desired. The only thing I changed to make these images was a
patch to bsdinstall to replace dist-file extraction with 'pkg install
userland kernel pkg ...'.

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Rodney W. Grimes]

> On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
> freebsd-...@gndrsh.dnsmgr.net> wrote:
> 
> > >
> > > Correct, this is ZFS only. And it's something we're using specific to
> > FreeNAS / TrueOS, which is why I didn't originally mention it as apart of
> > our CFT.
> >
> > Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
> > calling this FreeBSD pkg base when it is not was wrong,
> > and miss leading.
> >
> 
> Sorry, I disagree.
Which is fine.

> This pkg base is independent of the ZFS tool we're using
> to wrangle boot-environments. Hence why it wasn't mentioned in the CFT.
> These base packages work the same as existing in-tree pkg base on UFS, no
> difference. If anything are probably safer due to being able to update all
> of userland in single extract operation, so you don't have out of order
> extraction of libc or some such.

You missed the major string change and focused on the edge,
No comment on calling iXsystems :stuff: FreeBSD instead of FreeNAS/TrueOS?

That was the major point of my statement, your miss leading the user
community, you yourself said this would never be imported into FreeBSD
base, so I see no reason that it should be called "FreeBSD package Base",
as it is not, that is a different project.

> > > For UFS, there will need to be additional care taken when doing updates.
> > >
> > > --
> > > Kris Moore
> > > Vice President of Engineering
> > > iXsystems, Inc
> > > Ph: (408) 943-4100
> > > Ph: (408) 943-4101
> > > The Groundbreaking TrueNAS M-Series -
> > > Enterprise Storage & Servers Driven By Open Source
> > >
> > > -Original Message-
> > > From: Goran Meki? 
> > > Sent: Monday, April 29, 2019 9:43 AM
> > > To: Kris Moore 
> > > Cc: Emmanuel Vadot ; FreeBSD Stable <
> > freebsd-sta...@freebsd.org>; FreeBSD Current ;
> > freebsd-pkgbase@freebsd.org; freebsd-...@freebsd.org;
> > freebsd-hack...@freebsd.org; freebsd-po...@freebsd.org
> > > Subject: Re: CFT: FreeBSD Package Base
> > >
> > > On Mon, Apr 29, 2019 at 09:25:05AM -0400, Kris Moore wrote:
> > > > We've written our own tool "sysutils/sysup" in GO which handles this.
> > > > It performs updates using Boot-Environments to ensure that
> > > > kernel/world are updated at same time.
> > >
> > > If I'm right, UFS doesn't support boot environments, so how would it
> > work for UFS based installs?
> > >
> > > I personally feel GO is a bit ackward choice of language for something
> > that practically should be part of base. At least I would expect OS
> > update/upgrade not to require any external package.
> > >
> > > Regards,
> > > meka
> > >
> > > ___
> > > freebsd-curr...@freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "
> > freebsd-current-unsubscr...@freebsd.org"
> > >
> > >
> >
> > --
> > Rod Grimes
> > rgri...@freebsd.org
> >

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Cy Schubert]

In message <201904291441.x3tefmid072...@gndrsh.dnsmgr.net>, "Rodney W. 
Grimes"
writes:
> > On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
> > freebsd-...@gndrsh.dnsmgr.net> wrote:
> > 
> > > >
> > > > Correct, this is ZFS only. And it's something we're using specific to
> > > FreeNAS / TrueOS, which is why I didn't originally mention it as apart of
> > > our CFT.
> > >
> > > Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
> > > calling this FreeBSD pkg base when it is not was wrong,
> > > and miss leading.
> > >
> > 
> > Sorry, I disagree.
> Which is fine.
>
> > This pkg base is independent of the ZFS tool we're using
> > to wrangle boot-environments. Hence why it wasn't mentioned in the CFT.
> > These base packages work the same as existing in-tree pkg base on UFS, no
> > difference. If anything are probably safer due to being able to update all
> > of userland in single extract operation, so you don't have out of order
> > extraction of libc or some such.
>
> You missed the major string change and focused on the edge,
> No comment on calling iXsystems :stuff: FreeBSD instead of FreeNAS/TrueOS?
>
> That was the major point of my statement, your miss leading the user
> community, you yourself said this would never be imported into FreeBSD
> base, so I see no reason that it should be called "FreeBSD package Base",
> as it is not, that is a different project.

Taking the last comment on this thread to ask a question and maybe 
refocus a little.

The discussion about granularity begs the question, why pkgbase in the 
first place? My impression was that it allowed people to select which 
components they wanted to either create a lean installation or mix and 
match base packages and ports (possibly with flavours to install in 
/usr rather than $LOCALBASE) such that maybe person A wanted a stock 
install while person B wanted to replace, picking a random example, BSD 
tar with GNU tar. Isn't that the real advantage of pkgbase?

If OTOH it's binary updates V 2.0, what's the point? I'm a little 
rhetorical here but you get my point. If I want ipfw instead pf or 
ipfilter instead of the others I should have the freedom. Similarly if 
I want vim instead of vi I should have the choice to install vim as 
/usr/bin/vi. Otherwise all the effort to replace binary updates makes 
no sense.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX: Web:  http://www.FreeBSD.org

The need of the many outweighs the greed of the few.


___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

RE: CFT: FreeBSD Package Base

[kris]

> -Original Message-
> From: Cy Schubert 
> Sent: Monday, April 29, 2019 3:31 PM
> To: Rodney W. Grimes 
> Cc: Kris Moore ; FreeBSD Stable  sta...@freebsd.org>; freebsd-po...@freebsd.org; Goran Mekić
> ; freebsd-hack...@freebsd.org; FreeBSD Current
> ; freebsd-pkgbase@freebsd.org; freebsd-
> p...@freebsd.org; Emmanuel Vadot 
> Subject: Re: CFT: FreeBSD Package Base
> 
> In message <201904291441.x3tefmid072...@gndrsh.dnsmgr.net>, "Rodney
> W.
> Grimes"
> writes:
> > > On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
> > > freebsd-...@gndrsh.dnsmgr.net> wrote:
> > >
> > > > >
> > > > > Correct, this is ZFS only. And it's something we're using
> > > > > specific to
> > > > FreeNAS / TrueOS, which is why I didn't originally mention it as
> > > > apart of our CFT.
> > > >
> > > > Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
> > > > calling this FreeBSD pkg base when it is not was wrong, and miss
> > > > leading.
> > > >
> > >
> > > Sorry, I disagree.
> > Which is fine.
> >
> > > This pkg base is independent of the ZFS tool we're using to wrangle
> > > boot-environments. Hence why it wasn't mentioned in the CFT.
> > > These base packages work the same as existing in-tree pkg base on
> > > UFS, no difference. If anything are probably safer due to being able
> > > to update all of userland in single extract operation, so you don't
> > > have out of order extraction of libc or some such.
> >
> > You missed the major string change and focused on the edge, No comment
> > on calling iXsystems :stuff: FreeBSD instead of FreeNAS/TrueOS?
> >
> > That was the major point of my statement, your miss leading the user
> > community, you yourself said this would never be imported into FreeBSD
> > base, so I see no reason that it should be called "FreeBSD package
> > Base", as it is not, that is a different project.
> 
> Taking the last comment on this thread to ask a question and maybe refocus
> a little.
> 
> The discussion about granularity begs the question, why pkgbase in the
first
> place? My impression was that it allowed people to select which components
> they wanted to either create a lean installation or mix and match base
> packages and ports (possibly with flavours to install in /usr rather than
> $LOCALBASE) such that maybe person A wanted a stock install while person
> B wanted to replace, picking a random example, BSD tar with GNU tar. Isn't
> that the real advantage of pkgbase?
> 
> If OTOH it's binary updates V 2.0, what's the point? I'm a little
rhetorical here
> but you get my point. If I want ipfw instead pf or ipfilter instead of the
others
> I should have the freedom. Similarly if I want vim instead of vi I should
have
> the choice to install vim as /usr/bin/vi. Otherwise all the effort to
replace
> binary updates makes no sense.
> 
> 

That is a fantastic point. The way we've been doing it is with the
os/userland meta-pkg. Using ZoL as an example, we build userland with the
ZFS options disabled, then added a ZOL option to userland, which makes
sysutils/zol a depend of userland meta-pkg.

Over time I can see this becoming a trend, were we replace bits of base (by
setting WITHOUT_*) and injecting the ports version of those bits via regular
pkg depends. Good candidates would be tools like svn / git, mailers,
compilers, shells, editors, etc.

Ironically this was an issue in the current pkg base implementation that led
us to flattening out the userland package. We found that run-time removal of
specific packages just flat out didn't work. I.E. pkg delete FreeBSD-zfs
didn't work without re-compiling all the things in advance using the proper
WITHOUT_* flags. Same for trying to remove RADIUS support, or others. Too
many things tended to change in seemingly un-related packages, you'd almost
need a full set of flavors for a lot of WITH/WITHOUT combinations for that
to be viable. 

Additionally, this was why we combined base/ports into a single pkg repo. By
building both through poudriere, it makes it possible to properly interject
depends and upgrade in lock-step. Really changes the paradigm of what is
base/ports in a positive way IMHO.









___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Creating jails from packagebase

[Martin Jakob]

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Joe Maloney]

With CFT version you chose to build, and package individual components such as 
sendmail with a port option.  That does entirely solve the problem of being 
able to reinstall sendmail after the fact without a rebuild of the userland 
(base) port but perhaps base flavors could solve that problem assuming flavors 
could extend beyond python.

Joe Maloney
Quality Engineering Manager / iXsystems
Enterprise Storage & Servers Driven By Open Source

> On Apr 29, 2019, at 3:31 PM, Cy Schubert  wrote:
> 
> In message <201904291441.x3tefmid072...@gndrsh.dnsmgr.net>, "Rodney W. 
> Grimes"
> writes:
>>> On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
>>> freebsd-...@gndrsh.dnsmgr.net> wrote:
>>> 
> 
> Correct, this is ZFS only. And it's something we're using specific to
 FreeNAS / TrueOS, which is why I didn't originally mention it as apart of
 our CFT.
 
 Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
 calling this FreeBSD pkg base when it is not was wrong,
 and miss leading.
 
>>> 
>>> Sorry, I disagree.
>> Which is fine.
>> 
>>> This pkg base is independent of the ZFS tool we're using
>>> to wrangle boot-environments. Hence why it wasn't mentioned in the CFT.
>>> These base packages work the same as existing in-tree pkg base on UFS, no
>>> difference. If anything are probably safer due to being able to update all
>>> of userland in single extract operation, so you don't have out of order
>>> extraction of libc or some such.
>> 
>> You missed the major string change and focused on the edge,
>> No comment on calling iXsystems :stuff: FreeBSD instead of FreeNAS/TrueOS?
>> 
>> That was the major point of my statement, your miss leading the user
>> community, you yourself said this would never be imported into FreeBSD
>> base, so I see no reason that it should be called "FreeBSD package Base",
>> as it is not, that is a different project.
> 
> Taking the last comment on this thread to ask a question and maybe 
> refocus a little.
> 
> The discussion about granularity begs the question, why pkgbase in the 
> first place? My impression was that it allowed people to select which 
> components they wanted to either create a lean installation or mix and 
> match base packages and ports (possibly with flavours to install in 
> /usr rather than $LOCALBASE) such that maybe person A wanted a stock 
> install while person B wanted to replace, picking a random example, BSD 
> tar with GNU tar. Isn't that the real advantage of pkgbase?
> 
> If OTOH it's binary updates V 2.0, what's the point? I'm a little 
> rhetorical here but you get my point. If I want ipfw instead pf or 
> ipfilter instead of the others I should have the freedom. Similarly if 
> I want vim instead of vi I should have the choice to install vim as 
> /usr/bin/vi. Otherwise all the effort to replace binary updates makes 
> no sense.
> 
> 
> -- 
> Cheers,
> Cy Schubert 
> FreeBSD UNIX: Web:  http://www.FreeBSD.org
> 
>   The need of the many outweighs the greed of the few.
> 
> 
> ___
> freebsd-curr...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Garrett Wollman]

< 
said:

> The discussion about granularity begs the question, why pkgbase in the 
> first place? My impression was that it allowed people to select which 
> components they wanted to either create a lean installation or mix and 
> match base packages and ports (possibly with flavours to install in 
> /usr rather than $LOCALBASE) such that maybe person A wanted a stock 
> install while person B wanted to replace, picking a random example, BSD 
> tar with GNU tar. Isn't that the real advantage of pkgbase?

No.  The "real" advantage of pkgbase is that it allows the distributor
of a customized version of the operating system to support binary-only
updates, without all the (non-trivial) infrastructure of running a
custom FreeBSD-update builder and distribution server.

Consider my position: I have about 30 servers (and another ~10 jails)
that all run the same local build of FreeBSD.  Right now, the only
reliable way to update them is to NFS-mount /usr/src and /usr/obj from
my build server, and run a (slow) "make installworld".  It would
literally save me hours out of every upgrade (or base-system security
fix) to be able to install compressed binary packages downloaded over
http, and I'd have better security because binary packages are
signed.

For my use case, I don't much care what the granularity is, so long as
I can safely upgrade (or update) the kernel independently of the
userland and independently of third-party packages -- just two
packages (kernel and userland) would suffice, although I'd probably
prefer the runtime libraries to be in a separate package just for
safety.

I'm not distributing packages to third parties, I just want to be able
to install and upgrade my packages on my fleet of servers and jails
quickly and safely.  This is not the entirety of the use cases the
project as a whole needs to support, but it's a major *end-user* use
case.  (And I've said as much in various surveys.)

-GAWollman

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

RE: CFT: FreeBSD Package Base

[Cy Schubert]

On April 29, 2019 1:09:59 PM PDT, k...@ixsystems.com wrote:
>
>
>> -Original Message-
>> From: Cy Schubert 
>> Sent: Monday, April 29, 2019 3:31 PM
>> To: Rodney W. Grimes 
>> Cc: Kris Moore ; FreeBSD Stable > sta...@freebsd.org>; freebsd-po...@freebsd.org; Goran Mekić
>> ; freebsd-hack...@freebsd.org; FreeBSD Current
>> ; freebsd-pkgbase@freebsd.org; freebsd-
>> p...@freebsd.org; Emmanuel Vadot 
>> Subject: Re: CFT: FreeBSD Package Base
>> 
>> In message <201904291441.x3tefmid072...@gndrsh.dnsmgr.net>, "Rodney
>> W.
>> Grimes"
>> writes:
>> > > On Mon, Apr 29, 2019 at 10:09 AM Rodney W. Grimes <
>> > > freebsd-...@gndrsh.dnsmgr.net> wrote:
>> > >
>> > > > >
>> > > > > Correct, this is ZFS only. And it's something we're using
>> > > > > specific to
>> > > > FreeNAS / TrueOS, which is why I didn't originally mention it
>as
>> > > > apart of our CFT.
>> > > >
>> > > > Then please it is "CFT: FreeNAS/TrueOS pkg base, ZFS only",
>> > > > calling this FreeBSD pkg base when it is not was wrong, and
>miss
>> > > > leading.
>> > > >
>> > >
>> > > Sorry, I disagree.
>> > Which is fine.
>> >
>> > > This pkg base is independent of the ZFS tool we're using to
>wrangle
>> > > boot-environments. Hence why it wasn't mentioned in the CFT.
>> > > These base packages work the same as existing in-tree pkg base on
>> > > UFS, no difference. If anything are probably safer due to being
>able
>> > > to update all of userland in single extract operation, so you
>don't
>> > > have out of order extraction of libc or some such.
>> >
>> > You missed the major string change and focused on the edge, No
>comment
>> > on calling iXsystems :stuff: FreeBSD instead of FreeNAS/TrueOS?
>> >
>> > That was the major point of my statement, your miss leading the
>user
>> > community, you yourself said this would never be imported into
>FreeBSD
>> > base, so I see no reason that it should be called "FreeBSD package
>> > Base", as it is not, that is a different project.
>> 
>> Taking the last comment on this thread to ask a question and maybe
>refocus
>> a little.
>> 
>> The discussion about granularity begs the question, why pkgbase in
>the
>first
>> place? My impression was that it allowed people to select which
>components
>> they wanted to either create a lean installation or mix and match
>base
>> packages and ports (possibly with flavours to install in /usr rather
>than
>> $LOCALBASE) such that maybe person A wanted a stock install while
>person
>> B wanted to replace, picking a random example, BSD tar with GNU tar.
>Isn't
>> that the real advantage of pkgbase?
>> 
>> If OTOH it's binary updates V 2.0, what's the point? I'm a little
>rhetorical here
>> but you get my point. If I want ipfw instead pf or ipfilter instead
>of the
>others
>> I should have the freedom. Similarly if I want vim instead of vi I
>should
>have
>> the choice to install vim as /usr/bin/vi. Otherwise all the effort to
>replace
>> binary updates makes no sense.
>> 
>> 
>
>That is a fantastic point. The way we've been doing it is with the
>os/userland meta-pkg. Using ZoL as an example, we build userland with
>the
>ZFS options disabled, then added a ZOL option to userland, which makes
>sysutils/zol a depend of userland meta-pkg.
>
>Over time I can see this becoming a trend, were we replace bits of base
>(by
>setting WITHOUT_*) and injecting the ports version of those bits via
>regular
>pkg depends. Good candidates would be tools like svn / git, mailers,
>compilers, shells, editors, etc.
>
>Ironically this was an issue in the current pkg base implementation
>that led
>us to flattening out the userland package. We found that run-time
>removal of
>specific packages just flat out didn't work. I.E. pkg delete
>FreeBSD-zfs
>didn't work without re-compiling all the things in advance using the
>proper
>WITHOUT_* flags. Same for trying to remove RADIUS support, or others.
>Too
>many things tended to change in seemingly un-related packages, you'd
>almost
>need a full set of flavors for a lot of WITH/WITHOUT combinations for
>that
>to be viable. 
>
>Additionally, this was why we combined base/ports into a single pkg
>repo. By
>building both through poudriere, it makes it possible to properly
>interject
>depends and upgrade in lock-step. Really changes the paradigm of what
>is
>base/ports in a positive way IMHO.

I don't think we want to disable parts of base. We should build base packages 
that can be optionally replaced by ports with flavour @base.

Ports could be built with flavours @base, with $PREFIX /usr, or @ports (or some 
better name), with $PREFIX /usr/local.

Pkgbase could build the base packages negating the need for WITHOUT_* or the 
base packages could be excluded from base using WITHOUT_* and new ports be used 
to build those same packages. In this case, IMO, the implementation doesn't 
matter at the moment but getting straight what we want is the important point 
here.

If this is indeed the goal, IMO should be, we can bikeshed the implementation 
details in anot

Re: CFT: FreeBSD Package Base

[Cy Schubert]

On April 29, 2019 1:50:00 PM PDT, Garrett Wollman  wrote:
>< said:
>
>> The discussion about granularity begs the question, why pkgbase in
>the 
>> first place? My impression was that it allowed people to select which
>
>> components they wanted to either create a lean installation or mix
>and 
>> match base packages and ports (possibly with flavours to install in 
>> /usr rather than $LOCALBASE) such that maybe person A wanted a stock 
>> install while person B wanted to replace, picking a random example,
>BSD 
>> tar with GNU tar. Isn't that the real advantage of pkgbase?
>
>No.  The "real" advantage of pkgbase is that it allows the distributor
>of a customized version of the operating system to support binary-only
>updates, without all the (non-trivial) infrastructure of running a
>custom FreeBSD-update builder and distribution server.
>
>Consider my position: I have about 30 servers (and another ~10 jails)
>that all run the same local build of FreeBSD.  Right now, the only
>reliable way to update them is to NFS-mount /usr/src and /usr/obj from
>my build server, and run a (slow) "make installworld".  It would
>literally save me hours out of every upgrade (or base-system security
>fix) to be able to install compressed binary packages downloaded over
>http, and I'd have better security because binary packages are
>signed.
>
>For my use case, I don't much care what the granularity is, so long as
>I can safely upgrade (or update) the kernel independently of the
>userland and independently of third-party packages -- just two
>packages (kernel and userland) would suffice, although I'd probably
>prefer the runtime libraries to be in a separate package just for
>safety.
>
>I'm not distributing packages to third parties, I just want to be able
>to install and upgrade my packages on my fleet of servers and jails
>quickly and safely.  This is not the entirety of the use cases the
>project as a whole needs to support, but it's a major *end-user* use
>case.  (And I've said as much in various surveys.)
>
>-GAWollman

An anaconda-like installer for freebsd could do that. Also a perfect job for 
cfengine or ansible. Deploy and use a playbook to enforce policy.

You don't need to break up base into packages (not arguing against packaging) 
to gain the benefits of configuration management.

As for updating, freebsd-update is mostly there to accomplish your requirement 
without pkgbase. Which begs the question,  if we're simply replacing 
freebsd-update and it does most of what we want why the extra effort? Unless we 
want to solve more than just this problem? Which BTW I think we do.

I've seen pkgbase as a building block to build an anaconda-like installer 
complete with scripting language. The ability to pick and choose packages as 
many Linux distros do is one part of it.


-- 
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert 
FreeBSD UNIX:  Web: http://www.FreeBSD.org

The need of the many outweighs the greed of the few.
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Rodney W. Grimes]

> On April 29, 2019 1:50:00 PM PDT, Garrett Wollman  
> wrote:
> >< > said:
> >
> >> The discussion about granularity begs the question, why pkgbase in
> >the 
> >> first place? My impression was that it allowed people to select which
> >
> >> components they wanted to either create a lean installation or mix
> >and 
> >> match base packages and ports (possibly with flavours to install in 
> >> /usr rather than $LOCALBASE) such that maybe person A wanted a stock 
> >> install while person B wanted to replace, picking a random example,
> >BSD 
> >> tar with GNU tar. Isn't that the real advantage of pkgbase?
> >
> >No.  The "real" advantage of pkgbase is that it allows the distributor
> >of a customized version of the operating system to support binary-only
> >updates, without all the (non-trivial) infrastructure of running a
> >custom FreeBSD-update builder and distribution server.
> >
> >Consider my position: I have about 30 servers (and another ~10 jails)
> >that all run the same local build of FreeBSD.  Right now, the only
> >reliable way to update them is to NFS-mount /usr/src and /usr/obj from
> >my build server, and run a (slow) "make installworld".  It would
> >literally save me hours out of every upgrade (or base-system security
> >fix) to be able to install compressed binary packages downloaded over
> >http, and I'd have better security because binary packages are
> >signed.
> >
> >For my use case, I don't much care what the granularity is, so long as
> >I can safely upgrade (or update) the kernel independently of the
> >userland and independently of third-party packages -- just two
> >packages (kernel and userland) would suffice, although I'd probably
> >prefer the runtime libraries to be in a separate package just for
> >safety.
> >
> >I'm not distributing packages to third parties, I just want to be able
> >to install and upgrade my packages on my fleet of servers and jails
> >quickly and safely.  This is not the entirety of the use cases the
> >project as a whole needs to support, but it's a major *end-user* use
> >case.  (And I've said as much in various surveys.)
> >
> >-GAWollman
> 
> An anaconda-like installer for freebsd could do that. Also a perfect job for 
> cfengine or ansible. Deploy and use a playbook to enforce policy.

https://anaconda-installer.readthedocs.io/en/latest/

> You don't need to break up base into packages (not arguing against packaging) 
> to gain the benefits of configuration management.
> 
> As for updating, freebsd-update is mostly there to accomplish your 
> requirement without pkgbase. Which begs the question,  if we're simply 
> replacing freebsd-update and it does most of what we want why the extra 
> effort? Unless we want to solve more than just this problem? Which BTW I 
> think we do.
> 
> I've seen pkgbase as a building block to build an anaconda-like installer 
> complete with scripting language. The ability to pick and choose packages as 
> many Linux distros do is one part of it.

What seems to be confusing here is that TrueOS/FreeNAS's
"package base" and the work that has been on going IN
the FreeBSD base system for 2+ (3?) years are 2 
different things with different goal sets and this
CFT has very much muddied that water as to what is
what.

Is there an advocation by iXsystems and TrueOS to replace
what is in the base system now with this new Go implementation
in ports?

Are they orthagonal?  If so can we please rename one?


-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"

Re: CFT: FreeBSD Package Base

[Joe Maloney]

What you describe is the sysup tool for managing updates with boot environments 
which is not part of the CFT ISO.  The pkg upgrade command is used to update 
the base packages.  Sysup is not necessary unless you want a wrapper to create 
boot environments.  

Having said that I cannot describe the problems I’ve had for several years with 
FreeBSD’s pkg base without muddying the waters with this CFT.  So I sort of 
agree it should be called “planned pkg-base”, or “actually has a chance at 
being integratabtle pkg-base”.  Try the ISO I think you will like it.

Joe Maloney
Quality Engineering Manager / iXsystems
Enterprise Storage & Servers Driven By Open Source

> On Apr 29, 2019, at 10:41 PM, Rodney W. Grimes 
>  wrote:
> 
>> On April 29, 2019 1:50:00 PM PDT, Garrett Wollman  
>> wrote:
>>> <>>  said:
>>> 
 The discussion about granularity begs the question, why pkgbase in
>>> the 
 first place? My impression was that it allowed people to select which
>>> 
 components they wanted to either create a lean installation or mix
>>> and 
 match base packages and ports (possibly with flavours to install in 
 /usr rather than $LOCALBASE) such that maybe person A wanted a stock 
 install while person B wanted to replace, picking a random example,
>>> BSD 
 tar with GNU tar. Isn't that the real advantage of pkgbase?
>>> 
>>> No.  The "real" advantage of pkgbase is that it allows the distributor
>>> of a customized version of the operating system to support binary-only
>>> updates, without all the (non-trivial) infrastructure of running a
>>> custom FreeBSD-update builder and distribution server.
>>> 
>>> Consider my position: I have about 30 servers (and another ~10 jails)
>>> that all run the same local build of FreeBSD.  Right now, the only
>>> reliable way to update them is to NFS-mount /usr/src and /usr/obj from
>>> my build server, and run a (slow) "make installworld".  It would
>>> literally save me hours out of every upgrade (or base-system security
>>> fix) to be able to install compressed binary packages downloaded over
>>> http, and I'd have better security because binary packages are
>>> signed.
>>> 
>>> For my use case, I don't much care what the granularity is, so long as
>>> I can safely upgrade (or update) the kernel independently of the
>>> userland and independently of third-party packages -- just two
>>> packages (kernel and userland) would suffice, although I'd probably
>>> prefer the runtime libraries to be in a separate package just for
>>> safety.
>>> 
>>> I'm not distributing packages to third parties, I just want to be able
>>> to install and upgrade my packages on my fleet of servers and jails
>>> quickly and safely.  This is not the entirety of the use cases the
>>> project as a whole needs to support, but it's a major *end-user* use
>>> case.  (And I've said as much in various surveys.)
>>> 
>>> -GAWollman
>> 
>> An anaconda-like installer for freebsd could do that. Also a perfect job for 
>> cfengine or ansible. Deploy and use a playbook to enforce policy.
> 
> https://anaconda-installer.readthedocs.io/en/latest/ 
> 
> 
>> You don't need to break up base into packages (not arguing against 
>> packaging) to gain the benefits of configuration management.
>> 
>> As for updating, freebsd-update is mostly there to accomplish your 
>> requirement without pkgbase. Which begs the question,  if we're simply 
>> replacing freebsd-update and it does most of what we want why the extra 
>> effort? Unless we want to solve more than just this problem? Which BTW I 
>> think we do.
>> 
>> I've seen pkgbase as a building block to build an anaconda-like installer 
>> complete with scripting language. The ability to pick and choose packages as 
>> many Linux distros do is one part of it.
> 
> What seems to be confusing here is that TrueOS/FreeNAS's
> "package base" and the work that has been on going IN
> the FreeBSD base system for 2+ (3?) years are 2 
> different things with different goal sets and this
> CFT has very much muddied that water as to what is
> what.
> 
> Is there an advocation by iXsystems and TrueOS to replace
> what is in the base system now with this new Go implementation
> in ports?
> 
> Are they orthagonal?  If so can we please rename one?
> 
> 
> -- 
> Rod Grimes 
> rgri...@freebsd.org 
> ___
> freebsd-pkgbase@freebsd.org  mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase 
> 
> To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org 
> "

___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send a

Re: CFT: FreeBSD Package Base

[Franco Fichtner]

> On 30. Apr 2019, at 2:25 AM, Cy Schubert  wrote:
> 
> As for updating, freebsd-update is mostly there to accomplish your 
> requirement without pkgbase. Which begs the question,  if we're simply 
> replacing freebsd-update and it does most of what we want why the extra 
> effort? Unless we want to solve more than just this problem? Which BTW I 
> think we do.

There's hbsd-update, pfSense-upgrade and opnsense-update precisely
because pkgbase was pulled ever since it's first ETA in 11.0 and
the subsequent non-delivery.  freebsd-update is not the answer.

Speaking for opnsense-update[1] specifically it's a shell script
with a bit of bloat for handling package updates as normal and
otherwise use base.txz and kernel.txz and arbitrary signatures
in pkg (built upon the idea of pkg.txz.sig) to verify their signature
directly linked to the fingerprints of the pkg repo used -- all well
under 1000 LOC counting whitespaces for readability and battle-
proven since 2015.

My point is simply this:

The sole focus on "the" pkgbase that will solve all issues and the
unwillingness to cooperate on "pkgbase" at the same time has made the
situation much much worse than it should have been for FreeBSD users
and blocked the road for other types of improvements in the update
area for a number of years now.

It's time to step up instead of further dividing the topic so I
want to thank Kris for his effort very much.


Cheers,
Franco

[1] 
https://raw.githubusercontent.com/opnsense/update/master/update/opnsense-update.sh
___
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscr...@freebsd.org"