Re: [HEADSUP] Deprecation of the ftp support in pkg
On 2022-01-23 10:19, Patrick M. Hausen wrote: Hi all, I did not really have an opinion on this, since we never used FTP, but I was a bit surprised by the suggestion to use SSH instead. It never occurred to us that anything but HTTP(S) was possible. We simply run Nginx in a jail serving the packages that Poudriere produces for us. Setup time/effort: 5 minutes. Now after this comment: Am 22.01.2022 um 09:35 schrieb Chris : I find it's less "housekeeping" to use ftp(1) setup through inetd(8) for pkg repos, than via ssh. I understand the appeal of FTP. Maybe this discussion is focusing on the wrong topic. Perhaps we should consider including a light weight way to serve HTTP(S) in base? Like Lighttpd, which as far as I know comes with a BSD 3-clause equivalent license. But then the general tendency has been to remove network services from base rather than introduce them. Like e.g. BIND. So I really have no idea what the general opinion is, just wanted to throw in that IMHO HTTPS is the best protocol to the task and if some way to serve that could be included in base, I for one would appreciate that. OTOH Chris, what's keeping you from installing a web server just serving static files? Different environments/ different requirements. But habit as much as anything else. Ftp is trivial, has always been available. So I never even need to think about it. I perform mass installs/upgrades in large networks. There is no overhead using ftp either through a one-start | inetd. The clients are all started/used at will. It seems to me that removing features also removes value. IMHO the gain from the removal of transports as trivial as ftp(1) bring little to the table for all concerned. But that's just me. :-) -- Chris Kind regards, Patrick -- punkt.de GmbH Patrick M. Hausen .infrastructure Kaiserallee 13a 76133 Karlsruhe Tel. +49 721 9109500 https://infrastructure.punkt.de i...@punkt.de AG Mannheim 108285 Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein 0xBDE49540.asc Description: application/pgp-keys
Re: [HEADSUP] Deprecation of the ftp support in pkg
On 2022-01-23 12:08, Stefan Esser wrote: Am 23.01.22 um 18:12 schrieb Jose Quinteiro: On 1/23/22 09:06, tech-lists wrote: On Sun, Jan 23, 2022 at 08:55:09AM -0800, Jose Quinteiro wrote: You can run HTTP on a non-standard port. For example, 8080 is commonly used. As an added bonus, this means that the HTTP server need not run as root. Unless I'm mistaken, there is no web server in base. There is though, an ftp server. Touche. I wouldn't mind having Thttpd in base. https://www.acme.com/software/thttpd/ An interesting idea, we have it as a port in www/thttpd. While we generally rather move base components that are not generally required to ports, this might be a case where it makes sense to replace one base system component by another one currently only available as a port ... The thttpd binary is smaller than the ftpd currently in base, The binaries may be smaller. But I would argue that the "holes" potentially created are *larger*. :-) An otherwise interesting proposition. and it might make sense to replace ftpd by this simple httpd: $ size /usr/libexec/ftpd textdata bss dec hex filename 10590142765984 116161 1c5c1 /usr/libexec/ftpd $ size work/stage/usr/local/sbin/* textdata bss dec hex filename 5469 656 1126237185d work/stage/usr/local/sbin/makeweb 7453 760 1208333208d work/stage/usr/local/sbin/thtpasswd 8118483362280 91800 16698 work/stage/usr/local/sbin/thttpd The sources are distributed under a 2 clause BSD copyright. Maybe having a HTTP server in base is more useful than a FTP server, today ... Regards, STefan -- Chris 0xBDE49540.asc Description: application/pgp-keys
Re: [HEADSUP] Deprecation of the ftp support in pkg
On 2022-01-23 23:06, Alexander Leidinger wrote: Quoting "Patrick M. Hausen" (from Sun, 23 Jan 2022 19:19:57 +0100): Am 22.01.2022 um 09:35 schrieb Chris : I find it's less "housekeeping" to use ftp(1) setup through inetd(8) for pkg repos, than via ssh. I understand the appeal of FTP. Maybe this discussion is focusing on the wrong topic. Perhaps we should consider including a light weight way to serve HTTP(S) in base? Like Lighttpd, which as far as I know comes with a BSD 3-clause equivalent license. But then the general tendency has been to remove network services from base rather than introduce them. Like e.g. BIND. So I really have no idea what the general opinion is, just wanted to throw in that IMHO HTTPS is the best protocol to the task and if some way to serve that could be included in base, I for one would appreciate that. Personally I think that a http(s) server is not needed in base, as long as we don't need it for something in base (and I wouldn't mind if ftpd would get removed from base for the same reason). Installing a package is easy enough (no matter if for http or ftp). I think in this thread it was mentioned that someone didn't want to install 3rd party software for this. I still can't wrap my head around that part...: It's not the removal from base || adding 3rd party "stuff". It's the removal of the *ability* to *use* ftp(1) as a transport with pkg(8). - we talk about a tool which is used to exclusively install 3rd party software (pkg) - this tool itself is installed like a 3rd party software (package/port, the system-pkg is only a bootstrap) - and the complete context is serving 3rd party software from a tool which builds 3rd party software locally (poudriere) or at least a downloaded subset from FreeBSD - poudriere is also not in the base system but installed like a 3rd party software - maintaining a list of software you are interested in (be it for poudriere or for automated downloads of FreeBSD packages into a local repo) seems more effort to me, than setting up one more 3rd party software for local file distribution I wanted to propose here to include config snippeds for apache/nginx/thttp for such an use-case to decrease the configuration burden, but this seems to be already the case (at least for nginx/apache). I can't see within 30sec any docs about this in the poudriere wiki, so maybe adding a prominent pointer to it there might be an improvement? Bye, Alexander. -- Chris 0xBDE49540.asc Description: application/pgp-keys
Re: [HEADSUP] Deprecation of the ftp support in pkg
On Mon, Jan 24, 2022 at 12:12:20AM -0800, Chris wrote: > On 2022-01-23 10:19, Patrick M. Hausen wrote: > > Hi all, > > > > I did not really have an opinion on this, since we never used FTP, > > but I was a bit surprised by the suggestion to use SSH instead. > > > > It never occurred to us that anything but HTTP(S) was possible. > > We simply run Nginx in a jail serving the packages that Poudriere > > produces for us. Setup time/effort: 5 minutes. > > > > Now after this comment: > > > > > Am 22.01.2022 um 09:35 schrieb Chris : > > > I find it's less "housekeeping" to use ftp(1) setup through inetd(8) > > > for pkg repos, than > > > via ssh. > > > > I understand the appeal of FTP. > > Maybe this discussion is focusing on the wrong topic. Perhaps > > we should consider including a light weight way to serve HTTP(S) > > in base? Like Lighttpd, which as far as I know comes with a BSD > > 3-clause equivalent license. > > > > But then the general tendency has been to remove network services > > from base rather than introduce them. Like e.g. BIND. > > > > So I really have no idea what the general opinion is, just wanted > > to throw in that IMHO HTTPS is the best protocol to the task and > > if some way to serve that could be included in base, I for one would > > appreciate that. > > > > OTOH Chris, what's keeping you from installing a web server just > > serving static files? > Different environments/ different requirements. But habit as much as > anything else. > Ftp is trivial, has always been available. So I never even need to think > about it. > I perform mass installs/upgrades in large networks. There is no overhead > using ftp > either through a one-start | inetd. The clients are all started/used at will. > It seems to me that removing features also removes value. IMHO the gain from > the > removal of transports as trivial as ftp(1) bring little to the table for all > concerned. But that's just me. :-) > based on the discussion I am adding right now, a new protocol: tcp:// which uses the protocol we made on top of ssh way simple than ftp and capsicumized on the server side), but without the requirement for a ssh connection. This will enable people with performance concern but still willing to have data encrypted to use spiped or socat for example as a transport. And for other a simple inetd will work. on inetd.conf pkg stream tcp nowait nobody /usr/local/sbin/pkg -- -o SSH_RESTRICT_DIR=/mypackages ssh and define in /etc/services "pkg" to the port you want pkg to be serving its files to. for the repo on the pkg side: tcp://url:port/mypackages What do you think? For me as a maintainer it is very few lines of code to add, the protocol itself being already written, way easier to suppoer and it provides an interesting feature by the ability to serve via spiped or inetd. Best regards, Bapt
Re: [HEADSUP] Deprecation of the ftp support in pkg
Hi, I just wanted to chime in on Alexander's (netchild@) mail and I fully agree. Looking at base we already have a lot of contrib and since we need to adapt each software project to our build framework a lot tends to get dated quickly and currently is so I don't see the benefit importing more at all. I would also like to highly advice against importing software which is considered dead upstream, https://en.wikipedia.org/wiki/Thttpd . If anything, something like https://github.com/emikulic/darkhttpd or similar which would be very easy to maintain and is active Best regards, Daniel
Re: [HEADSUP] Deprecation of the ftp support in pkg
On 24/01/2022 08:59, Baptiste Daroussin wrote: On Mon, Jan 24, 2022 at 12:12:20AM -0800, Chris wrote: On 2022-01-23 10:19, Patrick M. Hausen wrote: Hi all, based on the discussion I am adding right now, a new protocol: tcp:// which uses the protocol we made on top of ssh way simple than ftp and capsicumized on the server side), but without the requirement for a ssh connection. This will enable people with performance concern but still willing to have data encrypted to use spiped or socat for example as a transport. And for other a simple inetd will work. on inetd.conf pkg stream tcp nowait nobody /usr/local/sbin/pkg -- -o SSH_RESTRICT_DIR=/mypackages ssh and define in /etc/services "pkg" to the port you want pkg to be serving its files to. for the repo on the pkg side: tcp://url:port/mypackages What do you think? Not having a preference re: ftp I think this tcp:// option looks very useful. As an aside has anyone written up how to setup a repo over ssh? I've had a quick google with no luck. I'm sure its not too tricky and an hour or so I could work it out but if anyone has already done it that would be great. Thanks, Vince For me as a maintainer it is very few lines of code to add, the protocol itself being already written, way easier to suppoer and it provides an interesting feature by the ability to serve via spiped or inetd. Best regards, Bapt
Re: [HEADSUP] Deprecation of the ftp support in pkg
On Mon, Jan 24, 2022 at 12:10:11PM +, Vincent Hoffman-Kazlauskas wrote:
>
>
> On 24/01/2022 08:59, Baptiste Daroussin wrote:
> > On Mon, Jan 24, 2022 at 12:12:20AM -0800, Chris wrote:
> > > On 2022-01-23 10:19, Patrick M. Hausen wrote:
> > > > Hi all,
> > > >
>
>
>
> > >
> >
> > based on the discussion I am adding right now, a new protocol: tcp:// which
> > uses
> > the protocol we made on top of ssh way simple than ftp and capsicumized on
> > the
> > server side), but without the requirement for a ssh connection.
> >
> > This will enable people with performance concern but still willing to have
> > data
> > encrypted to use spiped or socat for example as a transport.
> >
> > And for other a simple inetd will work.
> >
> > on inetd.conf
> >
> > pkg stream tcp nowait nobody /usr/local/sbin/pkg -- -o
> > SSH_RESTRICT_DIR=/mypackages ssh
> >
> >
> > and define in /etc/services "pkg" to the port you want pkg to be serving its
> > files to.
> >
> > for the repo on the pkg side:
> >
> > tcp://url:port/mypackages
> >
> > What do you think?
> >
> Not having a preference re: ftp I think this tcp:// option looks very
> useful. As an aside has anyone written up how to setup a repo over ssh? I've
> had a quick google with no luck. I'm sure its not too tricky and an hour or
> so I could work it out but if anyone has already done it that would be
> great.
>
> Thanks,
> Vince
on the client side one need a to be allow to ssh to the server, for the given
user (it exists technics to allow anonssh, like it is done for git on the
freebsd infrastructure if needed).
on the server side one need pkg to be installed (reminder pkg works on linux and
osx if needed so the server does not need to be a freebsd server)
in the pkg.conf of the server side I do recommand setting SSH_RESTRICT_DIR to a
path "upper" from the path of the packages.
Let say the packages are in /usr/local/freebsd/packages/
one can set SSH_RESTRICT_DIR to /usr/local/packages
on the client side:
myrepo: {
url: ssh://user@host/${ABI}
}
that is all.
the doc part of the url represent the path under this SSH_RESTRICT_DIR (relative
to).
Note that pkg will open that directory if on freebsd can use capcisum to get
restricted to it.
Under linux and osx, it will make sure via path comparison to never escape it.
Now with the new tcp:// which I just committed it is exactly the same except
that one does not need to setup ssh.
Best regards,
Bapt
Re: pkgbase checksums
On 21.01.22 19:49, Baptiste Daroussin wrote:
17 janv. 2022 16:22:31 Rosenke :
On 17.01.22 16:20, Baptiste Daroussin wrote:
On Mon, Jan 17, 2022 at 04:17:46PM +0100, Rosenke wrote:
On 17.01.22 16:12, Rosenke wrote:
On 13.01.22 13:57, Baptiste Daroussin wrote:
On Thu, Jan 13, 2022 at 11:33:55AM +0100, Stefan Esser wrote:
…
We have @config for base were pkg will do a 3 way merge, just those
files are
probably not marked as @config and should.
(@config is native while @sample is external and define only in the
ports tree)
if they are marked as such, then it means we have a bug in pkg
checksum as it
should not check the checksum of the files marked @config
Best regards,
Bapt
I tried it again with pkg 1.17.5 and FreeBSD 12.3 packages but got the
same Error. The @config section is included in the packages:
pkg info -R FreeBSD-clibs
config: [
"/etc/nsswitch.conf",
"/etc/libmap.conf",
"/etc/hosts",
"/etc/protocols",
"/etc/netconfig",
"/etc/hosts.equiv",
"/etc/rpc",
"/etc/mac.conf",
"/etc/shells",
"/etc/networks"
]
In total these are now 9 cheksum mismatches in the pkgbase jail:
=== root@dsssrvt4j1 (pts/2) ~ 19(3) -> pkg check -sa
Checking all packages: 1%
FreeBSD-clibs-12.2.s20220114103031: checksum mismatch for /etc/hosts
FreeBSD-clibs-12.2.s20220114103031: checksum mismatch for /etc/shells
Checking all packages: 78%
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for /.profile
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
/etc/locate.rc
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
/etc/login.conf
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
/etc/sysctl.conf
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
/etc/syslog.conf
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for /etc/ttys
FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
/root/.profile
Checking all packages: 100%
Whoops, copied the wrong output:
=== root@dsssrvt4j1 (pts/1) ~ 1(3) -> pkg check -sa
Checking all packages: 1%
FreeBSD-clibs-12.3.s20220117084939: checksum mismatch for /etc/hosts
FreeBSD-clibs-12.3.s20220117084939: checksum mismatch for /etc/shells
Checking all packages: 78%
FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/locate.rc
FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/login.conf
FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/motd
FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/sysctl.conf
FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/syslog.conf
FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/ttys
Checking all packages: 100%
Here we are ;), the bug is on my side then :D, we pkg devs (I said we because
maybe manu will be faster than me on this) will dive in the code and fix.
Best regards
Bapt
Thank you very much. The concept of pkgbase is great, loving it on jails with
zfs so far.
--
Mit freundlichen Grüßen
Henrik Rosenke / Operating
Data-Service GmbH
Beethovenstr. 2a
23617 Stockelsdorf
Tel. +49 451/49001-34 Fax. +49 451/49001-26
Email:rose...@dssgmbh.de /www.dssgmbh.de
Amtsgericht Lübeck, HRB 318 BS
Geschäftsführer: Wilfried Paepcke, Dr. Andreas Longwitz,
Dr. Hans-Martin Rasch, Dr. Uwe Szyszka
Fixed in git
Bapt
Thanks, when will the next pkg Version be released?
Tried to manually port back the change in pkg.c but 1.17.5 is missing
the whole hashing structure and i dont want to port back all of this.
[00:01:17] pkg.c:1522:7: error: implicit declaration of function
'pkghash_get_value' is invalid in C99
[-Werror,-Wimplicit-function-declaration]
[00:01:17] pkghash_get_value(pkg->config_files_hash, f->path) == NULL) {
[00:01:17] ^
[00:01:17] pkg.c:1522:30: error: no member named 'config_files_hash' in
'struct pkg'; did you mean 'config_files'?
[00:01:17] pkghash_get_value(pkg->config_files_hash, f->path) == NULL) {
[00:01:17] ^
[00:01:17] config_files
--
Mit freundlichen Grüßen
Henrik Rosenke / Operating
Data-Service GmbH
Beethovenstr. 2a
23617 Stockelsdorf
Tel. +49 451/49001-34 Fax. +49 451/49001-26
Email:rose...@dssgmbh.de /www.dssgmbh.de
Amtsgericht Lübeck, HRB 318 BS
Geschäftsführer: Wilfried Paepcke, Dr. Andreas Longwitz,
Dr. Hans-Martin Rasch, Dr. Uwe Szyszka
Re: pkgbase checksums
On Mon, Jan 24, 2022 at 03:11:52PM +0100, Rosenke wrote:
>
>
> On 21.01.22 19:49, Baptiste Daroussin wrote:
> > 17 janv. 2022 16:22:31 Rosenke :
> >
> > > On 17.01.22 16:20, Baptiste Daroussin wrote:
> > > > On Mon, Jan 17, 2022 at 04:17:46PM +0100, Rosenke wrote:
> > > > > On 17.01.22 16:12, Rosenke wrote:
> > > > > > On 13.01.22 13:57, Baptiste Daroussin wrote:
> > > > > > > On Thu, Jan 13, 2022 at 11:33:55AM +0100, Stefan Esser wrote:
> > > > > > > > …
> > > > > > > We have @config for base were pkg will do a 3 way merge, just
> > > > > > > those
> > > > > > > files are
> > > > > > > probably not marked as @config and should.
> > > > > > >
> > > > > > > (@config is native while @sample is external and define only in
> > > > > > > the
> > > > > > > ports tree)
> > > > > > >
> > > > > > > if they are marked as such, then it means we have a bug in pkg
> > > > > > > checksum as it
> > > > > > > should not check the checksum of the files marked @config
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Bapt
> > > > > > I tried it again with pkg 1.17.5 and FreeBSD 12.3 packages but got
> > > > > > the
> > > > > > same Error. The @config section is included in the packages:
> > > > > > pkg info -R FreeBSD-clibs
> > > > > > config: [
> > > > > > "/etc/nsswitch.conf",
> > > > > > "/etc/libmap.conf",
> > > > > > "/etc/hosts",
> > > > > > "/etc/protocols",
> > > > > > "/etc/netconfig",
> > > > > > "/etc/hosts.equiv",
> > > > > > "/etc/rpc",
> > > > > > "/etc/mac.conf",
> > > > > > "/etc/shells",
> > > > > > "/etc/networks"
> > > > > > ]
> > > > > >
> > > > > > In total these are now 9 cheksum mismatches in the pkgbase jail:
> > > > > > === root@dsssrvt4j1 (pts/2) ~ 19(3) -> pkg check -sa
> > > > > > Checking all packages: 1%
> > > > > > FreeBSD-clibs-12.2.s20220114103031: checksum mismatch for /etc/hosts
> > > > > > FreeBSD-clibs-12.2.s20220114103031: checksum mismatch for
> > > > > > /etc/shells
> > > > > > Checking all packages: 78%
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /.profile
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /etc/locate.rc
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /etc/login.conf
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /etc/sysctl.conf
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /etc/syslog.conf
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /etc/ttys
> > > > > > FreeBSD-runtime-12.2.s20220114103031: checksum mismatch for
> > > > > > /root/.profile
> > > > > > Checking all packages: 100%
> > > > > >
> > > > > Whoops, copied the wrong output:
> > > > > === root@dsssrvt4j1 (pts/1) ~ 1(3) -> pkg check -sa
> > > > > Checking all packages: 1%
> > > > > FreeBSD-clibs-12.3.s20220117084939: checksum mismatch for /etc/hosts
> > > > > FreeBSD-clibs-12.3.s20220117084939: checksum mismatch for /etc/shells
> > > > > Checking all packages: 78%
> > > > > FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for
> > > > > /etc/locate.rc
> > > > > FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for
> > > > > /etc/login.conf
> > > > > FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/motd
> > > > > FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for
> > > > > /etc/sysctl.conf
> > > > > FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for
> > > > > /etc/syslog.conf
> > > > > FreeBSD-runtime-12.3.s20220117084939: checksum mismatch for /etc/ttys
> > > > > Checking all packages: 100%
> > > > >
> > > > Here we are ;), the bug is on my side then :D, we pkg devs (I said we
> > > > because
> > > > maybe manu will be faster than me on this) will dive in the code and
> > > > fix.
> > > >
> > > > Best regards
> > > > Bapt
> > > Thank you very much. The concept of pkgbase is great, loving it on jails
> > > with zfs so far.
> > >
> > > --
> > > Mit freundlichen Grüßen
> > > Henrik Rosenke / Operating
> > >
> > > Data-Service GmbH
> > > Beethovenstr. 2a
> > > 23617 Stockelsdorf
> > > Tel. +49 451/49001-34 Fax. +49 451/49001-26
> > > Email:rose...@dssgmbh.de /www.dssgmbh.de
> > >
> > > Amtsgericht Lübeck, HRB 318 BS
> > > Geschäftsführer: Wilfried Paepcke, Dr. Andreas Longwitz,
> > > Dr. Hans-Martin Rasch, Dr. Uwe Szyszka
> > Fixed in git
> >
> > Bapt
> Thanks, when will the next pkg Version be released?
> Tried to manually port back the change in pkg.c but 1.17.5 is missing the
> whole hashing structure and i dont want to port back all of this.
>
> [00:01:17] pkg.c:1522:7: error: implicit declaration of function
> 'pkghash_get_value' is invalid in C99
> [-Werror,-Wimplicit-function-declaration]
> [00:01:17] pkghash_get_value(pkg->config_files_hash, f->path) == NULL) {
> [00:01:17] ^
> [00:01:17] pkg.c:1522:30: e
Modular fetch design proposal: Was: [HEADSUP] Deprecation of the ftp support in pkg
On 24.01.22 09:12, Chris wrote:
On 2022-01-23 10:19, Patrick M. Hausen wrote:
Hi all,
I did not really have an opinion on this, since we never used FTP,
but I was a bit surprised by the suggestion to use SSH instead.
It never occurred to us that anything but HTTP(S) was possible.
We simply run Nginx in a jail serving the packages that Poudriere
produces for us. Setup time/effort: 5 minutes.
Now after this comment:
Am 22.01.2022 um 09:35 schrieb Chris :
I find it's less "housekeeping" to use ftp(1) setup through inetd(8)
for pkg repos, than
via ssh.
I understand the appeal of FTP.
Maybe this discussion is focusing on the wrong topic. Perhaps
we should consider including a light weight way to serve HTTP(S)
in base? Like Lighttpd, which as far as I know comes with a BSD
3-clause equivalent license.
But then the general tendency has been to remove network services
from base rather than introduce them. Like e.g. BIND.
So I really have no idea what the general opinion is, just wanted
to throw in that IMHO HTTPS is the best protocol to the task and
if some way to serve that could be included in base, I for one would
appreciate that.
OTOH Chris, what's keeping you from installing a web server just
serving static files?
Different environments/ different requirements. But habit as much as
anything else.
Ftp is trivial, has always been available. So I never even need to
think about it.
I perform mass installs/upgrades in large networks. There is no
overhead using ftp
either through a one-start | inetd. The clients are all started/used
at will.
It seems to me that removing features also removes value. IMHO the
gain from the
removal of transports as trivial as ftp(1) bring little to the table
for all
concerned. But that's just me. :-)
Have you ever looked into a FTP protocol parser and what's required to
get different FTP configurations through the NAT infested networks of
today? FTP is an ugly protocol from the beginning of time that should
have been put down decades ago. Even without pipelining HTTP saves
several network round trips and poudriere already generates HTML and
JSON status updates during builds as read only web ui.
This thread has shown that users have deployed complex, fragile
workarounds the limited protocol selection offered by pkg. I recommend
adding a clean and official extension interface spawning fetch helper
processes from a well known location outside of $PATH derived from the
URI schema (e.g. ${PREFIX}/libexec/pkg/fetch-${SCHEMA}). To keep helpers
simple and small they would be started in an execution environment
(working directory, environment variables, minimal set of inherited file
descriptors) to be prepared by pkg expecting the repository URI as first
(and only?) argument. Reading a stream of pairs of file name (e.g. the
package hash stored in the repository) and relative path per line to
fetch from standard input into the inherited working directory allowing
users to add their own transport helpers similar to git.
To support progress updates and allow pkg to start the installation of
fetch packages as soon as possible helpers could write lines with
"${BYTES_FETCHED} ${BYTES_TOTAL} ${FILE}" to standard output
periodically. A (permanent) transfer failure could be encoded by a
negative $BYTES_FETCHED and a successfully completed transfer as
$BYTES_FETCHED == $BYTES_TOTAL. If the helper doesn't know the file size
it should be allowed to use negative $BYTES_TOTAL values in all but the
last progress update (per fetched file). All transfers not reported as
successfully completed or permanently failed are implicitly confirmed by
exiting with EX_OK. Other exit codes implicitly fail all unconfirmed
transfers. Pkg should clean up the working directory after the the
helper has exited to delete partially transferred files (and anything
else the helper may have left taking care not to follow symlinks). Pkg
should apply resource limits and drop privileges (when running as root)
before exec()ing into the helper. Well written helpers can use capsicum
to provide further defense in depth.
The package repository already contains the the expected package sizes.
As an optimization for dealing with out of sync mirrors the known file
sizes can be matched against positive file sizes reported by helpers to
fail quickly.
Refactoring all supported protocols to use this interface would reduce
the complexity of pkg itself.
This design can be further extended with more features (and potential
for bugs) until we end up with something similar to the git annex
external special remote protocol
(https://git-annex.branchable.com/design/external_special_remote_protocol/)
if there are enough relevant use cases justifying the additional
complexity in pkg and its file transfer helpers.
Re: Deprecation of the ftp support in pkg
On 24.01.22 10:56, Helge Oldach wrote: Jose Quinteiro wrote on Mon, 24 Jan 2022 07:40:03 +0100 (CET): On 1/23/22 13:12, Helge Oldach wrote: Stefan Esser wrote on Sun, 23 Jan 2022 21:08:30 +0100 (CET): Am 23.01.22 um 18:12 schrieb Jose Quinteiro: On 1/23/22 09:06, tech-lists wrote: On Sun, Jan 23, 2022 at 08:55:09AM -0800, Jose Quinteiro wrote: You can run HTTP on a non-standard port. For example, 8080 is commonly used. As an added bonus, this means that the HTTP server need not run as root. Unless I'm mistaken, there is no web server in base. There is though, an ftp server. Touche. I wouldn't mind having Thttpd in base. https://www.acme.com/software/thttpd/ An interesting idea, we have it as a port in www/thttpd. Does thttpd support SSL in the meantime? I suspect that is an option people would ask for instantly. It does not. This is by design. It is really meant to be as small and simple as possible. www/webfs is significantly simpler and smaller (5k LOC compared to thttpd's 17k) and lacks SSL as well. I'd say importing NetBSD's or OpenBSD's httpd would be the better option then - www/obhttpd is in ports already. Apart from serving pkg purposes (as discussed here) it will bring in a resonable http server and feature parity with *BSD. Can the OpenBSD httpd be relied on to work with the OpenSSL version included in base? One of the problems with OpenBSD daemons is that an ever increasing number of them dropped support for the complex and error prone OpenSSL API and replaced them with the simpler libtls API calls only available as part of LibreTLS. There is also a trend among recently (re-)written daemons to rely heavily on pledge() and unveil() for security going as far as replacing the elaborate priv-sep designs used previously by OpenBSD for their daemons.
FreeBSD ports you maintain which are out of date
Dear port maintainer, The portscout new distfile checker has detected that one or more of your ports appears to be out of date. Please take the opportunity to check each of the ports listed below, and if possible and appropriate, submit/commit an update. If any ports have already been updated, you can safely ignore the entry. You will not be e-mailed again for any of the port/version combinations below. Full details can be found at the following URL: http://portscout.freebsd.org/po...@freebsd.org.html Port| Current version | New version +-+ mail/clawsker | 1.3.4 | 1.3.5 +-+ sysutils/google-compute-engine-oslogin | 20191018.00 | 20220113.00 +-+ www/py-fastapi | 0.70.0 | 0.73.0 +-+ If any of the above results are invalid, please check the following page for details on how to improve portscout's detection and selection of distfiles on a per-port basis: http://portscout.freebsd.org/info/portscout-portconfig.txt Reported by:portscout!
Re: Modular fetch design proposal: Was: [HEADSUP] Deprecation of the ftp support in pkg
On Mon, Jan 24, 2022 at 03:46:39PM +0100, Jan Bramkamp wrote:
> On 24.01.22 09:12, Chris wrote:
> > On 2022-01-23 10:19, Patrick M. Hausen wrote:
> > > Hi all,
> > >
> > > I did not really have an opinion on this, since we never used FTP,
> > > but I was a bit surprised by the suggestion to use SSH instead.
> > >
> > > It never occurred to us that anything but HTTP(S) was possible.
> > > We simply run Nginx in a jail serving the packages that Poudriere
> > > produces for us. Setup time/effort: 5 minutes.
> > >
> > > Now after this comment:
> > >
> > > > Am 22.01.2022 um 09:35 schrieb Chris :
> > > > I find it's less "housekeeping" to use ftp(1) setup through
> > > > inetd(8) for pkg repos, than
> > > > via ssh.
> > >
> > > I understand the appeal of FTP.
> > > Maybe this discussion is focusing on the wrong topic. Perhaps
> > > we should consider including a light weight way to serve HTTP(S)
> > > in base? Like Lighttpd, which as far as I know comes with a BSD
> > > 3-clause equivalent license.
> > >
> > > But then the general tendency has been to remove network services
> > > from base rather than introduce them. Like e.g. BIND.
> > >
> > > So I really have no idea what the general opinion is, just wanted
> > > to throw in that IMHO HTTPS is the best protocol to the task and
> > > if some way to serve that could be included in base, I for one would
> > > appreciate that.
> > >
> > > OTOH Chris, what's keeping you from installing a web server just
> > > serving static files?
> > Different environments/ different requirements. But habit as much as
> > anything else.
> > Ftp is trivial, has always been available. So I never even need to think
> > about it.
> > I perform mass installs/upgrades in large networks. There is no overhead
> > using ftp
> > either through a one-start | inetd. The clients are all started/used at
> > will.
> > It seems to me that removing features also removes value. IMHO the gain
> > from the
> > removal of transports as trivial as ftp(1) bring little to the table for
> > all
> > concerned. But that's just me. :-)
>
> Have you ever looked into a FTP protocol parser and what's required to get
> different FTP configurations through the NAT infested networks of today? FTP
> is an ugly protocol from the beginning of time that should have been put
> down decades ago. Even without pipelining HTTP saves several network round
> trips and poudriere already generates HTML and JSON status updates during
> builds as read only web ui.
>
>
> This thread has shown that users have deployed complex, fragile workarounds
> the limited protocol selection offered by pkg. I recommend adding a clean
> and official extension interface spawning fetch helper processes from a well
> known location outside of $PATH derived from the URI schema (e.g.
> ${PREFIX}/libexec/pkg/fetch-${SCHEMA}). To keep helpers simple and small
> they would be started in an execution environment (working directory,
> environment variables, minimal set of inherited file descriptors) to be
> prepared by pkg expecting the repository URI as first (and only?) argument.
> Reading a stream of pairs of file name (e.g. the package hash stored in the
> repository) and relative path per line to fetch from standard input into the
> inherited working directory allowing users to add their own transport
> helpers similar to git.
>
> To support progress updates and allow pkg to start the installation of fetch
> packages as soon as possible helpers could write lines with
> "${BYTES_FETCHED} ${BYTES_TOTAL} ${FILE}" to standard output periodically. A
> (permanent) transfer failure could be encoded by a negative $BYTES_FETCHED
> and a successfully completed transfer as $BYTES_FETCHED == $BYTES_TOTAL. If
> the helper doesn't know the file size it should be allowed to use negative
> $BYTES_TOTAL values in all but the last progress update (per fetched file).
> All transfers not reported as successfully completed or permanently failed
> are implicitly confirmed by exiting with EX_OK. Other exit codes implicitly
> fail all unconfirmed transfers. Pkg should clean up the working directory
> after the the helper has exited to delete partially transferred files (and
> anything else the helper may have left taking care not to follow symlinks).
> Pkg should apply resource limits and drop privileges (when running as root)
> before exec()ing into the helper. Well written helpers can use capsicum to
> provide further defense in depth.
>
> The package repository already contains the the expected package sizes. As
> an optimization for dealing with out of sync mirrors the known file sizes
> can be matched against positive file sizes reported by helpers to fail
> quickly.
>
> Refactoring all supported protocols to use this interface would reduce the
> complexity of pkg itself.
>
> This design can be further extended with more features (and potential for
> bugs) until we end up with something similar to the git annex external
> special remote protocol
> (ht
Re: Modular fetch design proposal: Was: [HEADSUP] Deprecation of the ftp support in pkg
Isn't that just reinventing libfetch which is already in the base install? Cheers, Freddie Typos due to smartphone keyboard.
Re: [HEADSUP] Deprecation of the ftp support in pkg
On 2022-01-24 03:00, Daniel Engberg wrote: Hi, I just wanted to chime in on Alexander's (netchild@) mail and I fully agree. Looking at base we already have a lot of contrib and since we need to adapt each software project to our build framework a lot tends to get dated quickly and currently is so I don't see the benefit importing more at all. I would also like to highly advice against importing software which is considered dead upstream, https://en.wikipedia.org/wiki/Thttpd . If anything, something like https://github.com/emikulic/darkhttpd or similar which would be very easy to maintain and is active FWIW I'd also like to vote +1 on NO additions. I only vote in favor of ftp in this thread because we have a million year... OK 30 plus years of track record for it and it just works. Tho I must admit I find @bapt's recent tcp proposal an interesting and appealing idea. :-) Best regards, Daniel -- Chris 0xBDE49540.asc Description: application/pgp-keys
Re: [HEADSUP] Deprecation of the ftp support in pkg
On Mon, Jan 24, 2022 at 09:29:45AM -0800, Chris wrote: > On 2022-01-24 03:00, Daniel Engberg wrote: > > Hi, > > > > I just wanted to chime in on Alexander's (netchild@) mail and I fully agree. > > Looking at base we already have a lot of contrib and since we need to > > adapt each > > software project to our build framework a lot tends to get dated quickly and > > currently is so I don't see the benefit importing more at all. I would > > also like > > to highly advice against importing software which is considered dead > > upstream, > > https://en.wikipedia.org/wiki/Thttpd . > > > > If anything, something like https://github.com/emikulic/darkhttpd or > > similar which > > would be very easy to maintain and is active > > FWIW I'd also like to vote +1 on NO additions. I only vote in favor of ftp > in this > thread because we have a million year... OK 30 plus years of track record > for it and > it just works. Tho I must admit I find @bapt's recent tcp proposal an > interesting and > appealing idea. :-) The proposal is now in anyway ;) I am just struggling on the name of the scheme: tcp:// or pkg+tcp:// (with a rename of ssh into pkg+ssh:// :D) Best regards, Bapt
Re: [HEADSUP] Deprecation of the ftp support in pkg
On 2022-01-24 10:08, Baptiste Daroussin wrote: On Mon, Jan 24, 2022 at 09:29:45AM -0800, Chris wrote: On 2022-01-24 03:00, Daniel Engberg wrote: > Hi, > > I just wanted to chime in on Alexander's (netchild@) mail and I fully agree. > Looking at base we already have a lot of contrib and since we need to > adapt each > software project to our build framework a lot tends to get dated quickly and > currently is so I don't see the benefit importing more at all. I would > also like > to highly advice against importing software which is considered dead > upstream, > https://en.wikipedia.org/wiki/Thttpd . > > If anything, something like https://github.com/emikulic/darkhttpd or > similar which > would be very easy to maintain and is active FWIW I'd also like to vote +1 on NO additions. I only vote in favor of ftp in this thread because we have a million year... OK 30 plus years of track record for it and it just works. Tho I must admit I find @bapt's recent tcp proposal an interesting and appealing idea. :-) The proposal is now in anyway ;) Uh, Oh. ;-) I am just struggling on the name of the scheme: tcp:// or pkg+tcp:// (with a rename of ssh into pkg+ssh:// :D) +1 for tcp:// Best regards, Bapt -- Chris P.S. You must REALLY hate ftp(1) to have gone to all this trouble. ;-) 0xBDE49540.asc Description: application/pgp-keys
sed trouble when UNAME_* is set in jail (fwd)
Sorry for resend colleagues, seem to lost my subscription list -- Sincerely, D.Marck[DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer:ma...@freebsd.org ] --- *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- woo...@woozle.net *** --- -- Forwarded message -- Date: Mon, 24 Jan 2022 22:29:35 From: Dmitry Morozovsky To: bdrew...@freebsd.org Cc: freebsd-po...@freebsd.org Subject: sed trouble when UNAME_* is set in jail Bryan, colleagues, I understand that building for obsoleted systems is not supported at all, but there are *cases* -- you all know! ;) I have a jail on my build system, nested-built on a previous major; on the case, it's stable/10; for a poudriere, it is defined as null-mounted from the build jail: 10-amd64 stable/10amd64 null 2022-01-20 13:10:41 /X/j10a for some other reasons, there're definition in login.conf like :setenv=MAIL=/var/mail/$,BLOCKSIZE=K,UNAME_r=10.4-STABLE,UNAME_v=FreeBSD 10.4-STABLE #0:\ however, bulk.sh/common.sh are unhappy with this breaking with sed: 1: "s/,UNAME_r.*:/:/ ; s/:\ ...": bad flag in substitute command: ',' I tracked this down to update_version_env() in common.sh, but then stuck did I miss something trivial? Thank you in advance! -- Sincerely, D.Marck[DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer:ma...@freebsd.org ] --- *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- woo...@woozle.net *** ---
Re: [HEADSUP] Deprecation of the ftp support in pkg
On Mon, 24 Jan 2022, Baptiste Daroussin wrote: > > [...] Tho I must admit I find @bapt's recent tcp proposal an > > interesting and appealing idea. :-) > > The proposal is now in anyway ;) That was fast; excellent service, sir! I now have even more reason to upgrade my ancient server (FTP is forbidden on my LAN; SSH was trivial to set up). > I am just struggling on the name of the scheme: tcp:// or pkg+tcp:// > (with a rename of ssh into pkg+ssh:// :D) I'm only a minnow here, but that "+" looks pretty ugly; are there any other methods that contain a "+" symbol? It looks like a sort of a port variant (in MacOS terms, which I also use). -- Dave
Re: sed trouble when UNAME_* is set in jail (fwd)
>
> From: Dmitry Morozovsky
> Sent: Mon Jan 24 20:45:11 CET 2022
> To:
> Subject: sed trouble when UNAME_* is set in jail (fwd)
>
> I have a jail on my build system, nested-built on a previous major; on the
> case, it's stable/10; for a poudriere, it is defined as null-mounted from the
> build jail:
>
> 10-amd64 stable/10amd64 null 2022-01-20 13:10:41 /X/j10a
>
> for some other reasons, there're definition in login.conf like
>
> :setenv=MAIL=/var/mail/$,BLOCKSIZE=K,UNAME_r=10.4-STABLE,UNAME_v=FreeBSD
> 10.4-STABLE #0:\
>
> however, bulk.sh/common.sh are unhappy with this breaking with
>
> sed: 1: "s/,UNAME_r.*:/:/ ; s/:\ ...": bad flag in substitute command: ','
>
> I tracked this down to update_version_env() in common.sh, but then stuck
> did I miss something trivial?
`sed' is complaining about a bad substitution flag. Here's the expected
behavior:
$ awk '/sed/ && /UNAME/ {print NR ": " $0}' /usr/local/share/poudriere/common.sh
2860: sed -i "" -e "s/,UNAME_r.*:/:/ ; s/:\(setenv.*\):/:\1${login_env}:/" \
$ cat testfile
:setenv=MAIL=/var/mail/$,BLOCKSIZE=K,UNAME_r=10.4-STABLE,UNAME_v=FreeBSD
10.4-STABLE #0:\
$ sed -e "s/,UNAME_r.*:/:/ ; s/:\(setenv.*\):/:\1${login_env}:/" testfile
:setenv=MAIL=/var/mail/$,BLOCKSIZE=K:\
Try running the sed command by hand one piece at a time to troubleshoot what's
breaking.
$ sed "s//,UNAME_r.*:/:/" ${mnt}/etc/login.conf
$ sed "s/:\(setenv.*\):/:\1${login_env}:/" ${mnt}/etc/login.conf
$ echo $login_env
--
Sent with https://mailfence.com
Secure and private email
Re: anki latest?
Pau Amma writes: > o/~ > Whatever became of anki? > Has anyone heard of it? > Is it broken? Ignored? > Perhaps deprecated? > Is a dependency > Broken? > o/~ Seems fine. FreshPorts nowadays provides package availability table. https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/anki-2.1.48_3.pkg http://www.ipv6proxy.net/go.php?u=http://beefy6.nyi.freebsd.org/data/122amd64-default/a9f0602be17e/logs/anki-2.1.48_3.log https://pkg.freebsd.org/FreeBSD:13:amd64/latest/All/anki-2.1.48_3.pkg http://www.ipv6proxy.net/go.php?u=http://beefy16.nyi.freebsd.org/data/130amd64-default/814fc6db724a/logs/anki-2.1.48_3.log https://pkg.freebsd.org/FreeBSD:14:amd64/latest/All/anki-2.1.48_3.pkg http://www.ipv6proxy.net/go.php?u=http://beefy18.nyi.freebsd.org/data/main-amd64-default/p0ee469549a2a_se92ffd9b62/logs/anki-2.1.48_3.log > Is something wrong with the package builders? I just noticed that a > long-deferred pkg upgrade would delete anki, which I use daily, so > after checking the port hadn't gone away, Run "pkg check -d" or beware of dragons. pkg(8) always tries to fix inconsistencies, so it may try to remove existing packages or install unrelated stuff because something was forcefully removed/reinstalled some time ago. For example, mixing ports and packages can easily lead to such a situation. > I checked https://pkg-status.freebsd.org/builds?server=beefy6, and for > 122amd64 default, it states: queued 5639, built 5195, failed 47, > skipped 73, and ignored 324. That sounds very low. (And I'm not sure > it explains why anki is gone from packages. Is a dependency broken?) Better use the following workflow: 1. Open https://pkg-status.freebsd.org/?all=1&type=package 2. Type desired major version + architecture in Search field e.g., 130amd64 3. Look at Status column for stopped:done 4. Look at Build column then click on poudriere icon (smoking powder keg) 5. In poudriere page type desired package name in *every* Search field 6. PROFIT: anki is green (built) on amd64 and yellow (ignored) on i386
poudriere and MAX_EXECUTION_TIME_PACKAGE vs. NOHANG_TIME
I've been experimenting with WITH_DEBUG= port builds via poudriere-devel --including some testing of bulk -a . For something like devel/llvm13 the .pkg file generated is huge (multi-GiBytes) and takes more time to produce than the rest of the llvm13 build did. This is with that being the only active builder (no otherwise significant activity on the ThreadRipper 1950X in use for the experiments). If other builders or other activity leads to load averages noticeably above the hardware thread count (32 in the context), that adds to how long teh .pkg file generation takes. What I noticed was that setting MAX_EXECUTION_TIME_PACKAGE effectively did no good for the load average case if the NOHANG_TIME was shorter: it stops for the NOHANG_TIME during the .pkg generation instead. In part this is the lack of having any output (progress messaging?) while the .pkg file is being generated, although may be one could argue that the package phase possibly should not have NOHANG_TIME applied at all. I'm only noting the relationship and need to understand it when setting the figures --and the need for large figures if WITH_DEBUG= is to be in use for various ports. I've not checked if there is any other phases/activities that might have similar issues possible. I just actually ran into having devel/llvm* builders stop for NOHANG_TIME . === Mark Millard marklmi at yahoo.com
Re: [HEADSUP] Deprecation of the ftp support in pkg
On 2022-01-24 10:16, Chris wrote: On 2022-01-24 10:08, Baptiste Daroussin wrote: On Mon, Jan 24, 2022 at 09:29:45AM -0800, Chris wrote: On 2022-01-24 03:00, Daniel Engberg wrote: > Hi, > > I just wanted to chime in on Alexander's (netchild@) mail and I fully agree. > Looking at base we already have a lot of contrib and since we need to > adapt each > software project to our build framework a lot tends to get dated quickly and > currently is so I don't see the benefit importing more at all. I would > also like > to highly advice against importing software which is considered dead > upstream, > https://en.wikipedia.org/wiki/Thttpd . > > If anything, something like https://github.com/emikulic/darkhttpd or > similar which > would be very easy to maintain and is active FWIW I'd also like to vote +1 on NO additions. I only vote in favor of ftp in this thread because we have a million year... OK 30 plus years of track record for it and it just works. Tho I must admit I find @bapt's recent tcp proposal an interesting and appealing idea. :-) The proposal is now in anyway ;) Uh, Oh. ;-) I am just struggling on the name of the scheme: tcp:// or pkg+tcp:// (with a rename of ssh into pkg+ssh:// :D) +1 for tcp:// OTOH maybe a pkg:// proto would be even better still (https://www.iana.org/protocols/apply) Best regards, Bapt -- Chris P.S. You must REALLY hate ftp(1) to have gone to all this trouble. ;-) 0xBDE49540.asc Description: application/pgp-keys
Re: poudriere and MAX_EXECUTION_TIME_PACKAGE vs. NOHANG_TIME
[Just a resend with a corrected email address.] On 2022-Jan-24, at 22:05, Mark Millard wrote: I've been experimenting with WITH_DEBUG= port builds via poudriere-devel --including some testing of bulk -a . For something like devel/llvm13 the .pkg file generated is huge (multi-GiBytes) and takes more time to produce than the rest of the llvm13 build did. This is with that being the only active builder (no otherwise significant activity on the ThreadRipper 1950X in use for the experiments). If other builders or other activity leads to load averages noticeably above the hardware thread count (32 in the context), that adds to how long teh .pkg file generation takes. What I noticed was that setting MAX_EXECUTION_TIME_PACKAGE effectively did no good for the load average case if the NOHANG_TIME was shorter: it stops for the NOHANG_TIME during the .pkg generation instead. In part this is the lack of having any output (progress messaging?) while the .pkg file is being generated, although may be one could argue that the package phase possibly should not have NOHANG_TIME applied at all. I'm only noting the relationship and need to understand it when setting the figures --and the need for large figures if WITH_DEBUG= is to be in use for various ports. I've not checked if there is any other phases/activities that might have similar issues possible. I just actually ran into having devel/llvm* builders stop for NOHANG_TIME . === Mark Millard marklmi at yahoo.com
