Re: We need to do something about build times
Hi, Many interesting thoughts about current situation, here's my take on it and I'm also trying to catch up. Some ports do require quite a lot of CPU time and memory to build, I fully understand that not everyone is using the latest hardware available but there's little we can do about upstream projects overall since the ports tree is after all intended for packaging not maintaining. By that I mean there will likely be little to no action taken if you report upstream that lets say Rust don't build with 12Gbyte of RAM. I do agree that if we can consolidate dependencies it's for the better in terms of build times, disk footprint (for installed ports, might not be much of an issue these days) and potential overlapping libraries/dependencies. What we need to accept is that 4 cores and 16Gb RAM isn't a "powerhouse" that will build whatever you throw at it, that's the reality. We can't adapt ports to some arbitrary "hardware requirement", that's not viable nor do we have anywhere near the manpower to do so however if we can streamline the build with little effort by all means, go for it. What I have concerns about is adapting port options that by default potentially will harm performance and disk footprint in favour of "it builds faster". I would at least like to claim that we primarily use FreeBSD to run applications not to build ports. If we can combine both that would be great but the former should be prioritized over latter. If we can make packages smaller in size we should because it benefits "everyone" however if it is of a very little gain and adds a lot of time to the build it should be evaluated. For example we do apply LTO and a few other optimizations to all Rust (Cargo based) ports because it greatly reduces binary size and in some cases also increases peformance while increasing total build by a bit. There are already other distros looking into enabling LTO "treewide" so we're likely going to head there too eventually. Porters Handbook covers unbundling and is also a policy for several other distros. I think we do a pretty good about about it in general however some projects makes it very hard or impossible to unbundle heavy dependencies, all we can do here is to discuss it with upstream. While we could "maintain" our own set of patches it's for certain a recipe for ports getting abandoned due to maintence burden sooner than later. In short, please upstream and have a discussion with upstream about such concerns. We already miss out in some cases quite significant performance enhancements with pre-built packages for ports that do not support run-time detection of hardware capatibilities so we probably need to look at difference target branches for each arch futher down the road adding more overall load. Some observations on my end, Without carefully measuring I do find that my Poduriere instances in many cases spends quite a bit of time not building but spending quite a bit of time in configure stage and struggle to scale over ~4-6 jobs. What we can do per port level that improves build times and scaling in general: If upstream uses GNU Autotools, use upstream release archives as they usually contains a configure script ready to run which means that you can avoid USES= autoreconf which is slow and adds unncessary dependencies. If upstream provides an alternative to GNU Autotools and/or gmake, libtool such as CMake or Meson it will likely run configure stage faster and build quicker, sometimes by quite a bit especially if you have a system with many "slow" cores (such as ARM but even x86 to some degree). On some ports you can cut down build times ~40-50% because of better scaling and processing, avoiding USES= autoreconf and friends. You likely wont see that much of a difference overall but 10-20% isn't impossible (slow systems such as RockPro64 tend to see better improvements). There might be some quirks switching build framework that you might need to work out with upstream but it's usually worth the effort in the end. Not saying that it'll compensate for spending 8h build LLVM/Clang but if we can "convert" lets say100 ports or more we will probably reduice the overall build time by quite a bit. If dependencies are unbundled you can save I/O and processing time by not extracting. Example: https://cgit.freebsd.org/ports/tree/net-mgmt/netdata/Makefile#n32 Best regards, Daniel (diizzy@)
Re: We need to do something about build times
> On Oct 28, 2023, at 12:32 AM, Moin Rahman wrote: > > > >> On Oct 27, 2023, at 11:37 PM, Tatsuki Makino >> wrote: >> >> Moin Rahman wrote on 2023/10/28 01:50: >>> But by no means it reduces the build >>> time of texlive-texmf. Hope everyone enjoys that. :) >>> >> >> There are updates in the current port tree that cause packages rebuilt by >> glib updates to be rebuilt again :) >> >>> [00:01:19] [Dry Run] Deleting texlive-base-20230313_3.pkg: new dependency: >>> print/ghostscript10 >>> [00:01:22] [Dry Run] Deleting texlive-texmf-20230313.pkg: missing >>> dependency: texlive-base-20230313_3 >>> [00:01:39] [Dry Run] Ports to build: ... print/texlive-base >>> print/texlive-texmf ... >> >> As far as I can see, it seems that it is possible to avoid rebuilding >> texlive-texmf by manually running poudriere bulk with print/texlive-base. >> However, something else may cause texlive-texmf package to be removed, so it >> would be better to proceed gradually from libXdmcp, libxcb or something. >> >> I enjoy manual manipulation to minimize the time spent on rebuilding :) >> >> Regards. >> > > I believe you do not have the latest tree. I have removed the build time > dependency to texlive-base. > > This is my latest build: > > https://pkg.bofh.network/data/latest-per-pkg/texlive-texmf/20230313/124i386-default.log > > And there is not call to texlive-base itself. > > Kind regards, > Moin(bofh@ with tex@ hats on) > In contrary to my previous comment I think that somehow poudriere detected the change that it no longer depends on print/texlive-base and hence rebuilt the pkg so that it's properly processed in the pkg and removes the dependency on print/texlive-base. So in the previous pkg repo it had a BUILD time dependency on print/texlive-base but after the latest build it is no longer there. Maybe there are still some place of improvements in poudriere's change detection mechanism specially BUILD_DEPENDS. :P Kind regards, Moin(bofh@ with all hats off) signature.asc Description: Message signed with OpenPGP
Re: We need to do something about build times
Moin Rahman wrote on 2023/10/28 23:10: >> I believe you do not have the latest tree. I have removed the build time >> dependency to texlive-base. >> >> And there is not call to texlive-base itself. > > In contrary to my previous comment I think that somehow poudriere detected > the change that it no longer depends on print/texlive-base and hence rebuilt > the pkg so that it's properly processed in the pkg and removes the dependency > on print/texlive-base. > > So in the previous pkg repo it had a BUILD time dependency on > print/texlive-base but after the latest build it is no longer there. Maybe > there are still some place of improvements in poudriere's change detection > mechanism specially BUILD_DEPENDS. :P I tried a command that intentionally rebuilds texlive-texmf. For example poudriere bulk -j ... -n -C print/texlive-base audio/libopenshot-audio [00:00:48] [Dry Run] (-C) Will delete existing package: libopenshot-audio-0.3.2.pkg [00:00:48] [Dry Run] (-C) Will delete existing package: texlive-base-20230313_3.pkg [00:00:52] [Dry Run] Deleting tex-formats-20230313.pkg: missing dependency: texlive-base-20230313_3 [00:00:52] [Dry Run] Deleting texlive-texmf-20230313.pkg: missing dependency: texlive-base-20230313_3 These are still packages created by the old port tree and were targeted for removal. [00:00:53] [Dry Run] Would build 2 packages using 2 builders [00:00:53] [Dry Run] Ports to build: audio/libopenshot-audio print/texlive-base However, they are not targeted for rebuilding because they do not have dependencies in the new tree. If an old texlive-texmf package is still removed, all packages that depend on it may be removed. By rebuilding only there, the dependency graph will be reconnected as if nothing had happened. poudriere bulk -j ... -C print/texlive-texmf poudriere bulk -j ... -C print/tex-formats It seems that texlive-texmf has been separated from its frequently rebuilt dependencies. Thank you very much.
Unmaintained FreeBSD ports which are out of date
Dear port maintainers, The portscout new distfile checker has detected that one or more unmaintained ports appears to be out of date. Please take the opportunity to check each of the ports listed below, and if possible and appropriate, submit/commit an update. Please consider also adopting this port. If any ports have already been updated, you can safely ignore the entry. An e-mail will not be sent again for any of the port/version combinations below. Full details can be found at the following URL: http://portscout.freebsd.org/po...@freebsd.org.html Port| Current version | New version +-+ games/tomenet | 4.6.0 | 4.9.1 +-+ If any of the above results are invalid, please check the following page for details on how to improve portscout's detection and selection of distfiles on a per-port basis: http://portscout.freebsd.org/info/portscout-portconfig.txt Reported by:portscout!
