Re: We need to do something about build times
On Sat, 28 Oct 2023 16:10:11 +0200 Moin Rahman wrote: > > On Oct 28, 2023, at 12:32 AM, Moin Rahman wrote: > > > > > > > >> On Oct 27, 2023, at 11:37 PM, Tatsuki Makino > >> wrote: > >> > >> Moin Rahman wrote on 2023/10/28 01:50: > >>> But by no means it reduces the build > >>> time of texlive-texmf. Hope everyone enjoys that. :) > >>> > >> > >> There are updates in the current port tree that cause packages rebuilt by > >> glib updates to be rebuilt again :) > >> > >>> [00:01:19] [Dry Run] Deleting texlive-base-20230313_3.pkg: new > >>> dependency: print/ghostscript10 > >>> [00:01:22] [Dry Run] Deleting texlive-texmf-20230313.pkg: missing > >>> dependency: texlive-base-20230313_3 > >>> [00:01:39] [Dry Run] Ports to build: ... print/texlive-base > >>> print/texlive-texmf ... > >> > >> As far as I can see, it seems that it is possible to avoid rebuilding > >> texlive-texmf by manually running poudriere bulk with print/texlive-base. > >> However, something else may cause texlive-texmf package to be removed, so > >> it would be better to proceed gradually from libXdmcp, libxcb or something. > >> > >> I enjoy manual manipulation to minimize the time spent on rebuilding :) > >> > >> Regards. > >> > > > > I believe you do not have the latest tree. I have removed the build time > > dependency to texlive-base. > > > > This is my latest build: > > > > https://pkg.bofh.network/data/latest-per-pkg/texlive-texmf/20230313/124i386-default.log > > > > And there is not call to texlive-base itself. > > > > Kind regards, > > Moin(bofh@ with tex@ hats on) > > > > In contrary to my previous comment I think that somehow poudriere detected > the change that it no longer depends on print/texlive-base and hence rebuilt > the pkg so that it's properly processed in the pkg and removes the dependency > on print/texlive-base. > > So in the previous pkg repo it had a BUILD time dependency on > print/texlive-base but after the latest build it is no longer there. Maybe > there are still some place of improvements in poudriere's change detection > mechanism specially BUILD_DEPENDS. :P > > Kind regards, > Moin(bofh@ with all hats off) And maybe indirect dependencies, too. ;-( Not sure it's already done or not, maybe pkg database should record... *ACTUAL fetch depends *ACTUAL patch depends *ACTUAL build depends *ACTUAL lib depends *ACTUAL run depends *And possibly, ACTUAL test depends regardless it's explicitly specified in ports skeleton (including via uses etc.) or not. For example, for lib depends, If port A depends on port B, and port B depends on port C in each Makefile, If nothing in port A directly links libraries in port C, no need to record port C as dependency of port A. If anything in port A directly links libraries in port C, port C SHALL be recorded as lib depends of port A, even if it is NOT explicitly described as *_DEPENDS in port A's Makefile. These would vary with how OPTIONS are set / how FLAVOURs are set. If this is possible, it would largely help maintainers, with the cost of huge amount of `make package` time. At worst, BUILD_DEPENDS should be recorded in pkg database and some option to retrieve them, like %r and %d. I strongly suspect the lack of this functionality affects mis-behaviour of poudriere on deciding build order and combination. -- Tomoaki AOKI
Re: We need to do something about build times
Tomoaki AOKI wrote on 2023/10/29 20:51: > On Sat, 28 Oct 2023 16:10:11 +0200 > Moin Rahman wrote: >> >> Maybe there are still some place of improvements in poudriere's change >> detection mechanism specially BUILD_DEPENDS. :P >> > > And maybe indirect dependencies, too. ;-( > > At worst, BUILD_DEPENDS should be recorded in pkg database and some > option to retrieve them, like %r and %d. > I strongly suspect the lack of this functionality affects mis-behaviour > of poudriere on deciding build order and combination. > I don't see any problem in detecting dependencies by poudriere. The various other *_DEPENDS that have been added are only conditional BUILD_DEPENDS or RUN_DEPENDS. For example, LIB_DEPENDS: detect using libname and set both BUILD_DEPENDS and RUN_DEPENDS. FETCH_DEPENDS: BUILD_DEPENDS used when distfiles are missing I think this problem is more of a RUN_DEPENDS problem. It is that ports that waste time when rebuilt frequently have been included in the following structure. [updated popular port like glib] <--+-- run-depend -- [ zero or some port ] <-- run-depend -- [ commonly used in build-time dependencies like doxygen ] <--+ | | | +-- build-depend --+ | | +--- run-depend --+-- [ port to be rebuilt as it lost popular port ] It is very confusing :) It seems that any dependencies, whether build or run, will be included in this graph, and that any ports where the run dependencies is broken will be rebuilt. In this regard, poudriere already has a method for pseudo-testing. As I have already written previously, it will be as follows poudriere bulk -j ... -C updated/upstream-popular-port affected-by-upstream-update/port-to-queue Here are some examples... But I can't think of a good example right off the bat :) First, there is a recent common poudriere bulk -j ... -n -C graphics/vulkan-headers devel/py-qt5-pyqt Never forget the -n option when this is performed. Otherwise, disaster will befall the already built package :) This reproduces vulkan updates. This will delete a lot of qt5-*. However, as you can see from the terminal logs, only qt5-gui is rebuilt, avoiding the loss of qt5-webkit. Another. poudriere bulk -j ... -n -C multimedia/aom multimedia/gstreamer1-plugins-core aom has also been updated rather frequently. However, while rebuilding ffmpeg is unavoidable, rebuilding that much will avoid rebuilding gstreamer1 and the ports that use it. poudriere already has the option -S to not detect dependencies. But this is too much. It also makes it difficult to find packages whose dependencies have been cut off. What should be done in poudriere to make this problem smaller is not a review of dependency detection methods. The decision to delete the package should be delayed. When a copy (hardlink) is made in the .building directory, unwanted packages are removed by not being processed. It would be nice to have the ability to stop doing that, copy all packages and then re-examine them to see if they need to be rebuilt when they are rebuilt. It doesn't sound too easy... Regards.
Unmaintained FreeBSD ports which are out of date
Dear port maintainers, The portscout new distfile checker has detected that one or more unmaintained ports appears to be out of date. Please take the opportunity to check each of the ports listed below, and if possible and appropriate, submit/commit an update. Please consider also adopting this port. If any ports have already been updated, you can safely ignore the entry. An e-mail will not be sent again for any of the port/version combinations below. Full details can be found at the following URL: http://portscout.freebsd.org/po...@freebsd.org.html Port| Current version | New version +-+ cad/ifcopenshell| 0.6.0 | blenderbim-231029 +-+ multimedia/vapoursynth | R64 | r65 +-+ If any of the above results are invalid, please check the following page for details on how to improve portscout's detection and selection of distfiles on a per-port basis: http://portscout.freebsd.org/info/portscout-portconfig.txt Reported by:portscout!
Re: We need to do something about build times
Daniel Engberg wrote on 2023/10/28 17:48:
> If upstream uses GNU Autotools, use upstream release archives as they
> usually contains a configure script ready to run which means that you
> can avoid USES= autoreconf which is slow and adds unncessary
> dependencies.
This has the following problem
Apply patches to {configure.ac,Makefile.am} and run autoreconf
versus
Apply patches to {configure,Makefile.in} and run ./configure immediately
Incidentally, some ports (e.g. security/heimdal*) do not know which it is.
> If dependencies are unbundled you can save I/O and processing time by
> not extracting.
>
> Example:
> https://cgit.freebsd.org/ports/tree/net-mgmt/netdata/Makefile#n32
With -X (or --exclude-from) of tar, a list of unwanted files can be made into a
file.
If the file is subject to SUB_FILES, the options also allow selection of
unnecessary files.
... I have thought about this for a moment.
It was impossible because the timing when SUB_FILES is extracted is much later
than when it is possible to do so.
Regards.
