[Bug 263353] lang/python38: fix excessive ld.lld memory use with LTO in port options
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 --- Comment #1 from Dan Kotowski --- I'm seeing the same problem even on a 10C/20T Xeon with 32GB DDR3 + 64GB NVMe swap. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.
FreeBSD ports you maintain which are out of date
Dear port maintainer, The portscout new distfile checker has detected that one or more of your ports appears to be out of date. Please take the opportunity to check each of the ports listed below, and if possible and appropriate, submit/commit an update. If any ports have already been updated, you can safely ignore the entry. You will not be e-mailed again for any of the port/version combinations below. Full details can be found at the following URL: http://portscout.freebsd.org/pyt...@freebsd.org.html Port| Current version | New version +-+ devel/py-pytest | 7.1.1 | 7.1.2 +-+ If any of the above results are invalid, please check the following page for details on how to improve portscout's detection and selection of distfiles on a per-port basis: http://portscout.freebsd.org/info/portscout-portconfig.txt Reported by:portscout!
Problem reports for python@FreeBSD.org that need special attention
To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status |Bug Id | Description +---+--- Open|205308 | devel/py-pip and devel/py-virtualenv don't aggree Open|240343 | x11-themes/plasma5-breeze-gtk: Fails to build if Open|242896 | lang/python*: Fail to package in poudriere (testp Open|256699 | lang/python38: Fails to configure when IPv6 not s Open|257362 | lang/python3: Add link for python3-embed.pc where Open|258192 | devel/py-pyinstaller: Fails to run on 3.8+. Fix i New |261505 | textproc/py-sphinx: Update to 4.4.0 In Progress |255025 | textproc/py-chardet: Update to 4.0.0 Open|257353 | lang/python38: Intermittently fails to build unde Open|261703 | Mk/bsd.default-versions.mk: Switch default Python In Progress |241416 | [NEW PORT] lang/python38 Open|234981 | graphics/py-wand: Add DOCS option Pass MAINTAINER Open|260448 | [NEW] devel/py-aiosignal: Project to manage callb New |262250 | [exp-run] devel/py-setuptools: Update to 61.0.0 Open|262109 | Mk/Uses/python.mk: Improve CMake/Python integrati Open|224115 | devel/py-babel directory name != port name In Progress |258195 | lang/python38: Update to 3.8.12 New |231555 | Mk/Uses/python.mk: Add USE_PYTHON=pytest 18 problems total for which you should take action.
[Bug 263353] lang/python38: fix excessive ld.lld memory use with LTO in port options
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Kubilay Kocak changed: What|Removed |Added Keywords|patch |needs-qa Status|New |Open --- Comment #2 from Kubilay Kocak --- @Matthias Can you test a build with -flto=thin -g please? LTO is well known to require significantly more resources during link time, and is precisely why a =thin model/option exists. Everything else equal, failing to link due to insufficient resources isn't a bug, with or without -g. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.
[Bug 263353] lang/python38: fix excessive ld.lld memory use with LTO in port options
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Kubilay Kocak changed: What|Removed |Added Severity|Affects Only Me |Affects Some People -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python38: fix excessive ld.lld memory use with LTO in port options
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 --- Comment #3 from Kubilay Kocak --- @Matthias Just realised that poyrts set --with-lto=thin already. Can you attach a full build log (compressed if necessary), so we can take a look at the compile invocations -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python38: fix excessive ld.lld memory use with LTO in port options
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 --- Comment #4 from Matthias Andree --- Created attachment 233456 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=233456&action=edit failing poudriere build log with LTO and optimization, showing excess -g. Kubilay, Python@ guys, the issue is an excess "-g" in spite of NOT setting WITH_DEBUG, not LTO. The FreeBSD ports Mk/* stuff properly omits -g, but the Python 3.8 build stuff adds it specifically when LTO is in place. In the log, find lots of "-flto -g" (search for it, literally, of course without the quotes). This goes unnoticed because everything's stripped in the stage area, so no stage-qa complaints, just excess disk space use. The culprit is the upstream configure stuff, not something in FreeBSD. I am attaching my log. But I seriously ask everyone to respect my debugging and solution and not fuss with distractions and excursions. Trust me to be able to read my Poudriere error logs. All these questions are wasting my time. So, unless there are substantiated vetos with better solutions brought up, I may commit my patch 14 days after my proposal, or with maintainer approval, whichever comes first. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python38: fix excessive ld.lld memory use with LTO in port options
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Kubilay Kocak changed: What|Removed |Added See Also||https://bugs.freebsd.org/bu ||gzilla/show_bug.cgi?id=2619 ||74 --- Comment #5 from Kubilay Kocak --- (In reply to Matthias Andree from comment #4) That's not how this works Matthias. Just like you'd like people to respect your time, everyone else also appreciates you respecting theirs, particularly as they spent time and effort on your bug reports. There are multiple considerations here: 1) with-lto supports =thin argument, but it was only set in python311 in ports 9a31e1b6d3 because support for =thin was not backported to earlier branches. I asked in comment 1 for a test of whether you can link with =thin -g as that (thin), along with decoupling our LTo build from Pythons lto option is the most desirable target state 2) Bug 261974 added support via Pythons build system flag for LTO, not using our own toolchain flags. This means we were necessarily coupled to what Python does with this flag. That commit landed somewhat prematurely prematurely and we are now dealing with that. 2) Adding -g in the LTO was added a while ago [1][2], and earlier than 3.8, and affects multiple Python port versions 3) The definition and scope of 'debug' in freebsd ports isn't an exact overlap with upstreams definitions of 'debug' (not just compile args). 4) Python without compelling arguments/cases to the contrary prefers to remain as close to upstream as possible, and where changes need to be made, changes are submitted for upstream inclusion/improvement unless that is impossible. We're not asking that you care about all of the above, but these are the things we need to consider, along with the overhead of carrying local patches, where alternatives may (and do) exist. [1] https://github.com/python/cpython/commit/1bb9dd337ed5aa9eafc8e2ce017ceedf044145e3 [2] https://github.com/python/cpython/issues/74530 -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 261974] lang/python311: Enable Link Time Optimization (via autoconf)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261974 Kubilay Kocak changed: What|Removed |Added See Also||https://bugs.freebsd.org/bu ||gzilla/show_bug.cgi?id=2633 ||53 -- You are receiving this mail because: You are on the CC list for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Kubilay Kocak changed: What|Removed |Added Summary|lang/python38: fix |lang/python3*: Fails to |excessive ld.lld memory use |link with LTO: Python |with LTO in port options|includes unconditionally ||adds -g to --with-lto -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Kubilay Kocak changed: What|Removed |Added Keywords||needs-patch -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Matthias Andree changed: What|Removed |Added Keywords|needs-patch | --- Comment #6 from Matthias Andree --- koobs@, we cannot work like that and leave ports regressed and non-building just because someone wants to have the perfect fix, possibly sitting the fix out until the cows come home (*) I don't buy your PGO consideration: What piece of the code would enable PGO anyways? The configure stage suggests to --enable-optimizations (which we do not do) to enable PGO, "make configure" is sufficient to see that. We don't see the regression that prompted the upstream change, and given the nature of the latter, I contend that it's trying to downstream (as in Python) fix compiler bugs. And now you're talking to me about unfitting patches. I fail to see what benefit -ftlto=thin would bring, because it does not take any of your objectsion away; but feel free to run it and see how far it reduces size of the wkrdir and peak linker memory use, and if it really achieves the same goals, commit before maintainer timeout. I am not wasting more time. (*) I do not care about minimal deviation from upstream on a life support branch (which is what Python 3.7 and 3.8 are, security fix only). The build regression is real, and the easy fix would have been to just revert the offending commit and possibly bump PORTEPOCH, which I haven't done although I could perfectly have invoked the "fix broken build" blanket just as well. We can always refine fixes later, but barring a better solution in due time (end April), we need a solution and we need to get the short-term fix in place and get it out of the (time-)critical path. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 --- Comment #7 from Matthias Andree --- TL;DR: none of your objections are relevant, or would be remedied with -flto=thin. The only item you mentioned that is half-way relevant is (3) but still we "port" ports by adjusting them to our conventions WRT compiler flags. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Matthias Andree changed: What|Removed |Added Keywords|needs-qa|patch, patch-ready -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 Kubilay Kocak changed: What|Removed |Added Keywords|patch, patch-ready, |needs-patch, needs-qa |regression | Flags|maintainer-feedback?(python |maintainer-feedback+ |@FreeBSD.org) | --- Comment #8 from Kubilay Kocak --- (In reply to Matthias Andree from comment #6) The issue applies to multiple python ports, you are welcome to provide a patch against all ports for consideration. The issue here is not about a perfect fix, its about your assumption that asking for additional details, testing or questions means someone is looking for a perfect fix, or in some manner, to block you. This is not the case. Please leave your frustrations at the Bugzilla door. In the mean time, since the LTO patch landed prematurely, it *is* indeed an option, and the report here is a symptom of the implementation of said option, you are welcome to provide a separate patch removing LTO from OPTIONS_DEFAULT (for all python ports), which we're happy to approve once provided. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 263353] lang/python3*: Fails to link with LTO: Python includes unconditionally adds -g to --with-lto
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263353 --- Comment #9 from Kubilay Kocak --- (In reply to Kubilay Kocak from comment #8) Said patch for removal from all python ports could omit python311, as it uses (supports) -flto=thin. Additionally testing with thin/-g combinations remain useful to determine the contribution of each in the failure cases, to inform and assist resolution upstream. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.
