Re: python 3 subprocess performance
Kubilay Kocak writes:
> On 12/04/2019 8:41 pm, Dima Pasechnik wrote:
>
>> On Fri, Apr 12, 2019 at 9:46 AM Alexander Zagrebin wrote:
>>>
>>> В Fri, 12 Apr 2019 09:36:13 +0200
>>> Dima Pasechnik пишет:
>>>
On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin
wrote:
>
> В Thu, 11 Apr 2019 17:32:42 +0200
> Jan Bramkamp пишет:
>
>> The reason is that that python does something stupid (tm). It
>> tries to close all file descriptors (except a few whitelisted
>> ones) up to the maximum file descriptor number. It does this by
>> asking the kernel for the maximum possible number and closing
>> everything it doesn't want to keep. Some time later someone came
>> up with an optimization (read the open file descriptors
>> from /dev/fd). All of this pain and suffering is caused by good
>> old Ulrich Drepper braindamage:
>> https://sourceware.org/bugzilla/show_bug.cgi?id=10353.
>>
>> Most Linux distros have lower default file descriptor limits than
>> FreeBSD making this workaround less painful. The correct solution
>> would be to teach python3 about closefrom(2).
>
> Thank you for hint and testing!
>
> Indeed the problem is in closing more than 400,000 file descriptors
> in loop. It seems that all current versions of Python are affected.
> Python2 uses False as default value for the close_fds parameter of
> the Popen constructor, so this issue is mostly not visible.
> Python3 has changed this default to True.
>
> As Jan Bramkamp suggested, I've wrote simple patch to fix an issue
> (see attached file). It seems the problem has gone.
The attachment has been stripped out. Could you paste the diff into
the message?
>>>
>>> Yes, sure.
>>>
>>> --- Modules/_posixsubprocess.c.orig 2018-12-24 00:37:14.0
>>> +0300 +++ Modules/_posixsubprocess.c 2019-04-12
>>> 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@
>>> _close_fds_by_brute_force(long start_fd, }
>>> start_fd = keep_fd + 1;
>>> }
>>> +#if defined(__FreeBSD__)
>>> +closefrom(start_fd);
>>> +#else
>>> if (start_fd <= end_fd) {
>>> for (fd_num = start_fd; fd_num < end_fd; ++fd_num) {
>>> close(fd_num);
>>> }
>>> }
>>> +#endif
>>> }
>>>
If this is a Python issue, shouldn't this be reported upstream, on
https://bugs.python.org ?
>>>
>>> May be. Rather, it is a FreeBSD-specific optimization.
>>
>> Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)),
>> OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is
>> there, I just checked)
FreeBSD was late to the party. Even DragonFly got closefrom() before. ;)
Note, closefrom() looks non-atomic on Solaris.
http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/gen/closefrom.c
> Issue exists for this:
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221700
Oh, nice. I've recently it hit via www/firefox build.
https://bugzilla.mozilla.org/show_bug.cgi?id=1507655
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Python27 install problem - Undefined symbol rl_callback_sigcleanup
This is a recent 11-STABLE jail but it may contain cruft from a previous 10 (or maybe even earlier) install. Portstree updated today. I've done the mergemaster bit and readline-8.0.0 is present, rl_callback_sigcleanup eists in /usr/local/include/readline/readline.h. Some relevant parts of the build: ... *** WARNING: renaming "readline" since importing it failed: build/lib.freebsd-11.2-STABLE-amd64-2.7/readline.so: Undefined symbol "rl_callback_sigcleanup" ... Failed to build these modules: readline ... make install ===> Installing for python27-2.7.16_1 ===> Checking if python27 is already installed actual-package-depends: dependency on /lib/libreadline.so.8 not registered (normal if it belongs to base) ===> Registering installation for python27-2.7.16_1 pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/readline.so:No such file or directory *** Error code 74 ... ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: PR 23672 - java/eclipse update
Hi Kurt, Am 13.04.19 um 19:57 schrieb Kurt Jaeger: > Hi! > > While test-building java/eclipse in poudriere from > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236792 > > I run into this problem: During the build, git is used to > create a local git repo. > > This works in current, because the build inside poudriere runs as root > and therefore, this works: > > git init > git config --global user.email "ecli...@freebsd.org" > git config --global user.name "Eclipse" > git add . > > It fails on 12.0 and 11.2, because the build is run as > user nobody with > > home: /nonexistent > error: could not lock config file /nonexistent/.gitconfig: No such file or > directory > error: could not lock config file /nonexistent/.gitconfig: No such file or > directory > > Any hints how I can force git to use existing directory as $HOME > so that git does not fail ? > Many thanks for the update. Eclipse 4.11 builds and installs fine! Now I am trying to also update several tools inside ... Best wishes, Rainer ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
INDEX build failed for 11.x
INDEX build failed with errors: Generating INDEX-11 - please wait..--- describe.accessibility --- --- describe.arabic --- --- describe.archivers --- --- describe.astro --- --- describe.audio --- --- describe.benchmarks --- --- describe.biology --- --- describe.cad --- --- describe.chinese --- --- describe.comms --- --- describe.converters --- --- describe.databases --- --- describe.deskutils --- --- describe.devel --- --- describe.dns --- --- describe.editors --- --- describe.emulators --- --- describe.finance --- --- describe.french --- --- describe.ftp --- [...] --- describe.hebrew --- --- describe.hungarian --- --- describe.irc --- --- describe.japanese --- --- describe.java --- make[5]: "/home/indexbuild/tindex/ports/java/eclipse-windowbuilder/Makefile" line 26: Cannot open /home/indexbuild/tindex/ports/java/eclipse-windowbuilder/../eclipse/Makefile.plugins make[5]: Fatal errors encountered -- cannot continue===> java/eclipse-windowbuilder failed *** [describe.java] Error code 1 make[2]: stopped in /home/indexbuild/tindex/ports make[2]: stopped in /home/indexbuild/tindex/ports *** Error code 1 Stop. make[1]: stopped in /home/indexbuild/tindex/ports *** Error code 1 Stop. make: stopped in /home/indexbuild/tindex/ports 1 error Committers on the hook: adridg amdmi3 antoine araujo brnrd ehaupt farrokhi kai pi Most recent SVN update was: Updating '.': Utextproc/py-yapf/Makefile Utextproc/py-yapf/distinfo Udevel/subversive/Makefile Udevel/py-mdv/Makefile Ddevel/py-mdv/files/patch-mdv_markdownviewer.py Adevel/py-mdv/files/extra-patch-setup.py Udevel/py-mdv/distinfo Udevel/adacurses/Makefile Udevel/adacurses/pkg-descr Ujava/eclipse-pydev/Makefile Ujava/intellij/pkg-plist Djava/eclipse/Makefile.plugins Ujava/eclipse/Makefile Ujava/eclipse/distinfo Djava/eclipse/files/patch-aggregator Djava/eclipse/files/patch-freebsd_natives Djava/eclipse/files/amd64-patch-freebsd_natives Djava/eclipse/files/patch-submodules Djava/eclipse/files/i386-patch-freebsd_natives Ujava/eclipse/files/eclipse.in Ajava/eclipse/files/patch-eclipse-platform-parent Ajava/eclipse/files/patch-eclipse.pde.build Ajava/eclipse/files/patch-eclipse.pde.ui Ajava/eclipse/files/patch-eclipse.platform.releng Ajava/eclipse/files/patch-eclipse.platform.releng.prereqs.sdk Ajava/eclipse/files/patch-eclipse.platform.releng.tychoeclipsebuilder Ajava/eclipse/files/patch-eclipse.platform.resources Ajava/eclipse/files/patch-eclipse.platform.runtime Ajava/eclipse/files/patch-eclipse.platform.swt Ajava/eclipse/files/patch-eclipse.platform.swt.binaries Ajava/eclipse/files/patch-eclipse.platform.team Ajava/eclipse/files/patch-eclipse.platform.ua Ajava/eclipse/files/patch-eclipse.platform.ui Ajava/eclipse/files/patch-production Ajava/eclipse/files/patch-rt.equinox.bundles Ajava/eclipse/files/patch-rt.equinox.framework Ajava/eclipse/files/patch-rt.equinox.p2 Ujava/eclipse/pkg-descr Ajava/eclipse/pkg-plist Ajava/eclipse/scripts/post-extract Ajava/eclipse/scripts/pre-build UU java/eclipse/scripts/pre-patch Usecurity/py-msoffcrypto-tool/Makefile Usecurity/py-msoffcrypto-tool/distinfo Uwww/youtube_dl/Makefile Uwww/youtube_dl/distinfo Udatabases/mariadb102-server/Makefile Udatabases/mariadb102-server/distinfo Udatabases/mariadb103-client/Makefile Udatabases/mariadb103-server/Makefile Udatabases/mariadb103-server/distinfo Udatabases/pg_repack/Makefile Udatabases/pg_repack/distinfo Uarchivers/py-python-snappy/Makefile Uarchivers/py-python-snappy/distinfo Umath/rocs/Makefile Dnet/fping/files Unet/fping/Makefile Unet/fping/distinfo Unet/radvd/Makefile Unet/radvd/distinfo Updated to revision 498894. ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
INDEX now builds successfully on 11.x
___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Can two ports have master-slave relatioship across categories?
Hi,
Source Han Sans is a font that supports 4 East Asian languages at the same
time. Thus initially, I have split the font into four when porting;
chinese/sourcehansans-{sc,tc}-otf, {japanese,korean}/sourcehansans-otf,
respectively.
These four have all their own Makefile, pkg-plist, distinfo, ..., which now I
think are great redundancy. This is what master and slave ports are for, right?
Question is: can these four have a master-slave relationship across their
respective categories?
I have schemed the Porter's Handbook, but it does not mention anything about
redefining essential variables such as CATEGORIES, etc.
Thank you,
--
Hyun "Harold" Hwang
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Can two ports have master-slave relatioship across categories?
On Sun, Apr 14, 2019 at 10:36 AM Hyun Hwang wrote:
>
> Hi,
>
> Source Han Sans is a font that supports 4 East Asian languages at the same
> time. Thus initially, I have split the font into four when porting;
> chinese/sourcehansans-{sc,tc}-otf, {japanese,korean}/sourcehansans-otf,
> respectively.
> These four have all their own Makefile, pkg-plist, distinfo, ..., which now I
> think are great redundancy. This is what master and slave ports are for,
> right?
>
> Question is: can these four have a master-slave relationship across their
> respective categories?
> I have schemed the Porter's Handbook, but it does not mention anything about
> redefining essential variables such as CATEGORIES, etc.
Hi Harold,
Absolutely! There's essentially no limit to what can be redefined.
Just use CATEGORIES?= in the master.
# Adam
--
Adam Weinberger
ad...@adamw.org
https://www.adamw.org
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: Can two ports have master-slave relatioship across categories?
Cool! I'll start working on "enslaving" other three.
Thanks!
--
Hyun "Harold" Hwang
On Sunday, April 14, 2019, 1:08 PM (UTC-04:00), Adam Weinberger
wrote:
> On Sun, Apr 14, 2019 at 10:36 AM Hyun Hwang wrote:
> >
> > Hi,
> >
> > Source Han Sans is a font that supports 4 East Asian languages at the same
> > time. Thus initially, I have split the font into four when porting;
> > chinese/sourcehansans-{sc,tc}-otf, {japanese,korean}/sourcehansans-otf,
> > respectively.
> > These four have all their own Makefile, pkg-plist, distinfo, ..., which now
> > I think are great redundancy. This is what master and slave ports are for,
> > right?
> >
> > Question is: can these four have a master-slave relationship across their
> > respective categories?
> > I have schemed the Porter's Handbook, but it does not mention anything
> > about redefining essential variables such as CATEGORIES, etc.
>
> Hi Harold,
>
> Absolutely! There's essentially no limit to what can be redefined.
> Just use CATEGORIES?= in the master.
>
> # Adam
>
>
> --
> Adam Weinberger
> ad...@adamw.org
> https://www.adamw.org
>
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
make delete-old-libs is your friend
But I forgot that, and ended up with both /lib/libreadline.so.8 AND /usr/local/lib/libreadline.so.8 on my machine, leading to woe when compiling the latest lang/python36. Unfortunately, the base version readline was quite a bit older than the one from ports. Nevertheless, the ports version is explicitly linked as libreadline.so.8, the same as the old base version, so python36 tried linking to the base version and failed because it did not contain the new(ish) function rl_callback_sigcleanup. There's no question I shot myself in the foot by not deleting the old libraries (an omission I have now remedied after a fair amount of thrashing around to see what was wrong), but it might have made my life a little easier if the port devel/readline linked itself as libreadline.so.9 instead of 8. Is there a recommended practice (or should there be one) to change the .so version when simultaneously moving a base library to ports and a new version? -- George signature.asc Description: OpenPGP digital signature
Re: make delete-old-libs is your friend
On Sun, Apr 14, 2019 at 6:26 PM George Mitchell wrote: > > But I forgot that, and ended up with both /lib/libreadline.so.8 > AND /usr/local/lib/libreadline.so.8 on my machine, leading to woe > when compiling the latest lang/python36. Unfortunately, the base > version readline was quite a bit older than the one from ports. > > Nevertheless, the ports version is explicitly linked as > libreadline.so.8, the same as the old base version, so python36 > tried linking to the base version and failed because it did not > contain the new(ish) function rl_callback_sigcleanup. > > There's no question I shot myself in the foot by not deleting > the old libraries (an omission I have now remedied after a fair > amount of thrashing around to see what was wrong), but it might > have made my life a little easier if the port devel/readline > linked itself as libreadline.so.9 instead of 8. Is there a > recommended practice (or should there be one) to change the .so > version when simultaneously moving a base library to ports and a > new version? -- George libreadline.so is at version 8 because the current readline is 8.0. libreadline.so.9 won't come until readline-9.0 is released. We really can't deviate from that, because it'd still be v8 software; it'd be like AT&T's ridiculous claim that they invented 5G by discovering the number 5. # Adam -- Adam Weinberger ad...@adamw.org https://www.adamw.org ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: make delete-old-libs is your friend
On 4/14/19 9:07 PM, Adam Weinberger wrote: > On Sun, Apr 14, 2019 at 6:26 PM George Mitchell > wrote: >> >> But I forgot that, and ended up with both /lib/libreadline.so.8 >> AND /usr/local/lib/libreadline.so.8 on my machine, leading to woe >> when compiling the latest lang/python36. Unfortunately, the base >> version readline was quite a bit older than the one from ports. >> >> Nevertheless, the ports version is explicitly linked as >> libreadline.so.8, the same as the old base version, so python36 >> tried linking to the base version and failed because it did not >> contain the new(ish) function rl_callback_sigcleanup. >> >> There's no question I shot myself in the foot by not deleting >> the old libraries (an omission I have now remedied after a fair >> amount of thrashing around to see what was wrong), but it might >> have made my life a little easier if the port devel/readline >> linked itself as libreadline.so.9 instead of 8. Is there a >> recommended practice (or should there be one) to change the .so >> version when simultaneously moving a base library to ports and a >> new version? -- George > > libreadline.so is at version 8 because the current readline is 8.0. > libreadline.so.9 won't come until readline-9.0 is released. We really > can't deviate from that, because it'd still be v8 software; it'd be > like AT&T's ridiculous claim that they invented 5G by discovering the > number 5. > > # Adam > > You leave me no choice but to rail at whoever allowed "v4" readline to be installed as /lib/libreadline.so.8 back when it was in base. (I'd rather rail at that person than have to face my own omission.) -- George signature.asc Description: OpenPGP digital signature
2.32 assertion fail elflink.c:2935 on RPi2
Attempts to build www/firefox on rpi2 fail during compilation of llvm60, with /usr/local/bin/ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935 There don't seem to be any other error messages in the build output. Swap usage peaks at 16%, vm.pageout_oom_seq=2048 . The machine is running FreeBSD www.zefox.com 11.2-STABLE FreeBSD 11.2-STABLE #2 r345473: Sun Mar 24 08:57:56 PDT 2019 b...@www.zefox.com:/usr/obj/usr/src/sys/RPI2 arm with ports at 498696. Thanks for reading and any suggestions. bob prohaska ___ freebsd-ports@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ports To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
Re: 2.32 assertion fail elflink.c:2935 on RPi2
On 15 Apr 2019, at 06:30, bob prohaska wrote: > > Attempts to build www/firefox on rpi2 fail during compilation of > llvm60, with > > /usr/local/bin/ld: BFD (GNU Binutils) 2.32 assertion fail elflink.c:2935 > > There don't seem to be any other error messages in the build output. > Swap usage peaks at 16%, vm.pageout_oom_seq=2048 . > > The machine is running > FreeBSD www.zefox.com 11.2-STABLE FreeBSD 11.2-STABLE #2 r345473: Sun Mar 24 > 08:57:56 PDT 2019 b...@www.zefox.com:/usr/obj/usr/src/sys/RPI2 arm > > with ports at 498696. > > Thanks for reading and any suggestions. Likely a BFD ld bug, but see: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237068 -Dimitry signature.asc Description: Message signed with OpenPGP
