Jumping in to note that the python3native patch mentioned above worked for us (removing the env var). We did not test any target packages.
-Glenn On Fri, Nov 13, 2020 at 5:45 AM Alexander Kanavin <alex.kana...@gmail.com> wrote: > I think I have to go back to my original patch. Using a wrapper for native > python is problematic for two reasons: > - thwarts target python build as target python needs native python, but > *without* the environment variable set > - adds an unconditional dependency on target python to a few items, even > if those items only want to run some scripts > > I think we're better off separating the two classes: python3native for > those who need native python, python3targetconfig for those > who also want to use target python config. > > Alex > > On Thu, 12 Nov 2020 at 21:02, Alexander Kanavin via lists.openembedded.org > <alex.kanavin=gmail....@lists.openembedded.org> wrote: > >> Right, I think this is better too. The only problematic bit is that >> situations where target _sysconfigdata is needed can produce really >> unhelpful errors (e.g. meson hides this behind a generic 'python is not >> working' message), and it is not obvious that these errors can be resolved >> by adding a target python dependency. >> >> Alex >> >> On Thu, 12 Nov 2020 at 20:39, Martin Jansa <martin.ja...@gmail.com> >> wrote: >> >>> Do you remember kergoth's take on this? From "[RFC PATCH] devtool: >>> remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack" thread and >>> https://lists.yoctoproject.org/g/yocto/topic/74637733#49571 >>> >>> >>> https://github.com/openembedded/openembedded-core/compare/master...kergoth:python-sysconfig >>> >>> I think wrapper is a bit better, because it will also catch cases where >>> host python3 is being called from e.g. do_compile (e.g. by full path - >>> after debugging ROS catkin in last few days, some components are really >>> creative how to find their preferred python) even when python3native is >>> inherited. And also doesn't need another bbclass to be inherited in cases >>> where target _sysconfigdata is really needed. >>> >>> On Thu, Nov 12, 2020 at 8:26 PM Alexander Kanavin < >>> alex.kana...@gmail.com> wrote: >>> >>>> Thanks - I have a preliminary fix here (needs thorough testing): >>>> >>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=akanavin/package-version-updates&id=d6b6a6f67cc967c0ead6dbab3c95435b7ca85246 >>>> >>>> Alex >>>> >>>> On Thu, 12 Nov 2020 at 19:46, Glenn Stroz <gst...@uber.com> wrote: >>>> >>>>> Hi Alex, >>>>> >>>>> We have a custom fetcher implemented as a bbclass that calls out to a >>>>> python script. That is why we see these issues with native. Here's a >>>>> sample bbclass and recipe that reproduces the issue: >>>>> >>>>> impfail.bbclass >>>>> ``` >>>>> inherit python3native >>>>> >>>>> do_unpack[noexec] = "1" >>>>> do_patch[noexec] = "1" >>>>> do_install[noexec] = "1" >>>>> >>>>> do_fetch() { >>>>> python3 -c "import pydoc" >>>>> } >>>>> ``` >>>>> >>>>> classrunner_0.0.1.bb >>>>> ``` >>>>> SUMMARY = "Python runner" >>>>> LICENSE = "CLOSED" >>>>> inherit impfail >>>>> ``` >>>>> >>>>> -Glenn >>>>> >>>>> >>>>> On Thu, Nov 12, 2020 at 4:52 AM Alexander Kanavin < >>>>> alex.kana...@gmail.com> wrote: >>>>> >>>>>> A quick fix may be to add a target python3 dependency to your recipe, >>>>>> yes ugly, but I can't figure out how to provide target python >>>>>> configuration >>>>>> to native python used in the context of building target recipes >>>>>> otherwise. >>>>>> >>>>>> Alex >>>>>> >>>>>> >>>>>> On Thu, 12 Nov 2020 at 10:49, Alexander Kanavin < >>>>>> alex.kana...@gmail.com> wrote: >>>>>> >>>>>>> On Thu, 12 Nov 2020 at 02:23, Glenn Stroz via lists.openembedded.org >>>>>>> <gstroz=uber....@lists.openembedded.org> wrote: >>>>>>> >>>>>>>> We've been upgrading our work to Dunfell and have encountered an >>>>>>>> issue with python3-native. >>>>>>>> >>>>>>>> One of our python3-native scripts uses pydoc and that in turn >>>>>>>> eventually pulls in python's sysconfig. However, this is failing to >>>>>>>> import >>>>>>>> with the following stack trace: >>>>>>>> >>>>>>>> ``` >>>>>>>> import pydoc >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/pydoc.py", >>>>>>>> line 366, in <module> >>>>>>>> class Doc: >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/pydoc.py", >>>>>>>> line 396, in Doc >>>>>>>> def getdocloc(self, object, >>>>>>>> basedir=sysconfig.get_path('stdlib')): >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>>>> line 519, in get_path >>>>>>>> return get_paths(scheme, vars, expand)[name] >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>>>> line 509, in get_paths >>>>>>>> return _expand_vars(scheme, vars) >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>>>> line 172, in _expand_vars >>>>>>>> _extend_dict(vars, get_config_vars()) >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>>>> line 558, in get_config_vars >>>>>>>> _init_posix(_CONFIG_VARS) >>>>>>>> File >>>>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>>>> line 428, in _init_posix >>>>>>>> _temp = __import__(name, globals(), locals(), >>>>>>>> ['build_time_vars'], 0) >>>>>>>> ModuleNotFoundError: No module named '_sysconfigdata' >>>>>>>> ``` >>>>>>>> >>>>>>>> I did a little digging and the following commit seems to reference >>>>>>>> related work: >>>>>>>> >>>>>>>> >>>>>>>> https://github.com/openembedded/openembedded-core/commit/02714c105426b0d687620913c1a7401b386428b6 >>>>>>>> >>>>>>>> I see that there is a patch added which adds the following to >>>>>>>> `sysconfig.py`: >>>>>>>> >>>>>>>> ``` >>>>>>>> if 'STAGING_LIBDIR' in os.environ: >>>>>>>> >>>>>>>> sys.path.append(os.environ['STAGING_LIBDIR']+'/python-sysconfigdata') >>>>>>>> ``` >>>>>>>> >>>>>>>> However, STAGING_LIBDIR does not contain this path: >>>>>>>> >>>>>>>> ``` >>>>>>>> ls >>>>>>>> /home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot/usr/lib/python-sysconfigdata >>>>>>>> >>>>>>>> ls: cannot access >>>>>>>> '/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot/usr/lib/python-sysconfigdata': >>>>>>>> No such file or directory >>>>>>>> ``` >>>>>>>> >>>>>>>> It appears as if this should really be pointing >>>>>>>> to STAGING_LIBDIR_NATIVE which does exist: >>>>>>>> >>>>>>>> ``` >>>>>>>> ls >>>>>>>> /home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python-sysconfigdata >>>>>>>> _sysconfigdata.py >>>>>>>> ``` >>>>>>>> >>>>>>>> The referenced commit made efforts to combine the native and cross >>>>>>>> python3 bbclass files into one. However, I think this may be something >>>>>>>> that >>>>>>>> needs to be treated differently. >>>>>>>> >>>>>>>> Anyone have any thoughts on how to proceed? >>>>>>>> >>>>>>> >>>>>>> Can you share the exact context where this is happening? Is your >>>>>>> script coming from a recipe and can you show that recipe? >>>>>>> >>>>>>> Alex >>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> >> >> >>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#144547): https://lists.openembedded.org/g/openembedded-core/message/144547 Mute This Topic: https://lists.openembedded.org/mt/78197661/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
