Right, I think this is better too. The only problematic bit is that situations where target _sysconfigdata is needed can produce really unhelpful errors (e.g. meson hides this behind a generic 'python is not working' message), and it is not obvious that these errors can be resolved by adding a target python dependency.
Alex On Thu, 12 Nov 2020 at 20:39, Martin Jansa <martin.ja...@gmail.com> wrote: > Do you remember kergoth's take on this? From "[RFC PATCH] devtool: remove > _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack" thread and > https://lists.yoctoproject.org/g/yocto/topic/74637733#49571 > > > https://github.com/openembedded/openembedded-core/compare/master...kergoth:python-sysconfig > > I think wrapper is a bit better, because it will also catch cases where > host python3 is being called from e.g. do_compile (e.g. by full path - > after debugging ROS catkin in last few days, some components are really > creative how to find their preferred python) even when python3native is > inherited. And also doesn't need another bbclass to be inherited in cases > where target _sysconfigdata is really needed. > > On Thu, Nov 12, 2020 at 8:26 PM Alexander Kanavin <alex.kana...@gmail.com> > wrote: > >> Thanks - I have a preliminary fix here (needs thorough testing): >> >> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=akanavin/package-version-updates&id=d6b6a6f67cc967c0ead6dbab3c95435b7ca85246 >> >> Alex >> >> On Thu, 12 Nov 2020 at 19:46, Glenn Stroz <gst...@uber.com> wrote: >> >>> Hi Alex, >>> >>> We have a custom fetcher implemented as a bbclass that calls out to a >>> python script. That is why we see these issues with native. Here's a >>> sample bbclass and recipe that reproduces the issue: >>> >>> impfail.bbclass >>> ``` >>> inherit python3native >>> >>> do_unpack[noexec] = "1" >>> do_patch[noexec] = "1" >>> do_install[noexec] = "1" >>> >>> do_fetch() { >>> python3 -c "import pydoc" >>> } >>> ``` >>> >>> classrunner_0.0.1.bb >>> ``` >>> SUMMARY = "Python runner" >>> LICENSE = "CLOSED" >>> inherit impfail >>> ``` >>> >>> -Glenn >>> >>> >>> On Thu, Nov 12, 2020 at 4:52 AM Alexander Kanavin < >>> alex.kana...@gmail.com> wrote: >>> >>>> A quick fix may be to add a target python3 dependency to your recipe, >>>> yes ugly, but I can't figure out how to provide target python configuration >>>> to native python used in the context of building target recipes otherwise. >>>> >>>> Alex >>>> >>>> >>>> On Thu, 12 Nov 2020 at 10:49, Alexander Kanavin <alex.kana...@gmail.com> >>>> wrote: >>>> >>>>> On Thu, 12 Nov 2020 at 02:23, Glenn Stroz via lists.openembedded.org >>>>> <gstroz=uber....@lists.openembedded.org> wrote: >>>>> >>>>>> We've been upgrading our work to Dunfell and have encountered an >>>>>> issue with python3-native. >>>>>> >>>>>> One of our python3-native scripts uses pydoc and that in turn >>>>>> eventually pulls in python's sysconfig. However, this is failing to >>>>>> import >>>>>> with the following stack trace: >>>>>> >>>>>> ``` >>>>>> import pydoc >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/pydoc.py", >>>>>> line 366, in <module> >>>>>> class Doc: >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/pydoc.py", >>>>>> line 396, in Doc >>>>>> def getdocloc(self, object, basedir=sysconfig.get_path('stdlib')): >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>> line 519, in get_path >>>>>> return get_paths(scheme, vars, expand)[name] >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>> line 509, in get_paths >>>>>> return _expand_vars(scheme, vars) >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>> line 172, in _expand_vars >>>>>> _extend_dict(vars, get_config_vars()) >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>> line 558, in get_config_vars >>>>>> _init_posix(_CONFIG_VARS) >>>>>> File >>>>>> "/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python3.8/sysconfig.py", >>>>>> line 428, in _init_posix >>>>>> _temp = __import__(name, globals(), locals(), >>>>>> ['build_time_vars'], 0) >>>>>> ModuleNotFoundError: No module named '_sysconfigdata' >>>>>> ``` >>>>>> >>>>>> I did a little digging and the following commit seems to reference >>>>>> related work: >>>>>> >>>>>> >>>>>> https://github.com/openembedded/openembedded-core/commit/02714c105426b0d687620913c1a7401b386428b6 >>>>>> >>>>>> I see that there is a patch added which adds the following to >>>>>> `sysconfig.py`: >>>>>> >>>>>> ``` >>>>>> if 'STAGING_LIBDIR' in os.environ: >>>>>> >>>>>> sys.path.append(os.environ['STAGING_LIBDIR']+'/python-sysconfigdata') >>>>>> ``` >>>>>> >>>>>> However, STAGING_LIBDIR does not contain this path: >>>>>> >>>>>> ``` >>>>>> ls >>>>>> /home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot/usr/lib/python-sysconfigdata >>>>>> >>>>>> ls: cannot access >>>>>> '/home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot/usr/lib/python-sysconfigdata': >>>>>> No such file or directory >>>>>> ``` >>>>>> >>>>>> It appears as if this should really be pointing >>>>>> to STAGING_LIBDIR_NATIVE which does exist: >>>>>> >>>>>> ``` >>>>>> ls >>>>>> /home/gstroz/.cache/avos/_avos_gstroz/avos/yocto-dunfell/build/tmp/work/corei7-64-poky-linux/ebpf-daemon/2020.282.17-r0/recipe-sysroot-native/usr/lib/python-sysconfigdata >>>>>> _sysconfigdata.py >>>>>> ``` >>>>>> >>>>>> The referenced commit made efforts to combine the native and cross >>>>>> python3 bbclass files into one. However, I think this may be something >>>>>> that >>>>>> needs to be treated differently. >>>>>> >>>>>> Anyone have any thoughts on how to proceed? >>>>>> >>>>> >>>>> Can you share the exact context where this is happening? Is your >>>>> script coming from a recipe and can you show that recipe? >>>>> >>>>> Alex >>>>> >>>>> >>>> >> >> >>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#144534): https://lists.openembedded.org/g/openembedded-core/message/144534 Mute This Topic: https://lists.openembedded.org/mt/78197661/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
