Advice wanted: handling weird vendoring situation
Hi, I'm working towards packaging pydevd (which is a recursive dependency of Spyder via IPython), and it's a somewhat challenging package! I have hit an issue and would appreciate some thoughts on how best to handle it. Background: pydevd is a debugging package which can attach to running scripts. It is used by PyDev, PyCharm, VSCode and Spyder, and with Spyder, it is imported through debugpy, which in turn is imported into IPython. In order to ensure that the libraries it is using have not been modified by gevent, it uses vendored copies of various standard library modules rather than just saying "import inspect" etc. I thought I could address this issue by replacing the vendored copies of the library modules by symlinks to /usr/lib/python3.X/, but now I've hit another snag: some of these modules import other modules. For example: pydev_imps/_pydev_SimpleXMLRPCServer.py is a very old version of /usr/lib/python3.X/xmlrpc/server.py. It contains within it the following lines: from _pydev_imps import _pydev_xmlrpclib as xmlrpclib from _pydev_imps._pydev_xmlrpclib import Fault from _pydev_imps import _pydev_SocketServer as SocketServer from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer These libraries are: _pydev_imps._pydev_xmlrpclib -> xmlrpc.client _pydev_imps._pydev_SocketServer -> socketserver _pydev_imps._pydev_BaseHTTPServer -> http.server So what should I do? One solution is just to symlink from _pydev_SimpleXMLRPCServer.py to /usr/lib/python3.X/xmlrpc/server.py and not worry about the other modules. But that might break things in non-obvious ways, so I don't want to do that. Another possible solution is to update all of the vendored copies in _pydev_imps using the relevant /usr/lib/python3.X/* modules and making the same modifications to the imports to load local copies. But then we will have duplicate copies of standard library modules, which I also don't want to do. Perhaps another possibility is to have symlinks in the _pydev_imps directory to the standard library versions and then temporarily modify sys.path to look in _pydev_imps before looking in standard locations. I don't know whether this will work, though. (There is another snag, too, which is that the path will depend on the Python version. So I will probably have _pydev_imps/python_39/socketserver.py -> /usr/lib/python3.9/socketserver.py and _pydev_imps/python_310/socketserver.py -> /usr/lib/python3.10/socketserver.py But that's a simpler issue.) Any thoughts on these ideas would be much appreciated! Best wishes, Julian
Re: Bug#1005043: lintian: check that Python version numbers are not 0.0.0
Hi Julian (2022.02.07_06:26:38_+) > I'm a little confused by this. Have a look at > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005039 against > python3-iniconfig. It has a very straightforward debian/rules, using > pybuild, and its setup.py script has "use_scm_version=True", but it > still produces a python package with version number 0.0.0. > > I have tried this in an environment where I have > python3-setuptools-scm installed, by the way (even though the package > does not Build-Depends on it). I'm using dh-python version 5.20220119 That's the issue, it *needs* to Build-Depend on that (pybuild only exports the PRETEND environment variable when there is a Build-Dependency). Committed a fix to git. SR -- Stefano Rivera http://tumbleweed.org.za/ +1 415 683 3272
Re: Bug#1005043: lintian: check that Python version numbers are not 0.0.0
On Tue, Feb 08, 2022 at 12:26:01AM +, Stefano Rivera wrote: > Hi Julian (2022.02.07_06:26:38_+) > > I'm a little confused by this. Have a look at > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005039 against > > python3-iniconfig. It has a very straightforward debian/rules, using > > pybuild, and its setup.py script has "use_scm_version=True", but it > > still produces a python package with version number 0.0.0. > > > > I have tried this in an environment where I have > > python3-setuptools-scm installed, by the way (even though the package > > does not Build-Depends on it). I'm using dh-python version 5.20220119 > > That's the issue, it *needs* to Build-Depend on that (pybuild only > exports the PRETEND environment variable when there is a > Build-Dependency). > > Committed a fix to git. Ah, thanks Stefano! Best wishes, Julian
Re: Advice wanted: handling weird vendoring situation
On Mon, Feb 07, 2022 at 09:27:28PM +, Julian Gilbey wrote: > [...] > > I thought I could address this issue by replacing the vendored copies > of the library modules by symlinks to /usr/lib/python3.X/, but now > I've hit another snag: some of these modules import other modules. > For example: > > pydev_imps/_pydev_SimpleXMLRPCServer.py > is a very old version of /usr/lib/python3.X/xmlrpc/server.py. It > contains within it the following lines: > > from _pydev_imps import _pydev_xmlrpclib as xmlrpclib > from _pydev_imps._pydev_xmlrpclib import Fault > from _pydev_imps import _pydev_SocketServer as SocketServer > from _pydev_imps import _pydev_BaseHTTPServer as BaseHTTPServer > [...] > Perhaps another possibility is to have symlinks in the _pydev_imps > directory to the standard library versions and then temporarily modify > sys.path to look in _pydev_imps before looking in standard locations. > I don't know whether this will work, though. I realise now that this "nice" solution won't work, as the standard library code says: import socketserver so modifying sys.path will just change the value of sys.modules["socketserver"]. However, the vendored code instead loads this module to sys.modules["_pydev_imps._pydev_SocketServer"] or something like that, deliberately avoiding interfering with sys.modules["socketserver"]. Ho hum. Julian
