Properly splitting Python "-doc" packages
Hi, I was working on the "boostrap-vz" package and noticed something really annoying when creating a "boostrap-vz-doc"[1] binary package with its Sphinx documentation: the actual Python files that composes the application weren't being packaged on the main "boostrap-vz" one. I had to add "usr/lib/python*/dist-packages/bootstrapvz/" to its "debian/boostrap-vz.install"[2] to fix this, but I feel this is wrong. Looking at other Python packages that have separate "-doc" packages, like "python-django", they don't seem to need this. Also, its "*.egg-info/" folder also went missing, as we can see with "debdiff": Files in first .deb but not in second - -rw-r--r-- root/root /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/PKG-INFO -rw-r--r-- root/root /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/dependency_links.txt -rw-r--r-- root/root /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/entry_points.txt -rw-r--r-- root/root /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/requires.txt -rw-r--r-- root/root /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info/top_level.txt This broke its entry point scripts (at least I finally find out the cause of the "pkg_resources.DistributionNotFound" error), so I had to either add custom ones[3] or add "usr/lib/python2.7/dist-packages/bootstrap_vz-*.egg-info/" to its "debian/boostrap-vz.install" file as well. Any ideas on what I missed here? Regards, Tiago. [1]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=899e841f89d17418de77e5d7f56ff48627415e79 [2]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=7d0cf538f8e806f83529b3b7cad9af3ee93cca42 [3]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=d8ec5b17af1f96d9b1221963abf5abc3ef087900 -- Tiago "Myhro" Ilieve Blog: https://blog.myhro.info/ GitHub: https://github.com/myhro LinkedIn: https://br.linkedin.com/in/myhro Montes Claros - MG, Brasil
Re: Properly splitting Python "-doc" packages
[Tiago Ilieve, 2016-04-14] > I was working on the "boostrap-vz" package and noticed something > really annoying when creating a "boostrap-vz-doc"[1] binary package > with its Sphinx documentation: the actual Python files that composes > the application weren't being packaged on the main "boostrap-vz" one. > I had to add "usr/lib/python*/dist-packages/bootstrapvz/" to its > "debian/boostrap-vz.install"[2] to fix this, but I feel this is wrong. > Looking at other Python packages that have separate "-doc" packages, > like "python-django", they don't seem to need this. that's because python-django is using --buildsystem=pybuild in dh; if you add export PYBUILD_NAME=bootstrapvz it will install .py / egg-info files into python-bootstrapvz binary package (please add python-bootstrapvz binary package, BTW) > Also, its "*.egg-info/" folder also went missing, as we can see with > "debdiff": "usr/lib/python*/dist-packages/bootstrapvz/" in bootstrap-vz.install doesn't match /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info, add this line if you want egg-info to be included: /usr/lib/python*/dist-packages/bootstrap_vz-*.egg-info/ -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Properly splitting Python "-doc" packages
Hi Piotr, On 14 April 2016 at 10:52, Piotr Ożarowski wrote: > that's because python-django is using --buildsystem=pybuild in dh; if you add > > export PYBUILD_NAME=bootstrapvz > > it will install .py / egg-info files into python-bootstrapvz binary package > (please add python-bootstrapvz binary package, BTW) I had made a test with Pybuild, but ended up facing the same problem. Now I see that the problem is that the variable "PYBUILD_NAME" should have a value of "bootstrap-vz" (notice the hyphen), not "boostrapvz". Probably because this has to be consistent with whatever is the package name defined in "setup.py", right? Anyway, if the package is named "python-boostrapvz" it ended up properly packaging the files, but as I've mentioned here a couple times before[1][2], I don't want to call any Python application package as "python-*". bootstrap-vz is a CLI application and should end up in a package named "bootstrap-vz". The problem is that if I do this, I'll and up with an empty binary package package again. Is there a way to tell Pybuild that the Python files should end up in the package "bootstrap-vz" and not "python-bootstrap-vz"? > "usr/lib/python*/dist-packages/bootstrapvz/" in bootstrap-vz.install > doesn't match /usr/lib/python2.7/dist-packages/bootstrap_vz-0.9.5.egg-info, > add this line if you want egg-info to be included: > > /usr/lib/python*/dist-packages/bootstrap_vz-*.egg-info/ If I can fix the package name mentioned above, Pybuild will add the "*.egg-info/" folder as well. Regards, Tiago. [1]: https://lists.debian.org/debian-python/2016/04/msg00017.html [2]: https://lists.debian.org/debian-python/2016/04/msg00027.html -- Tiago "Myhro" Ilieve Blog: https://blog.myhro.info/ GitHub: https://github.com/myhro LinkedIn: https://br.linkedin.com/in/myhro Montes Claros - MG, Brasil
Re: Properly splitting Python "-doc" packages
[Tiago Ilieve, 2016-04-14] > On 14 April 2016 at 10:52, Piotr Ożarowski wrote: > > that's because python-django is using --buildsystem=pybuild in dh; if you > > add > > > > export PYBUILD_NAME=bootstrapvz > > > > it will install .py / egg-info files into python-bootstrapvz binary package > > (please add python-bootstrapvz binary package, BTW) > > I had made a test with Pybuild, but ended up facing the same problem. > Now I see that the problem is that the variable "PYBUILD_NAME" should > have a value of "bootstrap-vz" (notice the hyphen), not "boostrapvz". > Probably because this has to be consistent with whatever is the > package name defined in "setup.py", right? PYBUILD_NAME is used to guess the Debian binary package name (and that's THE only thing it is used for), whatever is in setup.py doesn't matter. If you use PYBUILD_NAME=foo, it will install into: debian/python-foo/# python2.X and python2.X-dbg debian/python3-foo/ # python3.X and python3.X-dbg debian/pypy-foo/ # pypy that's all, there's no more magic with PYBUILD_NAME > Anyway, if the package is named "python-boostrapvz" it ended up that's the correct binary package name for public Python 2.X "boostrapvz" module, "boostrap_vz" is egg/dist name which we ignore. For module name "boostrap_vz" correct binary package names would be python-boostrap-vz > properly packaging the files, but as I've mentioned here a couple > times before[1][2], I don't want to call any Python application > package as "python-*". bootstrap-vz is a CLI application and should > end up in a package named "bootstrap-vz". The problem is that if I do then PLEASE PLEASE PLEASE do not install as public modules - install into /usr/share/boostrap-vz/ f.e. with this pybuild args: export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/boostrap-vz/ --install-scripts=/usr/share/boostrap-vz/ and /usr/share/boostrap-vz/boostrap-vz /usr/bin/boostrap-vz /usr/share/boostrap-vz/boostrap-remote /usr/bin/boostrap-remote /usr/share/boostrap-vz/boostrap-server /usr/bin/boostrap-server in debian/boostrap-vz.links > this, I'll and up with an empty binary package package again. Is there > a way to tell Pybuild that the Python files should end up in the > package "bootstrap-vz" and not "python-bootstrap-vz"? see PYBUILD_INSTALL_ARGS above (and use PYBUILD_DESTDIR=debian/bootstrap-vz/ instead of PYBUILD_NAME) -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: Properly splitting Python "-doc" packages
Piotr, On 14 April 2016 at 12:33, Piotr Ożarowski wrote: > PYBUILD_NAME is used to guess the Debian binary package name (and that's > THE only thing it is used for), whatever is in setup.py doesn't matter. > > If you use PYBUILD_NAME=foo, it will install into: > > debian/python-foo/# python2.X and python2.X-dbg > debian/python3-foo/ # python3.X and python3.X-dbg > debian/pypy-foo/ # pypy > > that's all, there's no more magic with PYBUILD_NAME Got it. Thanks for the explanation. > then PLEASE PLEASE PLEASE do not install as public modules - install into > /usr/share/boostrap-vz/ f.e. with this pybuild args: > > export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/boostrap-vz/ > --install-scripts=/usr/share/boostrap-vz/ > > and > > /usr/share/boostrap-vz/boostrap-vz /usr/bin/boostrap-vz > /usr/share/boostrap-vz/boostrap-remote /usr/bin/boostrap-remote > /usr/share/boostrap-vz/boostrap-server /usr/bin/boostrap-server > > in debian/boostrap-vz.links > > see PYBUILD_INSTALL_ARGS above (and use PYBUILD_DESTDIR=debian/bootstrap-vz/ > instead of PYBUILD_NAME) Ok. As this seems to be considered very wrong, I've separated the package[1], between "bootstrap-vz" and "python-bootstrap-vz". The first one contains binaries/man pages/etc. and the later contains the library with everything packaged by Pybuild. I've tried to use "/usr/share/boostrap-vz" for it, but as it run as root it ends up writing "*.pyc" files in there, resulting in a non-clean package removal. I guess the split between application and library package is a better form of organization. Thanks for your help and your awesome work in Pybuild! Regards, Tiago. [1]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=1c075a7c300fa541abe2ccdcf8c5ab35128f7a76 -- Tiago "Myhro" Ilieve Blog: https://blog.myhro.info/ GitHub: https://github.com/myhro LinkedIn: https://br.linkedin.com/in/myhro Montes Claros - MG, Brasil
Re: Properly splitting Python "-doc" packages
[Tiago Ilieve, 2016-04-14]
> > then PLEASE PLEASE PLEASE do not install as public modules - install into
> > /usr/share/boostrap-vz/ f.e. with this pybuild args:
> >
> > export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/boostrap-vz/
> > --install-scripts=/usr/share/boostrap-vz/
> >
> > and
> >
> > /usr/share/boostrap-vz/boostrap-vz /usr/bin/boostrap-vz
> > /usr/share/boostrap-vz/boostrap-remote /usr/bin/boostrap-remote
> > /usr/share/boostrap-vz/boostrap-server /usr/bin/boostrap-server
> >
> > in debian/boostrap-vz.links
> >
> > see PYBUILD_INSTALL_ARGS above (and use PYBUILD_DESTDIR=debian/bootstrap-vz/
> > instead of PYBUILD_NAME)
>
> Ok. As this seems to be considered very wrong, I've separated the
> package[1], between "bootstrap-vz" and "python-bootstrap-vz". The
if you decide to go this way, please use python-bootstrapvz, not
python-bootstrap-vz (module name is bootstrapvz, not bootstrap-vz)
> first one contains binaries/man pages/etc. and the later contains the
> library with everything packaged by Pybuild.
>
> I've tried to use "/usr/share/boostrap-vz" for it, but as it run as
> root it ends up writing "*.pyc" files in there, resulting in a
I copy-pasted your typo in package name so dh_python2 didn't find the
right directory and didn't do its job.
See attached patch (now it uses private dir)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
diff --git a/debian/bootstrap-vz.links b/debian/bootstrap-vz.links
index 06894e2..fb6b9c0 100644
--- a/debian/bootstrap-vz.links
+++ b/debian/bootstrap-vz.links
@@ -1 +1,4 @@
usr/share/man/man1/bootstrap-vz-remote.1.gz usr/share/man/man1/bootstrap-vz-server.1.gz
+/usr/share/bootstrap-vz/bootstrap-vz /usr/bin/bootstrap-vz
+/usr/share/bootstrap-vz/bootstrap-vz-remote /usr/bin/bootstrap-vz-remote
+/usr/share/bootstrap-vz/bootstrap-vz-server /usr/bin/bootstrap-vz-server
diff --git a/debian/control b/debian/control
index 19a5392..cdcdd83 100644
--- a/debian/control
+++ b/debian/control
@@ -15,25 +15,10 @@ Homepage: https://github.com/andsens/bootstrap-vz
Vcs-Git: https://anonscm.debian.org/git/cloud/bootstrap-vz.git
Vcs-Browser: https://anonscm.debian.org/git/cloud/bootstrap-vz.git
-Package: python-bootstrap-vz
-Architecture: all
-Depends: ${misc:Depends},
- ${python:Depends}
-Breaks: bootstrap-vz (<< 0.9.10)
-Replaces: bootstrap-vz (<< 0.9.10)
-Description: Tool for creating Debian images for cloud platforms (Library)
- It's a software for creating Debian images to run on public or private clouds
- (Amazon, Google, OpenStack, Eucaliptus, etc).
- It's using bootstraping as a method for creating images. Images created by this
- tool are fully operational Debian Images for Cloud.
- .
- This package contains the Python library.
-
Package: bootstrap-vz
Section: admin
Architecture: all
Depends: debootstrap,
- python-bootstrap-vz (= ${binary:Version}),
${misc:Depends},
${python:Depends}
Recommends: qemu-utils
@@ -44,20 +29,18 @@ Suggests: bootstrap-vz-doc,
virtualbox-guest-additions-iso,
wget,
zerofree
-Description: Tool for creating Debian images for cloud platforms (CLI)
+Description: tool for creating Debian images for cloud platforms (CLI)
It's a software for creating Debian images to run on public or private clouds
(Amazon, Google, OpenStack, Eucaliptus, etc).
It's using bootstraping as a method for creating images. Images created by this
tool are fully operational Debian Images for Cloud.
- .
- This package contains the command-line binaries.
Package: bootstrap-vz-doc
Section: doc
Architecture: all
Depends: ${misc:Depends},
${sphinxdoc:Depends}
-Description: Tool for creating Debian images for cloud platforms (Docs)
+Description: tool for creating Debian images for cloud platforms (Docs)
It's a software for creating Debian images to run on public or private clouds
(Amazon, Google, OpenStack, Eucaliptus, etc).
It's using bootstraping as a method for creating images. Images created by this
diff --git a/debian/rules b/debian/rules
index 4d59a15..37d6146 100755
--- a/debian/rules
+++ b/debian/rules
@@ -1,7 +1,8 @@
#!/usr/bin/make -f
#export DH_VERBOSE=1
-export PYBUILD_NAME=bootstrap-vz
+export PYBUILD_DESTDIR=debian/bootstrap-vz
+export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/bootstrap-vz/ --install-scripts=/usr/share/bootstrap-vz/
%:
dh $@ --with python2,sphinxdoc --buildsystem=pybuild
@@ -27,11 +28,5 @@ override_dh_compress:
dh_compress --exclude "bootstrap-vz-doc/html/"
-override_dh_install:
- dh_install
- # Remove entry point scripts from the library package
- rm debian/python-bootstrap-vz/usr/bin/*
-
-
override_dh_installchangelogs:
dh_installchangelogs CHANGELOG.rst
signature.asc
Description: PGP signature
Re: Properly splitting Python "-doc" packages
Tiago Ilieve writes: > Ok. As this seems to be considered very wrong, I've separated the > package[1], between "bootstrap-vz" and "python-bootstrap-vz". The > first one contains binaries/man pages/etc. and the later contains the > library with everything packaged by Pybuild. Is there really a need for this separation? If the Python modules are installed to an application-private directory, then by definition they will not be publicly importable. So the Python libraries don't make much sense as a separately installable package. Why not simply have the application package also contain the private Python libraries it needs? For an example of how I've done this, see my ongoing refactor of ‘dput’ https://notabug.org/bignose/dput/src/packaging-upgrade/debian>. The pacakging uses Pybuild to manage the Python libraries and installs them to an application-private location. There is no separate Python library package, because ‘dput’ is the only binary package that would use them and it just installs them itself. -- \“A right is not what someone gives you; it's what no one can | `\ take from you.” —Ramsey Clark | _o__) | Ben Finney
Re: Properly splitting Python "-doc" packages
Piotr, On 14 April 2016 at 18:28, Piotr Ożarowski wrote: > if you decide to go this way, please use python-bootstrapvz, not > python-bootstrap-vz (module name is bootstrapvz, not bootstrap-vz) > > I copy-pasted your typo in package name so dh_python2 didn't find the > right directory and didn't do its job. > > See attached patch (now it uses private dir) I've reverted[1] the package split and make a few changes based on your patch. Nice to see that it wasn't working because of a typo and not anything more serious. I also like the idea of not hardcoding a Python version (even a Python 2 one) in some "install" file path, as now could be changed from "usr/lib/python2.7/dist-packages/" to "usr/share/". Thanks again for the help and the patch. - Ben, On 15 April 2016 at 00:57, Ben Finney wrote: > Is there really a need for this separation? If the Python modules are > installed to an application-private directory, then by definition they > will not be publicly importable. So the Python libraries don't make much > sense as a separately installable package. Mostly because of the problem that I faced earlier, where the package hadn't worked as I expected because of the "*.pyc" file. Turns out that this was caused by a typo spotted by Piotr. The package is now being properly in a private application directory. Regards, Tiago. [1]: https://anonscm.debian.org/git/cloud/bootstrap-vz.git/commit/?id=c25f5cc8dd70456fb25e87f68419c553059e -- Tiago "Myhro" Ilieve Blog: https://blog.myhro.info/ GitHub: https://github.com/myhro LinkedIn: https://br.linkedin.com/in/myhro Montes Claros - MG, Brasil
