Re: backport of dpkg (>= 1.17.2) and apt (>= 0.9.16.1) for build profiles
Hi, Quoting Philipp Kern (2014-07-25 14:45:17) > On Fri, Jul 25, 2014 at 02:19:38PM +0200, Johannes Schauer wrote: > > Maybe. We'd be equally (if not more) happy if SRM would reconsider their > > decision (expressed on #debian-release toward Helmut Grohne) that these > > patches > > are too intrusive for a stable update. > > There are no decisions expressed on IRC, at most tendencies. Get them on the > right list (debian-release@lists.d.o) and get them reviewed there. (Which > never > happened to far, aside Guillem's general question avoid feasibility, but > without a patchset to consider.) following this advice, let us propose the attached three minimal patches to dpkg, apt and python-apt for a stable updated. Fixing bug#744246 would only require apt and python-apt but maybe having dpkg not choke on it as well would also be desirable. The patches do not implement full profile support but just allow the new Build-Depends syntax to be parsed without throwing an error. If the new syntax is encountered, dependencies will be added as if no build profile was active. The patches include no way to change the activated build profiles to keep them as minimal as possible. Do you think the attached patches are minimal/safe enough to include them as a stable update? If these patches were added to stable, then https://bugs.debian.org/744246 could be resolved before jessie release. This in turn would mean that we can test build profiles before the jessie release. And this means that we can fix not only bootstrapping bugs in packages before the release but also possibly missing build profile related bugs in dpkg, apt, other build tools or even the buildd machinery. That way, possible bugs could be weeded out before the jessie release. This would also benefit our Google Summer of Code student for the "Bootstrappable Debian" project whose patches could then immediately be applied by package maintainers instead of his work only bearing fruit after the jessie release. Sorry at the deity and dpkg lists for posting those patches once again but small changes were required to the debian/changelog versions, the Build-Depends version of python-apt, the symbol version in apt and the newer version of dpkg in stable required some refreshing of the patch offsets. Thank you for your consideration. cheers, josch diff -Nru dpkg-1.16.15/debian/changelog dpkg-1.16.15+deb7u1/debian/changelog --- dpkg-1.16.15/debian/changelog 2014-06-05 20:36:49.0 + +++ dpkg-1.16.15+deb7u1/debian/changelog 2014-07-28 11:18:23.0 + @@ -1,3 +1,19 @@ +dpkg (1.16.15+deb7u1) stable; urgency=low + + * Non-maintainer upload. + * Minimal support for restriction list Build-Depends syntax extension + - the syntax is described at https://wiki.debian.org/BuildProfileSpec + - dpkg will correctly parse and interpret build profile annotations +in build dependencies of source packages + - it will not allow to set any build profiles and will always assume +that no build profile is set + - this allows dpkg to not throw an error when encountering source packages +that make use of build profiles in stable versions of Debian with dpkg +version 1.17.2 or later in which full support for build profiles +was implemented + + -- Johannes Schauer Mon, 28 Jul 2014 10:47:00 + + dpkg (1.16.15) wheezy-security; urgency=high [ Guillem Jover ] diff -Nru dpkg-1.16.15/man/deb-src-control.5 dpkg-1.16.15+deb7u1/man/deb-src-control.5 --- dpkg-1.16.15/man/deb-src-control.5 2014-06-05 20:01:05.0 + +++ dpkg-1.16.15+deb7u1/man/deb-src-control.5 2014-07-28 10:30:33.0 + @@ -177,8 +177,9 @@ of packages separated by vertical bar (or "pipe") symbols, "|". The groups are separated by commas. Commas are to be read as "AND", and pipes as "OR", with pipes binding more tightly. Each package name is -optionally followed by a version number specification in parentheses and an -architecture specification in square brackets. +optionally followed by a version number specification in parentheses, an +architecture specification in square brackets, and a profile specification +in angle brackets. The syntax of the .BR Build\-Conflicts , @@ -188,7 +189,8 @@ fields is a list of comma-separated package names, where the comma is read as an "AND". Specifying alternative packages using a "pipe" is not supported. Each package name is optionally followed by a version number specification in -parentheses and an architecture specification in square brackets. +parentheses, an architecture specification in square brackets, and a profile +specification in angle brackets. A version number may start with a ">>", in which case any later version will match, and may specify or omit the Debian packaging revision (separated @@ -200,6 +202,10 @@ separated by whitespace. Exclamation marks may be prepended to each of the names, meaning "NOT". +A profile specification consists of one or more pro
Re: backport of dpkg (>= 1.17.2) and apt (>= 0.9.16.1) for build profiles
I personally don't think it's a good idea to get such patches into
stable. Anyway if someone wants to proceed anyway, here are a few
things I spotted:
Johannes Schauer (2014-07-28):
> apt (0.9.7.9+deb7u2) wheezy-security; urgency=high
>
>* SECURITY UPDATE: apt-get source validation (closes: #749795)
> diff -Nru apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols
> apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols
> --- apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols 2013-03-01
> 10:51:21.0 +
> +++ apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols 2014-07-28
> 11:32:23.0 +
> @@ -369,6 +369,7 @@
> (c++)"debListParser::VersionHash()@Base" 0.8.0
> (c++)"debListParser::Architecture()@Base" 0.8.0
> (c++)"debListParser::ParseDepends(char const*, char const*,
> std::basic_string, std::allocator >&,
> std::basic_string, std::allocator >&,
> unsigned int&, bool const&, bool const&)@Base" 0.8.0
> + (c++)"debListParser::ParseDepends(char const*, char const*,
> std::basic_string, std::allocator >&,
> std::basic_string, std::allocator >&,
> unsigned int&, bool const&, bool const&, bool const&)@Base" 0.9.7.9+deb7u2
This is wrong.
> (c++)"debListParser::ParseDepends(pkgCache::VerIterator&, char const*,
> unsigned int)@Base" 0.8.0
> (c++)"debListParser::ParseProvides(pkgCache::VerIterator&)@Base" 0.8.0
> (c++)"debListParser::ArchitectureAll()@Base" 0.8.0
> diff -Nru python-apt-0.8.8.2/debian/changelog
> python-apt-0.8.8.2+deb7u1/debian/changelog
> --- python-apt-0.8.8.2/debian/changelog 2013-03-14 20:25:26.0
> +
> +++ python-apt-0.8.8.2+deb7u1/debian/changelog2014-07-28
> 11:46:26.0 +
> @@ -1,3 +1,19 @@
> +python-apt (0.8.8.2+deb7u1) stable; urgency=low
> +
> + * Non-maintainer upload.
> + * Minimal support for restriction list Build-Depends syntax extension
> + - the syntax is described at https://wiki.debian.org/BuildProfileSpec
> + - apt will correctly parse and interpret build profile annotations
> +in build dependencies of source packages
> + - it will not allow to set any build profiles and will always assume
> +that no build profile is set
> + - this allows apt to not throw an error when encountering source
> packages
> +that make use of build profiles in stable versions of Debian with apt
> +version 0.9.16.1 or later in which full support for build profiles
> +was implemented
> +
> + -- Johannes Schauer Mon, 28 Jul 2014 11:45:53 +
> +
> python-apt (0.8.8.2) unstable; urgency=low
>
>[ David Prévot ]
> diff -Nru python-apt-0.8.8.2/debian/control
> python-apt-0.8.8.2+deb7u1/debian/control
> --- python-apt-0.8.8.2/debian/control 2013-03-13 22:25:59.0 +
> +++ python-apt-0.8.8.2+deb7u1/debian/control 2014-07-28 11:46:59.0
> +
> @@ -10,7 +10,7 @@
> apt-utils,
> debhelper (>= 7.3.5),
> fakeroot,
> - libapt-pkg-dev (>= 0.8.11),
> + libapt-pkg-dev (= 0.9.7.9+deb7u3),
I'm pretty sure this a bad idea.
This happened not so long ago:
[12 Jun 2014] DSA-2958 apt - security update
Next apt update would mean python-apt is no longer buildable in stable.
> python-all-dev (>= 2.6.6-3~),
> python-all-dbg,
> python3-all-dev (>= 3.1.2-10~),
> diff -Nru python-apt-0.8.8.2/python/apt_pkgmodule.cc
> python-apt-0.8.8.2+deb7u1/python/apt_pkgmodule.cc
> --- python-apt-0.8.8.2/python/apt_pkgmodule.cc2013-03-13
> 22:31:33.0 +
> +++ python-apt-0.8.8.2+deb7u1/python/apt_pkgmodule.cc 2014-07-28
> 11:45:11.0 +
> @@ -186,7 +186,7 @@
> "only contains those dependencies for the architecture set in the\n"
> "configuration variable APT::Architecture";
> static PyObject *RealParseDepends(PyObject *Self,PyObject *Args,
> - bool ParseArchFlags, std::string name,
> + bool ParseArchFlags, bool
> ParseRestrictionsList, std::string name,
>bool debStyle=false)
> {
> std::string Package;
> @@ -210,7 +210,7 @@
>break;
>
>Start = debListParser::ParseDepends(Start,Stop,Package,Version,Op,
> - ParseArchFlags, StripMultiArch);
> + ParseArchFlags, StripMultiArch,
> ParseRestrictionsList);
>if (Start == 0)
>{
>PyErr_SetString(PyExc_ValueError,"Problem Parsing Dependency");
> @@ -243,11 +243,11 @@
> }
> static PyObject *ParseDepends(PyObject *Self,PyObject *Args)
> {
> - return RealParseDepends(Self, Args, false, "parse_depends");
> + return RealParseDepends(Self, Args, false, false, "parse_depends");
> }
> static PyObject *ParseSrcDepends(PyObject *Self,PyObject *Args)
> {
> - return RealParseDepends(Self, Args, true, "parse_src_depends");
> + return RealParseDepends(Self,
Re: backport of dpkg (>= 1.17.2) and apt (>= 0.9.16.1) for build profiles
Hi, Quoting Cyril Brulebois (2014-07-28 16:40:49) > > diff -Nru apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols > > apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols > > --- apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols 2013-03-01 > > 10:51:21.0 + > > +++ apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols 2014-07-28 > > 11:32:23.0 + > > @@ -369,6 +369,7 @@ > > (c++)"debListParser::VersionHash()@Base" 0.8.0 > > (c++)"debListParser::Architecture()@Base" 0.8.0 > > (c++)"debListParser::ParseDepends(char const*, char const*, > > std::basic_string, std::allocator >&, > > std::basic_string, std::allocator >&, > > unsigned int&, bool const&, bool const&)@Base" 0.8.0 > > + (c++)"debListParser::ParseDepends(char const*, char const*, > > std::basic_string, std::allocator >&, > > std::basic_string, std::allocator >&, > > unsigned int&, bool const&, bool const&, bool const&)@Base" 0.9.7.9+deb7u2 > > This is wrong. Why? And how would it be done "right"? > > diff -Nru python-apt-0.8.8.2/debian/control > > python-apt-0.8.8.2+deb7u1/debian/control > > --- python-apt-0.8.8.2/debian/control 2013-03-13 22:25:59.0 + > > +++ python-apt-0.8.8.2+deb7u1/debian/control 2014-07-28 11:46:59.0 > > + > > @@ -10,7 +10,7 @@ > > apt-utils, > > debhelper (>= 7.3.5), > > fakeroot, > > - libapt-pkg-dev (>= 0.8.11), > > + libapt-pkg-dev (= 0.9.7.9+deb7u3), > > I'm pretty sure this a bad idea. > > This happened not so long ago: > [12 Jun 2014] DSA-2958 apt - security update > > Next apt update would mean python-apt is no longer buildable in stable. This is correct. I am not aware of a correct way to express this dependency. > I know nothing about python bindings but anyway, it looks like you're > not updating doc strings. I can easily update all the necessary doc strings. But it seems that more fundamental questions should be solved first. > More importantly, what's the impact in packages using those functions? Were > packages identified, their code reviewed, and their features tested? I might lack the necessary understanding but given the changes expressed in the patches I cannot imagine in what way packages depending on apt or python-apt would start failing or having any different functionality besides not failing when faced with the new syntax. For python-apt, none of the exposed python functions gained new arguments and thus no code should break. The only difference now is, that it will be able to understand the new syntax. For libapt-pkg4.12, existing code will be calling ParseDepends in a way which sets ParseRestrictionsList to false and thus should not experience any functionality change at all. To be able to understand the new syntax as well, they'd have to explicitly call ParseDepends with ParseRestrictionsList=true. But this is no change in functionality from the status quo. If you want us to do any additional tests, I'll be glad to carry them out. cheers, josch -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140728161353.4150.65436@hoothoot
Re: backport of dpkg (>= 1.17.2) and apt (>= 0.9.16.1) for build profiles
Johannes Schauer (2014-07-28): > Quoting Cyril Brulebois (2014-07-28 16:40:49) > > > diff -Nru apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols > > > apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols > > > --- apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols 2013-03-01 > > > 10:51:21.0 + > > > +++ apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols 2014-07-28 > > > 11:32:23.0 + > > > @@ -369,6 +369,7 @@ > > > (c++)"debListParser::VersionHash()@Base" 0.8.0 > > > (c++)"debListParser::Architecture()@Base" 0.8.0 > > > (c++)"debListParser::ParseDepends(char const*, char const*, > > > std::basic_string, std::allocator >&, > > > std::basic_string, std::allocator >&, > > > unsigned int&, bool const&, bool const&)@Base" 0.8.0 > > > + (c++)"debListParser::ParseDepends(char const*, char const*, > > > std::basic_string, std::allocator >&, > > > std::basic_string, std::allocator >&, > > > unsigned int&, bool const&, bool const&, bool const&)@Base" 0.9.7.9+deb7u2 > > > > This is wrong. > > Why? > > And how would it be done "right"? Pretty sure 0.9.7.9+deb7u2 doesn't ship the symbol you pretend it does… (Ansgar told you where to look, by the way.) > > > diff -Nru python-apt-0.8.8.2/debian/control > > > python-apt-0.8.8.2+deb7u1/debian/control > > > --- python-apt-0.8.8.2/debian/control 2013-03-13 22:25:59.0 + > > > +++ python-apt-0.8.8.2+deb7u1/debian/control 2014-07-28 > > > 11:46:59.0 + > > > @@ -10,7 +10,7 @@ > > > apt-utils, > > > debhelper (>= 7.3.5), > > > fakeroot, > > > - libapt-pkg-dev (>= 0.8.11), > > > + libapt-pkg-dev (= 0.9.7.9+deb7u3), > > > > I'm pretty sure this a bad idea. > > > > This happened not so long ago: > > [12 Jun 2014] DSA-2958 apt - security update > > > > Next apt update would mean python-apt is no longer buildable in stable. > > This is correct. I am not aware of a correct way to express this dependency. Well, as usual, >= foo together with << bar? > > I know nothing about python bindings but anyway, it looks like you're > > not updating doc strings. > > I can easily update all the necessary doc strings. But it seems that more > fundamental questions should be solved first. Right. > > More importantly, what's the impact in packages using those functions? Were > > packages identified, their code reviewed, and their features tested? > > I might lack the necessary understanding but given the changes expressed in > the > patches I cannot imagine in what way packages depending on apt or python-apt > would start failing or having any different functionality besides not failing > when faced with the new syntax. > > For python-apt, none of the exposed python functions gained new arguments and > thus no code should break. The only difference now is, that it will be able to > understand the new syntax. > > For libapt-pkg4.12, existing code will be calling ParseDepends in a way which > sets ParseRestrictionsList to false and thus should not experience any > functionality change at all. To be able to understand the new syntax as well, > they'd have to explicitly call ParseDepends with ParseRestrictionsList=true. > But this is no change in functionality from the status quo. > > If you want us to do any additional tests, I'll be glad to carry them out. I read this as “no tests have been performed”. Not quite what I'd expect for things as critical as a new apt in stable… Mraw, KiBi. signature.asc Description: Digital signature
Re: backport of dpkg (>= 1.17.2) and apt (>= 0.9.16.1) for build profiles
Hi, Quoting Cyril Brulebois (2014-07-28 18:38:38) > Johannes Schauer (2014-07-28): > > Quoting Cyril Brulebois (2014-07-28 16:40:49) > > > > diff -Nru apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols > > > > apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols > > > > --- apt-0.9.7.9+deb7u2/debian/libapt-pkg4.12.symbols 2013-03-01 > > > > 10:51:21.0 + > > > > +++ apt-0.9.7.9+deb7u3/debian/libapt-pkg4.12.symbols 2014-07-28 > > > > 11:32:23.0 + > > > > @@ -369,6 +369,7 @@ > > > > (c++)"debListParser::VersionHash()@Base" 0.8.0 > > > > (c++)"debListParser::Architecture()@Base" 0.8.0 > > > > (c++)"debListParser::ParseDepends(char const*, char const*, > > > > std::basic_string, std::allocator > > > > >&, std::basic_string, > > > > std::allocator >&, unsigned int&, bool const&, bool const&)@Base" > > > > 0.8.0 > > > > + (c++)"debListParser::ParseDepends(char const*, char const*, > > > > std::basic_string, std::allocator > > > > >&, std::basic_string, > > > > std::allocator >&, unsigned int&, bool const&, bool const&, bool > > > > const&)@Base" 0.9.7.9+deb7u2 > > > > > > This is wrong. > > > > Why? > > > > And how would it be done "right"? > > Pretty sure 0.9.7.9+deb7u2 doesn't ship the symbol you pretend it does… you are right, I wrongly updated the version when I rebased the patch from the one prepared for backports to the current stable version of apt. But that is trivially fixed. Is there anything else wrong with that line? > (Ansgar told you where to look, by the way.) Which message of Ansgar are you referring to? > > > > diff -Nru python-apt-0.8.8.2/debian/control > > > > python-apt-0.8.8.2+deb7u1/debian/control > > > > --- python-apt-0.8.8.2/debian/control 2013-03-13 22:25:59.0 > > > > + > > > > +++ python-apt-0.8.8.2+deb7u1/debian/control 2014-07-28 > > > > 11:46:59.0 + > > > > @@ -10,7 +10,7 @@ > > > > apt-utils, > > > > debhelper (>= 7.3.5), > > > > fakeroot, > > > > - libapt-pkg-dev (>= 0.8.11), > > > > + libapt-pkg-dev (= 0.9.7.9+deb7u3), > > > > > > I'm pretty sure this a bad idea. > > > > > > This happened not so long ago: > > > [12 Jun 2014] DSA-2958 apt - security update > > > > > > Next apt update would mean python-apt is no longer buildable in stable. > > > > This is correct. I am not aware of a correct way to express this dependency. > > Well, as usual, >= foo together with << bar? My problem with that solution is: << than what version? Also, apt starts shipping the proper symbol with version 0.9.16.1 and the only way to retrieve a version before 0.9.16.1 is from snapshot.debian.org. So is a << even necessary? If it is, then less than what version? > I read this as “no tests have been performed”. Not quite what I'd expect for > things as critical as a new apt in stable… We are quite confident that apt, python-apt and dpkg itself work as expected as those were thoroughly tested. As for their reverse dependencies: Of the reverse dependencies of libapt-pkg4.12 only qapt uses the ParseDepends function. But it doesnt do so for parsing source control data so there will be no behaviour change. Of the reverse dependencies of python-apt, ParseSrcDepends is used by none. The rest use ParseDepends which only acts on binary packages and thus does not have a change in functionality. Of the reverse dependencies of libdpkg-perl only libsbuild-perl and xapt make use of the changed deps_parse function. Because they do not pass the reduce_profiles argument they will not be able to parse the new syntax. We did test sbuild but we did not test xapt. What other tests would you like me to run? cheers, josch -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20140728181959.4150.26090@hoothoot
Debian binary diff
Hi all, In my work we are using debian obviously... But the most important part is that is an embedded system very particular. We are using ARM architecture and the constraint harder that we have is the bandwidth. It's really important to us reduce the amount of data when we update our system. So I made an ugly and nasty hack that allow from two debian packages build a third one with only the changed files. The unchanged files are moved in the preinst script to a temporal folder and are move back to the original path in the postinst script. I know, you must be thinking WTF? Such a mess... Well, yes probably it is. Because of that I'm here telling the problem maybe there is another approach to do this. Summarizing, I need to update my system only with the changed files using a debian package and try to not break the debian database and the filesystem neither. If you are interested on keep reading... The current procedure is, build the current package from the filesystem using dpkg-repack (just in case if something goes wrong, great tool BTW tx joeyh) and after that, install the package built with the differences of two debian packages (two specific versions). This method is not robust at all because I need to move files and if in the middle of the process the system loses power (the energy is not reliable in the system, so probably it gonna happen from time to time) the filesystem can be broken. I've downloaded the dpkg source and I'll try to hack it a little bit to "support the .debdiff format". But first I want to know if someone has a better idea or can point me on a better direction. Cheers, -- TiN -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/53d6cc2f.3050...@sluc.org.ar
Re: Debian binary diff
Hi, Agustin Henze (2014-07-28): > Hi all, >In my work we are using debian obviously... But the most > important part is that is an embedded system very particular. We are > using ARM architecture and the constraint harder that we have is the > bandwidth. It's really important to us reduce the amount of data when > we update our system. I think you want to look at debdelta, which might be the right tool for you. Mraw, KiBi. signature.asc Description: Digital signature
Re: Debian binary diff
On 07/28/2014 07:34 PM, Cyril Brulebois wrote: > I think you want to look at debdelta, which might be the right tool for > you. Great! I'll take a look, seems to be the solution to my problem :) Thanks, -- TiN signature.asc Description: OpenPGP digital signature
