Re: Bug#571776: document symbols

2012-01-13 Thread Russ Allbery 
Raphael Hertzog  writes:
> On Mon, 02 Jan 2012, Russ Allbery wrote:

> [...]
>>  
>>shlibs files were the original mechanism for
>>handling library dependencies.  They are documented
>>in .  symbols files,
>>documented in this section, are recommended for most packages,
>>since they provide dependency information for each exported
>>symbol and therefore generate more accurate dependencies for
>>binaries that do not use symbols from newer versions of the
>>shared library.  However, shlibs files must be used
>>for udebs.  Packages which provide a symbols file
>>are not required to provide a shlibs file.
>>  

> In practice I don't think that we have any package providing a symbols
> files without a shlibs file.

Yes, but there was some discussion in the Policy bug asking why shlibs
files were required when they're not used if a symbols file is present,
and while I originally argued that keeping them both made sense, I came
around to that position after reviewing the bug discussion.  It doesn't
hurt anything, but there doesn't really seem to be any point in providing
shlibs if symbols is present.

> 
>>   If your source package builds only a
>>  single binary package that contains only compiled binaries and
>>  libraries (but no scripts) and is not multiarch, you can use a
>>  command such as:
>>  
>> dpkg-shlibdeps debian/tmp/usr/bin/* debian/tmp/usr/sbin/* \
>>   debian/tmp/usr/lib/*
>>  
>>  but normally finding all of the binaries is more
>>  complex.

> I would drop the part above (up to the 4 dashes that I added). This
> example will only mislead people IMO because debian/tmp/ is far from
> being the norm. debian/ is much more common for single binary
> packages. And as you said, it doesn't cover all cases.

Yeah, good point.  This was retained from the old shlibs documentation,
but at this point I don't think anyone really looks here to see how to do
this.  Dropped.

>>  binary package.
>>Again, dh_shlibdeps
>>and dh_gencontrol will handle all of this for
>>you if you're using debhelper.
>>  

> I would indicate in the footnote that dh_shlibdeps/dh_gencontrol use an
> alternate substvars file by default (debian/.substvars).

I now have:

binary package.
  Again, dh_shlibdeps
  and dh_gencontrol will handle all of this for
  you if you're using debhelper, including generating
  separate substvars files for each binary
  package and calling dpkg-gencontrol with the
  appropriate flags.


>>  
>>The symbols File Format
> [...]
>>  For our example, the zlib1g symbols file
>>  would contain:
>>  
>>  * Build-Depends-Package: zlib1g-dev
>>  
>>  (Don't forget the leading space.)

> What leading space are you referring to ?

I now have:

(Don't forget the space before the * so that it will
be parsed as part of the entry for that library.)

Due to the way that the formatting of Policy works, it's very hard to tell
that there's a space there, and unlike with symbols where the indentation
is fairly obvious, it's not completely obvious that it's required.

> [...]
>>  
>>The shlibs files present on the
>>  system
> [...]
>>
>>  DEBIAN/shlibs files in the "build
>>directory"
>> 
>>  
>>When packages are being built,
>>any debian/shlibs files are copied into the
>>control information file area of the temporary build
>>directory and given the name shlibs.  These
>>files give details of any shared libraries included in
>>the same package.
>>  
>>

> debian/shlibs is (no longer) a standard file... debhelper doesn't install
> such files, it generates shlibs files on the fly and I don't believe that
> any modern package still has debian/shlibs.

> So I would rewrite this paragraph to just say that DEBIAN/shlibs is the
> shlibs file produced by the current package build.

I now have:


  These files are generated as part of the package build
  process and staged for inclusion as control files in the
  binary packages being built.  They provide details of
  any shared libraries included in the same package.


> I would shorten the explanation here and drop the example of how to
> install a manually crafted shlibs file. This is far from being the norm
> and should not be difficult to find out alone if one really needs it.

I've replaced this section with just:


  Providing a shlibs file

  
To provide a shlibs file for a shared library

Re: Bug#571776: document symbols

2012-01-13 Thread Russ Allbery 
Russ Allbery  writes:

> I tried sending a unified diff, but the new sections are largely
> unreadable since they're intermixed with the old sections being removed.
> Hence, for review purposes, here are the symbols and shlibs sections in
> their entirety, followed by a diff for the changes elsewhere in Policy.
> You can also refer to branch bug571776-rra in the Policy repository.

Here is an updated diff with the changes from Raphael's review.  git diff
--patience actually generates a decent diff (I learn something new every
day!), so I didn't have to separate out the new sections.

diff --git a/policy.sgml b/policy.sgml
index 23c1913..b57dd76 100644
--- a/policy.sgml
+++ b/policy.sgml
@@ -840,10 +840,11 @@
Among those files are the package maintainer scripts
and control, the binary
package control file that contains the control fields for
-   the package.  Other control information files
-   include the shlibs
-   file used to store shared library dependency information
-   and the conffiles file that lists the package's
+   the package.  Other control information files include
+   the symbols file
+   or shlibs file
+   used to store shared library dependency information and
+   the conffiles file that lists the package's
configuration files (described in ).
   
 
@@ -5521,9 +5522,9 @@ Replaces: mail-transport-agent
  linked against the old shared library.  Correct versioning of
  dependencies on the newer shared library by binaries that use
  the new interfaces is handled via
- the shlibs
- system or via symbols files (see
- ).
+ the symbols system
+ or the shlibs
+ system.

 
   
@@ -5792,361 +5793,789 @@ Replaces: mail-transport-agent

   
 
+  
+   Dependencies between the library and other packages -
+   the symbols system
+
+   
+ If a package contains a binary or library which links to a
+ shared library, we must ensure that, when the package is
+ installed on the system, all of the libraries needed are also
+ installed.  These dependencies must be added to the binary
+ package when it is built, since they may change based on which
+ version of a shared library the binary or library was linked
+ with.  To allow these dependencies to be constructed, shared
+ libraries must provide either a symbols file or
+ a shlibs file, which provide information on the
+ package dependencies required to ensure the presence of this
+ library.  Any package which uses a shared library must use these
+ files to determine the required dependencies when it is built.
+   
+
+   
+ shlibs files were the original mechanism for
+ handling library dependencies.  They are documented
+ in .  symbols files,
+ documented in this section, are recommended for most packages,
+ since they provide dependency information for each exported
+ symbol and therefore generate more accurate dependencies for
+ binaries that do not use symbols from newer versions of the
+ shared library.  However, shlibs files must be used
+ for udebs.  Packages which provide a symbols file
+ are not required to provide a shlibs file.
+   
+
+   
+ When a package that contains any shared libraries or compiled
+ binaries is built, it must run dpkg-shlibdeps on
+ each shared library and compiled binary to determine the
+ libraries used and hence the dependencies needed by the
+ package.
+   dpkg-shlibdeps will use a program
+   like objdump or readelf to find the
+   libraries and the symbols in those libraries directly needed
+   by the binaries or shared libraries in the package.
+ 
+   
+
+   
+ We say that a binary foo directly uses a
+ library libbar if it is explicitly linked with that
+ library (that is, the library is listed in the
+ ELF NEEDED attribute, caused by adding -lbar
+ to the link line when the binary is created).  Other libraries
+ that are needed by libbar are
+ linked indirectly to foo, and the dynamic
+ linker will load them automatically when it
+ loads libbar.  A package should depend on the libraries
+ it directly uses, but not the libraries it indirectly uses.  The
+ dependencies for those libraries will automatically pull in the
+ other libraries.  dpkg-shlibdeps will handle this
+ logic automatically, but package maintainers need to be aware of
+ this distinction between directly and indirectly using a library
+ if they have to override its results for some reason.
+ 
+   A good example of where this helps is the following.  We could
+

Re: Bug#571776: document symbols

2012-01-13 Thread Raphael Hertzog 
On Fri, 13 Jan 2012, Russ Allbery wrote:
> >>For our example, the zlib1g symbols file
> >>would contain:
> >>
> >>  * Build-Depends-Package: zlib1g-dev
> >>
> >>(Don't forget the leading space.)
> 
> > What leading space are you referring to ?
> 
> I now have:
> 
>   (Don't forget the space before the * so that it will
>   be parsed as part of the entry for that library.)
> 
> Due to the way that the formatting of Policy works, it's very hard to tell
> that there's a space there, and unlike with symbols where the indentation
> is fairly obvious, it's not completely obvious that it's required.

There is no leading space before the "*". Just like "|" it must be
on the first column to differentiate with symbol definitions which do have
a leading space on their lines.

> Thanks for the review!

All the other corrections you made are fine. Thanks!

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120113193823.gd14...@rivendell.home.ouaza.com

Re: Bug#571776: document symbols

2012-01-13 Thread Raphael Hertzog 
On Fri, 13 Jan 2012, Russ Allbery wrote:
> +   
> + 
> +library-soname main-dependency-template
> +[ | alternative-dependency-template ]
> +[ ... ]
> +[ * field-name: field-value ]
> +[ ... ]
> + symbol minimal-version[ 
> id-of-dependency-template ]
> + 

I think this description adapted from the deb-symbols(5) manual page
mislead you into thinking that there were leading spaces before | or *
when in fact there are none.

I have updated the manual page to make it look like this now:

library-soname main-dependency-template
[| alternative-dependency-template]
[...]
[* field-name: field-value]
[...]
 symbol minimal-version [id-of-dependency-template]

> +   
> +libGL.so.1 libgl1
> + | libgl1-mesa-glx #MINVER#

Drop the leading space on that last line.

> + publicGlSymbol@Base 6.3-1
> + [...]
> + implementationSpecificSymbol@Base 6.5.2-7 1
> + [...]

> + For our example, the zlib1g symbols file
> + would contain:
> + 
> + * Build-Depends-Package: zlib1g-dev

And here too.

> + 
> + (Don't forget the space before the * so that it will
> + be parsed as part of the entry for that library.)

And that sentence is then useless (or needs to be reworded).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Pre-order a copy of the Debian Administrator's Handbook and help
liberate it: http://debian-handbook.info/liberation/


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120113195534.ge14...@rivendell.home.ouaza.com

Re: Bug#571776: document symbols

2012-01-13 Thread Russ Allbery 
Raphael Hertzog  writes:

> There is no leading space before the "*". Just like "|" it must be on
> the first column to differentiate with symbol definitions which do have
> a leading space on their lines.

Oh, then deb-symbols(5) is wrong for both * and |... oh, I see, I was
misreading how the syntax definition worked and the spaces around []
weren't literal.  Aie.

I'll fix.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87vcof8h7x@windlord.stanford.edu

Re: Bug#571776: document symbols

2012-01-13 Thread Russ Allbery 
Raphael Hertzog  writes:

> I think this description adapted from the deb-symbols(5) manual page
> mislead you into thinking that there were leading spaces before | or *
> when in fact there are none.

> I have updated the manual page to make it look like this now:

> library-soname main-dependency-template
> [| alternative-dependency-template]
> [...]
> [* field-name: field-value]
> [...]
>  symbol minimal-version [id-of-dependency-template]

Thanks, I've now fixed my draft text as well.

>> +  
>> +libGL.so.1 libgl1
>> + | libgl1-mesa-glx #MINVER#

> Drop the leading space on that last line.

Done.

>> +For our example, the zlib1g symbols file
>> +would contain:
>> +
>> + * Build-Depends-Package: zlib1g-dev

> And here too.

Done.

>> +
>> +(Don't forget the space before the * so that it will
>> +be parsed as part of the entry for that library.)

> And that sentence is then useless (or needs to be reworded).

Dropped.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehv35h9k@windlord.stanford.edu

Re: Bug#571776: document symbols

2012-01-13 Thread Charles Plessy 
Dear Russ and Raphaël,

here are some comments about the current patch.  I agree with the other changes
made subsequently in that thread.


> +   If a package contains a binary or library which links to a
> +   shared library, we must ensure that, when the package is
> +   installed on the system, all of the libraries needed are also
> +   installed.  These dependencies must be added to the binary
> +   package when it is built, since they may change based on which
> +   version of a shared library the binary or library was linked
> +   with.

I have a difficulty with that sentence : isn't the goal of the symbols system
to actually avoid that a dependancy changes when a package is built on a newer
version of a library ?  What rather changes is that the dependancy is modified
when a new version of the package makes new calls to the library, which may
have been implemented only in newer versions of the library.

If the misunderstanding is on my side, please consider it a hint that the
explanations can be expanded :)  [PS: after reading the whole patch, maybe
this has something to do with minimal-version ?]


> + 
> +   When a package that contains any shared libraries or compiled
> +   binaries is built, it must run dpkg-shlibdeps on
> +   each shared library and compiled binary to determine the
> +   libraries used and hence the dependencies needed by the
> +   package.
> + dpkg-shlibdeps will use a program
> + like objdump or readelf to find the
> + libraries and the symbols in those libraries directly needed
> + by the binaries or shared libraries in the package.
> +   
> + 

(I understand that this is carried over from 3.9.2, but…) Just running
dpkg-shlibdeps is not enough, as this program only populates debian/substvars.
This is actually documented later in this chapter, which makes this paragraph
redundant.  If its purpose is to lock the use of dpkg-shlibdeps and prevent
parallel implementations, it could be slimmed as follows.


  Packages that contain any shared libraries or compiled binaries must
  determine their dependencies by using the dpkg-shlibdeps
  program and the ${shlibs:Depends}
  substvar.  See  for detailed instructions.



> + 
> +   During the package build, if the package itself contains
> +   shared libraries with symbols files, they
> +   will be generated in these staging directories
> +   by dpkg-gensymbols.  symbols

With a linear reading, it is the first time dpkg-gensymbols appears in the
Policy.  How about linking to the "symbols" section, that gives a few more
words and refers to the manual page ? 


> +   
> + If your package contains any compiled binaries or shared
> + libraries, put a call to dpkg-shlibdeps into
> + your debian/rules file in the source package.

The footnote removed above could be added here:

  
dpkg-shlibdeps will use a program
like objdump or readelf to find the
libraries and the symbols in those libraries directly needed
by the binaries or shared libraries in the package.
  


> +   
> + This command puts the dependency information into
> + the debian/substvars file, which is then used
> + by dpkg-gencontrol.

I propose to mitigate this by adding “by default”, because when using
debhelper, which is very frequent, debian/substvars is actually not used.  (It
took me hours to understand why I could not populate new substvars in
debian/control by simply adding their values in debian/substvars in my
debhelper-managed packages).


> - 
> -   
> - dpkg-shlibdeps - calculates shared library
> - dependencies
> -   

I propose to keep a placeholder so that the next subsections are not
renumbered.  Once all these sections are trimmed, we could consider removing
the appendix altogether.


Have a nice day,

-- 
Charles Plessy
Tsurumi, Kanagawa, Japan


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20120114044940.gd9...@merveille.plessy.net

Re: Bug#571776: document symbols

2012-01-13 Thread Russ Allbery 
Charles Plessy  writes:

> here are some comments about the current patch.  I agree with the other
> changes made subsequently in that thread.

>> +  If a package contains a binary or library which links to a
>> +  shared library, we must ensure that, when the package is
>> +  installed on the system, all of the libraries needed are also
>> +  installed.  These dependencies must be added to the binary
>> +  package when it is built, since they may change based on which
>> +  version of a shared library the binary or library was linked
>> +  with.

> I have a difficulty with that sentence : isn't the goal of the symbols
> system to actually avoid that a dependancy changes when a package is
> built on a newer version of a library ?  What rather changes is that the
> dependancy is modified when a new version of the package makes new calls
> to the library, which may have been implemented only in newer versions
> of the library.

That's true, but somewhat deceptive because this can happen with no
changes to the client code.  In some cases, changes only to the library
with no changes to the client code can require tighter dependencies when
the client is built against the newer library.  The obvious example is
replacing a macro in a header with an actual function entry point.  In
this case, the client code will build and work fine with either the older
or the newer version of the library, but when built against the newer
library, it won't work with the older library.

That's what this is trying to capture here.  If the exact same code is
rebuilt against a newer version of a library, it may have to depend on
that newer version of the library.

> If the misunderstanding is on my side, please consider it a hint that
> the explanations can be expanded :)  [PS: after reading the whole patch,
> maybe this has something to do with minimal-version ?]

I'm not really sure what else to say, but I'm happy to add more if someone
can take a shot at letting me know what's needed

But I was intentionally trying to keep this vague and not get into
enumerating the reasons why dependencies may need to be tightened.  That's
really a topic for a whole separate guide on how shared libraries should
be packaged.

The rest of your changes look good, although I haven't had a chance to
incorporate them yet.  I'll take a look, hopefully soon, although I may
not have time this weekend (which is a long holiday weekend in the US).

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87boq6512a@windlord.stanford.edu

Re: Bug#571776: document symbols

2012-01-13 Thread Cyril Brulebois 
Russ Allbery  (13/01/2012):
> Yes, but there was some discussion in the Policy bug asking why shlibs
> files were required when they're not used if a symbols file is present,
> and while I originally argued that keeping them both made sense, I came
> around to that position after reviewing the bug discussion.  It doesn't
> hurt anything, but there doesn't really seem to be any point in providing
> shlibs if symbols is present.

Last I checked, one needs an shlibs file to get proper dependencies into
udebs, which explains why libx* packages still use some -V flag in their
dh_makeshlibs call.

Demo: xorg-server/experimental build-depends on libpciaccess (>= version-
in-unstable), and the symbols added in this version are really needed (see
below).

Rebuilding libpciaccess to drop the -V flag, and rebuilding xorg-server
against it generates this change in xserver-xorg-core-udeb's dependencies:
| Depends: […], [-libpciaccess0-udeb (>= 0.12.902),-] {+libpciaccess0-udeb,+} 
[…]

→ Fail.


“Those new symbols really are needed”:
| $ grep 902 ../lib/libpciaccess.git/debian/*symbols
|  pci_device_map_legacy@Base 0.12.902
|  pci_device_unmap_legacy@Base 0.12.902
| $ for i in $(find debian/xserver-xorg-core-udeb -type f); do if nm -D $i 
2>&1|grep -qs pci_device_'.*'map_legacy; then echo $i; fi; done
| debian/xserver-xorg-core-udeb/usr/lib/xorg/modules/libvgahw.so
| debian/xserver-xorg-core-udeb/usr/lib/xorg/modules/libint10.so

Diff in the libpciaccess package to exhibit that:
| -   dh_makeshlibs -V'libpciaccess0 (>= 0.12.902)' 
--add-udeb=$(PACKAGE)-udeb -- -c4
| +   dh_makeshlibs --add-udeb=$(PACKAGE)-udeb -- -c4

Mraw,
KiBi.


signature.asc
Description: Digital signature

Re: Bug#571776: document symbols

2012-01-13 Thread Russ Allbery 
Cyril Brulebois  writes:
> Russ Allbery  (13/01/2012):

>> Yes, but there was some discussion in the Policy bug asking why shlibs
>> files were required when they're not used if a symbols file is present,
>> and while I originally argued that keeping them both made sense, I came
>> around to that position after reviewing the bug discussion.  It doesn't
>> hurt anything, but there doesn't really seem to be any point in
>> providing shlibs if symbols is present.

> Last I checked, one needs an shlibs file to get proper dependencies into
> udebs, which explains why libx* packages still use some -V flag in their
> dh_makeshlibs call.

Yes, udebs are called out as an exception and it's still stated that
shlibs must be used for udebs.  They still are consistent with the
property above, though, since symbols aren't supported for udebs, or at
least that's what I understood.

-- 
Russ Allbery (r...@debian.org)   


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87obu63ey2@windlord.stanford.edu