[lfs-dev] Patches for iana-etc-2.30
Hi all.
I suggest two patches for the iana-etc, which solves the problem of
data updates from the IANA site. The first patch is from LinHES project and
can be found here:
http://linhes.org/projects/linhes/repository/revisions/master/entry/abs/core/iana-etc/newer.patch
I added second patch to place short protocols and services descriptions in
output files.
Data update must be done by command
make get
before make and make install.
Michael Uleysky
diff -Naur old/Makefile new/Makefile
--- old/Makefile 2008-03-06 05:02:40.0 +1100
+++ new/Makefile 2012-07-23 03:20:19.009174483 +1000
@@ -42,11 +42,11 @@
protocol-numbers.iana port-numbers.iana
protocol-numbers.iana:
- $(AWK) -f get.gawk -v file=protocol-numbers >protocol-numbers.iana
+ $(AWK) -f get.gawk -v file=protocol-numbers/protocol-numbers.xml >protocol-numbers.iana
rm -f protocol-numbers
port-numbers.iana:
- $(AWK) -f get.gawk -v file=port-numbers >port-numbers.iana
+ $(AWK) -f get.gawk -v file=service-names-port-numbers/service-names-port-numbers.xml >port-numbers.iana
rm -f port-numbers
protocol-numbers:
diff -Naur old/get.gawk new/get.gawk
--- old/get.gawk 2008-03-06 05:04:18.0 +1100
+++ new/get.gawk 2012-07-23 03:20:19.009174483 +1000
@@ -26,7 +26,7 @@
NR++
if (in_content) { print }
- if (/^$/) { in_content=1 }
+ if (/^\r?$/){ in_content=1 }
}
printf "\n" >"/dev/stderr"
diff -Naur old/protocols.gawk new/protocols.gawk
--- old/protocols.gawk 2006-11-15 05:20:26.0 +1100
+++ new/protocols.gawk 2012-07-23 03:22:13.479330458 +1000
@@ -1,32 +1,13 @@
-#!/usr/bin/gawk --re-interval -f
-# the above doesn't work (stupid kernel) but serves as documentation
-
-# Copyright (c) 2003-2004, 2006 Seth W. Klein
-# Licensed under the Open Software License version 3.0
-# See the file COPYING in the distribution tarball or
-# http://www.opensource.org/licenses/osl-3.0.txt
+#!/usr/bin/awk -f
BEGIN {
-if (strip == "yes") {
- strip = 1
- format = "%s\t%s\t%s\n"
- header_printed = 1
-} else {
- strip = 0
- print "# See also: protocols(5)" \
- ", http://www.sethwklein.net/projects/iana-etc/\n#";
- format = "%-12s %3s %-12s # %s\n"
- header_printed = 0
-}
+ print "# See the full IANA XML file at: /usr/share/iana-etc/protocol-numbers.iana\n"
+ FS="[<>]"
}
-{ sub(/\r/, "") }
-match($0, /^[ \t]+([0-9]+)[ \t]{1,5}([^ \t]+)(.*)/, f) {
-if ( ! header_printed) {
- printf format, "# protocol", "num", "aliases", "comments"
- header_printed = 1;
-}
-sub(/^[ \t]*/, "", f[3])
-printf format, tolower(f[2]), f[1], f[2], f[3]
-next
+
+{
+ if (/
-# Licensed under the Open Software License version 3.0
-# See the file COPYING in the distribution tarball or
-# http://www.opensource.org/licenses/osl-3.0.txt
+#!/usr/bin/awk -f
BEGIN {
-if (strip == "yes") {
- strip = 1
-} else {
- strip = 0
- print "# See also: services(5)" \
- ", http://www.sethwklein.net/projects/iana-etc/\n#";
-}
-while (getline <"port-aliases") {
- sub(/#.*/, "")
- if (/^[ \t]*$/) { continue }
- # 1:name 2:protocol 3:aliases
- match($0, /^[ \t]*([^ \t]+)[ \t]+([^ \t]+)[ \t]+(.*)$/, f)
- aliases[f[1] " " f[2]] = " " f[3]
-}
+ print "# See the full IANA XML file at: /usr/share/iana-etc/port-numbers.iana\n"
+ FS="[<>]"
}
-{ sub(/\r/, "") }
-# 1:name 2:ws3:port 4:range 5:proto 6:comment
-match($0, /(^[[:alnum:]][^ \t]+)([ \t]+)([0-9]+)(-[0-9]+)?\/([^ \t]+)(.*)/, f) \
-&& f[3] != "0" {
-# port 0 means unallocated, per port-numbers
-name = f[1]
-whitespace = f[2]
-port = f[3]
-protocol = f[5]
-comment = f[6]
-if (strip) {
- whitespace = "\t"
- comment = ""
-} else if (length(comment) > 0)
- sub(/^[ \t]*/, "&# ", comment)
-print name whitespace port "/" protocol aliases[name " " protocol] comment
-next
+
+{
+ if (/diff -urNd iana-etc-2.30/protocols.gawk iana-etc-2.30-michael/protocols.gawk
--- iana-etc-2.30/protocols.gawk 2014-01-29 19:14:56.153941452 +1100
+++ iana-etc-2.30-michael/protocols.gawk 2014-01-29 19:15:18.223973251 +1100
@@ -6,8 +6,9 @@
}
{
- if (/--
http://linuxfromscratch.org/mailman/listinfo/lfs-dev
FAQ: http://www.linuxfromscratch.org/faq/
Unsubscribe: See the above information page
Re: [lfs-dev] Patches for iana-etc-2.30
2014-01-31 Bruce Dubbs : > Michael Uleysky wrote: > > Hi all. > > > > I suggest two patches for the iana-etc, which solves the problem of > > data updates from the IANA site. > > What problem is that? > iana-etc includes some scripts for downloading recent version of protocols and services from IANA site. These scripts are unusable now because IANA changes addresses and formats of its data. It is not a problem if you happy with 5-years old data. Really, there is only one interesting change in new IANA data - port numbers for dccp and sctp protocols in services file. > > The first patch is from LinHES project and > > can be found here: > > > http://linhes.org/projects/linhes/repository/revisions/master/entry/abs/core/iana-etc/newer.patch > > I added second patch to place short protocols and services descriptions > in > > output files. > > Data update must be done by command > > make get > > before make and make install. > > Shouldn't this suggestion be made to the upstream maintainers? > It seems that upstream is dead. >-- Bruce > > -- > http://linuxfromscratch.org/mailman/listinfo/lfs-dev > FAQ: http://www.linuxfromscratch.org/faq/ > Unsubscribe: See the above information page > -- http://linuxfromscratch.org/mailman/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] Patches for iana-etc-2.30
2014-01-31 Bruce Dubbs : > Michael Uleysky wrote: > > 2014-01-31 Bruce Dubbs : > > > >> Michael Uleysky wrote: > >>> Hi all. > >>> > >>> I suggest two patches for the iana-etc, which solves the problem of > >>> data updates from the IANA site. > >> > >> What problem is that? > >> > > > > iana-etc includes some scripts for downloading recent version of > protocols > > and services from IANA site. > > Interesting. That is embedded in some gawk scripts, but don't install > those. I've never noticed them before. That said, have you tested the > scripts? I tried to follow them manually and I can't find the pages > that are specified, either old or in the patch. Note that we do not > install the gawk scripts at all. > Pages to download are specifies in Makefile. Of course, I test these scripts in old and patched variants. Old scripts are almost unusable, services file generated incorrectly. Patched version use xml as input instead of plain text and is very simple. > These scripts are unusable now because IANA > > changes addresses and formats of its data. It is not a problem if you > happy > > with 5-years old data. Really, there is only one interesting change in > new > > IANA data - port numbers for dccp and sctp protocols in services file. > > Wouldn't it just be easier to do a sed to the affected files when > needed? We do that for rpcbind now: > > sed -i 's/^sunrpc/rpcbind/' /etc/services > It can be done for protocols, because here changes are small, but not for services. Services from IANA are very large. Of course, it is mainly a proprietary junk, but I don't know how to separate really useful entries. For me, I add rpcbind as alias for sunrpc, sunrpc111/tcp<->rpcbind # SUN Remote Procedure Call sunrpc111/udp<->rpcbind # SUN Remote Procedure Call There also can be some other aliases, see, for example, services and protocols files from Gentoo baselayout, http://de-mirror.org/gentoo/distfiles/baselayout-2.2.tar.bz2 I don't know the authorized source of these aliases. And about CLFS patches. First patch, iana-etc-2.30-get_fix-1.patch, correct only download url. It is not enough, because format of IANA data also changed and services file can't be regenerated without changes in scripts. Second patch is just a snapshot of IANA data on 2012-06-08. I think, we may combine two ways in one: provide patches on scripts for people living on the edge, and provide some snapshot of data for regular use. >-- Bruce > -- > http://linuxfromscratch.org/mailman/listinfo/lfs-dev > FAQ: http://www.linuxfromscratch.org/faq/ > Unsubscribe: See the above information page > -- http://linuxfromscratch.org/mailman/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
Re: [lfs-dev] Patches for iana-etc-2.30
2014-02-02 Bruce Dubbs :
> Bruce Dubbs wrote:
>
> OK, I have been playing with this. If we create a tarball with the up
> to date services and protocols files along with the following script
> named update-iana-files, the install procedure would be merely:
>
> install -m644 -o root -g root services protocols /etc
> install -m744 -o root -g root update-iana-files /usr/sbin
>
> The update-iana-files is below. Comments?
>
>-- Bruce
>
>
> #! /bin/bash
>
> if [ $(id -u) -ne 0 ]; then
>echo "$0: You must be root to run this script"
>exit 1
> fi
>
> IANA_DIR=/usr/share/iana-etc
>
> mkdir -p $IANA_DIR
> cd $IANA_DIR
> rm *.xml
>
> wget http://www.iana.org/assignments/protocol-numbers/protocol-numbers.xml
>
> wget
>
> http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xml
>
>
> PROTOCOLS='BEGIN {
> print "# See the full IANA XML file at:
> /usr/share/iana-etc/protocol-numbers.xml\n"
> FS="[<>]"
> }
>
> {
> if (/ if (/ if (/ if (/ if (/<\/record/ && (v || n=="HOPOPT") && n) printf "%-15s %3i %-15s #
> %s\n", tolower(n),v,n,d
> }'
>
> gawk --re-interval -v strip=yes -- "$PROTOCOLS" protocol-numbers.xml >
> /etc/protocols
>
> SERVICES='BEGIN {
> print "# See the full IANA XML file at:
> /usr/share/iana-etc/service-names-port-numbers.xml\n"
> FS="[<>]"
> }
>
> {
> if (/ if (/ if (/ if (/ if (/ if (/Unassigned/ || /Reserved/ || /historic/) c=1
> if (/<\/record/ && n && u && p && !c) printf "%-15s %5i/%-5s # %s\n",
> n,u,p,d # services
> }'
>
> gawk -v strip=yes -- "$SERVICES" service-names-port-numbers.xml >
> /etc/services
> --
> http://linuxfromscratch.org/mailman/listinfo/lfs-dev
> FAQ: http://www.linuxfromscratch.org/faq/
> Unsubscribe: See the above information page
>
Last string may be replaced by
gawk -v strip=yes -- "$SERVICES" service-names-port-numbers.xml | sed
"s/sunrpc/rpcbind/" >
/etc/services
or like, because rpcbind in /etc/services is crucial for working rpcbind
portmapper and nfs server.
--
http://linuxfromscratch.org/mailman/listinfo/lfs-dev
FAQ: http://www.linuxfromscratch.org/faq/
Unsubscribe: See the above information page
Re: [lfs-dev] iana-files-1.0
2014-02-03 Bruce Dubbs :
> I am getting ready to change the iana file installation in LFS. The
> procedure will be to download:
>
>
> http://anduin.linuxfromscratch.org/sources/LFS/lfs-packages/conglomeration/iana-etc/iana-files-1.0.tar.xz
>
> After unpacking the install will be:
>
> install -m644 -u root -g root services protocols /etc
> install -m744 -u root -g root update* /usr/sbin
>
I add a simple Makefile for this.
> Two executable scripts are installed, update-iana-protocols and
> update-iana-services. These scripts create /usr/share/iana-etc if it
> does not exist and use wget (not available in LFS) to get the upstream
> xml files. The scripts then parse the xml files and update
> /etc/protocols and /etc/services as appropriate.
>
> As I write this, I'm thinking that the header of the installed files
> should perhaps also read:
>
> # To update this file, use the update-iana- script.
>
Also include in my patch.
>
> We might also want to just number the tarball iana-files-1.tar.xz and
> increment the number as needed without major/minor versions.
>
May be iana-files-$DATE_OF_UPDATE.tar.xz?
> Other tweaks can be made. Please let me know what you think.
>
I create a patch with some additions:
1) Simple Makefile for installation. I think, we must also include source
protocol-numbers.xml and service-names-port-numbers.xml in installation.
2) I add an option "restore" for update scripts. With this option services
and protocols will be regenerated from backup xml files.
diff -urNd iana-files-1.0/Makefile iana-files-1.0-michael/Makefile
--- iana-files-1.0/Makefile1970-01-01 10:00:00.0 +1000
+++ iana-files-1.0-michael/Makefile2014-02-03 20:43:46.571416045 +1100
@@ -0,0 +1,17 @@
+PREFIX =
+EXECPREFIX = /usr
+DATAPREFIX = /usr
+
+main: services protocols
+
+install-data: main protocol-numbers.xml service-names-port-numbers.xml
+ install -o root -g root -m644 services $(PREFIX)/etc/
+ install -o root -g root -m644 protocols $(PREFIX)/etc/
+ install -o root -g root -m644 -D protocol-numbers.xml
$(DATAPREFIX)/share/iana-etc/protocol-numbers.xml
+ install -o root -g root -m644 -D service-names-port-numbers.xml
$(DATAPREFIX)/share/iana-etc/service-names-port-numbers.xml
+
+install-bin: update-iana-protocols update-iana-services
+ install -o root -g root -m700 update-iana-protocols
$(EXECPREFIX)/sbin/
+ install -o root -g root -m700 update-iana-services $(EXECPREFIX)/sbin/
+
+install: install-data install-bin
diff -urNd iana-files-1.0/update-iana-protocols
iana-files-1.0-michael/update-iana-protocols
--- iana-files-1.0/update-iana-protocols2014-02-03 15:04:08.0
+1100
+++ iana-files-1.0-michael/update-iana-protocols2014-02-03
20:50:20.594435684 +1100
@@ -6,6 +6,9 @@
exit 1
}
+RESTORE=0
+if [ "$1" == "restore" ]; then RESTORE=1; fi
+
if [ $(id -u) -ne 0 ]; then
echo "$0: You must be root to run this script"
exit 1
@@ -18,18 +21,12 @@
DATE=$(date)
mkdir -p $IANA_DIR
-cd $IANA_DIR
-
-if [ -e $FILE ]; then
- mv -f $FILE ${FILE}.backup
-fi
-
-wget $HTTP/protocol-numbers/$FILE ||
- die "Could not download $FILE" $LINENO
+cd $IANA_DIR
PROTOCOLS="BEGIN {
print \"# See the full IANA XML file at: $IANA_DIR/$FILE\"
print \"# Updated $DATE\n\"
+ print \"# To update this file, use the update-iana-protocols script\n\"
print \"# protocol num aliases # comments\"
print \"# --- ---\"
@@ -45,6 +42,16 @@
printf \"%-15s %3i %-15s # %s\n\", tolower(n),v,n,d
}"
-gawk --re-interval -v strip=yes -- "$PROTOCOLS" $FILE > /etc/protocols
-echo "/etc/protocols updated"
+if [ $RESTORE -ne 0 ]; then
+ gawk --re-interval -v strip=yes -- "$PROTOCOLS" ${FILE}.backup >
/etc/protocols
+ echo "/etc/protocols restored"
+else
+ if [ -e $FILE ]; then
+ mv -f $FILE ${FILE}.backup
+ fi
+ wget $HTTP/protocol-numbers/$FILE ||
+ die "Could not download $FILE" $LINENO
+ gawk --re-interval -v strip=yes -- "$PROTOCOLS" $FILE > /etc/protocols
+ echo "/etc/protocols updated"
+fi
diff -urNd iana-files-1.0/update-iana-services
iana-files-1.0-michael/update-iana-services
--- iana-files-1.0/update-iana-services2014-02-03 15:04:08.0
+1100
+++ iana-files-1.0-michael/update-iana-services2014-02-03
20:52:28.159442043 +1100
@@ -6,6 +6,9 @@
exit 1
}
+RESTORE=0
+if [ "$1" == "restore" ]; then RESTORE=1; fi
+
if [ $(id -u) -ne 0 ]; then
echo "$0: You must be root to run this script"
exit 1
@@ -18,18 +21,12 @@
DATE=$(date)
mkdir -p $IANA_DIR
-cd $IANA_DIR
-
-if [ -e $FILE ]; then
- mv -f $FILE ${FILE}.backup
-fi
-
-wget $HTTP/service-names-port-numbers/$FILE ||
- die "Could not download $FILE" $LINENO
+cd $IANA_DIR
SERVICES="BEGIN {
print \"# See the full IANA XML file at: $IANA_DIR/$FILE\"
print \"# Updated $DATE\n\"
+ print \"# To update this file, use the update-iana-services script\"
print \"# The range for assigned ports managed by the IANA is 0-1023
Re: [lfs-dev] iana-files-1.0
2014-02-03 Bruce Dubbs : > Michael Uleysky wrote: > > 2014-02-03 Bruce Dubbs : > > > >> I am getting ready to change the iana file installation in LFS. The > >> procedure will be to download: > >> > >> > >> > http://anduin.linuxfromscratch.org/sources/LFS/lfs-packages/conglomeration/iana-etc/iana-files-1.0.tar.xz > >> > >> After unpacking the install will be: > >> > >> install -m644 -u root -g root services protocols /etc > >> install -m744 -u root -g root update* /usr/sbin > >> > > > > I add a simple Makefile for this. > > Why add a Makefile for such a simple install? > > > >> Two executable scripts are installed, update-iana-protocols and > >> update-iana-services. These scripts create /usr/share/iana-etc if it > >> does not exist and use wget (not available in LFS) to get the upstream > >> xml files. The scripts then parse the xml files and update > >> /etc/protocols and /etc/services as appropriate. > >> > >> As I write this, I'm thinking that the header of the installed files > >> should perhaps also read: > >> > >> # To update this file, use the update-iana- script. > >> > > Also include in my patch. > > > > > >> > >> We might also want to just number the tarball iana-files-1.tar.xz and > >> increment the number as needed without major/minor versions. > >> > > May be iana-files-$DATE_OF_UPDATE.tar.xz? > > Seems to be overkill since the only time I foresee this changing is if > the upstream xml format changes. We have been using files since 2006 > and this is the first time it has come up. > Ok, I agree with it. >> Other tweaks can be made. Please let me know what you think. >> > I create a patch with some additions: > 1) Simple Makefile for installation. I think, we must also include source > protocol-numbers.xml and service-names-port-numbers.xml in installation. I thought about that, but why is it needed? It's not in the old tarball > and we've never needed it. I'm not completely against this though. > For me is much simple to type make; make install than type long install bla-bla-bla strings. Make is just more familiar. Old iana-etc have a Makefile. > > 2) I add an option "restore" for update scripts. With this option > services > > and protocols will be regenerated from backup xml files. > > Again, this seems to be overkill. The tarball has the needed files if > that is needed. Generally the files are just in /etc and don't often > change. The only update we've ever needed is to change sunrpc to > rpcbind in the services file. > It can be dangerous situation, if someone try to update and leave with unusable files because IANA changes format again. In this case restore will be useful. Another comment. If we start adding options to the scripts, we need to > add at least a help option and perhaps man pages. I'm trying to KISS > this change. > I completely agree with you. My solution is not optimal and I also not very like it. Ok, do we really need such thing as update scripts, installed somewhere in the system? I think, not. So, lets just place protocols and services in /etc (we can do it without Makefile, of course) and leave update scripts only in the tarball for people who want update _before_ install. It is just a way as original iana-etc works. But in this case we need to set IANA_DIR=. and remove /etc in last two strings in scripts. -- http://linuxfromscratch.org/mailman/listinfo/lfs-dev FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
