Re: Creating jigdo files without loal file access
Richard Atterer wrote: Hi Patrick, On Mon, Mar 03, 2003 at 12:37:06PM +0100, Patrick Strasser wrote: In short, I want to produce jigdo files for the Debian/Hurd CD set, but I don't have access to, don't want to setup , and don't have the space for a mirror. My first reaction: This will be difficult! :) I know :->>> warning: I've wote this mail during two hours of recherche, so some parts may sound strange, but I didn't remove thema as they contain information needed to understand things. Also note that a problem with .jigdo files for Debian is that the mirror contents change. As soon as a file on the CD is no longer on the mirror, jigdo will fail. For this reason, it is mandatory with Debian CDs to set up a "fallback mirror", I'm aware of this effect. I have talked to Attila Nagy from fsn.hu, which host a fallback directory for the snapshots. He said setting up a fallback directory for Hurd packages would be no problem for two weeks buffering. I don't expect too much packages to change. As a second way I could set up a fallback directory by myself after some convincing our admin to leve me some space for a Good Reason. ;-) That's right. :-/ One possible trick is to re-use the [Parts] section of existing .jigdo files. For example, if you were to release a custom version of an official CD, you could copy the section from the official .jigdo files (or even more comfortable, use "jigdo-file make-template --merge"). I would be the first to produce Hurd jigdo file sets. So this is no way to go. - Obtain list of md5sums and paths on the server - Obtain list of md5sums needed by the .template of the loop-mounted image: "jigdo-file ls -t foo.template --hex | grep need-file" (4th column) Ok, I think I've exactly found what I'm looking for: Debian has a file called md5sums.gz on it's mirrors, about 5.6 Mb in size, which is exactly the output ot md5sum, which can be easily be run local. I'll just match the md5sums and thus will get all the files which are identical on the mirrors and on the mounted imgage, even with server path. Then I play arround with the pathes and hopefully get a nice jigdo file set (sometimes, somehow). - Find paths for the checksums (e.g. using the "join" utility) IMHO you really need a little help from someone with access to the server. Of course I can talk to people having access, but we additionaly a problem: Philip Charles, the creator of the Debian/Hurd CDs, is located in New Zealand, and has only low bandwith capacity. He manages to fetch all necessary packages, but uploading 5 CDs takes much to long on his connection. So he gives the images to a friend, who has better connectivity and loads them up. This is quite complicated and takes time. That's why I'm investigating a way to build jigdo file sets without a big local filesystem, which is expensive to keep up to date; Philip would then build his CDs, make jigdo fiel sets, which are much smaller, and hopefully load tehm up himself, aving a lot of time. At the initial upload server someone runs jigdo and blows them up to full size. So all the CD creator does is managing the meta-information (CD structure and special packages/files). Thus he does not move around data, but just the additional data for oraganisation. I like this idea :-) Greatings Patrick -- Engineers motto: cheap, good, fast: choose any two Patrick Strasser Student of Telematik, Techn. University Graz, Austria -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Creating jigdo files without loal file access
On Tue, 4 Mar 2003, Patrick Strasser wrote: > Philip Charles, the creator of the Debian/Hurd CDs, is located in New > Zealand, and has only low bandwith capacity. He manages to fetch all > necessary packages, but uploading 5 CDs takes much to long on his > connection. So he gives the images to a friend, who has better > connectivity and loads them up. This is quite complicated and takes > time. That's why I'm investigating a way to build jigdo file sets > without a big local filesystem, which is expensive to keep up to date; > Philip would then build his CDs, make jigdo fiel sets, which are much > smaller, and hopefully load tehm up himself, aving a lot of time. At the > initial upload server someone runs jigdo and blows them up to full size. > > So all the CD creator does is managing the meta-information (CD > structure and special packages/files). Thus he does not move around > data, but just the additional data for oraganisation. I like this idea :-) So do I. I can always give prior notice of when I am freezing the mirror to produce the CDs, then the CD creator can freeze theirs at the same time. Boot floppies remain constant thoughout the K (or whatever) series, however baseGNU.tgz and its associated floppies change with each release, say 35-40 MB. There are some packages I have collected in addition to those on alpha.gnu.ord which with alpha make up the local fs. Andrew Mitchel is uploading the K2 images at the moment. May be Andrew and I could be given access to a suitable machine and do the job ourselves? Phil. -- Philip Charles; 39a Paterson Street, Abbotsford, Dunedin, New Zealand +64 3 488 2818Fax +64 3 488 2875Mobile 025 267 9420 [EMAIL PROTECTED] - preferred. [EMAIL PROTECTED] I sell GNU/Linux & GNU/Hurd CDs. See http://www.copyleft.co.nz -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Creating jigdo files without loal file access
On Tue, Mar 04, 2003 at 12:33:24PM +0100, Patrick Strasser wrote: > Richard Atterer wrote: > >it is mandatory with Debian CDs to set up a "fallback mirror", > > I'm aware of this effect. I have talked to Attila Nagy from fsn.hu, > which host a fallback directory for the snapshots. He said setting up a > fallback directory for Hurd packages would be no problem for two weeks > buffering. I don't expect too much packages to change. OK. If you need more, I think we could also set up a fallback directory on gluck.d.o which can be specific to the Hurd CDs, i.e. it'll ensure successful downloads for as long as you like. > Of course I can talk to people having access, but we additionaly a > problem: > > Philip Charles, the creator of the Debian/Hurd CDs, is located in New > Zealand, and has only low bandwith capacity. He manages to fetch all > necessary packages, but uploading 5 CDs takes much to long on his > connection. So he gives the images to a friend, who has better > connectivity and loads them up. This is quite complicated and takes time. But then isn't the obvious way to solve the issue the following: Philip generates the CD images on a Debian server (using nohup/screen to log off while mkisofs runs), and the image is piped into jigdo-file to generate jigdo/template files on that server. Subsequently, Phil can download just the jigdo/template files and recreate the image locally for testing. The only problem might be that the templates could get too big to download over modem. A typical size is 5..10 MB for a full-size CD image. (The official Debian CD template files are bigger because they deliberately put some files' data in the template.) > That's why I'm investigating a way to build jigdo file sets without a big > local filesystem, which is expensive to keep up to date; Philip would > then build his CDs, make jigdo fiel sets, which are much smaller, and > hopefully load tehm up himself, aving a lot of time. At the initial > upload server someone runs jigdo and blows them up to full size. Oh, in that case I should mention the following possibility: If that "blowing up" happens on a machine with a full Debian mirror (i.e. the files needed by the template are all /somewhere/, but you don't know where), just do this: jigdo-file --cache ~/.jigdo-cache.db make-image \ -t foo.template -i output.iso /my/mirror Only the template file is required by this, not the jigdo file. Philip would have to take care that the template file created by him only references files which are available on the machine where "jigdo-file make-image" is run. Cheers, Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: New vendor procedure - assistance required
On Sun, Mar 02, 2003 at 04:57:12PM -0800, Dafydd Walters wrote: > I've followed the procedure exactly as described on the page "Adding new > vendors to the CD vendors list" and sent an email (a couple of weeks > ago) to [EMAIL PROTECTED], but I still have not heard anything. > > Any feedback would be appreciated. Craig Small is in charge of updating the vendor info. Craig, are you there? :-) Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Creating jigdo files without loal file access
Richard Atterer wrote: Hi Patrick, On Mon, Mar 03, 2003 at 12:37:06PM +0100, Patrick Strasser wrote: In short, I want to produce jigdo files for the Debian/Hurd CD set, but I don't have access to, don't want to setup , and don't have the space for a mirror. My first reaction: This will be difficult! :) I know :->>> warning: I've wote this mail during two hours of recherche, so some parts may sound strange, but I didn't remove thema as they contain information needed to understand things. Also note that a problem with .jigdo files for Debian is that the mirror contents change. As soon as a file on the CD is no longer on the mirror, jigdo will fail. For this reason, it is mandatory with Debian CDs to set up a "fallback mirror", I'm aware of this effect. I have talked to Attila Nagy from fsn.hu, which host a fallback directory for the snapshots. He said setting up a fallback directory for Hurd packages would be no problem for two weeks buffering. I don't expect too much packages to change. As a second way I could set up a fallback directory by myself after some convincing our admin to leve me some space for a Good Reason. ;-) That's right. :-/ One possible trick is to re-use the [Parts] section of existing .jigdo files. For example, if you were to release a custom version of an official CD, you could copy the section from the official .jigdo files (or even more comfortable, use "jigdo-file make-template --merge"). I would be the first to produce Hurd jigdo file sets. So this is no way to go. - Obtain list of md5sums and paths on the server - Obtain list of md5sums needed by the .template of the loop-mounted image: "jigdo-file ls -t foo.template --hex | grep need-file" (4th column) Ok, I think I've exactly found what I'm looking for: Debian has a file called md5sums.gz on it's mirrors, about 5.6 Mb in size, which is exactly the output ot md5sum, which can be easily be run local. I'll just match the md5sums and thus will get all the files which are identical on the mirrors and on the mounted imgage, even with server path. Then I play arround with the pathes and hopefully get a nice jigdo file set (sometimes, somehow). - Find paths for the checksums (e.g. using the "join" utility) IMHO you really need a little help from someone with access to the server. Of course I can talk to people having access, but we additionaly a problem: Philip Charles, the creator of the Debian/Hurd CDs, is located in New Zealand, and has only low bandwith capacity. He manages to fetch all necessary packages, but uploading 5 CDs takes much to long on his connection. So he gives the images to a friend, who has better connectivity and loads them up. This is quite complicated and takes time. That's why I'm investigating a way to build jigdo file sets without a big local filesystem, which is expensive to keep up to date; Philip would then build his CDs, make jigdo fiel sets, which are much smaller, and hopefully load tehm up himself, aving a lot of time. At the initial upload server someone runs jigdo and blows them up to full size. So all the CD creator does is managing the meta-information (CD structure and special packages/files). Thus he does not move around data, but just the additional data for oraganisation. I like this idea :-) Greatings Patrick -- Engineers motto: cheap, good, fast: choose any two Patrick Strasser Student of Telematik, Techn. University Graz, Austria
Re: Creating jigdo files without loal file access
On Tue, 4 Mar 2003, Patrick Strasser wrote: > Philip Charles, the creator of the Debian/Hurd CDs, is located in New > Zealand, and has only low bandwith capacity. He manages to fetch all > necessary packages, but uploading 5 CDs takes much to long on his > connection. So he gives the images to a friend, who has better > connectivity and loads them up. This is quite complicated and takes > time. That's why I'm investigating a way to build jigdo file sets > without a big local filesystem, which is expensive to keep up to date; > Philip would then build his CDs, make jigdo fiel sets, which are much > smaller, and hopefully load tehm up himself, aving a lot of time. At the > initial upload server someone runs jigdo and blows them up to full size. > > So all the CD creator does is managing the meta-information (CD > structure and special packages/files). Thus he does not move around > data, but just the additional data for oraganisation. I like this idea :-) So do I. I can always give prior notice of when I am freezing the mirror to produce the CDs, then the CD creator can freeze theirs at the same time. Boot floppies remain constant thoughout the K (or whatever) series, however baseGNU.tgz and its associated floppies change with each release, say 35-40 MB. There are some packages I have collected in addition to those on alpha.gnu.ord which with alpha make up the local fs. Andrew Mitchel is uploading the K2 images at the moment. May be Andrew and I could be given access to a suitable machine and do the job ourselves? Phil. -- Philip Charles; 39a Paterson Street, Abbotsford, Dunedin, New Zealand +64 3 488 2818Fax +64 3 488 2875Mobile 025 267 9420 [EMAIL PROTECTED] - preferred. [EMAIL PROTECTED] I sell GNU/Linux & GNU/Hurd CDs. See http://www.copyleft.co.nz
Re: Creating jigdo files without loal file access
On Tue, Mar 04, 2003 at 12:33:24PM +0100, Patrick Strasser wrote: > Richard Atterer wrote: > >it is mandatory with Debian CDs to set up a "fallback mirror", > > I'm aware of this effect. I have talked to Attila Nagy from fsn.hu, > which host a fallback directory for the snapshots. He said setting up a > fallback directory for Hurd packages would be no problem for two weeks > buffering. I don't expect too much packages to change. OK. If you need more, I think we could also set up a fallback directory on gluck.d.o which can be specific to the Hurd CDs, i.e. it'll ensure successful downloads for as long as you like. > Of course I can talk to people having access, but we additionaly a > problem: > > Philip Charles, the creator of the Debian/Hurd CDs, is located in New > Zealand, and has only low bandwith capacity. He manages to fetch all > necessary packages, but uploading 5 CDs takes much to long on his > connection. So he gives the images to a friend, who has better > connectivity and loads them up. This is quite complicated and takes time. But then isn't the obvious way to solve the issue the following: Philip generates the CD images on a Debian server (using nohup/screen to log off while mkisofs runs), and the image is piped into jigdo-file to generate jigdo/template files on that server. Subsequently, Phil can download just the jigdo/template files and recreate the image locally for testing. The only problem might be that the templates could get too big to download over modem. A typical size is 5..10 MB for a full-size CD image. (The official Debian CD template files are bigger because they deliberately put some files' data in the template.) > That's why I'm investigating a way to build jigdo file sets without a big > local filesystem, which is expensive to keep up to date; Philip would > then build his CDs, make jigdo fiel sets, which are much smaller, and > hopefully load tehm up himself, aving a lot of time. At the initial > upload server someone runs jigdo and blows them up to full size. Oh, in that case I should mention the following possibility: If that "blowing up" happens on a machine with a full Debian mirror (i.e. the files needed by the template are all /somewhere/, but you don't know where), just do this: jigdo-file --cache ~/.jigdo-cache.db make-image \ -t foo.template -i output.iso /my/mirror Only the template file is required by this, not the jigdo file. Philip would have to take care that the template file created by him only references files which are available on the machine where "jigdo-file make-image" is run. Cheers, Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯
Re: New vendor procedure - assistance required
On Sun, Mar 02, 2003 at 04:57:12PM -0800, Dafydd Walters wrote: > I've followed the procedure exactly as described on the page "Adding new > vendors to the CD vendors list" and sent an email (a couple of weeks > ago) to [EMAIL PROTECTED], but I still have not heard anything. > > Any feedback would be appreciated. Craig Small is in charge of updating the vendor info. Craig, are you there? :-) Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯
