On Jul 14, 2013, at 3:43 PM, Craig Rodrigues wrote: On Sat, Jul 13, 2013 at 10:54 PM, Teske, Devin <devin.te...@fisglobal.com<mailto:devin.te...@fisglobal.com>> wrote:
I assume that poudiere builds packages from ports. Yes. That's not how we build package repositories here (and would expect that there are many more like us). How do you build packages if you are not using FreeBSD ports? Two spectrums... When developer A says he needs xerces, I go to: (for an active release, such as 8.4 or 9.1) ftp://ftp.freebsd.org/pub/FreeBSD/releases/{arch}/X.Y-RELEASE/packages/All<ftp://ftp.freebsd.org/pub/FreeBSD/releases/%7Barch%7D/X.Y-RELEASE/packages/All> (for an in-active release, such as 8.1 or 9.0) ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/{arch}/X.Y-RELEASE/packages/All<ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/%7Barch%7D/X.Y-RELEASE/packages/All> Meanwhile, when either developer B or integrator A says "I have software X that I want to package", we use "pkgbase" to build the FreeBSD package, the RedHat RPM, etc. Do you have your own Makefiles which resemble FreeBSD ports, or are you doing something completely different? That's essentially what pkgbase is. Except instead of building makefiles that build the 3rd-party software, we rely on the 3rd-party compilation process to produce the binaries and have a configuration file (pkgbase.conf) which describes the vendor layout. The vendor layout can be anything and do anything and prepare anything. But the pkgbase.conf says "fileA" in the vendor layout goes to "src/fileA" (where "src/" looks exactly like the un-packed tar-ball, *minus* +CONTENTS -- which is generated from PLIST when you say "make"). Since we're using CVS, people that are familiar with CVS would think it's quite cute that it does a "cvs admin -ko" on files that are pulled from the vendor branch (resulting in keyword expansion to retain its vendor location -- so if a problem happens in the field, we know that the file that should be edited -- based on its keyword expansion -- is over in the vendor "depend" tree which, upon modification, will cause a cascading rebuild of packages for releases we actively maintain [through a sliding window]). So it's very similar to ports, but we need the mechanism of being able to (for the benefit of developer A/B and integrator A above) freeze the software distribution of software X. The ports tree manages that through a collection of centrally located distfiles (which acts as a fall-back for when vendor distfiles dry up). But that method doesn't work so well for us because when we find a bug we often fix it ourselves in our own local branch and then communicate the fix back the vendor (leaving it up to them to incorporate it). The next time around on our migration to a new release, *if* we have made a local patch, producing a custom version, *and* the vendor has not taken a patch, we'll _first_ re-evaluate the new version with a documented test-case or (if that fails) we'll keep the patch in-place. The ports tree moves too fast for us. And many times we find a bug, we end up performing cost/benefit analysis only to find that we can live with the bug *if* the problem has been fixed already. In rare cases, we'll merge a single patch from a future revision to get things going. But we *won't* just blindly take a new version simply because it fixes a bug (as it may introduce new bugs -- unwinding many man-hours of many testers and developers attempting to lead up to a release of a wholly contained full-working software entity). I hope this helps to explain a little more about how we take FreeBSD releases, vet them, and them blast them out to thousands of machines over many customers after many months of testing. Now to say that we don't use ports would be a lie. Sometimes a developer asks for a package that is not available on the release we're shipping. This happened recently. So I went into the ports tree, built the package on 8.1 (which had 99% binary packages from 8.1-RELEASE -- see above URLs) and then I said "make package" and threw it in the repository. ASIDE: For diligence, "make describe" iirc generates my INDEX entry to append. We expect to start with an empty directory, go grab packages that we want (one by one) from FTP. Then put those tarballs into a directory. There's no additional step because we download the INDEX file to the repository too. Local repository is built. What FTP site are you grabbing packages from, the FreeBSD ftp site (or mirrors) or one of your own? We grab from either ftp.f.o or ftp-archive.f.o and plop into an NFS-based repo (which has a structure mirroring that of the ftp{,-archive}.f.o structure). How are you generating the INDEX file? Well, since we prefer to ship 100% with the packages that were cut with X.Y-RELEASE, the INDEX file that's up on the ftp.f.o site does fine (except when we have to inject things from ports -- for example, when developer A needs software X which isn't available on X.Y for the binary release packages -- in-which case we have "make describe"). However, as previously mentioned, the tool that mass-installs the architecture (and release) specific packages out of the repo was written by a different developer (before I came to this company) and *it* doesn't care about INDEX. Do you append to the INDEX file for every package which you sucessfully download from FTP, or are you downloading a previously generated INDEX file? The latter (except for rare cases that I want a newly-injected port package to be browsable/usable by sysinstall -- which uses INDEX). What does your local repository of packages consist of if you are not using FreeBSD ports? The goal each release that we cycle through is indeed to wipe the local repository completely clean of any/all packages that we ourselves do not author/maintain. But it just doesn't happen. As previously mentioned, when we start to track a new FreeBSD release for development/testing purposes [in our labs], we start with binary release packages. When we find that a package (either a trusty one that we've known and loved for over a dozen years; OR a new one that we're still feeling out) doesn't operate quite right, here's what we do... We slurp it into pkgbase (it has a method for doing this; requires only a "./unpack.sh -o bad_package.tbz" -- which not only unpacks the tarball, but puts it in the specific origin sub-directory, reverses the +CONTENTS file into a PLIST, and assembles the Makefile system into the unpacked directory so that you could, if so-desired, go in and say "make" and get the exact same product as what you unpacked; after you say "unpack" you then go in and say "make import" and it imports it into the revision control system -- CVS for us). Once slurped-in, we then tack-on a "_1" to the version in the PLIST file, go fix the problem, commit both the fix and the bumped version back to CVS. Then we say "make tag" and it tags the files with the version specified in the PLIST, and then finally... "make" which produces the *fixed* "bad_package.tbz" which can then be put into the repository. We then go tack-on "_1" to the version that is specified in the arch-specific config for our tool, and that architecture now installs (and will update-to) the fixed package. Packages break. We choose to deal with it silently within our own release cycle. If we notice that something silly hasn't been fixed in the time it took us to jump to the next release, we often release our fix for it (because obviously nobody else came up with a better fix). But, more-oft than not, what happens is that the breakage was a silly thing and somebody fixed it right away (but we couldn't take that fix because it was post-release and we couldn't take the risk of bringing in other things). ASIDE: Even if we did take the fix from above, we'd have to have our own local system *like* what I'm describing so that we can maintain accurate audit-history for our clients that demand a break-fix mentality rather than a break-upgrade mentality. But I want bsdconfig to work with local repositories without having poudriere. You don't have to use poudriere to build a package repository. It is just an optional tool that is there for you to try out, and it works quite well when building a package repository based off of FreeBSD ports. If you don't want to use it, you don't have to. I was *quite* sad when oldports.org<http://oldports.org> went away (not that I used it much, as I could always check out historical revisions of the port Makefiles, etc.). It offered a very simple service... download a tarball of a snapshot of any port for any given revision. For poudriere (or ports for that matter) to work for something like $work, it would have to accommodate inordinate durations of being frozen *and* work well with a static versioning system. And as I was typing that, I think I realized a way to do it... Each release of FreeBSD comes with a "ports.tgz" distribution. Would I be wrong in assuming that this essentially represents the very snapshot-in-time of the ports tree in-which the binary release packages were built? I think I've heard others mention that this is not always guaranteed to be. That aside, I'm not so sure that it would be that beneficial to build the packages ourselves when we so-rarely (and prefer not-to) have the need to remaster packages. In other words, we like leveraging the time someone else took to compile the packages. I think so too. But right now a *lot* of unanswered questions. Possibly, but I think that you are supporting a workflow that people on this mailing list don't fully understand. If you could send out a separate e-mail describing exactly what your workflow is, step-by-step, that might help you get better answers to your questions. Ok, that's a good idea. Hopefully people haven't [already] muddled the two topics of bsdconfig and the separate topic of my workflow @ $work (which requires me to understand the nitty gritty details of +MANIFEST). As you say, I should really spin the latter off into a thread of its own (I gave the disclaimer at the beginning that the former was a side-"thing", but it seems to have taken over this thread which was supposed to be "hey, let's go pkg on bsdconfig"). This thread should stay specific to "pkg_install is dead, long live pkg -- dteske needs help answering some questions about pkg for bsdconfig integration with it". Questions still "out-there" in the thread in that arena are: What are the mirrors (actual addresses) -- one post in this thread asks "hey, is this the right _first-step_" in which I proceed to shift away from FTP and toward HTTP (removing all the swedish, japanese, etc. FTP mirrors and adding at-a-minimum an IPv4 and IPv6 master HTTP mirror). What's the most efficient way (in a UI) to display a list of packages (we've covered rquery; we've covered how rquery works; but I'm thinking to myself... if an "rquery" requires an "update", could I perhaps have the UI somehow present something better than a "waiting for pkg update" box -- e.g., if I go fetch the DB and put it where pkg expects it, I could give progress bars, transfer speeds, completion time, etc. etc. in dialog style). The FreeBSD ports team has been migrating to pkgng for the past few years, so a lot of people understand the FreeBSD ports/packages workflow. because it has been discussed for a few years on multiple mailing lists, and at various BSD conferences like BSDCan. Indeed. However, I'm looking to create the UI for pkgng, and we're bound to run into areas of discussion that haven't been discussed. I know that some have worked on UIs for pkgng, but those UIs have enjoyed a level of integration that I cannot. All the suggestions to use libpkg are not that helpful because the code is not in C. Some of the confusion with others on this list is that you are working on a tool called "bsdconfig" and committing it into FreeBSD-CURRENT, but the workflow you are trying to support is slightly different than what the FreeBSD ports/packages teams have been supporting and adapting their tools and workflow towards. The topic of my own personal quest to understand +MANIFEST and the tarball contents is entirely separate from the bsdconfig topic. How we ended up down this road is by-way of ignoring the sign posted, which said (as a pre-qualifier to the text that introduced this subject) "This is going to be important not for bsdconfig, but $work" (highlight "not for bsdconfig"). Here is the original quote (which garnered flame, but folks like Matthew Seaman took me seriously and -- through it should have been pushed to a new thread -- was answering this little foot-note which I had at the bottom of my July 13th, 10:48P PDT response in this thread): (begin quote) On Jul 13, 2013, at 10:48 PM, Teske, Devin wrote: Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). (end quote) So just to clear things up... [again] bsdconfig needs to blaze the trail on building a UI around pkg. My own personal quest to understand +MANIFEST so I can update my "pkgbase" (which is like ports)... ... not related to bsdconfig. I think that if you better describe your workflow in a separate e-mail, then folks can better recommend how to adapt bsdconfig to the new pkg tools. ... You may need to be open to using things like libpkg inside bsdconfig, if that is the best solution for bsdconfig inside FreeBSD. Not C. I agree with you that having a pkgng transition document on the wiki would be useful for systems integrators. That was actually one of my original points is all. And the very first reply was about ... "8/9 packages?" My response to "pkg_install is dead, long live pkg" (paraphrasing) was actually just about wanting a transition document (of which I would then digest in my integration of bsdconfig). However, I was more than happy to turn the thread into a "let's get bsdconfig updated" thread. I know that many people build products based on FreeBSD and leverage the pkg_XXX tools for building their own products. I have done this myself in the past. Clarifying the transition path for system integrators would be very helpful. I'm still trying to figure out how our eccentricities here @ work can translate back. pkgbase solves our problems (in a nut-shell) because we're slow and we need something that allows us to dance around problems without worrying about what the out-side world is doing. When we come up for air, we rally around the latest software, and then go back under water and don't see the light of day until we've delivered a solid product. It almost seems like *everybody* else wants to do the opposite. Which is whenever they hit a problem, go immediately to the ports three, update, and see if the problem goes away. Unfortunately, with over a million lines of source-code in our product and touching everything from kernel source to user-land _and_ running on 3 classes of hardware (server, workstation, and pedestal), __AND__ interfacing with USB devices, high-end SCSI scanners, and many custom pieces of hardware... ... when we do come up for air in-between major releases, we have zero time to waste in regression testing. Maybe you can help write the transition doc, since you have the perspective? I think that when bsdconfig has been updated to work with pkg... that would be an *excellent* time for me to jump in on a doc-writing event. It will be fresh in my mind *and* we can go through subversion revision history to see exactly what was changed (for example, the communicated list of mirrors would be worth documenting in a Wiki alone, imho). -- Devin _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
