On Thu, Apr 14, 2016 at 5:40 PM Nicholas McCarthy <namcc...@gmail.com> wrote:
> Yeah, this is one of the things I like most about Dockerfiles... the self > documenting aspect. However... have you noticed how many times you end up > dealing with Docker images apart from their Dockerfiles? For a lot of > useful images, it's difficult (maybe impossible?) to track the Dockerfile > down... I've heard it's pretty easy to reverse engineer from the > intermediate layers, but still. With pybombs, the catalog is part of the > image at the command line. Also, for any OOT, there might just be one guy > who's an expert at building the project (the guy who wrote the recipe, > presumably). Anyone can use the resulting recipe to install, so if you > want to support a Docker image with dependencies in the pybombs recipe > world, you don't have to learn how to install them... just use the work > already in pybombs. > Distribute a Dockerfile in the root folder of each OOT. Then you don't have to have any expert build, you just do "docker build ." and pow, it's recreated anytime, anywhere. Whether you use pybombs or you just put the dependencies and build process in the Dockerfile shouldn't matter, the entire recipe should be in there for anyone to recreate. > > Yeah, I should run a real test pitting compression versus > non-compression.... I may have gotten confused with pre-existing image > layers and the source-stripping feature (which definitely helps a lot). If > I can avoid the whole tar and untar, that'd make everything much easier. > The manual decompression takes a lot longer than the Docker decompression, > though, so maybe bzip2 is just doing a better job? This will have to wait > until I have a really good connection, tho. > > Okay, that's true... it does very much depend on what you're > rebuilding.... So it makes sense to isolate code that changes a lot from > stable code in layers. To me, this means installing all the dependencies > for uhd and gnuradio in a layer, then uhd, then gnuradio... Does it make > sense to layer any more? (I mean, obviously, it could matter theoretically, > but in practice? Am I missing a volatile dependency?) > It means that if you're doing daily Gnuradio builds you'll have to pull the whole layer down each day. If you only want daily builds of your OOT module, then no, there's no reason you'd want to layer any more finely grained than that. The incremental cost of updating an OOT should be just a few MB. --n > > Cheers, > Nick M. > > On Thu, Apr 14, 2016 at 7:48 PM Nick Foster <bistrom...@gmail.com> wrote: > >> On Thu, Apr 14, 2016 at 3:54 PM Nicholas McCarthy <namcc...@gmail.com> >> wrote: >> >>> Three points to make about this OOT Docker example. >>> 1) Everything you're doing manually here can be accomplished using >>> pybombs2... and pybombs2 makes itself quite amenable to existing within the >>> Docker image both as an interface to what's already installed on the image >>> and a handy way to install new OOTs and dependencies on top of the image. >>> I would favor using pybombs2 as a basis for gnuradio Docker images... at >>> least that's what I'm doing for myself. >>> >> >> The only argument I have against this is that not doing so (i.e., >> cataloging the dependencies manually in the Dockerfile) sort of obviates >> the need for pybombs, in this particular case. The dependencies and build >> procedures are either encoded in the individual Dockerfiles, which are >> simply part of the OOT project itself, or they're centralized into pybombs >> recipes. Their usefulness overlaps somewhat. >> >> Where I see a clear win for pybombs in a Docker recipe is when you want >> to compile and install multiple OOTs in a single Docker image -- it's >> painful to "stack" targets to create a catchall image (i.e., uhd -> >> gnuradio -> gr-air-modes -> gr-ais...), and annoying to cut and paste >> Dockerfile recipes to concatenate them. It's not an issue from a production >> standpoint since half the reason to use Docker in production is to isolate >> services from each other, one image per service (if all your services >> depend on the same base layer, there's little/no image size overhead). But >> if you're putting together a single catchall image for folks to muck around >> with, a pybombs base layer seems like a great way to do that and still keep >> a sane-ish Dockerfile. >> >> >>> 2) Supporting a pybombs2 Docker image is a great way to ensure that >>> pybombs2 builds from a very, very minimal initial Linux installation. >>> Since pybombs2 is newish and experiencing lots of activity, the project >>> might benefit from anchoring itself to nightly Docker builds. >>> 3) How to handle OOTs could be a bit more complicated if you try, as I >>> have, to reduce image size using the pybombs2 "deploy" mechanism. Deploy >>> lets you do two nice things... it optionally strips all the src out of your >>> build automatically (a significant weight reduction step, as you note, but >>> installed .h files remain... sort of like a dev package). It also >>> compresses the rest of your build. As long as you install and then >>> compress in the same Docker RUN call, your UFS layer is small, too. The >>> result saves GBs over the wire (I think?). When you're using Docker to >>> deploy code, image sizes over the wire can be a big deal, and I'd love for >>> it to be convenient enough to start each day off by docker pulling the >>> latest build. The "pybombs" image I posted is a vanilla pybombs2 install >>> with uhd forcebuilt and the source stripped (so it's pretty much a complete >>> gnuradio build, gui and all)... it's ~1.5G to download (Again, I think? >>> Sometimes you have hidden image layers that make things appear smaller than >>> they actually are.) That's big in Docker world from what I've seen, but >>> it's not much of an inconvenience. ~2.5G for the version retaining the >>> source code is a little more glaringly bad. Fortunately, unless you're >>> working on code in the uhd or gnuradio repos, the 1.5G version is totally >>> fine. >>> >> >>> For me, these are the "natural" GR versions to support via Docker... I'd >>> be interested in a version more like yours that strips the gui and other >>> modules, too. Maybe it's worth cutting uhd in an image, but 200MB doesn't >>> sound worthwhile at that cost. >>> >>> I'm interested in experiences/opinions regarding the use of pybombs >>> deploy to compress between intermediate UFS layers. Am I somehow >>> overestimating the savings in image size? Does it seem too complicated, >>> and/or am I overlooking an easier way? Maybe no one cares about image >>> size? (Actually, I know some people who definitely do care, or at least >>> they claim to care. I'm pretty sure I will, too.) >>> >> >> I'd be really interested to see what the OTW savings from deploy are! >> That seems pretty useful, generally speaking. But remember that Docker >> Registry already implements image compression over the wire for each of the >> UFS layers it's sending. So I'd be a little concerned that the additional >> compression step isn't helping actual transfer times, although optionally >> removing the src will certainly help. >> >> >>> >>> I don't think building the distribution in explicit layers is very >>> helpful. It's something that seems useful when you hear a description of >>> the UFS, but it seems to wind up being a waste of time. One or several GR >>> images makes sense as a basis for images building OOTs, but what the hell >>> are you going to do with an image that's run pybombs install boost except >>> run pybombs install gnuradio? >>> >> >> Only the incremental layers which have changed need to be sent over the >> wire. This can be pretty painful if you're building Gnuradio monolithically >> (i.e., in a single RUN command), and means building a distribution in >> explicit layers has possibly the largest effect on over-the-wire >> requirements for users who need daily Docker builds. >> >> --n >> >> >>> >>> Cheers, >>> Nick M. >>> >>> >>> On Thu, Apr 14, 2016 at 2:16 PM Nick Foster <bistrom...@gmail.com> >>> wrote: >>> >>>> I think it would be really helpful for the GNU Radio project to support >>>>>> a standard, basic gnuradio docker install with uhd and grc enabled as >>>>>> well >>>>>> as an example or two to demonstrate sane ways to run OOT modules on top >>>>>> of >>>>>> that image. As Ben mentioned, Docker seems like a pretty >>>>>> energy-efficient >>>>>> way to approach support for systems like Windows and OSX going forward. >>>>>> Not having used boot2docker personally, I won't say that it's necessarily >>>>>> time to retire the live usb image, but I think Docker may evolve quickly >>>>>> into a pretty obvious replacement, if it hasn't already. I also >>>>>> appreciate >>>>>> GNU Radio looking for ways to support users and potential users >>>>>> attempting >>>>>> to build and deploy applications that reach beyond the immediate >>>>>> environment of GNU Radio and its core devs. >>>>>> >>>>> >>>> As far as OOT modules, that's easy. For instance, a Dockerfile for >>>> gr-air-modes could look like this (this is untested, don't get any ideas): >>>> >>>> FROM bistromath/gnuradio:3.7.8 >>>> MAINTAINER bistrom...@gmail.com version: 0.1 >>>> RUN apt-get install -y python-zmq >>>> WORKDIR /opt/gr-air-modes >>>> COPY . /opt/gr-air-modes >>>> RUN mkdir build \ >>>> && cd build \ >>>> && cmake ../ \ >>>> && make -j4 \ >>>> && make install >>>> >>>> ...that's more or less the whole thing, although this particular >>>> example is broken for a couple of reasons (no Qt in my base layer, other >>>> missing prerequisites). It might be nice to include a Dockerfile template >>>> in the OOT example. The nice part about doing OOT modules in this manner is >>>> that Gnuradio users could potentially never have to compile Gnuradio -- >>>> just write their OOT and base its Dockerfile upon a precompiled Gnuradio >>>> base layer. Another benefit is bitrot is all but eliminated, as you're >>>> basing your module on top of a versioned base layer rather than master. >>>> >>>> >>>>> >>>>>> One problem we have to face, though, is image size. I'm trying to >>>>>> tackle that problem by compressing the install for transactions over the >>>>>> wire and then uncompressing locally for applications (using pybombs2, of >>>>>> course). This is all a little awkward for docker distribution, but lots >>>>>> of >>>>>> things in docker are a little awkward. Developers could build on top by >>>>>> untarring the prefix, pybombs installing extra recipes (possibly custom >>>>>> recipes) and then using the deploy command again, all within the same >>>>>> Docker "RUN" section. Locally, if you docker build applications >>>>>> beginning >>>>>> with the same commands to untar the image, then all applications can take >>>>>> advantage of that layer (you'll have to untar the base image only one >>>>>> time >>>>>> regardless of how many applications use the base image). Alternatively, >>>>>> you can docker run with cmd or entry set to untar the image (and then, >>>>>> presumably, you'll want to commit the running container locally so you >>>>>> don't have to untar again). >>>>>> >>>>> >>>>>> >>>>> Does anyone have a better idea for bringing image size down without >>>>>> making it impossible to build and deploy OOTs? Those Bistromath images >>>>>> are >>>>>> pretty tiny... I haven't really looked into the Alpine base image, >>>>>> either. >>>>>> >>>>>> >>>>> >>>> The Docker image I put up on Docker Hub is small-ish because it only >>>> includes these components (and their prerequisites): >>>> >>>> -- * python-support >>>> -- * testing-support >>>> -- * volk >>>> -- * gnuradio-runtime >>>> -- * gr-blocks >>>> -- * gnuradio-companion >>>> -- * gr-fft >>>> -- * gr-filter >>>> -- * gr-analog >>>> -- * gr-digital >>>> -- * gr-channels >>>> -- * gr-uhd >>>> -- * gr-utils >>>> -- * gr-wxgui >>>> >>>> It could be a lot smaller if I removed the GR build files (292MB), GR >>>> source files (88MB), and UHD source/build (200MB). That would cut it down >>>> to somewhat more than half its current size. I like having them there >>>> because if I'm working inside the environment I can compile changes >>>> incrementally. For a pure deployment system, though, they're unnecessary. >>>> >>>> It's possible, albeit messy, to build a Gnuradio distribution in layers >>>> and tag the individual layers separately. Because each command in a >>>> Dockerfile produces an incremental UFS layer, if you can break the >>>> compilation of Gnuradio into separate commands for each component in the >>>> Dockerfile, then you can tag the various incremental layers to build >>>> different composite Gnuradio distributions. It's probably simpler just >>>> to provide a "bells-'n-whistles" version and a "bare bones" version. >>>> >>>> If you like, I can see just how small I can reasonably get things. I'd >>>> argue, though, that a one-time, couple-of-GB download is a reasonable >>>> compromise for the convenience of versioned distribution in all but niche >>>> applications (embedded or offline come to mind). In other words, the >>>> benefit of getting the wire size down probably doesn't outweigh the effort >>>> for most people. >>>> >>>> --n >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Discuss-gnuradio mailing list >>>>>> Discuss-gnuradio@gnu.org >>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Discuss-gnuradio mailing list >>>>> Discuss-gnuradio@gnu.org >>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio >>>>> >>>>
_______________________________________________ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
