[yocto] Managing multiple builds
Hi, I've been struggling a bit with this question. I want to use Yocto to build two+ products with separate dev/prod images for each (dev including debug-tweaks, etc.). I've ruled out separate image recipes because my dev builds need ENABLE_UART on my RaspberryPi and that needs to be set at the conf level (I've got it set conditionally in my base dist conf). Multiconfig looked promising, but I'm not happy about how long the parsing takes to start a build. "--postread" looked nice, but I've barely seen it mentioned so I'm worried that it's not well supported. Basically, what do most people do for controlling their builds? Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Managing multiple builds
It'll be multiple software loads (with different required kernel modules and things like) over at least two machines. These products will do very different things but I want to use the same base and take advantage of OE's sstate, etc. So I want to be clear, I can figure out how to piece things together with bash scripts to accomplish what I want, but I feel like there's a "correct" way to manage multiple builds (that use different distros, machines, environment variables) that I'm missing. (And perhaps I've already stumbled on the "correct way" but it's not as elegant as I'd like so I'm not happy with it yet.) Problem number one is that the build directory doesn't seem to be something that is intended to be source controlled. I found the TEMPLATECONF flag last night so that answers my question of how to source control my sstate mirror locations and etc. I can write instructions that are basically "git pull poky, git pull the layers, 'TEMPLATECONF=... source oe_init_build_env', bitbake add_layers" But then what if you want to build product a or product b? I think what I may have just settled on is to make sure I've got my distros set up correctly (one distro per product) and add in my own DEVBUILD flag to my template local.conf.sample which will do things like turning on debug_tweaks, ENABLE_UART and adding dropbear/etc. And that'll let me just configure things by setting DISTRO, MACHINE and DEVBUILD=1/0. I'm ruling out multiconf. I don't want my builds to take five minutes before I find out that I have a typo in a recipe and I can get the same behavior by just controlling the above. I'll probably set the tmpdir to something like "tmp_DISTRO_MACHINE" to avoid different configs stepping on each other. Does this seem like the proper way to do things? On Wed, Feb 13, 2019 at 11:18 PM Mike Looijmans wrote: > Two products sounds like two machines. Just create a machine.conf for each > product (even if they use similar hardware), then you don't need overrides > elsewhere. > > OE/Yocto is smart enough to figure out what needs to be (re)built. Some OE > projects build the same image(s) for over 40 machines (and counting)... > > On 14-02-19 01:34, Timothy Froehlich wrote: > > > > Hi, I've been struggling a bit with this question. I want to use Yocto > to > > build two+ products with separate dev/prod images for each (dev > including > > debug-tweaks, etc.). I've ruled out separate image recipes because my > dev > > builds need ENABLE_UART on my RaspberryPi and that needs to be set at > the conf > > level (I've got it set conditionally in my base dist conf). Multiconfig > looked > > promising, but I'm not happy about how long the parsing takes to start a > > build. "--postread" looked nice, but I've barely seen it mentioned so > I'm > > worried that it's not well supported. > > > > Basically, what do most people do for controlling their builds? > > Tim Froehlich > > Embedded Linux Engineer > > tfroehl...@archsys.io <mailto:tfroehl...@archsys.io> > > 215-218-8955 > > > > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Managing multiple builds
I'm considering that. It's the kernel build (cmdline) and the image build (RaspberryPi, the config.txt) recipes from the meta-raspberrypi layer. Setting it in a distro.conf, gated behind a "DEVBUILD=1/0" flag should work, I think. Also, please see my response to Mike Looijmans's reply. On Thu, Feb 14, 2019 at 1:00 AM Alexander Kanavin wrote: > On Thu, 14 Feb 2019 at 01:35, Timothy Froehlich > wrote: > > Hi, I've been struggling a bit with this question. I want to use Yocto > to build two+ products with separate dev/prod images for each (dev > including debug-tweaks, etc.). I've ruled out separate image recipes > because my dev builds need ENABLE_UART on my RaspberryPi and that needs to > be set at the conf level (I've got it set conditionally in my base dist > conf). Multiconfig looked promising, but I'm not happy about how long the > parsing takes to start a build. "--postread" looked nice, but I've barely > seen it mentioned so I'm worried that it's not well supported. > > > > Which recipes use the ENABLE_UART setting? You might want to write two > variants of those, and include them into images accordingly. > > Alex > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Managing multiple builds
Thank you, that looks like a very good idea. On Thu, Feb 14, 2019 at 8:28 AM Ulf Samuelsson wrote: > Den 2019-02-14 kl. 10:00, skrev Alexander Kanavin: > > On Thu, 14 Feb 2019 at 01:35, Timothy Froehlich > wrote: > >> Hi, I've been struggling a bit with this question. I want to use Yocto > to build two+ products with separate dev/prod images for each (dev > including debug-tweaks, etc.). I've ruled out separate image recipes > because my dev builds need ENABLE_UART on my RaspberryPi and that needs to > be set at the conf level (I've got it set conditionally in my base dist > conf). Multiconfig looked promising, but I'm not happy about how long the > parsing takes to start a build. "--postread" looked nice, but I've barely > seen it mentioned so I'm worried that it's not well supported. > >> > > > > Which recipes use the ENABLE_UART setting? You might want to write two > > variants of those, and include them into images accordingly. > > > > Alex > > > > Some guys I know solved the problem a little bit differently, and they > did not mind to share. > I created a meta-layer with that and some other ideas. > > * https://github.com/emagii/meta-map-sheriffco > > An example recipe is: > == > SUMMARY = "" > DESCRIPTION = "" > > inherit license-mit > > SRC_URI = " \ > file://class/class-recipe.c \ > file://class/Makefile \ > " > > EXTRA_OEMAKE_class-development = 'CFLAGS="${CFLAGS} -DDEVELOPMENT"' > EXTRA_OEMAKE_class-production = 'CFLAGS="${CFLAGS} -DPRODUCTION"' > EXTRA_OEMAKE_class-release = 'CFLAGS="${CFLAGS} -DRELEASE"' > > do_install-class-development () { > install -d ${D}${bindir} > install -m 0755 class ${D}${bindir}/development > } > > do_install-class-production () { > install -d ${D}${bindir} > install -m 0755 class ${D}${bindir}/production > } > > do_install-class-release () { > install -d ${D}${bindir} > install -m 0755 class ${D}${bindir}/release > } > > BBCLASSEXTEND = "development production release" > == > You now have the same recipe which builds four package variants. > > , > -development, > -production, > -release, > > The base package is not included in any image. > In the development image, you install the -development and so on. > == > Three bbclasses are used, one for each special package. > > Here is the development.bbclass > == > # Class for use in BBCLASSEXTEND to make it easier to have a single > recipe that > # build and generate packages separately for development and normal images. > # > # Usage: > # BBCLASSEXTEND = "development" > > CLASSOVERRIDE .= ":class-development" > > python development_virtclass_handler () { > # Do nothing if this is inherited, as it's for BBCLASSEXTEND > if "development" not in (d.getVar('BBCLASSEXTEND') or ""): > bb.error("Don't inherit development, use BBCLASSEXTEND") > return > > # Restore BPN > bpn = d.getVar('BPN') > newbpn = bpn.replace('-development', '') > d.setVar('BPN', newbpn) > > # Use default FILESPATH searching for patches and files > filespath = d.getVar('FILESPATH') > newfilespath = filespath.replace('-development', '') > d.setVar('FILESPATH', newfilespath) > } > > addhandler development_virtclass_handler > development_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise" > > == > > -- > Best Regards > Ulf Samuelsson > -- > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Yocto General purpose evaluation
Sounds like you should be using an embedded Linux, since only including the minimum required software will help keep your image small and power usage down. Removing audio, video and usb, including from your kernel config, should drastically reduce your size. You should consider the value of a read only FS, since I'd assume you'll have random power loss due to solar being your primary power source. Yocto/Buildroot will be much better with that. For Yocto vs Buildroot, here's a good place to start for your research: https://lwn.net/Articles/682540/. In my experience, Buildroot is easy, simple, and great as long as you're building one thing. Yocto is much more powerful and if you need more than what Buildroot can provide then you're better off investing your time learning Yocto instead of learning how to modify Buildroot, IMO. On Mon, Feb 18, 2019 at 10:33 AM wrote: > Hello, > > > > I’m looking to evaluate the general purpose/utility of a custom build > Yocto embedded OS. I’m trying to get my head around the benefits of using > such an OS. I’m sure it’s on case by case basis so let me provide my > background and requirements. > > > > Background/requirements > > Currently using Debian OS on my prototype build based on a Beaglebone > green wireless. > > Custom software is constructed in java. > > System utilizes 9 axis accelerometer, vibration sensor, Bluetooth, WiFi, > LTE-M, GNSS, power management (through BB I2c and GPIO channels) > > It runs on a battery and solar. > > Power usage is important. Standby is used and still needs to listen for > vibration sensor, Bluetooth and LTE-m cellular events. > > Only constrained by power usage and data usage. Processor or memory usage > are unconstrained except for in relation to power usage. > > Minimizing data usage if an OS update is needed would be an advantage. > > No audio , video, USB or GUI is used. > > > > > > So would there be a significant benefit seen by using a custom Yocto build > or would a GUIless version of Debian be just as effective? Also considering > Buildroot if that would be just as effective and simpler to execute. If > anyone has any other thoughts or concerns I would love to discuss. Thank > you for your time. > > > > Thank you, > > Andrew Rudd > > President, Aktyon > > 352-256-8086 > > andrew.r...@aktyon.com > > [image: AktyonLogoAndFontSignature] > > > -- > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Storing Sstate in S3 success stories?
I've been spending a bit too long this past week trying to build up a reproducable build infrastructure in AWS and I've got very little experience with cloud infrastucture and I'm wondering if I'm going in the wrong direction. I'm attempting to host my sstate_cache as a mirror in a private S3 bucket, and I believe I have everything configured properly, including exposing the bucket to http requests, since I can wget files that I've previously synced up to the bucket. However if I add in the SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) and barely seems to get anything. The EC2 instance is in the same region as the S3 bucket, roles have been configured properly to allow access, etc. I'm not looking for help debugging this, I just want to know whether I'm right that hosting my sstate in an S3 bucket should work. I've only been able to find one mention of it being done with no reproduction hints. -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Storing Sstate in S3 success stories?
Well, based on the responses above I did some more research and it didn't seem like the file sizes should be causing problems on the scale that I was seeing so I investigated further. I realized that despite my build/sstate directory getting slowly larger, it wasn't actually getting the files and was leaving empty files behind. I tried just changing the https in my SSTATE_MIRRORs line just http and it worked perfectly, pulling down a half gig of sstate before I could tell it was even working. So there was likely some other misconfiguration in our AWS account that caused the https to fail (despite being able to wget individual files using https). Thanks for the responses! On Tue, Feb 26, 2019 at 12:19 AM Erik Hoogeveen wrote: > Hi Timothy, > > The S3 protocol is HTTP(S) based, the overhead per object is quite > significant. This is not much of a problem for large files but the > sstate_cache contains mostly lots of really small files. I think in this > case you’re better of storing the cache on a secondary EBS volume that you > can attach as a regular block device to the EC2 instance. You can swtich on > deletion protection to make it survive EC2 termination. > > Since EBS volumes are quite a bit more expensive than S3 buckets you > could make snapshots to transfer the state between build runs in stead then > you can destroy the EBS volume when nothing is running. > > Alls the documentation about EBS is here > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html > > Cheers, > Erik > On 26 Feb 2019, 02:45 +0100, Timothy Froehlich , > wrote: > > I've been spending a bit too long this past week trying to build up a > reproducable build infrastructure in AWS and I've got very little > experience with cloud infrastucture and I'm wondering if I'm going in the > wrong direction. I'm attempting to host my sstate_cache as a mirror in a > private S3 bucket, and I believe I have everything configured properly, > including exposing the bucket to http requests, since I can wget files that > I've previously synced up to the bucket. However if I add in the > SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) > and barely seems to get anything. The EC2 instance is in the same region as > the S3 bucket, roles have been configured properly to allow access, etc. > > I'm not looking for help debugging this, I just want to know whether I'm > right that hosting my sstate in an S3 bucket should work. I've only been > able to find one mention of it being done with no reproduction hints. > > -- > ___ > yocto mailing list > yocto@yoctoproject.org > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.yoctoproject.org%2Flistinfo%2Fyocto&data=02%7C01%7C%7Cdc66c2ab9da44272220b08d69b8c2262%7C84df9e7fe9f640afb435%7C1%7C0%7C636867423506178146&sdata=TjwcnaFaZQpk9iuxL16%2FosjuqEH2S7aXB16JjBIDGko%3D&reserved=0 > > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Storing Sstate in S3 success stories?
This doesn't seem to be an issue. I have multiple files with plus signs in their names that made it back down to my local cache without requiring a rebuild (including the whole Linux kernel) On Tue, Feb 26, 2019 at 11:35 AM Brian Walsh wrote: > On Mon, Feb 25, 2019 at 8:46 PM Timothy Froehlich > wrote: > > > > I've been spending a bit too long this past week trying to build up a > reproducable build infrastructure in AWS and I've got very little > experience with cloud infrastucture and I'm wondering if I'm going in the > wrong direction. I'm attempting to host my sstate_cache as a mirror in a > private S3 bucket, and I believe I have everything configured properly, > including exposing the bucket to http requests, since I can wget files that > I've previously synced up to the bucket. However if I add in the > SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM) > and barely seems to get anything. The EC2 instance is in the same region as > the S3 bucket, roles have been configured properly to allow access, etc. > > > > I'm not looking for help debugging this, I just want to know whether I'm > right that hosting my sstate in an S3 bucket should work. I've only been > able to find one mention of it being done with no reproduction hints. > > > > A lot of the files end up with plus signs in the name. This causes > problems with retrieving files through http access with S3. S3 > translates all plus signs to spaces, even those in the file path. So > if my-file_v1.0+g1241876 actually exists as named in S3 an http > request for that file will trigger the server to look for > "my-file_v1.0 g1241876" > > I ran into this problem trying to host an opkg repository in S3 for > upgrading. > > It may mostly work for you but there will be many files that it will > never be able to find in your S3 hosted sstate. > > Maybe this has been fixed by AWS. I noticed the problem a year or two ago. > > > https://stackoverflow.com/questions/36734171/how-to-decide-if-the-filename-has-a-plus-sign-in-it#36758133 > https://news.ycombinator.com/item?id=15398804 > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Wic can't find mkfs.btrfs
Been slogging through this one for a bit, would appreciate any helpful suggestions. I'm using mender.io's layer which includes a class that builds a full SD card image including multiple partitions. I'm trying to turn one of those partitions into a btrfs partition and my current error is "A native program mkfs.btrfs... was not found". Wic's source code comments suggest that it's looking for native utils in STAGING_DIR_NATIVE, which bitbake -e says is in workrecipe-sysroot, and when I checked, it does indeed include mkfs.ext4/vfat but not btrfs. I haven't yet found the mechanism that copies these files into the location (thought occurs to me to check the image's logs itself) so I don't know what mechanism is failing to copy the mkfs.btrfs but is working for mkfs.ext*. Any pointers? -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Wic can't find mkfs.btrfs
Ok, I think I'm along the path to figuring this out. Realized that btrfs-tools-native:do_populate_sysroot needed to be explicitly run by something. On Tue, Mar 19, 2019 at 12:40 PM Timothy Froehlich wrote: > Been slogging through this one for a bit, would appreciate any helpful > suggestions. > > I'm using mender.io's layer which includes a class that builds a full SD > card image including multiple partitions. I'm trying to turn one of those > partitions into a btrfs partition and my current error is "A native program > mkfs.btrfs... was not found". Wic's source code comments suggest that it's > looking for native utils in STAGING_DIR_NATIVE, which bitbake -e says is in > workrecipe-sysroot, and when I checked, it > does indeed include mkfs.ext4/vfat but not btrfs. I haven't yet found the > mechanism that copies these files into the location (thought occurs to me > to check the image's logs itself) so I don't know what mechanism is failing > to copy the mkfs.btrfs but is working for mkfs.ext*. Any pointers? > > -- > Tim Froehlich > Embedded Linux Engineer > tfroehl...@archsys.io > 215-218-8955 > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] How to add task only if flag set
I'm working with a read only file system and I'm working on splitting packages so that their config dirs are symlinked in from the persistent data partition. But I'd like to be able to enable or disable this optionally, so that I can still build an image with a r/w file system and a single partition for easier development. I've got a class file with a new task that moves config files to /data and symlinks them back, but I can't figure out how to conditionally enable it. Ideally it'd be enabled only if read-only-rootfs is set in IMAGE_FEATURES. Here's the class if it helps: ORIGINAL_FILES ??= "file1 file2..." SYMLINK_NAMES ??= "sym1 sym2..." do_symlink_to_data () { install -d ${D}/${PERSIST_DIR} ORIG_ARR=(${ORIGINAL_FILES}) SYMLINK_ARR=(${SYMLINK_NAMES}) for i in ${!ORIG_ARR[*]}; do ORIG=ORIG_ARR[i] SYMLINK=SYMLINK_ARR[i] mv ${D}/$ORIG ${D}/${PERSIST_DIR}/$SYMLINK ln -sr ${D}/${PERSIST_DIR}/$SYMLINK ${D}/$ORIG done } FILES_${PN}_append_mender-image += " ${SYMLINK_NAMES}" addtask symlink_to_data after do_install -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Read-only filesystems: Move directory to RW directory and symlink back
Is there a recommended way of doing this? Right now I have a ROOTFS_POSTPROCESS_CMD that moves the directories and symlinks them back, but I'm not sure if I'm doing it exactly right or if there's something built-in. The function just does a bunch of this: install -d ${D}/${persist_dir} mv ${D}/${ORIG} ${D}/${persist_dir}/${LINKNAME} ln -sr ${D}/${persist_dir}/${LINKNAME} ${D}/${ORIG} -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] System service
Looks like it's dropping the service in ${sysconfdir}/init.d which resolves to /etc/init.d. I'm not sure that systemd won't look into init.d for services. The standard place to put them is ${D}/${systemd_system_unitdir}, which resolves to /lib/systemd/system. Also, check the "SYSTEMD_AUTO_ENABLE_pn-ofono" variable for that recipe. The systemd class sets it to "enable" by default, but something else could be overriding it and setting it to "disable". Alternatively, confirm that the service isn't actually starting up and failing. It may be that you're restarting it once the system is fully booted and everything ofono needs is now ready. On Thu, May 2, 2019 at 7:55 AM JH wrote: > Hi, > > The official ofono.inc recipe defines SYSTEMD_SERVICE_${PN} = > "ofono.service", but when I built the image, the system service does > not start ofono daemon automatically, I have to run the ofono daemon > manually. What I could be missing here? Here is the ofono recipes, > sorry if I need to ask ofono mailing list. > > $ cat ofono.inc > HOMEPAGE = "http://www.ofono.org"; > SUMMARY = "open source telephony" > DESCRIPTION = "oFono is a stack for mobile telephony devices on Linux. > oFono supports speaking to telephony devices through specific drivers, > or with generic AT commands." > LICENSE = "GPLv2" > LIC_FILES_CHKSUM = "file://COPYING;md5=eb723b61539feef013de476e68b5c50a \ > > > file://src/ofono.h;beginline=1;endline=20;md5=3ce17d5978ef3445def265b98899c2ee" > > inherit autotools pkgconfig update-rc.d systemd bluetooth > gobject-introspection-data > > DEPENDS = "dbus glib-2.0 udev mobile-broadband-provider-info" > > INITSCRIPT_NAME = "ofono" > INITSCRIPT_PARAMS = "defaults 22" > > PACKAGECONFIG ??= "\ > ${@bb.utils.filter('DISTRO_FEATURES', 'systemd', d)} \ > ${@bb.utils.contains('DISTRO_FEATURES', 'bluetooth', 'bluez', '', d)} \ > " > PACKAGECONFIG[systemd] = > "--with-systemdunitdir=${systemd_unitdir}/system/,--with-systemdunitdir=" > PACKAGECONFIG[bluez] = "--enable-bluetooth, --disable-bluetooth, ${BLUEZ}" > > EXTRA_OECONF += "--enable-test" > > SYSTEMD_SERVICE_${PN} = "ofono.service" > > do_install_append() { > install -d ${D}${sysconfdir}/init.d/ > install -m 0755 ${WORKDIR}/ofono ${D}${sysconfdir}/init.d/ofono > } > > PACKAGES =+ "${PN}-tests" > > RDEPENDS_${PN} += "dbus" > RRECOMMENDS_${PN} += "kernel-module-tun mobile-broadband-provider-info" > > FILES_${PN} += "${systemd_unitdir}" > FILES_${PN}-tests = "${libdir}/${BPN}/test" > RDEPENDS_${PN}-tests = "python3 python3-dbus" > RDEPENDS_${PN}-tests += "${@bb.utils.contains('GI_DATA_ENABLED', > 'True', 'python3-pygobject', '', d)}" > > Thank you. > > - jupiter > -- > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > -- Tim Froehlich Embedded Linux Engineer tfroehl...@archsys.io 215-218-8955 -- ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto