Thanks Ross for your answers. I'm still working on fully understanding shared state so I appreciate your help. Let me follow up on both your answers in a single message.
On Mon, Apr 23, 2018 at 6:38 PM, Burton, Ross <ross.bur...@intel.com> wrote: > On 23 April 2018 at 16:23, Iván Castell <icast...@nayarsystems.com> wrote: >> [...] >> >> Related with that shared state cache, I found some information on a e-book >> (search the text on google to find the source): >> >> "Sharing a shared state cache is possible; however, it needs to be >> approached with care. Not all changes are detected by the shared state cache >> implementation, and when this happens, some or all of the cache needs to be >> invalidated. This can cause problems when the state cache is being shared. [...] > Very curious as to what book said that, because *any* example of that > happening is a bug in the recipe itself. I wouldn't listen to it: the YP > autobuilder has a shared sstate for three distributions * four architectures > * two libc implementations and doesn't have problems. > > Ross > I think when talking about shared state there can be 2 aspects of sharing 1.) sharing over time: Several invocations of bitbake in the same environment (DISTRO, MACHINE etc) 2.) sharing over distros etc. Number 1 is an incremental build (as opposed to clean or full build). Without any doubt that is useful (or really mandatory) for development work. You build, you make little changes, you build, you test, you make more changes... You share state from the previous build, because making a full clean build every time would lead to completely inacceptable cycle times. But when you are "done" with your changes you make an integration build and to my experience that should be a clean (aka full) build. You intentionally do not share any state from a previous build, you don't make an incremental build, because there is always the risk of broken recipes. I mean if there were no broken software, we would all be unemployed by now... Let me give a simple example I experienced my self recently: I added creation of a new file in some task. In my incremental developer build it all worked fine. When I put into a clean integration build the task failed. It turned out that the directory I wrote my new file to in task A was created by a different task B. In my development build task B had already been executed long before I even started to make the recipe modificationto to create a new file, so task A always succeeded. In the clean build task A happened to run first and failed. A simple missing dependency / incorrect task ordering. With parallel clean builds you might not always find it, but with incremental builds it can go undetected forever. It's not only "some book" that mentions the risk of "bad" state, but also the Yocto documentaton: https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#concepts-tips-and-tricks (Especially 4.5.4.2. Invalidating Shared State. Of course the invalidating is easy, but finding what you need to invalidate is hard.) Of course the less something is tested the bigger the risk is. So the bitbake system should be rather fine, commonly used Poky recipes probably too. But when it comes to random layers and our own local recipes and bbappends I would expect the risk to grow a lot. So do you really tell us in the message above that above that Yocto project runs incremental integration builds? Another way to think about it: Shared state is used to to avoid rebuilds. So how could I benefit from having state information in some SSTATE_DIR already? I have to build it anyway, because my build area is empty to begin with. The 2nd aspect, sharing over distros was covered in the other message. On Mon, Apr 23, 2018 at 6:10 PM, Burton, Ross <ross.bur...@intel.com> wrote: > On 20 April 2018 at 11:47, Uwe Geuder <jrswdna...@snkmail.com> wrote: >> But can you share state between distros? Isn't the purpose of distros to >> use different options (variable settings) so the state would always be >> different? > > If the input to the recipe is different then the hashes would be > different so the content won't conflict. You can definitely share > sstate between DISTROs. > > Ross Under the assumption that all recepies were perfect (which they probably aren't in every point of history, see above) I fully believe that the build system would work perfectly when I share the SSTATE_DIR over DISTRO borders (or diferent MACHINE settings). But do I really have any commmon data that would lead to any saving? As you say the hashes would be different. In my thinking all "expensive" state info should be different, because a different DISTRO (or different MACHINE) would lead to different hashes for bascially every item. So in the best case I end up with several distinct sets of state data stored under the same top directory, but that doesn't seem to result in any saving. In the worst case an item goes into the same file, so I end up with "cache thrashing". Build A calculates the state and stores it, build B finds the hashes not matching and puts its new info there. Build A finds the hashes not matching and... Or does the saving come on the native part of the build? There I would guess that common state could occur more commonly. I think it would really help to have a hello world type of Yocto build, where you could manually read through the sstate data to understand how it works exactly. In a real world sized build the amount of data does not really help to understand the concepts. Does anybody have a recommendation what would be the smallest possible Yocto build that makes some sense? Regards, Uwe Geuder Neuro Event Labs Oy Tampere, Finland uwe.gex...@neuroeventlabs.com (Bot check: fix one obvious typo) -- _______________________________________________ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
