Hi Martin, apk is quite different compared to other package managers. A traditional package manager download a package, saves it to a local cache, verifies it and then extract it. That's at least 3 reads and 2 writes of the data. Apk does this on the fly, so 1 read and 1 write of the data. I believe that it's a so fundamental design choice that opkg would have it very hard to compete for speed. The speed advantage is even bigger when installing over network since the extraction and verification is done wile the network is waiting for data. That means that apk has usually installed the package when an ordinary package manager just has downloaded it and should start to verify and install it.
So there's no secret sauce or awesome optimizations, it's "just" an other architectural design. With that said, there's still some low hanging fruit for apk, for example package generation (my patch is doing 2 reads and 2 writes of all data) and index generation (duplicate reads and one extraction too much) that could have significant speedup. But let's make it possible to use apk at all before starting to tune it. More information: https://www.youtube.com/watch?v=sIG2P9k6EjA BR Fredrik ________________________________________ From: Martin Jansa <martin.ja...@gmail.com> Sent: Tuesday, June 30, 2020 11:54 PM To: Fredrik Gustafsson Cc: Ross Burton; OE-core; tools-cfpbuild-internal Subject: Re: [OE-core] Add package managers as a plugin On Tue, Jun 30, 2020 at 07:01:23PM +0000, Fredrik Gustafsson wrote: > Hi Ross, > those 5 seconds will increase to minutes for my use case and we build a lot > hence I hope this will save us a lot of computer power and engineering time. > For example I've sent numbers on building a bigger image > (core-image-sato-sdk-ptest) > to Alex and Alex: > out.apk.1: 1:13.35 > out.apk.2: 1:13.51 > out.apk.3: 1:13.23 > out.apk.4: 1:14.07 > out.apk.5: 1:13.00 > out.deb.1: 3:49.37 > out.deb.2: 3:50.77 > out.deb.3: 3:51.39 > out.deb.4: 3:53.40 > out.deb.5: 3:53.99 > out.ipk.1: 2:38.99 > out.ipk.2: 2:39.07 > out.ipk.3: 2:35.34 > out.ipk.4: 2:36.15 > out.ipk.5: 2:34.55 > out.rpm.1: 1:58.61 > out.rpm.2: 1:59.42 > out.rpm.3: 1:59.70 > out.rpm.4: 1:58.96 > out.rpm.5: 1:58.11 > > Also consider that if we've a target without python with limited space, rpm > is out > of the question. So the real comparison would be between ipk and apk. Let's > say we can save 80 seconds in each build. Now multiply that with the number of > builds, and we do a lot (I guess we might approach 100 000 builds/week in the > next year). Then this will save 92.5 days build time each week. This is impressive. Have you tried to profile where opkg spends most of the time compared to apk? It would be nice to know if there is something fundamentally different in how they handle packages or if apk is just better optimized. Now with opkg better maintained (thanks Alejandro!) there might be some relatively low hanging optimalizations which could be implemented there as well (for people who didn't implement the rest to switch to apk yet :)). I know at LGE we have some patches (some related to performance as well) which unfortunately weren't ever upstreamed :/, some are at: https://github.com/webosose/meta-webosose/tree/master/meta-webos/recipes-devtools/opkg/opkg Cheers,
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#140152): https://lists.openembedded.org/g/openembedded-core/message/140152 Mute This Topic: https://lists.openembedded.org/mt/75057633/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
