On Fri, Nov 25, 2011 at 08:59:06PM +0100, Koen Kooi wrote: > > Op 25 nov. 2011, om 19:15 heeft Koen Kooi het volgende geschreven: > > > > > Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven: > > > >> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote: > >>> In OE-classic the opkg-make-index cache was working pretty well, > >>> do_rootfs only spent a few seconds doing opkg-make-index on > >>> incremental builds. In the OE-core world the situation is different, > >>> opkg-make-index will reindex every package on each do_rootfs run, so > >>> after a while building images starts taking *really* long, on my > >>> systemd (work/ on ssd, deploy on rotating media) it now takes > >>> ±7minutes just do refresh the opkg indices. > >>> > >>> Since I can't blame sstate for this I suspect that pseudo gets in the > >>> way of the naive caching logic. Is there a simple way to turn off > >>> pseudo when running opkg-make-index? > > > > With a relatively small deploy: > > > > real 6m14.611s > > user 2m45.909s > > sys 0m43.674s > > > >> You could try: > >> > >> PSEUDO_DISABLED=1 opkg-make-index > > > > real 5m25.920s > > user 2m41.279s > > sys 0m41.681s > > > >> or even > >> > >> PSEUDO_UNLOAD=1 opkg-make-index > > > > real 5m5.699s > > user 2m37.283s > > sys 0m37.508s > > > > Now I need to double check to see if the patches are really having effect > > on the opkg caching instead of just not having pseudo overhead. > > Retesting without PSEUDO changes: > > real 5m18.412s > user 2m40.736s > sys 0m41.374s > > So the PSEUDO changes don't seem to do much :(
I can confirm this change here too:
OE om-gta02@shr ~/shr-core $ time /usr/bin/python2.7 \
/OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
-r /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
-p /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
-l /OE/shr-core/tmp/deploy/ipk/armv4t/Packages.filelist \
-m /OE/shr-core/tmp/deploy/ipk/armv4t/
real 11m47.949s
user 4m13.848s
sys 0m17.917s
It hurts even more when someone is building ie for spitz and
linux-kexecboot depends on initramfs image which calls another
opkg-make-index with every image build.
If I replace opkg-make-index from oe-core with old oe-classic version, I
get a lot of messages like:
"Lost field license, MIT-X"
but the time is similar
real 10m23.953s
user 4m10.940s
sys 0m15.833s
And if I do the same completely in shr-unstable (based on OE-classic)
OE om-gta02@shr ~/shr-unstable $ time /usr/bin/python2.7 \
/OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
-r /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
-p /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
-l /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages.filelist \
-m /OE/shr-unstable/tmp/deploy/ipk/armv4t/
real 0m6.469s
user 0m6.056s
sys 0m0.400s
Another difference seems to be in Packages.filelist which were empty in
OE-classic:
OE om-gta02@shr ~/shr-core $ find tmp/deploy/ipk/ -name Packages.filelist -exec
wc -l {} \;
1173 tmp/deploy/ipk/palmpre2/Packages.filelist
1254 tmp/deploy/ipk/palmpre/Packages.filelist
107346 tmp/deploy/ipk/armv4t/Packages.filelist
110557 tmp/deploy/ipk/armv7a-vfp-neon/Packages.filelist
0 tmp/deploy/ipk/Packages.filelist
1365 tmp/deploy/ipk/nokia900/Packages.filelist
812 tmp/deploy/ipk/all/Packages.filelist
1643 tmp/deploy/ipk/om_gta02/Packages.filelist
OE om-gta02@shr ~/shr-core $ cd ../shr-unstable/
OE om-gta02@shr ~/shr-unstable $ find tmp/deploy/ipk/ -name Packages.filelist
-exec wc -l {} \;
0 tmp/deploy/ipk/armv7a/Packages.filelist
0 tmp/deploy/ipk/om-gta01/Packages.filelist
0 tmp/deploy/ipk/om-gta02/Packages.filelist
0 tmp/deploy/ipk/palmpre2/Packages.filelist
0 tmp/deploy/ipk/armv6/Packages.filelist
0 tmp/deploy/ipk/palmpre/Packages.filelist
0 tmp/deploy/ipk/armv4t/Packages.filelist
0 tmp/deploy/ipk/armv4/Packages.filelist
0 tmp/deploy/ipk/armv6-novfp/Packages.filelist
0 tmp/deploy/ipk/Packages.filelist
0 tmp/deploy/ipk/htcdream/Packages.filelist
0 tmp/deploy/ipk/nokia900/Packages.filelist
0 tmp/deploy/ipk/all/Packages.filelist
0 tmp/deploy/ipk/armv7/Packages.filelist
But that's not the cause, because even without Packages.filelist
generation it takes about 3 min (better but it still writes info for
every package).
Weird is that there is only small diff between new and old
opkg.py/opkg-make-index
--- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg.py 2011-05-27
18:35:03.000000000 +0200
+++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg.py 2011-09-01
16:36:35.000000000 +0200
@@ -145,6 +145,7 @@
self.priority = None
self.tags = None
self.fn = fn
+ self.license = None
if fn:
# see if it is deb format
@@ -319,6 +320,12 @@
def get_section(self, section):
return self.section
+ def set_license(self, license):
+ self.license = license
+
+ def get_license(self, license):
+ return self.license
+
def get_file_list(self):
if not self.fn:
return []
@@ -425,6 +432,7 @@
if self.description: out = out + "Description: %s\n" %
(self.description)
if self.oe: out = out + "OE: %s\n" % (self.oe)
if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
+ if self.license: out = out + "License: %s\n" % (self.license)
if self.priority: out = out + "Priority: %s\n" % (self.priority)
if self.tags: out = out + "Tags: %s\n" % (self.tags)
out = out + "\n"
(reverse-i-search)`uNr': diff -^Cr ~/common/conf/shr-core/ conf/
--- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index
2011-05-27 18:35:03.000000000 +0200
+++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index
2011-11-29 12:32:39.000000000 +0100
@@ -197,7 +197,15 @@
names = packages.packages.keys()
names.sort()
for name in names:
- for fn in packages[name].get_file_list():
+ try:
+ fnlist = packages[name].get_file_list()
+ except OSError, e:
+ sys.stderr.write("Package %s disappeared on us!\n" % (name))
+ continue
+ except IOError, e:
+ sys.stderr.write("Package %s disappeared on us!\n" % (name))
+ continue
+ for fn in fnlist:
(h,t) = os.path.split(fn)
if not t: continue
if not files.has_key(t): files[t] = name+':'+fn
Regards,
--
Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com
signature.asc
Description: Digital signature
_______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
