Greetings,
Here is the post-mortem and lessons learned review for Linaro release 11.09.
Thanks to the teams who have contributed to this.
Release Review: 2011.09
Android
Highlights
Earlier Toolchain release date was very helpful.
Tracking toolchain tip was helpful.
LTs were more responsive to bugs.
Issues
BPs can sometimes be unclear. What to do and and why.
Suggestions:
Make clear where the BP comes from - traceability to
requirement.
Be prudent with Headline and Done criteria.
Make sure BPs are gardened regulaly and WIs are updated.
LTs and WGs are not agreeing with all our plans that involve them.
Suggestions:
PM to add affected groups to each BP and Bug and talk one-on-one
with them
and their PM at the start of the cycle (maybe co-team kickoffs)?
Some tasks could not be completed in the cycle.
Suggestions:
If something doesn't work after a week of effort, lets regroup
on it.
Its hard to see which groups need to work on a BP.
Suggestions:
Assignee must assert that co-workers are subscribed and properly
bugged for progress.
Use the prescribed syntax in WI for describing who does what.
The PM should also be bugging the other teams PM.
iMX53 was difficult.
Suggestions:
We need to gear up with JTAG debuggers.
We need to get the iMX53 LT more engaged.
Sometimes we can't have our way (probably we'd have had less
trouble if we had
started with the LT kernel than with trying to build a
LT/jstultz hybrid).
Get SoC documentation.
Ask LT members for PoCs.
Work the community sites.
Read commit logs for PoCs.
Post on android-building and android-platform.
Blueprints that have unfinished items should not be marked as
'implemented' until a review is done.
This was the case with
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-snowball-a-release.
Work Items
https://blueprints.launchpad.net/linaro-android/+spec/enable-faketsd-on-leb-panda
done - But after release, noted in blueprint
This was a verification issue that fell through the cracks
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-integrate-glmark2
postponed items broken out into a new blueprint - not a bug since
the items need discussion before the work will be done
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-snowball-a-release
Many items here have been postponed, yet the BP is marked
implemented.
https://blueprints.launchpad.net/linaro-android/+spec/update-gerrit-and-lava-integration
postponed items - New blueprint created for 11.10:
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-enhance-gerrit-and-lava-integration
https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-fix-power-top
Bugs to be logged against
Developer Platform
Highlights
Production development cycle since there was more time for development.
CI for daily builds was started.
Evaluation of derived distro worked well.
Debug symbols for kernel delivered: makes tool building easier.
Issues
Work items should be more descriptive in order to be more
understandable.
Some components from the working groups were not delivered on Thursday
Communicating respin: getting better, but a defined process is needed.
Separate upstreaming activities from monthly cycle.
Work Items
Cross Buildable Nano PPA, Natty based did not make the release.
Rationale: technical development issues
Suggestions:
Better estimation and sizing effort
OMAP 4 SPL USB support at U-Boot did not make the release.
Rationale:
Partial Delivery?
Suggestions:
Star Rating process definition and documentation did not make the
release.
Rationale: This blueprint has been stagnating
Partial Delivery?
Suggestions:
CI with the Linux Linaro packages was split. Unfinished work items have
been re-targeted
to CI with the Linux Linaro packages - Jenkins and LAVA Integration for
release 11.10.
Issues: poorly planned and estimated, was put off track by resource
dependency.
Bug #865547 has been logged against Cross Buildable Nano PPA, Natty
based
This blueprint was planned poorly and badly estimated.
OMAP 4 SPL USB support at U-Boot has been moved to the backlog since it
has not been started and has been stagnating for a while.
The issue of enabling boot loader testing should be brought to TSC
concerning blueprint CI with the U-Boot-Linaro packages
Requires improved LAVA support
A possible solution could be SPL USB booting support rather than
JTAG
Added Bug #865573 to track postponed item from blueprint Change hwpacks
to use hwpacks v2
Generic Toolchain Improvements for 11.09 has been moved to the backlog
in order to re-plan
Bug 867825 has been logged against postponed item in Resyncing
U-Boot-Linaro with upstream U-Boot (11.09)
Bug 867938 has been logged against blueprint Native Gcc-Linaro packages
Bug 867935 has been logged against blueprint Native Gcc-Linaro packages
and Cross Native Gcc-Linaro packages
Power Management
Highlights
Team is well set with the process, no issue was reported.
Lessons Learned: each month PM and TL set and assign the blueprints and
work items in launchpad then tell the team about them. In another word
buffer the engineers from Launchpad modification.
Issues
Constantly in the planning mode, as soon as a month is over, the next
starts. Maintaining the plan is currently very expensive.
Toolchain
Highlights
More than one person can push release candidates builds, now.One can
push the build and another can analyze the results. Making use of the time
zone difference, this method works great, one can push the builds go home,
others can analyze the build when they login
Issues
Too many bugs before the release, need to address bugs before they
accumulate
Validation
Highlights
Issues
lava-dashboard and lava-server were released later than expected.
lava-scheduler-tool and linaro-python-dashboard-bundle weren't planned
to be released.
Blueprint changes:
https://blueprints.launchpad.net/lava-qatracker/+spec/linaro-platforms-o-lava-qatracker
(milestone target: 2011.09->2011.10, priority: high->medium)
https://blueprints.launchpad.net/lava-dispatcher/+spec/linaro-platforms-o-dispatcher-inject-kernel(priority:
high->low)
Infrastructure
Highlights
Linaro CI build service is now available for wider beta testing for
Linaro engineering teams.
Issues
https://blueprints.launchpad.net/linaro-ci/+spec/all-ci-jobs-to-bzr
Several blocked items - created bug 869419 to cover it
Several postponed items - covered in bug 859473
https://blueprints.launchpad.net/linaro-ci/+spec/jenkins-maintenance-upgrade-1not
started, retarget to 11.10
CI kernel fails to reboot successfully: Bug #860556, re-targeted to
11.10
Release
Highlights
Freescale and ST-Ericsson LT have notified their release, respinned
their hwpacks and asked for approval.
Overall the release is running smoothly. Each monthly release is going
better than the previous one.
Issues
Android: Linaro Android RC images announced later than expected. The
URLs to the images should
be communicated as we need them for the call for testing. Monday 16:00
UTC should be targeted.
Graphics WG: glproxy wasn't released as expected.
LT: Kernel has been released without notification. Tarball
created/released while a tag wasn't available.
I'm not even sure the source code tarball is the one used in the hwpack.
Developer Platform: TI LT kernel packaged from git tag. respinned
lt-panda hwpack without
approval. u-boot-linaro was released later than expected. last minute
integration issues with
u-boot-linaro (regression spotted with RC).
Validation: lava-dashboard and lava-server were released later than
expected. lava-scheduler-tool
and linaro-python-dashboard-bundle weren't planned to be released.
Toolchain WG: cortex-strings release delayed. It required to poke TL/PM
to get a status.
General:
Still a lot of work to do on QA and release testing. We need
lava-qatracker and mandatory testing.
Headlines came late in the cycle. They should be defined during the
planning phase, right after the release.
Ubuntu-build jobs should be scheduled at 16:00 UTC instead of 00:00
UTC.
Image respin should be communicated (including rationale, bugs,
etc...).
It's like domino: if components aren't released in time, Platform
have less time to resolve issues spotted on integration.
Release Team should be in the loop for each topic affecting the
release: delay, issues, respin, etc...
Lessons Learned
Monthly Release
Pm & TL do most of the Launchpad work (leg work)
better team acceptance of the process
continue doing until team is ready to handle themselves
roles
Make sure every role has a backup, has 2 people, that can do it
better coverage, when the main responsible is on leave
bugs
Too many bugs to handle before the release
should address bugs continuously and let them accumulate
Monthly Release
Post-mortem came late
There was not enough time to deliver the post-mortem quickly after the
release
Other issues delayed
Start the post-mortem process earlier within the release cycle
--
Linaro Release Team
Linaro.org | Open source software for ARM SoCs
_______________________________________________
linaro-dev mailing list
linaro-dev@lists.linaro.org
http://lists.linaro.org/mailman/listinfo/linaro-dev
