On Tue, Sep 19, 2017 at 6:51 PM Andrea Frittoli <andrea.fritt...@gmail.com> wrote:
> Dear Bats, > > thanks everyone for participating to the Queens PTG - in person and > remotely - and for making it a successful and enjoyable event. > > Here's a summary of what we discussed and achieved during the week. > This report is far from complete - please complement it with QA related > stories I may have missed. > > Unfortunately I ran out of bat stickers, I hope to have some more for the > next time we meet :) > > Pike Retrospective > ------------------------- > We experimented this time with running a retrospective [0]. > The input for the retrospective was almost exclusively from members of the > QA team, which means that either the retrospective was poorly advertised by > me, or that we did a nice job for the community during the QA cycle. I hope > the latter :) > > The outcome of the retrospective was mostly 'yay' for tasks completed, and > 'next cycle' for things we did not have time to work on. > A few extra things that came up were: > - bug triage and fix: we may need a bug czar and more automation to stay > on top of the bug queue > - elastic recheck categorisation: we may need an e-r czar to ensure we > don't leave gate failures and races uncategorised and accumulating > - meetings are mostly attended in the APAC time zone and very seldom > attended by non-qa folks. We will consider shortening / reducing them and > complementing them with QA office hours > > Monitoring the Gate > -------------------------- > At the beginning of the Pike cycle we had a number of stability issues in > the gate. > To prevent issues from accumulating over time we discussed a few ideas > about monitoring the status of the gate and providing more feedback to > reviewers to help catch patches that may introduce issues [1]. > There are a few things that can be done relatively easily (or that we > already have) in terms of data collection: job duration and failure rates, > aggregated dstat data, resources created by tests. > We miss OpenStack Health contributors to create new views for this data. > Links from gerrit and zuul dashboard into OpenStack Health would help > making the data discoverable. > > Tempest > ------------ > A large chunk of the patches required to mark test.py as a stable > interface for plugins was merged during the summit. > It was good to have them merged during PTG so it was easier to fix > resulting issues in Tempest plugins - at least Manila and Sahara needed a > small patch. > There are two patch series left [2][3] which should have very little > impact on plugins - we'll try to merge them as soon as possible to avoid > disruptions later in the Queens cycle. > > We worked with a few project teams on the goal to split Tempest plugins to > a dedicated repo. > The step by step process [4] linked to the goal includes an example > section - we hope to get more and more examples in there from team who > already went through the process. > > Devstack > ------------ > We discussed about devstack runtime. The time to setup an OpenStack cloud > using devstack seems to have increased over time [5][6] - it would be good > to investigate why and see if there is time we can save in the gate and on > each developer laptop :) Between August and September the average runtime > in the gate increased by about 200s. > > Upgrade Testing > ---------------------- > Rolling upgrade testing via grenade is important for project that seek > obtaining the support rolling upgrade tag [7]. > While the scope of Grenade is rather fixed, it should be possible to > support ordering (or relative ordering) in project updates. > > Policy Testing > ------------------ > We discussed what's next for the Queens cycle - support for multi-policy > testing is the largest chunk of work planned for now. > > stestr > ------- > The migration in ostestr to use stestr internally happened shortly before > the PTG [8] and we worked through the PTG to amend any deviation in > behaviour that this may have caused. > Next in the todo list is to run stestr natively in Tempest, bypassing > ostestr completely. > The plan is for this to lead the way for projects to gradually remove the > dependency to ostestr completely. > > HA Testing > --------------- > We talked quite a bit about HA and non-functional testing in general. > Non-functional testing is not a good fit for gate testing, since it's not > as reliable as functional / integration testing and it often produces > results which needs to be interpreted by a human being. > It also has strong dependencies to the deployment tooling and > architecture. Until now most of OpenStack non-functional testing has been > done by vendors and operators using downstream tools. > > SamP and guatamdivgi presented to the QA team a proposal for an Ansible > based framework for HA testing [9]. Plugins will allow to seamlessly port > different test scenario against different cloud architecture, thus > rendering the framework of interest as a general testing tool for > OpenStack. The same concept can be extended to non-functional testing in > general. > > It's not clear yet if any of this could run as part of OpenStack CI. We > hope to see a PoC after a couple of months in the Queens cycle. > The NFV ecosystem would be happy to see publicly available non-functional > tests for OpenStack as well - if we had a common framework for such tests > we'd have an opportunity to attract contributions into tests from them as > well. > > All things Zuul V3 > ----------------------- > Zuul v3 will provide a DB backend for test runs data, which we want to > integrate as a new data source for OpenStack Health, so that we can then > provide a full picture of all job runs in O-H. > > Job names will change with zuul v3. While we found a simple solution to > keep OpenStack Health data functional with Zuul v3 provided metadata, we > will still loose 6 months of historical data in the process of migration, > for which we don't have a solution yet. > > In terms of Zuul v3 jobs, we started working on Zuul v3 native Tempest > jobs [10]. this will be the basis for most jobs that run Tempest against > devstack. > The patch is still WIP, but it already runs the full Tempest run, with > only one test failure which I need to investigate. > > QA Help Room > -------------------- > The help room was helpful (heh, I couldn't resist) mostly on day 1. > It was good to have dedicated time allocated to answer questions, since it > gave us space to open the laptop and work on code directly to answer > specific questions. > > There were no questions asked during day 2. We used the time to join other > sessions and work on QA topics, but perhaps in Dublin we could consider > having a single help room day. > > Cross-Community > ------------------------ > NFV: > It was very interesting to discuss with the OPNFV community about CI and > test processes and architecture. > The aim on both sides to re-use / share tools and best practices, share > test results and avoid duplicate efforts where relevant. > > K8S: > We discussed with dmellado the possibility of having a basic k8s client > hosted in a Tempest plugin [11], that could be used by k8s related > OpenStack project to write OpenStack / k8s scenario tests. The main > advantages of a k8s client written in Tempest format and hosted by > OpenStack would be a uniform interface with the OpenStack Tempest client > and a stable testing API. > > QA Themes for Queens > -------------------------------- > We summarised the topics the QA team will focus on through the Queens > cycle in out queens priority etherpad [11]. > The main etherpad for QA at the PTG in Denver is available as well [12]. > > QA Team Pictures > ------------------------ > I'll forward the pictures as soon as I have a high-res copy available :) > Team Pictures: https://photos.app.goo.gl/R6cuI7lAD6YKzwfF2 > > Thanks for reading! > > Andrea Frittoli (andreaf) > > [0] https://etherpad.openstack.org/p/qa-pike-retrospective > [1] https://etherpad.openstack.org/p/qa-pike-gate-performance > [2] > https://review.openstack.org/#/q/topic:test_module_stable+(status:open+OR+status:merged > ) > [3] > https://review.openstack.org/#/q/topic:bp/consistent-service-method-names+status:open+owner:%22Ghanshyam+Mann+%253Cghanshyammann%2540gmail.com%253E%22 > > [4] https://etherpad.openstack.org/p/tempest-separate-plugin > [5] > http://lists.openstack.org/pipermail/openstack-dev/2017-September/122271.html > [6] > http://status.openstack.org/openstack-health/#/test/devstack?resolutionKey=day&duration=P6M > > [7] > https://governance.openstack.org/tc/reference/tags/assert_supports-rolling-upgrade.html > [8] > http://lists.openstack.org/pipermail/openstack-dev/2017-September/122135.html > > [9] https://etherpad.openstack.org/p/qa-queens-ptg-destructive-testing > [10] https://review.openstack.org/#/c/504246/ > [11] https://etherpad.openstack.org/p/qa-queens-priorities > [12] https://etherpad.openstack.org/p/qa-queens-ptg >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
