Igor, We have UI and CLI integration tests which use fake mode of Nailgun, and we can't avoid using fake threads for them. So I think we need to think how to fix fake threads instead. There is a critical bug <https://bugs.launchpad.net/fuel/+bug/1549750> which is the main reason of randomly failing UI tests. To fix it, we need to fix fake threads behaviour.
2016-03-16 17:06 GMT+03:00 Igor Kalnitsky <ikalnit...@mirantis.com>: > Hey Fuelers, > > As you might know recently we encounter a lot of random test failures > on CI, and they are still there (likely with a bit less probability). > A nature of that random failures is actually not a random, they are > happened because of so called fake threads. > > Fake threads, actually, ain't fake at all. They are native OS threads > that are designed to emulate Astute behaviour (i.e. catch RPC call and > respond with appropriate message). Since they are native threads and > we use SQLAlchemy's scoped_session, fake threads are using a separate > database session, hence - transaction. That leads to the following > issues: > > * Races. We don't know when threads are switched, therefore, we don't > know what's committed and what's not. Some Nailgun tests sends > something via RPC (catched by fake threads) and immediately checks > something. The issue is, we can't guarantee fake threads is already > committed produced result. That could be avoided by waiting for > 'ready' status of created nailgun task, however, it's better to simply > do not use fake threads in that case and simply call appropriate > Nailgun receiver's method directly in the test. > > * Deadlocks. It's incredibly hard to ensure the same order of database > locks in test + business code on one hand and fake thread code on > other hand. That's why we can (and we do) encounter deadlocks on CI, > when test case waits for lock acquired by fake thread, and fake thread > waits for lock acquired by test case. > > Fake threads are became a bottleneck of landing patches to master in > time, and we can't ignore it anymore. We have ~190 tests that use fake > threads, and fixing them all at once is a boring routine. So I kindly > ask Nailgun contrubitors to fix them as soon as we face them. Let's > file a bug on each file in CI, and quicly prepare a separate patch > that removes fake thread from failed test. > > Thanks in advance, > Igor > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Vitaly Kramskikh, Fuel UI Tech Lead, Mirantis, Inc.
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
