Generally each test should run independent of all the other tests. You should be able to run the tests in any order. Adding the --gtest_shuffle option when running the test should not fail.
You should also be able to add use the --gtest_filter option to limit the test to an individual tests or set of tests for debugging and verification. Now that I had said that I will point out that the policyengine tests don’t follow some of the basic rules of unit testing. The test must be run in order since early tests setup the policy for later tests. I don’t think this is the only test that has this problem. They are written more like test programs than unit tests. I only discovered this myself recently when I tried to clean up a memory leak in the unit tests. When I closed the db file opened by OCInit it caused a chain reaction of failures because the tests are not independent. The Policy Engine tests are not a good example of following good unit testing guidelines. There are other tests that are much better cbortests and stacktests for example. The top answers to this stack overflow question are useful for understanding good unit testing practices: https://stackoverflow.com/questions/61400/what-makes-a-good-unit-test Will these tests be run in the same process? Yes. Unless the developer of the test does something extra to ensure they don’t run in the same process. Will these tests be run sequentially and serially? Yes. Unless you add a run flag when launching the process like --gtest_shuffle the test is run top to bottom. Do the tests in policyengine.cpp run in an independent process? No there is no guarantee that policyengine will have no impact on unit test running from other modules. Since policy engine is writing to a db file that file could be affected by other processes. If it is trying to communicate with another process via IoTivity its possible it could find unexpected client/servers while running. I only recently discovered the issues with policyengine tests myself. Some of them can and should be fixed while others are unavoidable. Some of them can use reasonable programing tactics to avoid. Even given its limitations the policyengine test should be run every commit since they are working as expected and will catch braking changes. From: iotivity-dev-boun...@lists.iotivity.org [mailto:iotivity-dev-boun...@lists.iotivity.org] On Behalf Of Steve Saunders (CableLabs) Sent: Thursday, October 5, 2017 8:29 AM To: iotivity-dev@lists.iotivity.org Subject: [dev] Unit test Questions (depe Hi All, I would like to get a better feel for how the unit tests run with respect to dependence/independence among tests both within the same test module and across test modules. So, for example, I am working on a unit test module that looks like this — policyengine.cpp — TEST(PolicyEngine, ServerInit) { ... result = OCInit(NULL, 0, OC_SERVER); ... } TEST(PolicyEngineCore, Anonymous) { … } TEST(PolicyEngineSvr, Anonymous) { … } TEST(PolicyEngine, ServerStop) { ... result = OCStop(); ... } Questions: Will these tests be run in the same process, i.e will the OCInitI() in the first test apply to the state of the server in subsequent tests? I have some evidence that this is the case Will these tests be run sequentially and serially, i.e. can I assume that the test with OCInit() will be run first, and to completion, prior to the next test being run, and can I assume that all of the remaining tests will have completed before the last test with OCStop() is run? I have some evidence that this is NOT the case Do the tests in policyengine.cpp run in an independent process i.e. can I assume that the OCInit() / OCStop() will have no impact on unit tests running from other modules. I have some evidence that this is NOT the case You insights are appreciated. Kind Regards Steve
_______________________________________________ iotivity-dev mailing list iotivity-dev@lists.iotivity.org https://lists.iotivity.org/mailman/listinfo/iotivity-dev
