Hi Stephen, On 23 November 2015 at 18:45, Simon Glass <s...@chromium.org> wrote: > Hi Stephen, > > On 22 November 2015 at 10:30, Stephen Warren <swar...@wwwdotorg.org> wrote: >> On 11/21/2015 09:49 AM, Simon Glass wrote: >>> Hi Stephen, >>> >>> On 19 November 2015 at 12:09, Stephen Warren <swar...@wwwdotorg.org> wrote: >>>> >>>> On 11/19/2015 10:00 AM, Stephen Warren wrote: >>>>> >>>>> On 11/19/2015 07:45 AM, Simon Glass wrote: >>>>>> >>>>>> Hi Stephen, >>>>>> >>>>>> On 14 November 2015 at 23:53, Stephen Warren <swar...@wwwdotorg.org> >>>>>> wrote: >>>>>>> >>>>>>> This tool aims to test U-Boot by executing U-Boot shell commands >>>>>>> using the >>>>>>> console interface. A single top-level script exists to execute or attach >>>>>>> to the U-Boot console, run the entire script of tests against it, and >>>>>>> summarize the results. Advantages of this approach are: >>>>>>> >>>>>>> - Testing is performed in the same way a user or script would interact >>>>>>> with U-Boot; there can be no disconnect. >>>>>>> - There is no need to write or embed test-related code into U-Boot >>>>>>> itself. >>>>>>> It is asserted that writing test-related code in Python is simpler >>>>>>> and >>>>>>> more flexible that writing it all in C. >>>>>>> - It is reasonably simple to interact with U-Boot in this way. >>>>>>> >>>>>>> A few simple tests are provided as examples. Soon, we should convert as >>>>>>> many as possible of the other tests in test/* and test/cmd_ut.c too. >>>>>> >>>>>> >>>>>> It's great to see this and thank you for putting in the effort! >>>>>> >>>>>> It looks like a good way of doing functional tests. I still see a role >>>>>> for unit tests and things like test/dm. But if we can arrange to call >>>>>> all U-Boot tests (unit and functional) from one 'test.py' command that >>>>>> would be a win. >>>>>> >>>>>> I'll look more when I can get it to work - see below. >>>> >>>> ... >>>>> >>>>> made it print a message about checking the docs for missing >>>>> requirements. I can probably patch the top-level test.py to do the same. >>>> >>>> >>>> I've pushed such a patch to: >>>> >>>> git://github.com/swarren/u-boot.git tegra_dev >>>> (the separate pytests branch has now been deleted) >>>> >>>> There are also a variety of other patches there related to this testing >>>> infra-structure. I guess I'll hold off sending them to the list until >>>> there's been some general feedback on the patches I've already posted, but >>>> feel free to pull the branch down and play with it. Note that it's likely >>>> to get rebased as I work. >>> >>> OK I got it working thank you. It is horribly slow though - do you >>> know what is holding it up? For me to takes 12 seconds to run the >>> (very basic) tests. >> >> It looks like pexpect includes a default delay to simulate human >> interaction. If you edit test/py/uboot_console_base.py ensure_spawned() >> and add the following somewhere soon after the assignment to self.p: >> >> self.p.delaybeforesend = 0 >> >> ... that will more than halve the execution time. (8.3 -> 3.5s on my >> 5-year-old laptop). >> >> That said, even your 12s or my 8.3s doesn't seem like a bad price to pay >> for some easy-to-use automated testing. > > Sure, but my reference is to the difference between a native C test > and this framework. As we add more and more tests the overhead will be > significant. If it takes 8 seconds to run the current (fairly trivial) > tests, it might take a minute to run a larger suite, and to me that is > too long (e.g. to bisect for a failing commit). > > I wonder what is causing the delay? > >> >>> Also please see dm_test_usb_tree() which uses a console buffer to >>> check command output. >> >> OK, I'll take a look. >> >>> I wonder if we should use something like that >>> for simple unit tests, and use python for the more complicated >>> functional tests? >> >> I'm not sure that's a good idea; it'd be best to settle on a single way >> of executing tests so that (a) people don't have to run/implement >> different kinds of tests in different ways (b) we can leverage test code >> across as many tests as possible. >> >> (Well, doing unit tests and system level tests differently might be >> necessary since one calls functions and the other uses the shell "user >> interface", but having multiple ways of doing e.g. system tests doesn't >> seem like a good idea.) > > As you found with some of the tests, it is convenient/necessary to be > able to call U-Boot C functions in some tests. So I don't see this as > a one-size-fits-all solution. > > I think it is perfectly reasonable for the python framework to run the > existing C tests - there is no need to rewrite them in Python. Also > for the driver model tests - we can just run the tests from some sort > of python wrapper and get the best of both worlds, right? > > Please don't take this to indicate any lack of enthusiasm for what you > are doing - it's a great development and I'm sure it will help a lot! > We really need to unify all the tests so we can run them all in one > step. > > I just think we should aim to have the automated tests run in a few > seconds (let's say 5-10 at the outside). We need to make sure that the > python framework will allow this even when running thousands of tests.
BTW I would like to see if buildman can run tests automatically on each commit. It's been a long-term goal for a while. Regards, Simon _______________________________________________ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
