[Twisted-Python] Running tests in parallel with trial
Hi, is there a way to tell trial run groups of tests in parallel in different processes? Something like the "-jN" option in zope-testrunner. Or alternatively, what's the recommended way to make better use of multi-core CPUs when running a test suite with trial? Thanks, Free ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Running tests in parallel with trial
On Tue, Aug 10, 2010 at 3:58 PM, Free Ekanayaka wrote: > Hi, > > is there a way to tell trial run groups of tests in parallel in > different processes? Something like the "-jN" option in zope-testrunner. > > Or alternatively, what's the recommended way to make better use of > multi-core CPUs when running a test suite with trial? > In short, no. There's a longstanding ticket about doing this: http://twistedmatrix.com/trac/ticket/1784. It would be pretty easy to kludge something together that worked for parallelizing across cores if you're willing to sacrifice the debugger support and a bunch of the other advanced things that disttrial aim to achieve. * Add a way for trial to run tests based off a list of FQPNs in a file, say, --from-list * Write something that does: * trial --reporter=subunit --dry-run | subunit-ls > all-tests * split all-tests into N sublists however you fancy * trial --reporter=subunit --from-list=sublistN * Then something that aggregates those subunit streams into a real report. jml ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Running tests in parallel with trial
Hey, |--==> On Tue, 10 Aug 2010 16:07:56 +0100, Jonathan Lange said: JL> On Tue, Aug 10, 2010 at 3:58 PM, Free Ekanayaka wrote: >>Hi, >> >>is there a way to tell trial run groups of tests in parallel in >>different processes? Something like the "-jN" option in zope-testrunner. >> >>Or alternatively, what's the recommended way to make better use of >>multi-core CPUs when running a test suite with trial? >> JL> In short, no. There's a longstanding ticket about doing this: JL> http://twistedmatrix.com/trac/ticket/1784. Thanks for the pointer, it looks very promising. JL> It would be pretty easy to kludge something together that worked for JL> parallelizing across cores if you're willing to sacrifice the debugger JL> support and a bunch of the other advanced things that disttrial aim to JL> achieve. JL> * Add a way for trial to run tests based off a list of FQPNs in a JL> file, say, --from-list JL> * Write something that does: JL> * trial --reporter=subunit --dry-run | subunit-ls > all-tests JL> * split all-tests into N sublists however you fancy JL> * trial --reporter=subunit --from-list=sublistN JL> * Then something that aggregates those subunit streams into a real report. That would be indeed easy, and probably acceptable while waiting for disttrial to get its way into Twisted trunk. Cheers, Free ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Running tests in parallel with trial
On 03:07 pm, j...@mumak.net wrote: >On Tue, Aug 10, 2010 at 3:58 PM, Free Ekanayaka >wrote: >>Hi, >> >>is there a way to tell trial run groups of tests in parallel in >>different processes? Something like the "-jN" option in zope- >>testrunner. >> >>Or alternatively, what's the recommended way to make better use of >>multi-core CPUs when running a test suite with trial? > >In short, no. There's a longstanding ticket about doing this: >http://twistedmatrix.com/trac/ticket/1784. fwiw, the code in the branch basically works. The only problem it has is with doctests. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using cProfile instead of hotshot with twistd
This doesn't seem to work either. Also on 8.2.0. -J On Fri, Aug 6, 2010 at 9:42 PM, Paul Goins wrote: > Add --savestats. I'm on 8.2.0, but it probably should still work. > > - Paul Goins > > On 8/7/2010 8:20 AM, Jason J. W. Williams wrote: >> Hi, >> >> I'm having an issue using twistd with --profiler=cprofile. I was using >> hotshot, but there's a bug in loading big profile logs with it so I'm >> trying cProfile instead. >> >> The issue is that with --profiler=cprofile, the --profile= >> flag seems to be ignored. That is no profile file gets created. Any >> help is greatly appreciated. >> >> -J >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using cProfile instead of hotshot with twistd
On 08:55 pm, jasonjwwilli...@gmail.com wrote: >This doesn't seem to work either. Also on 8.2.0. This works for me: twistd --savestats --profiler cprofile -n --profile prof.stats web prof.stats is written and contains a binary dump of the profile data. This also works: twistd --profiler cprofile -n --profile prof.stats web and produces a prof.stats containing a plaintext dump of the profile. I tested on the 8.2 release branch, which should be the same as the 8.2 release. I used Python 2.6 on Ubuntu 9.10. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Using cProfile instead of hotshot with twistd
I'm not sure what's wrong then. Tried your command line and got nothing written. Tried it with 10.1 and same deal. I'll keep hacking at it. -J On Tue, Aug 10, 2010 at 3:20 PM, wrote: > On 08:55 pm, jasonjwwilli...@gmail.com wrote: >>This doesn't seem to work either. Also on 8.2.0. > > This works for me: > > twistd --savestats --profiler cprofile -n --profile prof.stats web > > prof.stats is written and contains a binary dump of the profile data. > This also works: > > twistd --profiler cprofile -n --profile prof.stats web > > and produces a prof.stats containing a plaintext dump of the profile. > > I tested on the 8.2 release branch, which should be the same as the 8.2 > release. I used Python 2.6 on Ubuntu 9.10. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
