> On Jan 12, 2015, at 12:49 PM, Tobias Oberstein <tobias.oberst...@tavendo.de> > wrote: > >>> Probably the test machine was changed, a new version of OpenSSL or >> pyOpenSSL, or something else? >> >> One of those things. There is no infrastructure in place for identifying >> events >> which impact the performance testing infrastructure. The only performance > > Yes, this is an important point: track changes in infrastructure (everything > that might > have an influence, but is outside the tested code). > >> testing environment is a very old mac mini still running Snow Leopard, which > > omg;) > >>> I'd say: the infrastructure aspects when doing performance tests do >> matter. To the degree that performance results are of very limited value at >> all, if the former aspects are not accounted for. >> >> I don't think the results that we have presently are worth much at all. My >> point was mostly that there is some infrastructure which is halfway usable, >> and so you don't have to start from scratch. If you could take over this > > You mean taking over the code "as is" > > http://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/files > > or the task in general (Twisted benchmarking)?
Both, I think. I'd really prefer it if you could start with the existing benchmarks, incrementally maintain them (perhaps beginning by porting to Github, we seem to be having pretty good success with travis) and probably eventually replace them wholesale, than replace them from the beginning. >> project (I am pretty sure at this point there is nobody to take it over >> *from*, > > We are currently developing performance test infrastructure for Crossbar.io - > naturally, > it is eating it's own dog food: the infrastructure is based on Crossbar.io > and WAMP to > orchestrate and wire up things in a distributed test setup. > > We could extend that to test at the Twisted(-only) level. Need to think about > how > that fits into "overall strategy", as the Crossbar.io perf. test stuff isn't > open-source. > > The testing hardware above (mac, no real network) is insufficient for what I > need. > I'm thinking about buying and setting up 2 more boxes for Linux. Keep in mind that a performance testing environment should be scalable. Others may have different environments they care about. Building your specific environment would be tremendously useful, but it would be even more useful to build it in a way that others can compare in their own hardware setups. > Rgd. Codespeed (https://github.com/tobami/codespeed), which seems to be used > by speedcenter.twistedmatrix.com: I have issues here as well. > > E.g. I need latency histograms, but this seems unsupported (benchmark results > can > only have avg/min/max/stddev). For me, this isn't "nice to have", but > essential. > Throughput is one thing. Constistent low latency a completely different. The > latter is > much much harder. > > But what is the "interface" between test cases from "twisted-benchmarks" to > codespeed? Codespeed runs the benchmark, and they print out this stuff: https://bazaar.launchpad.net/~twisted-dev/twisted-benchmarks/trunk/view/head:/benchlib.py#L12 POSTing them via JSON would be nicer, structured data is great. > This > > https://github.com/tobami/codespeed#saving-data > > seems to suggest, performance test results are HTTP/POSTed as JSON to > codespeed. > > And codespeed is then only responsible for visualization and web hosting, > right? > > I think we can find something better for that part. > >> (And if you care a lot about performance in a particular environment you >> could set it up in that environment and get attention for it :)). > > Yes, in particular that very last one is a factor to justify efforts;) > Anything like having > a promo logo or similar - that would be an argument to invest time and > material. > I will seriously contemplate .. need to align with strategy/available time. You should probably contact tsf@ with these concerns :). > We already host FreeBSD buildslaves for both Twisted and PyPy. That might be > another > synergy (hosting the latter on that same boxes). > >> You should also have a look at the existing benchmark suite and potentially >> look at maintaining / expanding that as well. > > I will try to integrate some of this into our upcoming perf. infrastructure. > > /Tobias > >> >> Thoughts? >> >> -glyph >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
