Dear Wiki user, You have subscribed to a wiki page or wiki category on "Ant Wiki" for change notification.
The following page has been changed by SteveLoughran: http://wiki.apache.org/ant/Proposals/EnhancedTestReports The comment on the change is: Look in detail at how badly XML-in-Exception.toString() get's processed ------------------------------------------------------------------------------ * No way to attach artifacts such as VMWare images to test results * Only one failure is allowed per test case. (JUnit requires there be only one failure per test case, but other test frameworks e.g. Selenium allow for the possibility of multiple failures per test case.) - Summary: it was good at the time, but as testing has got more advanced, we need to evolve the format (carefully) == Radical Alternatives == @@ -94, +93 @@ === Improved Logging information === - * Retain the current stdout+stderr logs, but also allow people to bind to custom log4j/commons-logging/java.util.logging back ends that grab the raw events and log them as structured XML events (host,process,thread,level,timestamp,text). These would be renderable at different levels. This is not something we'd do automatically as it can change application behaviour. We'd have to provide the custom back-ends for the loggers and offer a switch to turn this on in the task; the switch would set the properties for the forked process (and it would have to be forked) to log through our system. There's an + * Retain the current stdout+stderr logs, but also allow people to bind to custom log4j/commons-logging/java.util.logging back ends that grab the raw events and log them as structured XML events (host,process,thread,level,timestamp,text). These would be renderable at different levels. This is not something we'd do automatically as it can change application behaviour. We'd have to provide the custom back-ends for the loggers and offer a switch to turn this on in the task; the switch would set the properties for the forked process (and it would have to be forked) to log through our system. There's an - [http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/LogEntry.java?view=markup example class] in the smartfrog repository to capture log entries [http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/ThrowableTraceInfo.java?view=markup and accompanying exceptions] that can be captured serialized and shared between processes. + [http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/LogEntry.java?view=markup example ] in the smartfrog repository to capture log entries and accompanying[http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/xunit/src/org/smartfrog/services/xunit/serial/ThrowableTraceInfo.java?view=markup exceptions] that can be captured serialized and shared between processes. * XSL style sheets to merge output, ordering by clock (received time) and displaying log messages from different machines/levels in different colour * easy turn on/off display of different levels in viewers; maybe the test runner itself, though log4j.properties and similar can do this (actually, given that java.util logging is such a PITA to set up, we could add helper operations in the <java> task for all to use. @@ -141, +140 @@ MattDoar: I like the idea of being able to define my own test states, but I would like to see perhaps half a dozen common ones predefined. Perhaps: - * Pass + * Pass - * Fail + * Fail - * Error - unable to determine pass or fail, which includes timeout + * Error - unable to determine pass or fail, which includes timeout - * Skipped + * Skipped - * In Progress, aka not-yet-finished + * In Progress, aka not-yet-finished SteveLoughran: I like failure itself to be categorised. So every state can be a pass state, a warning or a failure. Warnings could include "passing, but took 50% longer than usual". Failures can include "tests passed but outside allowed time" "tests passed but memory consumption was over the limit", as well as simple "some assertion failed". Maybe we decouple result name from outcome, with a limited set of outcomes, but an unlimited set of outcome categories. @@ -197, +196 @@ What would this look like? Perhaps and element with URLs to related issues? - = Examples = + == Examples of existing tests == + + === multi-site tests === + + http://people.apache.org/~stevel/cddlm/interop/ current presentation of multi-host interop tests]. This directory contains the results of running the same client against three SOAP endpoints, implementing the set of tests agreed by a standard's working group. + 1. Some of the tests fail 'unimplemented' meaning they haven't been unimplemented; these should really be warnings or skipped tests whose absence is noted, and which all would have the tag 'unimplemented'. + 1. The data was collected on a single machine, 3 separate JVMs each with a different endpoint property set to test against a different remote endpoint. + 1. The tests throw faults whose toString() method return the SOAPFault. This is being embedded into the error text unescaped. For example, in [http://people.apache.org/~stevel/cddlm/interop/ourgrid/org/smartfrog/services/deployapi/test/system/alpine/deployapi/api/creation/0_Api_05_system_create_destroy_Test.html one test] the output appears as + {{{ + ns1:Server + Server Error + + Server + Error at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:473) + }}} + If you look at the source, it is more informative, showing that the XSL didnt strip the ?xml declaration, or escape any XML + {{{ + <code> + <?xml version="1.0" encoding="UTF-8"?><br/> + <Fault xmlns="http://schemas.xmlsoap.org/soap/envelope/";><br/> + <faultcode xmlns:ns1="http://xml.apache.org/axis/"; xmlns="">ns1:Server</faultcode><br/> + <faultstring xmlns="">Server Error</faultstring><br/> + <detail xmlns=""><br/> + <stackTrace xmlns:ns2="http://xml.apache.org/axis/"; xmlns="http://xml.apache.org/axis/";>Server<br/> + Error at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:473)<br/> + at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)<br/> + at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)<br/> + at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) <br/> + at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)<br/> + at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) <br/> + at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)<br/> + at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)<br/> + at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)<br/> + at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)<br/> + at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)<br/> + at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)<br/> + at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)<br/> + at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)<br/> + at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)<br/> + at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler + .processConnection(Http11Protocol.java:744)<br/> + at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)<br/> + at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)<br/> + at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)<br/> + at java.lang.Thread.run(Thread.java:595) </stackTrace><br/> + <hostname xmlns:ns3="http://xml.apache.org/axis/"; xmlns="http://xml.apache.org/axis/";>cddlm</hostname><br/> + </detail><br/> + </Fault><br/> + <br/><br/> + at org.smartfrog.projects.alpine.transport.http.HttpTransmitter.transmit(HttpTransmitter.java:184)<br/> + at org.smartfrog.projects.alpine.transport.Transmission.call(Transmission.java:128)<br/> + at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)<br/> + at java.util.concurrent.FutureTask.run(FutureTask.java:123)<br/> + at org.smartfrog.projects.alpine.transport.DirectExecutor.execute(DirectExecutor.java:32)<br/> + at org.smartfrog.projects.alpine.transport.TransmitQueue.transmit(TransmitQueue.java:106)<br/> + at org.smartfrog.projects.alpine.transport.Session.queue(Session.java:204)<br/> + at org.smartfrog.services.deployapi.alpineclient.model.PortalSession.beginCreate(PortalSession.java:109)<br/> + at org.smartfrog.services.deployapi.alpineclient.model.PortalSession.create(PortalSession.java:138)<br/> + at org.smartfrog.services.deployapi.test.system.alpine.deployapi.api.creation.Api_05_system_create_destroy_Test. + testCreateDestroySystem(Api_05_system_create_destroy_Test.java:39)<br/> + </code> + }}} + + What we are seeing here, then, is a Java Exception that contains a SOAPFault raised by Axis1.x at the far end of an HTTP link, but most of this is lost in the textual report. While adding custom XSL for SOAPFaults is probably out (esp. because JAX-WS 2.0 always tries to hide them), we should present the returned XML as (a) it's informative and (b) what we do now is a security risk. What if an Exception.toString() generated HTML with script tags? + + + + + = Examples of Extended tests = Examples of what a new Test Report could look like. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
