Hi James, you are right, this is another reason that parsing is slow. (See JENKINS-10097, JENKINS-10596)
Currently, when the parser detects relative names it guesses absolute paths. The current implementation is quite dumb and very slow for large projects. A volunteer tried to implement a faster solution for this problem, however that code still is not ready for a public release. Ull On 02/10/2012 04:11 PM, Nord, James wrote: > > Hi Ulli, > > > > I tried to find the offending lines in order to get a sample – but in > doing this I also went to look at the full detail of the error in the > source file and was met with the following: > > > > Copying the source file '[INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java' > from the workspace to the build folder > '/jenkinsdir/data/jobs/CMS/jobs/vcm/jobs/vcm-app/jobs/commit/builds/2012-02-08_15-04-06/workspace-files/1fb0ab88.tmp' > on the Hudson master failed. > > Seems that the path is relative, however an absolute path is required > when copying the sources. > > Is the file 'ClassificationMediaDataCreator.java' contained more than > once in your workspace? > > Is the file '[INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java' > a valid filename? > > If you are building on a slave: please check if the file is accessible > under '$HUDSON_HOME/[job-name]/[INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java' > > If you are building on the master: please check if the file is > accessible under '$HUDSON_HOME/[job-name]/workspace/[INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java' > > hudson.util.IOException2: remote file operation failed: [INFO] > [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java > at hudson.remoting.Channel@2f891745:cabci-cf-rh53-x64-10 > > at hudson.FilePath.act(FilePath.java:754) > > <snip> > > Caused by: java.io.FileNotFoundException: [INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java > (No such file or directory) > > at java.io.FileInputStream.open(Native Method) > > at java.io.FileInputStream.<init>(FileInputStream.java:138) > > <snip> > > > > I wonder if there are multiple reasons then why this is slow – trying > to find files that don’t exist by recursively looking form them – and > regexp parsing? > > As this was a release build the files will be contained twice in the > workspace – once for the initial checkout and once for the subsequent > release checkout. > > > > What I see in the console for javadoc issues (maven2 build) is > > > > [INFO] [INFO] Generating "JavaDocs" report. > > [INFO] [ERROR] Error fetching link: > http://someserver/maven_sites/somepath/package-list. Ignored it. > > [INFO] Loading source files for package a.b.c... > > [INFO] Loading source files for package x.y.z... > > [INFO] Constructing Javadoc information... > > [INFO] Standard Doclet version 1.6.0_22 > > [INFO] Building tree for all the packages and classes... > > [INFO] Generating /path/to/some/file.html... > > <above repeated for each file> > > [INFO] Building index for all the packages and classes... > > [INFO] Generating /path/to/some/file.html. > > <above repeated again > > > [INFO] Building index for all classes... > > [INFO] Generating /path/to/some/file.html. > > <above repeated again > > > [INFO] Note: Custom tags that could override future standard tags: > @todo. To avoid potential overrides, use at least one period character > (.) in custom tag names. > > [INFO] Note: Custom tags that were not seen: @todo > > [INFO] 3 warnings > > [INFO] [WARNING] Javadoc Warnings > > [INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ClassificationMediaDataCreator.java:65: > warning - @return tag has no arguments. > > [INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/ProgramClassificationAssociationDataGatherer.java:101: > warning - @param argument "em2" is not a parameter name. > > [INFO] [WARNING] > /jenkinsdir/workspace/jobname/target/checkout/som/path/to/a/file/classification/PublishClassificationDataCreator.java:30: > warning - @return tag has no arguments. > > > > > > Note the [WARNING] in this case is inside the [INFO] as the maven > invocation performing javadoc is the release maven process. > > > > So for this version of javadoc it looks like*“.**\w\.java: warning –“ > – so again I would think that “warning” could be used the same as > javadoc (although 1.7 and pre 1.6 vms may be different and it may be > different still if you don’t use maven?) > > > > The large log was actually a mistake on my part – as I forgot to add > “-Dmaven.test.redirectTestOutputToFile=true” and the tests run with > debug log enabled J > > So I can fix my immediate issue by sorting out my own mess – but I > thought there may be some other issue lurking in the background. > > > > /James > > > > *From:*jenkinsci-users@googlegroups.com > [mailto:jenkinsci-users@googlegroups.com] *On Behalf Of *Ullrich Hafner > *Sent:* 10 February 2012 13:04 > *To:* jenkinsci-users@googlegroups.com > *Subject:* Re: warnings plugin > > > > The parsing depends on the size of the file to read. Each line is > matched with a regular expression. The complexity of these expressions > is quite different for different parsers. > > The Java parser has an optimization step: it using a string matcher > for each line that detects the existence of the word "warning". Only > lines containing that text will be matched with the regular expression > afterwards. I don't remember the syntax of JavaDoc warnings, is there > a similar string that can be used? > > And 750MB is a really large log file :-) As a workaround you can try > to write the output of the javadoc processing to a different file. > > Ulli > > > > On 02/09/2012 05:32 PM, Nord, James wrote: > > Hi, > > > > We have the warnings plugin enabled – which looks for javac and > javadoc warnings in the maven2 type build. > > > > All is fine until I ran a release. The warnings plugin took about 9.5 > hours (yes nine and a half) to parse the console to find the javadoc > warnings!? > > A build of deploy site-deploy with the same configuration takes a few > minutes. > > > > The main difference is the maven jobs use > -Dmaven.test.redirectTestOutputToFile=true (and as such the console > was large ~700MB) > > However the javac error parsing completed in a few minutes… > > > > (<consoleLogParsers> > > <string>Java Compiler</string> > > <string>JavaDoc</string> > > </consoleLogParsers> > > ) > > > > The parser found 161 javadoc warnings – so not that many :-P > > > > Stack trace from the job is as follows > > "Executor #0 for xxxxxxxx: executing yyyycommit #5" Id=63 Group=main RUNNABLE > > at java.util.regex.Pattern$CharProperty.match(Pattern.java:3694) > > at java.util.regex.Pattern$Curly.match0(Pattern.java:4158) > > at java.util.regex.Pattern$Curly.match(Pattern.java:4132) > > at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556) > > at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556) > > at java.util.regex.Pattern$Branch.match(Pattern.java:4502) > > at java.util.regex.Pattern$GroupHead.match(Pattern.java:4556) > > at java.util.regex.Pattern$Branch.match(Pattern.java:4500) > > at java.util.regex.Pattern$Start.match(Pattern.java:3408) > > at java.util.regex.Matcher.search(Matcher.java:1198) > > at java.util.regex.Matcher.find(Matcher.java:591) > > at > hudson.plugins.warnings.parser.RegexpParser.findAnnotations(RegexpParser.java:81) > > at > hudson.plugins.warnings.parser.RegexpLineParser.parse(RegexpLineParser.java:82) > > at > hudson.plugins.warnings.parser.ParserRegistry.parse(ParserRegistry.java:273) > > at > hudson.plugins.warnings.WarningsPublisher.parseConsoleLog(WarningsPublisher.java:278) > > at > hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:244) > > at > hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:338) > > at > hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27) > > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682) > > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657) > > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:942) > > at > hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:604) > > at hudson.model.Run.run(Run.java:1400) > > at > hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:476) > > at > hudson.model.ResourceController.execute(ResourceController.java:88) > > at hudson.model.Executor.run(Executor.java:175) > > > > > > Is there something massively different between how javadoc and javac > warnings are tracked – I’m unclear why it took so long. > > > > Monitoring showed there was plenty of heap available and although this > is on a vm – it pretty much had one core of a X5690 dedicated to this > parsing (Jenkins itself has 2 vcpus – and the whole system was mostly > idle.) > > > > Jenkins 1.424.2 with 3.26 of the warnings plugin. > > > > /James > > > > ------------------------------------------------------------------------ > > > ************************************************************************************** > This message is confidential and intended only for the addressee. If > you have received this message in error, please immediately notify the > postmas...@nds.com <mailto:postmas...@nds.com> and delete it from your > system as well as any copies. The content of e-mails as well as > traffic data may be monitored by NDS for employment and security > purposes. To protect the environment please do not print this e-mail > unless necessary. > > NDS Limited. Registered Office: One London Road, Staines, Middlesex, > TW18 4EX, United Kingdom. A company registered in England and Wales. > Registered no. 3080780. VAT no. GB 603 8808 40-00 > ************************************************************************************** > > >
