Re: RE: Groovy 3 very slow stub generation

I tried to remove the following error alternative of `rparen`, parsing all groovy source files under nextflow module[1] with 4.0.0-SNAPSHOT takes only "14s". https://github.com/apache/groovy/blob/GROOVY_3_0_0_RC_3/src/antlr/GroovyParser.g4#L1232-L1234 *4.0.0-SNAPSHOT antlr4 parser: 14s* result-4.

Re: RE: Groovy 3 very slow stub generation

The current result of parsing all groovy source files under nextflow module[1] with 3.0.0-rc-3 and the distribution built from master. After some rules tweaked, about 39.8% time is saved util now(Formula: (103s - 62s) / 103s = 39.8%). *3.0.0-rc-3 costs 103s* result-3.log

Re: RE: Groovy 3 very slow stub generation

Hi Paolo, As you can see, Sam and I have been trying to improve the performance of the antlr4 parser recently. I believe the performance could be improved a lot, but antlr4 parser will be slower than antlr2 parser as a whole finally due to the following reasons(including but not limited to):

Re: RE: Groovy 3 very slow stub generation

What is the maximum number of lines of code that a source file should contain to prevent perf degradation? On Fri, Jan 31, 2020 at 8:16 PM Daniel.Sun wrote: > I find antlr4 parser parses *big* files much slower than antlr2 parser, > unfortunately nextflow project contains quite a few big files

RE: RE: Groovy 3 very slow stub generation

I find antlr4 parser parses *big* files much slower than antlr2 parser, unfortunately nextflow project contains quite a few big files... One of reasons is antlr2 uses LL(k) and antlr4 uses ALL. As we know, LL(k) will look ahead *limited* count of tokens, but ALL will look ahead *unlimited* count o

RE: RE: Groovy 3 very slow stub generation

Hi Eric, Thanks a lot for your help. The top time-consuming method `closure` is an internal method of the implementation of antlr4. It's hard for us to change. Luckily, Sam Harwell, the co-author of antlr4, is trying to help us tune the performance of the antlr4 parser, too. But Groovy's gra

RE: RE: Groovy 3 very slow stub generation

e.org Subject: RE: RE: Groovy 3 very slow stub generation Here is the flame graph for parsing all groovy source files under nextflow module[1] with the distribution built from master. profile.svg <https://urldefense.proofpoint.com/v2/url?u=http-3A__groovy.3

RE: RE: Groovy 3 very slow stub generation

Here is the flame graph for parsing all groovy source files under nextflow module[1] with the distribution built from master. profile.svg Cheers, Daniel.Sun [1] https://github.com/nextflow-io/nextflow/tree/master/modules/nextflow

RE: RE: Groovy 3 very slow stub generation

Here is the flame graph for parsing all groovy source files under nextflow module[1] with 3.0.0-rc-3 profile-3.svg Cheers, Daniel.Sun [1] https://github.com/nextflow-io/nextflow/tree/master/modules/nextflow - Apache Groovy co

RE: RE: Groovy 3 very slow stub generation

Here is the flame graph for parsing all groovy source files under nextflow module[1] with the distribution build from master branch[2] profile.svg Cheers, Daniel.Sun [1] https://github.com/nextflow-io/nextflow/tree/master/modules/ne

RE: RE: Groovy 3 very slow stub generation

I created a branch to reproduce the performance issue: https://github.com/apache/groovy/commits/nextflow-perf-issue Here is the test: https://github.com/apache/groovy/blob/nextflow-perf-issue/src/test/groovy/bugs/StubGenTest.groovy After you run the test, you will find Parrot parser runs slower a

RE: RE: Groovy 3 very slow stub generation

I'd be glad to take a look at the current performance of the antlr4 parser. Is there a specific test fixture that I can use? -Original Message- From: Daniel.Sun Sent: Monday, January 20, 2020 11:14 AM To: d...@groovy.incubator.apache.org Subject: RE: RE: Groovy 3 very slow

RE: RE: Groovy 3 very slow stub generation

Agreed. If time allowed, I prefer to profile and tune the performance. BTW, if you could do a favor, that would be great. Cheers, Daniel.Sun - Apache Groovy committer & PMC member Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-De

RE: RE: Groovy 3 very slow stub generation

How about running a profiler on the parser code? The antlr2 parser did not need parallel stub generation to get an acceptable result. -Original Message- From: Daniel.Sun Sent: Sunday, January 19, 2020 12:50 PM To: d...@groovy.incubator.apache.org Subject: Re: RE: Groovy 3 very slow

Re: RE: Groovy 3 very slow stub generation

This sounds great! On Sun, Jan 19, 2020 at 7:48 PM Daniel.Sun wrote: > Hi Paolo, > > I'm thinking about how to solve the performance issue: > > 1) Improve the performance of Parrot parser even if the groovy code is > parsed for the first time > 2) Generate stubs in parallel > > Cheers, > Da

Re: RE: Groovy 3 very slow stub generation

Hi Paolo, I'm thinking about how to solve the performance issue: 1) Improve the performance of Parrot parser even if the groovy code is parsed for the first time 2) Generate stubs in parallel Cheers, Daniel.Sun - Apache Groovy committer & PMC member Blog: http://blog.sunlan.me Twit

Re: RE: Groovy 3 very slow stub generation

Hi Daniel, Thanks for looking at this. I've reported the problem in the IntelliJ issue tracking. Let me know if I can help further. https://youtrack.jetbrains.com/issue/IDEA-230943 Cheers, Paolo On Sun, Jan 19, 2020 at 12:15 PM Daniel.Sun wrote: > Hi Paolo, > > When stubs are being generate

Re: RE: Groovy 3 very slow stub generation

Hi Paolo, When stubs are being generated, the first stubs generation will probably miss cache, but the performance of Parrot parser relies on its cache heavily, so it will run faster and faster while cache is filled fuller and fuller, which is the reason why antlr4 recommends NEVER clear its cache

Re: RE: Groovy 3 very slow stub generation

Generating stubs with antlr2 parser is much faster than parsing with antlr2 parser, if we can work out the cause, the performance issue of generating stubs with antlr4 can be solved. Assuming some operations are skipped. Here is the labs branch: https://github.com/apache/groovy/tree/nextflow-perf-

Re: RE: Groovy 3 very slow stub generation

Stubs] Generated 779 Java stub(s) 81.572s elapsed [FunctionalTests.java, groovy, misc, nextflow, test] D:\_TEMP\nextflow-master\modules\nextflow> ``` I am quite curious about the antlr2 parser result "7.977s" because "57.177s" cost when we parsed all the groovy source files wit

Re: RE: Groovy 3 very slow stub generation

Hi, Using "-Dgroovy.antlr4=false", it takes 38s, therefore, it is similar to groovy 2.5. When setting both "-Dgroovy.antlr4.cache.threshold=200" or "-Dgroovy.antlr4.cache.threshold=600", it is slow: 1m 41s the first, and 1m 48s the second. p On Fri, Jan 17, 2020 at 10:11 PM Daniel.Sun wrote:

Re: RE: Groovy 3 very slow stub generation

Here is the detailed result: results.zip Under Windows, we can set the JVM option by: set JAVA_OPTS=-Xms2g -Xmx2G -Dgroovy.antlr4.cache.threshold=600 then run `groovy Parse.groovy 2` or `groovy Parse.groovy 4` Cheers, Daniel.Sun

Re: RE: Groovy 3 very slow stub generation

I compared the time cost by antlr2 parser and Parrot parser via running the following script "Parse.groovy", which is placed under directory "nextflow-master\modules\nextflow\src\main\groovy\nextflow"[1]: The heap size is set to 2G, i.e. "-Xms2g -Xmx2g" 1) run antlr2 parser `groovy Parse.groovy

RE: Groovy 3 very slow stub generation

If you find the performance issue is gone when you disable the Parrot parser with `-Dgroovy.antlr4=false`, you can try to enable the Parrot parser again and apply `-Dgroovy.antlr4.cache.threshold=200` shown as follows. ``` compileGroovy { groovyOptions.fork = true groovyOptions.forkOption

RE: Groovy 3 very slow stub generation

FYI. ``` compileGroovy { groovyOptions.fork = true groovyOptions.forkOptions.jvmArgs += ["-Dgroovy.antlr4=false"] // disable the Parrot parser } ``` Cheers, Daniel.Sun - Apache Groovy committer & PMC member Blog: http://blog.sunlan.me Twitter: @daniel_sun -- Sent from: http://g

RE: Groovy 3 very slow stub generation

parsing. From: Paolo Di Tommaso Sent: Friday, January 17, 2020 3:34 AM To: dev@groovy.apache.org Cc: d...@groovy.incubator.apache.org Subject: Re: Groovy 3 very slow stub generation Hi Daniel, Groovy 2.5.8: https://gradle.com/s/eqjp6767u5n2o<https://urldefense.proofpoint.com/v2/url?u=ht

Re: Groovy 3 very slow stub generation

Hi Daniel, Groovy 2.5.8: https://gradle.com/s/eqjp6767u5n2o Groovy 3.0.0: https://gradle.com/s/orr64x7ilfwpu Regarding the slow IntelliJ build, sure I'll report to them however I saw clearly it stuck on the Groovy stub generation, and it works fine with the prior version. Therefore it makes me th

Re: Groovy 3 very slow stub generation

Hi Paolo, > The gradle build with groovy 2.5.8 compiles in 34 seconds > The gradle build with groovy 3.0.0-rc-3 takes 1m 46s Could you share the build-scan reports? > The build with IntelliJ more than 5 minutes, which makes it unusable :( As for the performance issue of intellij IDEA,

Re: Groovy 3 very slow stub generation

Hi Daniel, I've made the test you suggested: - The gradle build with groovy 2.5.8 compiles in 34 seconds - The gradle build with groovy 3.0.0-rc-3 takes 1m 46s - The build with IntelliJ more than 5 minutes, which makes it unusable :( p On Mon, Nov 18, 2019 at 4:04 PM Daniel.Sun wrote

Re: Groovy 3 very slow stub generation

Hi Paolo, Please do experiment through the following steps: 1) In your command line(e.g. under Windows OS, type `cmd` to open it), run `gradlew compileGroovy --no-build-cache` to compile with 3.0.0-rc-1 and 2.5.x 2) Compare the time cost by 3.0.0-rc-1 and 2.5.x Cheers, Daniel.Sun

Re: Groovy 3 very slow stub generation

Hi Daniel, No, I'm not delegating the build to Gradle. The compilation is done via IntelliJ. let me know if there's something that I can test. Cheers, Paolo On Mon, Nov 18, 2019 at 2:11 PM Daniel.Sun wrote: > Hi Paolo, > > If you build your project with `gradlew compileGroovy > --no-bu

Re: Groovy 3 very slow stub generation

Hi Paolo, If you build your project with `gradlew compileGroovy --no-build-cache`(assuming you are using gradle), it still takes much more time when using 3.0.0-rc-1 than 2.5.x? If yes, it is probably a groovy issue, otherwise it could be an intellij IDEA issue. Cheers, Daniel.Sun

Groovy 3 very slow stub generation

Dear all, I was giving a trying compiling a medium size project with Groovy 3 rc1 with IntelliJ. Though the compilation is successful, the stub generation process is insanely slow. It took more than 5 minutes to compile the project while