It sounds like you're making good progress. I agree with how you want to use the Antlr4 callbacks to construct a FalconAST.
- Gordon Sent from my iPad > On Jul 24, 2014, at 11:06 PM, "Christofer Dutz" <christofer.d...@c-ware.de> > wrote: > > Hi Gordon, > > I won't object that Shaoting Cai certainly knew what he was doing, but Antlr4 > is in some respects greatly different from 2 and 3. I bet if Antlr4 had been > available at the time Falcon was initiate, he would have used that. Not only > does it seem to allow much simpler grammars, it also generates an interface > for each grammar in which "enterXYZ" and "exitXYZ" callbacks are defined for > every rule in the grammar. I can hereby create an implementation of that > interface and construct the FalconAST in here directly. Usually you parse the > input and get an AntlrAST and use a treewalker in conjunction with that > Interface implementation, but you can also hook that up as parse-listener and > it's used by the initial parsing. This way I think I could remove several > tree traversals. > > I also got closer on the JBurg part. I was actually wrong with the runtime > and generator part. It's actually all just a Generator. I therefore have > create a jburg-maven-plugin that I can hook up into the build. The cool thing > about that ist hat every plugin has it's own classloader and therefore JBurg > can use whatever it likes without interfering with the rest. I am having one > really strange classloading error though that I still need to investigage and > as soon as that's finished I promissed tom to donate the plugin to JBurg. I > also had a small discussion with tom and he will be willing to update JBurg > to update JBurg to Antlr4 and Stringtemplate (The external stringtemplate, > not the Antlr internal version). After that I agreed to make JBurg build with > Maven ... so I guess we are having some progress on this :-) > > Chris > > -----Ursprüngliche Nachricht----- > Von: Gordon Smith [mailto:gsmit...@hotmail.com] > Gesendet: Freitag, 25. Juli 2014 04:10 > An: dev@flex.apache.org > Betreff: RE: AW: AW: AW: AW: AW: Falcon and Antlr4 > >> I am sort of confused what JBurg is actually working on ... Does it directly >> work on FalconAST or is there some sort of "ReducedFalconAST"? > > The input to the BURM (Bottom-Up Rewrite Machine) that is produced by JBurg > (Java Bottom-Up Rewrite Generator) is the FalconAST. For example, a BURM > pattern like > > Pattern addExpr > Op_AddID(expression l, expression r); > > in CMCPatterns.jbg, representing a '+' expression, applies to each subtree > headed by a BinaryOperatorPlusNode, because the getNodeId() method of this > node class returns Op_AddId. > >> ABC code (I guess this is the native Flash bytecode?) > > Right. ABC stands for ActionScript Byte Code, which is the byte code esecuted > by the ActionScript Virtual Machine (AVM) in the Flash Player and AIR. > >> CSS... I would like to simplify this to: >> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST] >> --[JBurg]--> SWF > > I won't object, however the CSS implementation was done for Falcon by a > former Adobe engineer (Shaoting Cai) who was well versed in Antlr, since he > had been a student of the professor who developed Antlr. (The AS lexer and > parser are much older, inherited from Flash Builder, and written by engineers > who were probably not as expert in Antlr, and were using older version of > Antlr that didn't have tree parsers.) So I tend to think that the "Antlr way" > may be to let Antlr create an AntlrAST automatically, and then transform it > into a more custom FalconAST with a tree parser. On the other hand, if this > is slower than having the parser produce the FalconAST directly from the > token stream, which seems likely, then I vote for going for speed. > >> AS... Here I would like to remove the RawASTTokenizer and replace that >> with an Antlr4 Lexer AS --[ASLexer]--> TokenStream --[ASParser]--> >> FalconAST --[JBurg]--> SWF > > This is what I had in mind, although again I'm not sure this would be > considered Antlr best practice. (As before, if Shaoting had done the AS > implementation, he probably would have produced an AntlrAST and then > transformed it into a FalconAST with a tree parser.) > >> Do we even need a dependency to JBurg itself to execute the code generated >> by JBurg? > > Yes. As you discovered, there is a runtime part, which determines which > sequence of reductions has the lowest 'cost' and therefore gets used for code > generation. > >> I will contact the JBurg project and check if it was possible to switch to >> the new Stringtemplate project (and if it was possible to split up JBurg >> into two modules "generator", and "runtime"). > > If Tom doesn't want to decouple JBurg from Antlr, one option would be for us > to fork JBurg and remove its dependency on the Antlr Stringtemplate. Is > another option for JBurg to continue using whatever old Antlr it has been > using and the AS and CSS lexer/parser to use the latest Antlr? > > - Gordon > > >> From: christofer.d...@c-ware.de >> To: dev@flex.apache.org >> Subject: AW: AW: AW: AW: AW: Falcon and Antlr4 >> Date: Thu, 24 Jul 2014 09:08:38 +0000 >> >> Ok ... so I created a JBurg Maven Plugin (Will contact the JBurg >> project lead for donating that) >> >> But (unfortunateley there's a but): >> - JBurg seems to consist of a Generator and a Runtime, but both are >> located in the same Jar (Even if this wouldn't really be a blocker) >> - JBurgs runtime part doesn't directly rely on Antlr AST classes, but >> unfortunateley Antlr Stringtemplate which is only bundled in Antlr3 >> distributions because it seems to have been created a project spinoff >> containing only the stringtemplate part >> (http://www.stringtemplate.org/) >> >> I will contact the JBurg project and check if it was possible to switch to >> the new Stringtemplate project (and if it was possible to split up JBurg >> into two modules "generator", and "runtime"). >> >> Chris >> >> >> ________________________________________ >> Von: Christofer Dutz <christofer.d...@c-ware.de> >> Gesendet: Donnerstag, 24. Juli 2014 09:57 >> An: dev@flex.apache.org >> Betreff: AW: AW: AW: AW: AW: Falcon and Antlr4 >> >> So let's just define some words that might help avoid confusion. >> >> A typical Antlr parser geneates a parse-tree consisting of Antlr objects ... >> let's call that AntlrAST. >> Falcon has it's own internal representation using its own objects ... let's >> call that FalconAST. >> Then I am sort of confused that JBurg is actually working on ... Does it >> directly work on FalconAST or is there some sort of "ReducedFalconAST"? >> >> Because I started with the CSS Parser, I saw that we use Antlr3 to generate >> a parser for CSS files. That outputs an AntlrAST which is fed into a second >> Antlr generated parser "CSSTree" which converts this AntlrAST into a >> FalconAST. After that I think (i'm not 100% sure) the FalconAST is passed in >> to the CSSReducer which sort of directly generates ABC code (I guess this is >> the native Flash bytecode?). >> >> ////////// >> // Css >> ////////// >> >> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> AntlrAST >> --[CSSTreeParser]--> FalconAST --[CSSReducer]--> [ReducedFalconAST] >> --[JBurg]--> SWF (Everything from the FalconAST still seems a little >> mystical to me though) >> >> I would like to simplyfy this to: >> >> CSS --[CSSLexer]--> TokenStream --[CSSParser]--> [FalconAST] >> --[JBurg]--> SWF >> >> ////////// >> // ActionScript >> ////////// >> >> For AS the procedure is different (As far as I got it together with your >> mail). We are using Antlr2 here: >> >> AS --[RawASTTokenizer]--> TokenStream --[ASParser]--> FalconAST >> --[JBurg]--> SWF >> >> Here I would like to remove the RawASTTokenizer and replace that with >> an Antlr4 Lexer >> >> AS --[ASLexer]--> TokenStream --[ASParser]--> FalconAST --[JBurg]--> >> SWF >> >> ////////// >> // JBurg >> ////////// >> >> In the meanwhile I think I had some progress. As far as I understand now >> JBurg is built by Antlr and uses Antlr to parse and generate output. The >> output however no longer has a reference to Antlr. Do we even need a >> dependency to JBurg itself to execute the code generated by JBurg? If not I >> would whip up a jburg-maven-plugin which wraps JBurgs execution. Then I >> think we shouldn't have any problems with Antlr versions. >> >> Currently a I want to do is do an Antlr4 POC for the CSS compilation. As >> soon as that's done, I would like to compare the performance to that of the >> existing one and only if that is at least as fast as the current solution I >> would proceed to the next part (If its not slower than the current solution >> I would prefer the simpler process). >> >> Chris >> >> >> ________________________________________ >> Von: Gordon Smith <gsmit...@hotmail.com> >> Gesendet: Mittwoch, 23. Juli 2014 18:05 >> An: dev@flex.apache.org >> Betreff: RE: AW: AW: AW: AW: Falcon and Antlr4 >> >> CSSTree.g isn't involved in compiling AS files. It's involved only in >> compiling CSS files. >> >> For AS compilation: We use JFlex to create RawASTokenizer.java from >> RawASTokenizer.lex. This is a lexer which takes an AS file as input and >> creates a sequence of ASToken objects as output. We use Antlr 3 to create >> ASParser.java from ASParser.g. This is a parser which takes a sequence of >> ASToken objects as output and creates an AST consisting of NodeBase objects. >> This is a custom AST, not the kind of generic Antlr AST that Antlr can >> create automatically. >> >> If Antlr 4 can tokenize as fast as JFlex can, then I think it would be great >> to have a unified Antlr 4 lexer/parser. However, I strongly recommend >> keeping Falcon's custom AST classes. >> >> I'm not an expert on JBurg and wasn't aware that it had a dependency on >> Antlr. Can you please show me where this depencency is? The "JBurg guys" are >> basically one person, Tom Harwood, who used to work at Adobe but no longer >> does. I don't know whether he is still supporting JBurg or not. >> >> As for eliminating JBurg... This is certainly doable, but would have serious >> implications. The main reason that Falcon does code generation using JBurg >> is in order to perform various code optimizations such as constant >> propagation. These code optimizations would be much harder to do without >> JBurg. Ideally, these optimizations would be done in the ActionScript >> Virtual Machine rather than in the compiler (and this was the plan for AS4, >> which Adobe killed) but since they are NOT done in the current AVM, if the >> compiler doesn't do them then the code will execute slower. However, if >> cross-compilation to JavaScript is the main goal for the future, then >> perhaps slower AVM execution is tolerable. >> >> - Gordon >> >> >>> From: christofer.d...@c-ware.de >>> To: dev@flex.apache.org >>> Subject: AW: AW: AW: AW: Falcon and Antlr4 >>> Date: Wed, 23 Jul 2014 12:54:15 +0000 >>> >>> Hi Eric, >>> >>> The question is definitely not silly. >>> >>> I'm not planning on changing anything here. The package tangling >>> seems to be related with JBurg needing access to some objects that >>> are not part of the interface but part of the internal model. I >>> would untangle this by extending the Interface to provide anything >>> JBurg needs so there shouldn't be any change in the Falcon AST. But >>> after all that's just an optimization. Currently I'm digging in the >>> inner workings of JBurg ... gee I sort of forgot why I looked in the >>> inner workings of Falcon in the first place ;-) >>> >>> I would however like to remove as many processing phases and model >>> conversions as possible as each one costs time and memory and makes things >>> more compilcated to understand. >>> >>> Chris >>> >>> ________________________________________ >>> Von: Erik de Bruin <e...@ixsoftware.nl> >>> Gesendet: Mittwoch, 23. Juli 2014 14:47 >>> An: dev@flex.apache.org >>> Betreff: Re: AW: AW: AW: Falcon and Antlr4 >>> >>> I'm not at all familiar with the inner workings of Falcon, but I >>> know that FalconJX relies heavily on the current Falcon AST. Will >>> the new AST be compatible with the old (that may be a silly >>> question, but it's the only one I know to ask)? >>> >>> EdB >>> >>> >>> >>> On Wed, Jul 23, 2014 at 2:36 PM, Christofer Dutz >>> <christofer.d...@c-ware.de> >>> wrote: >>> >>>> Ok ... so I used some idle time setting up a clean falcon-antlr4 >>>> module starting with the CSS parsing. >>>> >>>> As far as I can see it, Falcon is currently using Antlr3 to >>>> generate the lexer and a parser that generates an Antlr AST from >>>> that Then we generate a second parser CSSTree.g to convert that >>>> AST into the falcon-internal object structure, which is then >>>> passed to JBurg to generate the output. >>>> >>>> As you mentioned the ability to manipulate the AST. Are you >>>> talking about the first (antlr) AST or the one the Falcon AST? >>>> I would like to elminiate the second parsing step and generate the >>>> internal model directly from the first parser using Antlr4s >>>> Listeners. Are there any objections against this? I doubt that >>>> anyone is manipulating the Antlr AST directly, much more would I >>>> expect operations to be performed on the Falcon AST. >>>> >>>> I managed to get the CSS Lexer and Parser finished. The grammar is >>>> way more readable now and the general structure is much more >>>> straight forward :-) Now I would like to Build the internal Falcon >>>> object model for that. >>>> Guess I would have to contact the JBurg guys for continuing from >>>> then as I definitely need Antlr4 support but I would bet that >>>> either JBurg is dead or they are working or at least thinking >>>> about that. >>>> >>>> I did detect some ugly cyclic dependencies in the falcon classes >>>> though ... for example we have interfaces for defining the >>>> internal models interface and have implementations of these >>>> interfaces in the internal packages, but unfortunately the >>>> interfaces reference classes in the internal model ... I think >>>> that's bad design and would like to clean that up. >>>> >>>> Any objections? (Keep in mind ... it's all in a dedicated branch, >>>> but I don't want to make it hard to merge back as soon as I'm >>>> finished) >>>> >>>> There is no DOM or SAX involved in Falcon ... I was using a >>>> metaphor to explain what I was thinking about by using an example >>>> from the XML processing world. Here you can load an XML document >>>> to memory (DOM), transform that into a second memory >>>> representation and so on in contrast to SAX processing where >>>> parser generates a stream of events which can be processed >>>> allowing you to process petabyte big documents with as much ram as >>>> a calculator ;-) >>>> >>>> >>>> Chris >>>> >>>> >>>> ________________________________________ >>>> Von: Alex Harui <aha...@adobe.com> >>>> Gesendet: Mittwoch, 23. Juli 2014 08:48 >>>> An: dev@flex.apache.org >>>> Betreff: Re: AW: AW: AW: Falcon and Antlr4 >>>> >>>>> On 7/22/14 12:07 PM, "Christofer Dutz" <christofer.d...@c-ware.de> wrote: >>>>> >>>>> The way I understood JBurg is that it seems to have been designed >>>>> to manipulate ASTs produced by Antlr2 and Antlr3. >>>>> Therefore it has a compile and runtime dependency to Antlr (I >>>>> also sort of dislike them bundling the generator together with >>>>> the runtime stuff) but nevertheless ... I can't really use JBurg >>>>> with Antlr4 in my project not without splitting the module up but I think >>>>> that's rather ugly. >>>> I'm definitely not an expert on the subject. I'm surprised the >>>> AST has any Antlr-isms in it. Maybe Gordon can confirm. When I >>>> look at the AST, I don't see anything I find surprising. >>>> >>>>> >>>>> I too was thinking about making JBurg obsolete too ... the way I >>>>> understand Antlr4 it looked as if it provided a lot which could >>>>> make life a lot easier. For example currently the ASParser >>>>> creates the AST objects and JBurg then performs it's magic on >>>>> that AST tree. In Antlr4 I could create a Parser-Listener that >>>>> produces the AST object (Probably needed for IDE suppoer) but >>>>> could probably create a second listener that directly outputs >>>>> flash bytecode. I think this approach should be maximum >>>>> performant, maximum simple and use only a fragment of the memory >>>>> the current solutions need ... Sort of XML Dom processing (old AST >>>>> parsing) and SAX (Antlr4 direct Bytecode output). >>>> I have not played with SAX much nor Antlr, so I have no idea about >>>> parser/listener. I will say that there are times folks want to >>>> create the AST and then manipulate it before generating the >>>> output, so unless there is a huge performance win, making sure >>>> there is a way to do that would be preferred. >>>> >>>>> >>>>> But I also think that this would be quite an effort. Perhaps I >>>>> should split up my falcon-antlr4 branch even more and sort of >>>>> start with the CSS parser and as soon as that's up and running get the >>>>> output running ... >>>>> then we could compare the performance of both and see if it's >>>>> worth going down that path. Don't want to waste time I could be >>>>> working on Flexmojos, or the new Flex-Maven-Plugin for a less >>>>> performant solution. But if it was faster (I would expect it to be) it >>>>> could be quite breakthough ... >>>>> after all I have never seen several datastructure conversions be >>>>> faster than direct output. >>>> Up to you since you are doing the work. I just want to make sure >>>> you don't feel like you have to keep all of these subsystems like >>>> Jburg and Jflex around. >>>> >>>> -Alex >>>> >>>> >>> >>> >>> -- >>> Ix Multimedia Software >>> >>> Jan Luykenstraat 27 >>> 3521 VB Utrecht >>> >>> T. 06-51952295 >>> I. www.ixsoftware.nl >
