> > ASTs don't need to be built on a node-by-node basis (unless you meant > manifest-by-manifest basis >
Well, manifest-by-manifest where the manifest will vary depending on the node the compile job is depending upon. Reviewing the code I get the impression I just don't understand the existing parser[1] enough today to hold a valid opinion on this. But honestly, I'm not trying to troll. The goal of this particular initiative is to enable the C++ parser (i.e. > the frontend) to interop with the Ruby evaluation implementation (i.e. the > backend). The Puppet code is not being pre-compiled, but > pre-parsed/pre-validated; the C++ implementation will not (yet) evaluate > any Puppet code or load custom types or functions defined in Ruby. > How will this work with create_resources[2]? In compiler terminology, there's a "frontend" and a "backend". > In compiler terminology the frontend is a scanner composed of a parser and a lexer. The front-end validates the parse of the code as a side-effect. This is beyond the scope of the discussion of the PRFC and into a sizing competition about who's read Aho, Lam, Sethi and Ullman. The only point form this is that this is not compiling but a partial parsing. Some of my concerns cannot be raised until there is actual output to examine. The above seems to be confusing, understandably so, pre-compiling a > resource catalog with pre-parsing a manifest. In terms of a language like > Python, the "pre-compiled" pyc files are simply a representation of the > source that is more efficient to load and execute than having to parse the > Python source again > That is because the Java code and CPython code is completely compiled and ready to link in at runtime. In this the XPP proposal does not appear similar to .pyc files or Java bytecode. It does appear to me too be very similar to the Ecore technology[3] from Eclipse, and thus Geppetto as mentioned in the references on RGen in the prior art section. It does appears to be similar in how you write a Coffeescript parser for grammars in Atom or languages in Sublime Text. It is just that you plan to serialize the result to disk instead of displaying to the user. I suggest you read more about the CPython implementation of .pyc files in PEP 3147[4]. The PEP proposal is very well written, IMHO. It covers a lot of the questions that are being raised in comments on the PRFC. Like the discussion of not using a shadow filesystem for the files. An example from the PEP: will there be features like the ability to detect if pre-parsing is available or in use? Can I turn it off in code as a developer or must I always use the --no-xpp command line as a user? Would that even be a good idea? >From my understanding of the PRFC and XPP is a half-parsed file with compiler warnings mixed in. This brings to mind the use of this for created a blocking step in a code deployment process. I've already commented on that use in the document, tough. 1. https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser 2. https://docs.puppetlabs.com/puppet/latest/reference/function.html#createresources 3. http://puppet-on-the-edge.blogspot.com/2014/04/puppet-internals-introduction-to.html 4. https://www.python.org/dev/peps/pep-3147/ Jeremiah Powell Seagate Technology - U.S. Business Data Center http://www.seagate.com email : jeremiah.pow...@seagate.com phone: +1(405) 324-3238 WARNING: The information contained in this transmission may be privileged or confidential. It is intended only for the above-identified recipient(s). If you are not the intended recipient, please forward this transmission to the author and delete this transmission and all copies. Thank you. On Wed, Mar 30, 2016 at 2:37 PM, Peter Huene <peter.hu...@puppetlabs.com> wrote: > Hi Jeremiah, > > On Wed, Mar 30, 2016 at 11:56 AM, <jeremiah.pow...@seagate.com> wrote: > >> I write a lot of native types for our internal use at work. (I tell >> people that if you are just using Exec in Puppet that ansible is one search >> away in Google. ) Some of the Puppet code used with these types would be >> very challenging to "pre-compile" in any way. >> >> I think a lot of my questions are just because I don't consider a .pp >> file to usually be the unit of function in Puppet. I only consider the >> resource, which is usually implemented in Ruby, and everything else is >> dressing. >> >> Reading over this proposal raises questions about how it will fit into >> Puppet, a language and environment very different from Python, Java or >> other languages (Puppet 4 long leaving the domain-specific title by the >> wayside.) >> >> Since you have to build an AST on a node-by-node basis I am having a hard >> time telling the value of this verses something like a marker that >> indicates the code has been parser validated. If your code doesn't need >> node-by-node rebuild then perhaps the compiler could cache the results >> instead? I know the adage about about three hard problems in Computer >> Science still applies but what this is doing is creating and managing a >> cache. >> > > ASTs don't need to be built on a node-by-node basis (unless you meant > manifest-by-manifest basis); an AST is just a representation of the > manifest's source code. The XPP file format is simply an attempt to define > a serialization format for the AST itself so that, say, all the manifest > files in an environment could be parsed, validated, and the resulting ASTs > saved in a format that is faster to read when a catalog for a node is being > compiled all in one go and "upfront" (i.e. not during the catalog > compilation for any one node). > > In compiler terminology, there's a "frontend" and a "backend". The > frontend is responsible for producing an AST that the backend can evaluate > or generate code from (in the case of Puppet, the backend directly > evaluates ASTs to generate a catalog). Having a well-defined AST > serialization format means we could potentially swap out another > implementation for the compiler's frontend, which is one of the goals of > this project and a first step towards a new compiler implementation. > > >> Pre-compiling is translating one language into another because compiling >> is changing one programing language, usually higher level, into another, >> usually lower level. Puppet does something very different in my >> understanding given above. It sounds as if the goat of the C++ parser is >> not to produce a catalog of resources but instead a ruby program for the >> agent to run. >> > > The goal of this particular initiative is to enable the C++ parser (i.e. > the frontend) to interop with the Ruby evaluation implementation (i.e. the > backend). The Puppet code is not being pre-compiled, but > pre-parsed/pre-validated; the C++ implementation will not (yet) evaluate > any Puppet code or load custom types or functions defined in Ruby. > > The ultimate goal of the C++ implementation is to replace both the > frontend and the backend for catalog compilation, but that's a ways off > still as there needs to be design and implementation around loading custom > types and functions from Ruby source to maintain backwards compatibility. > > >> From the statements about a C++ Puppet parser is the target still just a >> collection of resources? Or is the goal to eventually spit out something >> other than what the Puppet server should send to a node? >> > > The target is still the same thing we send to agents now: a resource > catalog. That won't change when the C++ compiler implements the backend > too. > > >> Is the scope just 'closed' .pp files? That is classes where all >> variables can be resolved without inquiry to facts? The behavior of >> languages that support things like pre-compiled files is specific to how >> they do binding of missing data. While this proposal punts on the >> 'serialization' format the handling of binding is pretty central. That >> raises questions like: >> >> How can a compiled format deal with Puppet features that call APIs? Is >> this for the defined types? How will it deal with the fact that the code >> could radically change based on node data not even present at the time the >> pre-compile is done? >> >> What happens if a precompiled module that depends on PuppetDB information >> is moved to an environment without it? For that matter, is the format >> intended to move between Puppet installations or just very similar ones? >> >> My model of Puppet is that the compiled catalog is just an ordered list >> of resources which do not have any placeholders like variables or >> sub-dividable parts. Classes in this model are just names that help you >> find the resources. >> >> This format is relevant to recent discussions among people on IRC and in >> the mailing lists and groups about getting and storing the catalog for a >> node. This is historically something hard as an end user. >> >> The context of any given resource is the whole catalog and at a minimum >> requires the dependent and depending classes in the graph tree. Otherwise >> how does this deal with the unresolved parts? >> > > The above seems to be confusing, understandably so, pre-compiling a > resource catalog with pre-parsing a manifest. In terms of a language like > Python, the "pre-compiled" pyc files are simply a representation of the > source that is more efficient to load and execute than having to parse the > Python source again. This is analogous to what we're calling pre-parsed > XPP files: a representation of the Puppet source code that enables faster > evaluation (i.e. during resource catalog "compilation") and also enables > the existing Ruby implementation to load and evaluate even if a entirely > different parser implementation was used to generate them. > > Hope this helps. > > Peter > > >> >> On Wednesday, March 30, 2016 at 11:24:46 AM UTC-5, Eric Sorenson wrote: >>> >>> Hi, I've just posted a new Puppet RFC that describes pre-parsed and >>> pre-validated Puppet files, akin to '.pyc' files for Python. It's called >>> XPP and the doc is open for comments here: >>> >>> >>> https://docs.google.com/document/d/17SFn_2PJYcO5HjgA4R65a5ynR6_bng_Ak5W53KjM4F8/edit?usp=sharing >>> >>> Please comment inline on the doc, or come back to this thread if the >>> conversation gets too involved (more than about 4-5 replies in a comment >>> box on google docs becomes unwieldy) >>> >>> Once the commenting tapers off we'll incorporate changes into the spec >>> and post it as markdown in the puppet-rfc repo: >>> https://github.com/puppetlabs/puppet-rfc >>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_puppetlabs_puppet-2Drfc&d=CwMFaQ&c=IGDlg0lD0b-nebmJJ0Kp8A&r=XNTNYjt_GV8Csrb0reOpHgXix1pRe_p4VuXq7ewzo3I&m=HZ6NF4SJ1QEs-Fr0uTYIiMXFMHwUWsOD6a71A6a6Oto&s=6KK-JGc9X0PZxVDEAEWyJ-TO7Fy7g6KwSP9b1kfIrP4&e=> >>> >>> --eric0 >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-dev+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com >> <https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Developers" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-dev/ve0IdegWQck/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > puppet-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-dev/CACZQQfOhsBHcfjdoh9%3DyZS4YGUXQtnj6HRBTqLZiRr_CiueeNQ%40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-dev/CACZQQfOhsBHcfjdoh9%3DyZS4YGUXQtnj6HRBTqLZiRr_CiueeNQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAMYg32bfGn8Lj96mfN780BTOdG5FsCHP4Rxjy9zus2dn3naxnA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
