On Sun, Sep 1, 2013 at 10:53 AM, Dustin J. Mitchell <[email protected]>wrote:
> First, a vote against reverting to explicit loading -- the module > autoloader is *awesome* from a user perspective. > > Exactly. I don't think reverting to explicit loading would be a win for users. > Second, to the idea of doing a better job of invalidating caches > between runs, I don't think anyone (except perhaps Henrik, in the > comment that I, like Luke, can't quite parse) has considered the > impact of negative stat results. That is, if the parser caches an AST > for a file found in the third entry in the MODULEPATH, and a new file > is added in the first entry in the MODULEPATH that would have been > loaded for that symbol, then the cache needs to be invalidated. Yuck. > > I like the idea of a single serialized AST, but it has the indicated > downsides for 'puppet apply' users. I don't think implementing two > resolvers -- one with current semantics and one that just > de-serializes -- is a good idea unless the one with the current > semantics is used to build the AST before serialization. In which > case you've optimized manifest loading, but done nothing to fix the > fragility that started this whole thread. > > I think here (and below) you are getting to the core of the issue. We really need to define the complete semantics of loading. What exactly will a symbol resolve to in terms of files, and when will those files be checked for changes. > Most scripting languages have similar problems. Perl and Ruby just > try to open(1) a bajillion paths for every 'use' or 'require' > directive. Python does something similar, but looks for .pyc/.pyo > files too, which contain compiled bytecode. Python also has a > mandatory mapping from filename to module name, more similar to Puppet > than Perl or Ruby. In any of those cases, load order can cause funny > behavior, and they don't deal cleanly with files changing while a > program is running. That said, users are accustomed to the vagaries > of the lazy loading these languages implement, and there are > established ways to work around reloads or make loads faster. > Exactly. Perl handles this through 2 mechanisms: something that looks a bit like the Package Name -> File Name mapping in Java, and a, rarely used, mechanism to load specific files. It also provides along with this, some very specific and clear semantics around when it all happens (BEGIN, CHECK, INIT, END stages that appear when you really get into perl are what that is all about). > What those do *not* deal with is the need to re-read everything > repeatedly in the same process. To put it another way: in the vast > majority of cases, users expect a Python program to load all of its > source files once, near the beginning of its execution, and never look > again. But users expect their puppetmaster to parse each and every > manifest file anew on every catalog request. > > Yes. Those languages have tried again and again to come up with ways of doing that, but the runtimes just don't have the constraints in place to allow them to do hot swapping of code safely, without resorting to what amounts to a complete restart. In puppet, ignoring the ruby extensions, we can actually do the hot reloading (within reason). > But all of that is getting into questions of efficiency and response > to changes. Neither of those were the topic of the original post. > That post said that lazy loading is hard and ill-implemented, and that > the proposed solution was to switch to eager loading. I would counter > by saying that the lazy-loading paradigm is a good one, and the > current implementation is pretty effective from the perspective of > users. This is exactly the situation Python-2.x was in. The Python > devs took the existing behavior, defined it carefully, and then > rewrote the implementation to meet the same definition. It seems like > this is the right solution for Puppet, as well. And if there are > optimizations to be made in there, either invisible or user-initiated > (e.g., `puppet compile`), then plan the new implementation to support > those optimizations. > > Yeah, the lazy vs eager, I think is an implementation detail. The laziness should be an optimization strategy that the compiler/interpreter utilizes. I think I confused the issue by not making it clear, that I don't want to change the semantics of the puppet language at all. I just wanted to change the way that the internals dealt with finding and loading files from disk, but I didn't make that distinction clear. So, given the semantics for how symbols in puppet manifests relate to files on disk (which is mostly clear), and that puppet manifests do not represent "long running" applications which can hold state between multiple requests (once again, ignoring the ruby extensions), then we should be able to freely change the implementation between lazy and eager without users noticing too much. > Dustin > (sorry if that seems to ramble...) > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev. > For more options, visit https://groups.google.com/groups/opt_out. > -- Andrew Parker [email protected] Freenode: zaphod42 Twitter: @aparker42 Software Developer *Join us at PuppetConf 2014, September 23-24 in San Francisco* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
