On Fri, Sep 28, 2012 at 11:38 AM, jcbollinger <john.bollin...@stjude.org>wrote:
> > > On Thursday, September 27, 2012 11:56:28 AM UTC-5, Jeremy wrote: >> >> I've got a puppet module I've written to support deploying a custom PHP >> web application that a client has developed. The actual application and the >> fact that it's deployed within AWS is not the problem or important to the >> issue. I'm looking to see if I someone else can think of a better way to >> implement what I have done that is more efficient and improve the catalog >> rendering time. >> >> I've placed the module into a GitHub repository at ( >> https://github.com/jbouse/**puppet-app1<https://github.com/jbouse/puppet-app1>). >> The app is deployed based on the contents of a YAML file that is retrieved >> by the class. Initially deploying the sites defined was not an issue and >> was actually very fast. The problems got introduced in handling the >> dependent components as defined in the YAML file. I wrote several parser >> functions in an effort to identify only the unique components and versions >> needed and generate titles that would not conflict. I fully admit and >> consider what I've done a hack and I'm really open to hearing some >> suggestions on how to make it less of one. >> > > > I have been looking over your module, and so far I don't see any smoking > gun that would clearly explain why catalog compilation takes so long, but I > do have a guess (see below). The code seems generally clean, > well-organized, and nicely documented, but perhaps I have a few comments > that perhaps you will find helpful. In no particular order: > > - You use a lot of constructs of this form: "$somedir = inline_template > ("<%= File.join('${parent}', 'foo') %>")". That's a lot uglier and > heavier than "$somedir = "${parent}/foo", and it gains you nothing: the > template is always going to be evaluated on the master, which cannot run on > Windows, so the file separator is always going to be '/'. > - It is harmless, but unnecessary, for a File to explicitly 'require' > other File resources representing its parent or other ancestor directory. > Puppet will generate these relationships automatically if you do not > specify them explicitly. > - You evidently have a separate module "common" containing a > definition named "common::archive::tar-gz". Names in your Puppet > manifests should not contain the hyphen (-) -- it works in some places, in > some versions, but not in others. You would be wise to avoid it > altogether, perhaps by replacing it with an underscore (_). > - If there is one thing to be most suspicious of, it would be your > app1_deployment function's use of "YAML.load(open(args[0]))". Some of > the other code in your module leads me to suspect that the file it's > opening may not be local. (And if it were local, then you would probably > just put the data in the hiera store you are using for other data.) If you > are indeed retrieving that file over the network then the time to do so > could easily dominate your compilation times, and network slowness or > outage could make your compilations timeout or simply fail. > > > Good luck, > > John > John, Your observations were pretty much on target. The common module does have the define that handles retrieving and extracting the tarballs to a target directory and has worked perfectly for quite some time. I was re-designing the module that deploys the web app from their old single tarball to a multi-tarball deployment model so now it just gets called more. I hadn't heard of the issue with the hyphen but I'll take it under advisement and adjust. The use of the "YAML.load(open(args[0]))" call was in fact to support both local and network files. In this case I'm actually giving an authenticated S3 bucket URL to retrieve the file as the engineers releasing the code also upload the deployment YAML file to the S3 bucket. The tarballs that are deployed are also in the S3 bucket and also pass authenticated URLs in the catalog with an expiration equal to the catalog expiration time. I'd like to eventually modify it to include retrieving the deployment file and storing it locally only when it's been modified but want to keep it in S3 as it allows my Puppet master to operate as a blackbox that engineers have no access to. If I control the deployment file locally they claim I'm the bottleneck slowing them down so as long as I give them the means to update it and the process flow is error free and only problems encountered are when they screw up the deployment file contents accountability is maintainable. My thought on the "smoking gun" is in having to make the parser function calls to try and determine the unique components and unique versions in the case of a component with multiple versions needing to be deployed. This was the quickest way I could find to get the deployment file format converted and ensure that I only defined a resource once avoiding the duplicate resource definition errors. As a result I'm calling the 2 functions which have to iterate through the entire YAML content merging then sorting for unique values separately. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
