On Mon, 2013-05-20 at 14:51 +0100, Harry Percival wrote: > Interesting points both. I think it's insightful to think about the > line beween provisioning and deployment... The idea that deployment > shouldn't require root permissions is a good stake in the ground. > > > In the book I've decided I'm going to take people through spinning up > a server with Apache, and configuring two virtualhosts with mod_wsgi, > one for staging and one for production. We negotiate the gotchas, > including setting up static files, using absolute paths and getting > the permissions right for the (sqlite) database, etc. > > > Then, deployment is a matter of updating the source code, updating any > static files, and bouncing apache. > > > Provisioning: > - spinning up a server > - installing apache > - configuring apache > > Deployment: > -updating source code > -updating static files > -migrate database > -restart apache
> For those, the root/non-root perms thing holds pretty well, apart from > the "restart apache" bit... And, up to a certain point, what I talk > about under "provisioning" is stuff that might vary from platform to > platform, but stuff under "deployment" should be more universal... Interesting that the subject veers towards system design. I use FCGI myself and the restart can be entirely non-root. Provisioning then includes adding the FCGI reference into the server config. > For automating, I think they can be two separate scripts... Although > we might keep "spinning up a server" manual... > > What about testing? Obviously both scripts will be tested indirectly > via my functional tests (which I'm planning to run against staging, > and they will show up any, eg, CSS problems)... But I've been pretty > hot in the book on unit testing everything too. At PA, our deploy > scripts aren't unit tested, and as a result they're a bit of a mess. > Yup. That's where I'm coming from. But maybe there is a reasonable cut-off? If a deployment script does not have too many paths (which sounds reasonable) then one test (the just-do-it one) may be sufficient? > But, unit testing fabric scripts? Won't it end up being a series of > mocks? I find those kinds of tests a bit pointless... then again, > maybe if we'd had mocky unit tests for all our deploy scripts since > the beginning, perhaps they'd be better structured now... Thoughts? It _is_ possible to test deployment scripts (against a VM perhaps) to check that directories are created and so on, but I've tried to make sure that pretty much all of the files (config or other set-up, and static files) are part of the deployment itself and will be copies of whatever happened in the testing environment (perhaps, maybe, up to a point ... ) so testing might just boil down to "Oh looky here - there's a web site". But that might be a bit simplistic. Testing _will_ be required if the deployment updates a database - or is that part of migration? > > Also, question for django people: static files, as collected by > "manage.py collectstatic": in the repo, or not? > > > > > On 20 May 2013 13:03, Daniel Pope <lord.ma...@gmail.com> wrote: > It doesn't need to be SSH-based, there are orchestration > systems that are RPC or pubsub based (eg. Jenkins, > MCollective). > > As per the line between provisioning and deployment I've > argued in the past that provisioning should be for state that > spans many minor releases of an application. I also believe > it's preferable to require that deployment cannot use root > permissions, as this is less likely to impact independent > services hosted on the same machine. It also allows better > auditability - it prevents deployments making changes that are > rightly the domain of the provisioning system. > > On 20 May 2013 11:17, "MikedePlume" <m...@mikedeplume.com> > wrote: > On Sat, 2013-05-18 at 17:41 +0100, Andy Robinson > wrote: > > Daniel, who are you disagreeing with? Everyone > here agrees on > > automation, I think. > > > > - Andy > > As a small user (one or two servers, one or two > packages to deploy): > > I started by doing a lot of typing of commands, got > bored with that, > moved on to command line scripts and I now use a bunch > of (python) > scripts, some fabric based, to manage the server farm > api and the ssh > stuff. I think the point is that the underlying > process is always going > to involve ssh and some server farm api and it is the > automation of all > of that where the cornucopia of choice (?) is > displayed. > > From a teaching perspective (that's what the book is > about, I guess) I'd > recognise the basics of ssh and so on and then build > up from there. The > basic principles behind testing a fabric deploy should > be extendable to > other environments, I'd have thought? > > I must say, I'd love to see something about testing > bash scripts, and, > perhaps, where to draw the line (if at all) between > server provisioning > and ci. > > Cheers > > Mike S. > > > > > > > > On 18 May 2013 12:24, Daniel Pope > <lord.ma...@gmail.com> wrote: > > > I thoroughly disagree with those who are saying > that for small > > > installations, it's less time-consuming to do > things manually. A > > > deployment/provisioning system gives you > reproducibility - an executable > > > record of how to re-create a server configuration > or re-run a deployment > > > without missing anything. The number of times I've > spent 20 minutes > > > wondering what I've missed all add up - that is > why it automation breaks > > > even so rapidly. > > > > > > Of course, you (or your employers) might have > other reasons to want > > > automation: > > > > > > - so that your deployment/provisioning is subject > to automated testing, > > > version control, code review, etc > > > - so that you can build replicas of production > servers for development, > > > testing or disaster recovery > > > - so that you can smash and rebuild a compromised > or faulty machine without > > > wasting time > > > - so that you can deploy a dozen times a day and > get features or fixes into > > > the hands of your users > > > > > > > > > On 17 May 2013 15:39, M.-A. Lemburg > <m...@egenix.com> wrote: > > >> > > >> On 16.05.2013 17:46, Andy Robinson wrote: > > >> > Speaking as a relatively obsolete dinosaur, I > would suggest that if > > >> > you are going to discuss specific deployment > practices, you start with > > >> > the most fundamental ones: SSH, the unix shell > and so on. > > >> > > > >> > We have had issues over the years with people > coming in and > > >> > introducing sexy new deployment tools, but > ultimately they all just > > >> > run unix commands. Anyone managing a web > application in the > > >> > non-microsoft world is ultimately depending on > this. Some key skills > > >> > (assuming a Linux/Mac/Unix-ish environment): > > >> > - know about SSH keys and logging into remote > machines > > >> > - know the basics of at least one command line > editor (e.g. vi) > > >> > - basic shell knowledge: environment > variables, testing for existence > > >> > of files and directories etc > > >> > - how to interact with your database from the > command line, if you use > > >> > one (including dump and restore) > > >> > - how your web server works: starting, > stopping, configuration files, > > >> > where log files live and how it talks to Python > > >> > > > >> > Fabric may be useful if you want to control > half a dozen machines from > > >> > your desktop, and it might add a lot of value > if you want to control a > > >> > hundred of them. But to update one server, you > deploy by logging into > > >> > it and then running commands or short scripts. > > >> > > > >> > For example, we have a 'demo site' we rebuild > pretty often that uses > > >> > Django, MYSQL, Celery and a few other things. > It runs on plain > > >> > vanilla Ubuntu machines we build ourselves. > The sequence is... > > >> > > > >> > 1. Log in via SSH > > >> > 2. CD to correct directory > > >> > 3. activate virtual environment > > >> > 4. stop any celery worker processes > > >> > 5. stop web server processes (* in our setup, > we leave Apache running) > > >> > 6. pull latest code from mercurial - both the > app, and 3-4 libraries > > >> > it depends on > > >> > 7. run a management command to rebuild the > database > > >> > 8. run a smallish in-place test suite > > >> > 9. restart celery workers > > >> > 10.restart web server > > >> > 11. log out > > >> > > > >> > All of this after the login and CD can be > handled by a shell script on > > >> > the path of the server, so you can just run a > command called something > > >> > like > > >> > ./update_server > > >> > > > >> > More realistically, we tend to end up with a > management shell script > > >> > called 'server' with a bunch of > commands/arguments like 'stop / start > > >> > / restart / update-code-in-staging / > copy-live-data-to-staging / > > >> > run-health-checks / swap-live-and-staging' and > so on. SSH can execute > > >> > remote commands like this just fine with the > right arguments, if > > >> > actually logging in is too tedious. > > >> > > > >> > Production sites are complex and all > different. You might want to do > > >> > instantaneous swaps from live to staging (and > be able to back out fast > > >> > if stuff goes wrong); to switch DNS so the > world is looking at another > > >> > server while you update one; you might have > large databases to copy or > > >> > migrate that need significant time; it may or > may not be acceptable to > > >> > lose sessions and have downtime; and so on. > > >> > > > >> > > > >> > It takes less time to learn the fundamentals > than you will spend > > >> > debugging why your fancy new deployment tool > stopped working after > > >> > some Python dependency upgrade somewhere. And > it is less likely that > > >> > your new hires will disagree if you stick with > the lowest common > > >> > denominator. > > >> > > >> Fully agreed. > > >> > > >> The new devops tools are nice when it comes to > managing farms > > >> of VMs, where each VM is setup in more or less > the same way, > > >> and you want to reduce repetitive tasks, but for > a typical > > >> setup with just a few VMs/servers it'll take you > longer to > > >> write and (most importantly) test your devops > scripts than > > >> it would to write a few scripts that you can run > via SSH on > > >> the servers. > > >> > > >> No matter how smart you make your devops scripts, > Murphy's law > > >> is inevitably going to strike and humans are so > much better at > > >> parsing weird intermittent error messages than > machines ... > > >> still, after all these years :-) > > >> > > >> -- > > >> Marc-Andre Lemburg > > >> eGenix.com > > >> > > >> Professional Python Services directly from the > Source (#1, May 17 2013) > > >> >>> Python Projects, Consulting and Support ... > http://www.egenix.com/ > > >> >>> mxODBC.Zope/Plone.Database.Adapter ... > http://zope.egenix.com/ > > >> >>> mxODBC, mxDateTime, mxTextTools ... > http://python.egenix.com/ > > >> > > ________________________________________________________________________ > > >> 2013-05-07: Released mxODBC Zope DA 2.1.2 ... > http://egenix.com/go46 > > >> 2013-05-06: Released mxODBC 3.2.3 ... > http://egenix.com/go45 > > >> > > >> ::::: Try our mxODBC.Connect Python Database > Interface for free ! :::::: > > >> > > >> eGenix.com Software, Skills and Services GmbH > Pastor-Loeh-Str.48 > > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. > Marc-Andre Lemburg > > >> Registered at Amtsgericht Duesseldorf: > HRB 46611 > > >> > http://www.egenix.com/company/contact/ > > >> _______________________________________________ > > >> python-uk mailing list > > >> python-uk@python.org > > >> http://mail.python.org/mailman/listinfo/python-uk > > > > > > > > > > > > _______________________________________________ > > > python-uk mailing list > > > python-uk@python.org > > > http://mail.python.org/mailman/listinfo/python-uk > > > > > > > > > > > > _______________________________________________ > python-uk mailing list > python-uk@python.org > http://mail.python.org/mailman/listinfo/python-uk > > _______________________________________________ > python-uk mailing list > python-uk@python.org > http://mail.python.org/mailman/listinfo/python-uk > > > > > -- > ------------------------------ > Harry J.W. Percival > ------------------------------ > Twitter: @hjwp > Mobile: +44 (0) 78877 02511 > Skype: harry.percival > _______________________________________________ > python-uk mailing list > python-uk@python.org > http://mail.python.org/mailman/listinfo/python-uk _______________________________________________ python-uk mailing list python-uk@python.org http://mail.python.org/mailman/listinfo/python-uk
