On Wed, Feb 3, 2016 at 11:30 PM, Joshua Pokotilow <jpokoti...@gmail.com> wrote:
> At the startup where I work, we've written a lot of our server code in > Django. So far, we've adopted a "build it fast" mentality, so we invested > very little time in optimizing our code. A small amount of load testing has > revealed our codebase / infrastructure as it stands today needs to run > faster and support more users. > > We recently hired some new engineers who are extremely skeptical that we > should optimize our existing code. Their main concerns are: > > - We need to move to a service-oriented infrastructure because Django is > too monolithic (monolithic = technology lock-in & difficult to troubleshoot) > - It's too easy to write slow queries using the Django ORM > - It's hard to hire Django engineers > - While Instagram and DISQUS use Django to service large numbers of > people, they don't use it for any serious backend work > > After having worked with Django for the last 3 years, I'm a big believer > in it, and I believe it would scale. To defend my position, I've pointed > out to my colleagues that it's easy to identify bottlenecks with tools like > the Django Debug Toolbar and Yet Another Django Profiler. With my > colleagues present, I've isolated and fixed significant speed problems > inside of a few hours. I don't believe the Django ORM is inherently bad, > although I do think that coders who use it should Know What They're Doing. > Finally, I've referenced blog entries that talk about how Instagram and > Disqus use Django on the backend for backend-y tasks. > > Despite my best efforts, my colleagues are still pushing to have us > rewrite large portions of our infrastructure as separate services before we > try to fix them. For example, we have one slow REST endpoint that returns a > boatload of user data, and so there's talk about using a new microservice > for users in lieu of our existing Django models. Even if we are able to fix > bottlenecks we encounter in a timely fashion, my colleagues fear that > Django won't scale with the business. > My immediate reaction, knowing nothing about the site or it’s codebase - 1) There’s nothing they’re proposing that excludes Django from the mix. 2) From an engineering management perspective, the solution they’re proposing is much more concerning than the problems you’re describing. My suggestion for convincing management: Tell them that you can write Microservices in Django. Because you can. Build a minimal Django stack - something that just returns a static Hello World - and do some load testing. This will prove that Django can serve high load - or at least as much load as whatever technology they’re proposing. Tell them that Microservices is just a new word for something software engineers have been calling “High cohesion, low coupling” since the 1960s. The only difference is that this time, instead of using the low latency, high speed interface of a function call, we’re using the slow, unreliable transfer of HTTP. If you’re actually focussing on performance, it’s trivial to build a high cohesion, low coupling stack in *any* technology. All that Webservices do is enforce this by making the inter-module barrier obvious. Tell them that Microservices aren’t magic fairy sauce for speed. If the issue with your existing codebase is the speed of database queries, that problem isn’t going to go away by putting your code behind microservices. You’re just going to add the cost of inter-service HTTP transfer to the overhead of making a query. And if you’re putting something essential - like the user database - behind a service, then you’d better be prepared to add the round-trip time of a HTTP lookup onto Every. Single. Page. (Tell me again how this is good for performance?) Teach them about Second Systems Syndrome [1] [2]. [1] https://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_second-system_effect [2] http://coliveira.net/software/what-is-second-system-syndrome/ Tell them that while Django engineers might be hard to hire, they’re also relatively easy to grow from scratch. DjangoGirls proves you can take people with no experience in programming and make them competent Django developers. Take someone with a history in *any* programming language, and you can teach them Django; hire one or two Django experts to provide an internal knowledge and review, and you’re set. Lastly, tell them that despite their protestations, your site isn’t Instagram, Disqus, or anything like it. 99% of web sites are not in the top 1% of websites by traffic. Your website is *not* in the top 1%. It might be one day. But it isn’t now. And if you’re *ever* in a position where you might end up in the top 1% - I can *guarantee* that it will be accompanied by a metric buttload of engineers and money who will have a lot more experience in scaling large scale services than any of the people who are proposing microservices as a silver bullet. Now - I’m saying all this without having actually seen your code or knowing about your problem. It’s entirely possible that moving some of your code behind and SOA barrier would make good engineering sense. However, that doesn’t mean you need a re-write - it means you need a refactor. And, again, there’s nothing that precludes using Django for the services that get split out. Django is just a mechanism for mapping a HTTP request to a function that returns a HTTP response - and it’s one that has been repeatedly proven at *very* high scale. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAJxq84_RiJoruSJ1uSSqwf_oiguZNNA9YgOzjSUE5%3DaDjVJPdg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
