Can Django be used for a high traffic website with lots of writes to the database?

Lane Campbell Lifelong Entrepreneur

March 5th, 2015

I have spoken with different developers who have told me that Node.js is better suited for this kind of application. Seems like there are various opinions out there on what technology to use for different purposes. I know Instagram got started with Django so it seems logical that the framework is capable of handling high traffic websites. What is your opinion?

Steven Rubenstein

March 5th, 2015

It sounds like the different developers you spoke with know nothing about Python / Django. It is very scalable. There is admittedly greater overhead when using a framework such as Django or Rails, but they can definitely scale. For that matter, most technologies are scalable if you know how to use them properly.

Matthew Griffiths Technology Entrepreneur, CTO, Startup Executive

March 9th, 2015

Hi Dmitry, yes of cause (although I'll paraphrase due to time constraints):

Top line: Django on App Engine works and works pretty well. It's not without its issues but the combination of python/Django/Google offers a very stable and scalable solution for a decent slice of startups. There are of cause, cases where Django its self is not the right choice (see other comments in this thread) and there will be special cases where even if Django is the best web application framework, Google App Engine won't be the right deployment option.

In a bit more detail:

General Development
------------------------------
Worth noting that App Engine *does not* support Django1.7 which could be a huge pain as 1.7 is a great iteration. That said, 1.5 is supported and was (and still is) a very good choice for rapid web application development.

In addition, dependency management is non intuitive as App Engine does not support virtualenv or pip - workarounds are well documented.

Apart from that, you can get a pretty decent dev workflow for your team up and running pretty quickly including local env's (we didn't roll vagrant but I hear there is a project providing this), developer specific hosted AppEngine accounts and Dev and Staging etc.

CI
---
While App Engine suggests it supports github integration and push to deploy we never got this working and instead went for a Jenkins setup on a Google Cloud VM with Github integration running testing, per dev deployment and shared dev and staging integration. Nothing really specific here to app engine, just worth nothing that Jenkins is still the best choice in this area (in our experience).

Database
-------------
We went for Google Cloud Datastore using django appengine and after a few teething issues were impressed with the flexibility (no migration if managed correctly) and performance offered. We rolled our own management and BI console so I can't speak for the django admin site but I from previous experience, Django toolbox sorts out all but the most crazy no-sql things you can do with models.

Worth a quick aside to mention that fixtures are not (really) supported so if you have a ton of pre-fill data for the database you need to be inventive on how to get it loaded up.

Dev Ops
-----------
This is the real winner and the reason why AppEngine and Google Cloud is so hot. They have SO many dev ops features built into AppEngine that you really can have one of your good dev's running dev ops as a side project. You still have to be techy to get the best out of the dev ops suite but once setup, the exec/founder level reporting and monitoring is really good.

Integration
--------------
Worth noting that using App Engine makes it very easy to use and pay for Google services (maps, auth etc etc) if this is part of your stack then this can be a big win, especially if you just starting out as dev time can be shortened.

Conclusion
---------------
There are always cases for and against any tech. IMHO, Django over Google App Engine is great for statups that want to benefit from the scale that Google brings to play (and the pricing is pretty fair) and the huge win in Dev Ops. Yes you'll have issues integrating with App Engine (especially if moving from AWS etc) but thats not specific to App Engine :)

As always, PM me if you want to chat about any of this on a call etc.

Rohit Paliwal COO @ Uvaca, Inc.

March 6th, 2015

Short Answer: Nothing in your use case suggests that Django won't be able to serve your needs. In fact most mainstream frameworks are fine for your purpose, just pick one and go with it.

Longer answer:

Most web frameworks are fine and scalable if you put enough resources, you just have to make a decision and pick one that you are able to find people for. Given that you seems to be just starting, for you it's going to be a people-resource problem and not a technology problem.

To put things into perspective Twitter was built using ruby on rails (RoR), which is the ruby equivalent of python/Django, and while they did move away from RoR it wasn't till they had really high traffic and had found a product-market fit. Similarly, Facebook used PHP.

However, do pay attention to overall architecture to make sure the architecture is scalable and more importantly it can evolve as over period your architecture would be evolving anyway with your needs at the product evolves and you will most likely have to include different technologies anyway. At that point if the framework that you're chose initially doesn't meet your needs you will be in a much better situation to make deeper architectural decisions and make changes 'if needed'.

Always happy to chat more if needed, but you should be fine with either of Django, Node or almost anything else.

Jared Hardy Founding Director at Data Roads Foundation

March 6th, 2015

Google App Engine makes Django easy to use and learn -- it's one of the first full web stacks supported by GAE. If your new CTO knows any Python (and they all should by now) then they can pick up Django very easily.

If you are actually seeing real world database writes increasing exponentially, then you need to concentrate on the distributed database part more than the web backend in the stack, and Django supports all the best SQL and NoSQL database backends. If you're saying "we need lots of database writes" and your CTO candidate says "change the web platform", not "optimize the database for writes using X database in Y distributed configuration with Z hashing (or sharding)", then don't ever hire them -- they are clueless about how full web stacks actually work. I noted "real world" in the first sentence because reads often surprise web developers in how far and fast they outstrip writes after the bootstrap phase. Read loads can easily be handled by load balancers and caches on gateways, no matter what web stack you're using. Stacks or add-ons that compile into static HTML, CSS, ECMAScript, and/or JSON files have the most flexible CDN options.

Troy Gardner Chief Technology Officer, Chief Brewing Officer at Cloud9 Brewing Systems

March 5th, 2015

I've done both in production, both will work at scale. It's more of a question of the data load. If it's frequent writes you'll have to look into which database is the most appropriate, e.g. redis, mongo, cassandra, mysql etc. Which will be the real bottle neck as the server shouldn't really be storing anything persistently.

Django is pretty well put together, and as a web framework, fine for complicated web sites, when you hire a django developer you'll likely get a closer set of standards than in the javascript world.

Node.js is better for real time websocket things, and newer cooler but way more chaotic, like there aren't that many standard node.js developers, as there are a zillion libraries that reflect different incompatible philosophies. Even the appeal of using javascript on both client and server really isn't quite applicable, as client side has to deal with older browsers, and node can use experimental features of the language.

Anonymous

March 26th, 2015

Lane, I love what I have heard a few others here say and that is forget Django make sure your solving problems, not chasing technology stacks.

It isn't really a question of can Django or can Django not, it can. Django is also to choose Python, also capable. There are many families of frameworks sprouting from many families of languages any of them being quite capable in the right hands of helping you meet your awesome goals even with the volume and middle-ware solutions you have suggested. When the conversation starts with django, and app engine and cloudsql and elasticsearch I think there is a potential risk of focusing on solutions instead of focusing on problems. I helped a company many moons ago pull together something that smelled very similar to what you have outlined here using. We used Silverlight, Javascript, C#, WebAPI, SQL and sovren-resume-cv-parser and a hint of Flash to provide a great solution and I don't think it would have struggled to burden some serious growth and scaling would have been fairly simple. The BEST solutions often emerge from a talented motivated team doing the work. Perhaps you trust them to get you where you need to go and do so in a minimalist sort of a way so you can move quickly past the feeling of a "technology" problem, to having a sales or marketing problem in many weeks and short months instead of many months and short years. Just my two cents but then again I don't know much about Django or Rails! :)

Lane Campbell Lifelong Entrepreneur

March 6th, 2015

I have a tech background in interconnected networks, virtualization and most corporate technologies. I'm for all intensive purposes new to software development. We had a CTO who started building our product in Django before he had to drop out for health issues. I've been interviewing replacement CTO's but so few people know the Django framework and are open to working with it that I've started talking to developers in general. We have had more than one individual tell us now that Django wouldn't work for our needs. For all intensive purposes this is our application:

We anticipate having a few hundred thousand users that complete a profile.
We plan to use ElasticSearch for the search engine.
There will be triggers checking changes to these profiles and sending out emails.
We are planning on using Google App Engine and Google CloudSQL

There are obviously some more granular details regarding theses features. I'm trying to discern if the talent market is just unaware of the capabilities of Django or if it's really an issue with the framework. My take away thus far is that it's a talent market perspective problem and not an issue with the framework. Nothing we are building seems out of this world difficult to scale with Django. Thoughts?

J.P. Maxwell Internet Entrepreneur & Owner Tipit, LLC

March 7th, 2015

I would second what most have said here as Django being quite capable. In the end it is just a framework though and you need to engineer your scalability to your use. Also will second Google app engine. We do an amazing instant and infinitely scalable web application using Django non relational so we can leverage the native Google data store. Your database will always be a huge scaling pain point and utilizing their data store removes that almost entirely. Be careful of cost and price structure though depending on your use it may be a good or a really bad deal. Our app is super spikey and surgey getting massive traffic one minute and almost none the next. For this the automatic instant scaling and the price works great. Your mileage may vary.

Andy Terrel Fashion Metric CTO , NumFOCUS President

March 6th, 2015

Disqus ran through some hoops to get Django to scale a number of years ago. It really comes down to backend for the data and some updates to the code. The code has come forward quite a bit now so I imagine many of their hoops are no longer necessary.

http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views

Matthew Griffiths Technology Entrepreneur, CTO, Startup Executive

March 6th, 2015

Hi Lane,

I would second Tory's comments on the standards around Django. Its a pretty (very) mature framework that is super well supported and which promotes a pretty standard way of doing things. This can be a huge benefit when trying to bring on developers and can make it easy for you to assess potential developer competences etc. We (MetaLayer) have used Django for project of all scale with very little issue. Not to mention the fact that it comes with a host of community project for things like monitoring, A/B testing, social sign on etc etc.

That said, there are things that Django is not as good at that a tech like Node.js is. Some of this can be overcome with the correct choice of database engine (Django over Mongo works as does Django on Google Cloud over Google Datastore), but this won't be enough for some very specific cases (reach out if you want to chat about this).

In short, if you have a great tech co-founder then they will likely know the right mix of tech to delivery your USP, if not, then a choice like Django (or RoR etc) can save a non-tech exec a load of trouble.

Question stats