Jekyll is a blog-aware static site generator — heavily integrated with the social code sharing service GitHub — the move to which, was primarily motivated by a desire to embrace the brave new, post-CMS world we now find ourselves in. While WordPress is great, 130 outages over the past six months (totaling more than a day’s worth of downtime), left a bit to be desired in terms of hosting.
Don’t get me wrong. WordPress can be configured to fly given the right setup, and that’s exactly what I set out to do. I got the best of the best. I spun up a shiny new AWS box, got Nginx with microcache up and running, APC for opcode, page, and object cache, and even put everything behind Varnish.
The pitch is straightforward. It leads to simple, flexible, and reliable websites that allow for a renewed focus on what actually matters: the content. Dave Cole over at Development Seed (also powered by Jekyll) put it best:
In the past, building websites with features like consistent templates and lists of aggregated content meant setting up complex content management systems. These CMSs consisted of templating logic, application code, and content databases so they could assemble web pages each time they were requested by site visitors. They were complicated systems that depend on many separate applications working together, like a web server to route page requests to a PHP application that uses predefined page layout templates to format content that’s stored in a MySQL database. Serving a page request required at least three separate applications all working together — any one failing would bring down the system…
And then there’s cost. Putting aside the value of time for a moment, shared hosting’s going to run you in the ballpark of $7 a month; AWS starts at $14, plus the cost of bandwidth and storage; and Jekyll, if hosted by GitHub? Free.2
I stood up the three options side-by-side, and ran them through the rigors of a performance testing tool humorously called Siege, the results of which can be found below.
I’m still unpacking some of the boxes of bytes, so if you notice something that doesn’t seem right, feel free to open an issue, or better yet, like what you see, feel free to fork and contribute. Embracing somewhat of a crawl, walk, run, or fail-fast philosophy, next up is outputting the pages as JSON and relying on Backbone to do the heavy lifting.
The CMS is dead. Long live the CMS.
WARNING: Geek Content!
Edit (Feb 2014): With the recent improvements to GitHub Pages, I wanted to give the benchmark another go. Turns out the transaction rate has essentially doubled since my original test, and could be even higher if the benchmark hadn’t been rate-limited by GitHub’s automatic denial of service mitigation:
siege -c 20 -t 30S -b ben.balter.com
Transactions: 3600 hits Availability: 100.00 % Elapsed time: 29.91 secs Data transferred: 19.12 MB Response time: 0.13 secs Transaction rate: 120.36 trans/sec Throughput: 0.64 MB/sec Concurrency: 16.03 Successful transactions: 3600 Failed transactions: 0 Longest transaction: 4.28 Shortest transaction: 0.04
siege -c 20 -t 30S -b ben.balter.com
The first test was to benchmark the homepage, the most heavily trafficked page on the site. Given 30 seconds of continuous traffic from 20 concurrent users, Bluehost was able to serve a meager 40 users. AWS managed an impressive 2000 users during that same time period (a 50x performance improvement), and did so twice as fast. Enter Jekyll with more than 2600 users (65x increase), responding on average to each in less than a quarter of a second.
Homepage via shared Hosting (Bluehost)
Transactions: 40 hits Availability: 100.00 % Elapsed time: 29.54 secs Data transferred: 0.68 MB Response time: 0.57 secs Transaction rate: 1.35 trans/sec Throughput: 0.02 MB/sec Concurrency: 0.78 Successful transactions: 40 Failed transactions: 0 Longest transaction: 0.71 Shortest transaction: 0.47
Homepage via Varnish + Microcache + Page Cache + Object Cache (AWS)
Transactions: 1954 hits Availability: 100.00 % Elapsed time: 29.39 secs Data transferred: 13.63 MB Response time: 0.30 secs Transaction rate: 66.49 trans/sec Throughput: 0.46 MB/sec Concurrency: 19.80 Successful transactions: 1954 Failed transactions: 0 Longest transaction: 0.92 Shortest transaction: 0.06
Homepage via GitHub Pages
Transactions: 2629 hits Availability: 100.00 % Elapsed time: 29.42 secs Data transferred: 2.71 MB Response time: 0.22 secs Transaction rate: 89.36 trans/sec Throughput: 0.09 MB/sec Concurrency: 19.86 Successful transactions: 2629 Failed transactions: 0 Longest transaction: 1.38 Shortest transaction: 0.06
siege -c 20 -t 30S -b ben.balter.com/aaaaaaa/
The true challenge comes in not from serving a static front page (which is presumably cached by WordPress after the first request), but in what happens when it has to reach into the database to retrieve content, for example, when processing a page that doesn’t exist.3 Bluehost squeezed out a single response each second, AWS just over 50, and Jekyll didn’t flinch at 80.
404s via shared Hosting (Bluehost)
Transactions: 30 hits Availability: 21.43 % Elapsed time: 29.58 secs Data transferred: 0.19 MB Response time: 14.93 secs Transaction rate: 1.01 trans/sec Throughput: 0.01 MB/sec Concurrency: 15.14 Successful transactions: 0 Failed transactions: 110 Longest transaction: 22.88 Shortest transaction: 0.00
404s via Varnish + Microcache + Page Cache + Object Cache (AWS)
Transactions: 1567 hits Availability: 100.00 % Elapsed time: 29.13 secs Data transferred: 14.71 MB Response time: 0.37 secs Transaction rate: 53.79 trans/sec Throughput: 0.50 MB/sec Concurrency: 19.83 Successful transactions: 0 Failed transactions: 0 Longest transaction: 1.13 Shortest transaction: 0.00
404s via GitHub Pages
Transactions: 2373 hits Availability: 100.00 % Elapsed time: 29.82 secs Data transferred: 10.48 MB Response time: 0.25 secs Transaction rate: 79.58 trans/sec Throughput: 0.35 MB/sec Concurrency: 19.92 Successful transactions: 0 Failed transactions: 0 Longest transaction: 1.42 Shortest transaction: 0.00
The other concern was uptime. With the AWS route you may get the performance, but with all that complexity, it’s increasingly more like that something would go wrong, harder to diagnose and resolve, and unlike shared or managed hosting, if your site goes down at 3:00 am, the only person to call is yourself. (no thanks.)
With Jekyll, because the files are simply sitting on the server, absent a catastrophic failure, when things go wrong with Jekyll, it simply keeps serving the old content.4
It’s cheaper, it’s faster, it’s simpler, it’s worry free, and in my opinion, it’s the future. Welcome to the post-CMS world.
That’s free as in speech and free as in beer. ↩
Requesting a page that doesn’t exist will require WordPress to run multiple database queries to attempt to find the page, a request that would most likely not be cached in the event that the 404 was sent in error. ↩
GitHub’s build queue has been backing up every once in a while as of late, but if a change isn’t instantaneous, I’m okay with that. ↩
Prior to GitHub, Ben was a member of the inaugural class of Presidential Innovation Fellows where he served as entrepreneur in residence reimagining the role of technology in brokering the relationship between citizens and government. Ben has also served as a Fellow in the Office of the US Chief Information Officer within the Executive Office of the President where he was instrumental in drafting the President’s Digital Strategy and Open Data Policy, on the SoftWare Automation and Technology (SWAT) Team, the White House’s first and only agile development team, and as a New Media Fellow, in the Federal Communications Commission’s Office of the Managing Director. His paper, Towards a More Agile Government was published in the Public Contract Law Journal, arguing that Federal IT Procurement should be more amenable to modern, agile development methods. More about the author →