Understanding web performance improvement through “Welcome to Moe’s”
I recently went to Welcome to Moe’s for lunch with some of my co-workers and it was interesting to see how much their process resembled that of serving websites, especially when thinking in terms of load balancers, web servers and most importantly how the content is served to the user.
The bottleneck was obvious, if only it were this simple optimizing magento. Here is the story:
You see, the mechanics of building my lunch order are pretty simple. I stand in a long queue of people buying lunch. I move up the queue, and eventually I place my order with the first guy, who’s job it is to pull out a basket lined in tin foil and scribbles something down on the tin foil. You choose what ever you want off the menu (search and scan), Joey Jr hey it Moe’s Mondays after all… cool.
You then move up the line to the second guy. Now you start describing what you want: what type of meat, beans or rice, sour cream… you get the idea. This person might even pop your burrito in one of those industrial George Foreman things.
After that, you move to the third person whose in charge of adding more condiments! REALLY??? Now, if you’re lucky, your burrito making experience concludes here, but guess what? You still ain’t gettin your food. You see, we’ve arrived at our bottleneck. Although Moe’s had seven people preparing food, they’ve only the one cashier and she’s also busy frying tortillas. WTF!?!!
Why in the blue moon are you channeling the bulk of your workforce to where the process works best? Why would you ignore the most obvious bottleneck in the world? The first three people are lighting fast and they get that burrito rolling in no time. Then It takes as long to get to the register so I can go off and eat my food. If my food is ready why do I have to wait two more minutes to eat it?
See the problem? Although preparing the content is lighting fast, it takes too long to transfer to my stomach. The time to last byte is way too high. Sure, Moe’s can throw more employees at the problem but that won’t make it better, hell it’d probably make it worse.
So if we were to start naming layers in Moe’s, I’d say we have 4 web nodes (Nginx+PHP-FPM, 1 as a load balancer), 1 server with Solr, 1 server with memcache and 1 with MySQL. Sounds like a decent setup, right? So where’s the problem? Why would this setup be so bottleneck prone?
Ok, now what?
The answer is really complex, and by complex I mean I’d have an easier time explaining the unified field theory. But here is my interpretation on it:
The dye has been cast. We design and develop based on concepts and ideologies founded long ago before the Internet was the driving force of the modern world. Every single application is built on the idea of collecting, saving or serving data synchronously and using a confine set of general purpose tools that may not be best for the job at hand. Good web sites are not just about semantics or exceptionally well crafted code but more about how to better use the resources at hand.
What the heck does that mean?
In short… it means that today’s approach to web development is where the problem lies. PHP is as slow as honey on a turtle’s neck and there is no denying that fact, but I’d argue that every language when used inappropriately is very slow and PHP is the most misused language in this four dimensional brane. Sure, compiled languages have better throughput and therefore faster “speeds”, but that doesn’t mean they are more efficient. Companies like Google, Facebook, Twitter and Netflix have discovered this problem and are fixing on their own way. We should start using the KISS of the Mechanic approach and make sure we pick of all the available tools the right one for the task at hand. This includes using the right programming language for the job and to not bastardize a tool for something it was never meant to do.
How do we fix Moe’s problem?
For one, we could add more cashiers and more lines for parallel serving (processing). We could also go as far as using analytics to find out the best way to organize ourselves. You could even use smartphones as cashiers and have several “unused” employees working as temporary cashiers. Heck, you could even use PayPal or NFC to pay. Architectural changes are expensive, very expensive, but we can accommodate changes into our approach such as caching or predictive paths. You could even try to figure out who are the regulars and have prepaid options for them.
It’s all about the options and the willingness to improve. Never settle for a process or situation that may look as optimal because that is the only limitation you’ll have to make it better. In development there is always 2 or 3 ways to improve what you’ve done and speed things up.