We discussed this interesting blog post by Joseph Perla and building and scaling quickly internally this week, and I wanted to share some brief thoughts about some of his points. Below are some topics from his post with my thoughts and reactions:
Keep it simple
No argument here. This is often easier said than done, however. There's a lot of art and gut to this. Many product managers equate high simplicity with low value.
Get it right (have tests).
Good testing practices and culture are key, and this is fairly well-known in the industry at this point
Don't hide power
They wrote a library called Pebbles which gives an HTML way to express a few common ajax interaction patterns. Definitely there are smart folks in the camp of making a domain specific language (DSL) for every project. He's doing that here in at least his pebbles AJAX system and his python twitter parsing code.
Leave it to the client
This is the most dubious point in the article in my opinion. This strategy works OK if you only have a single interface, but how realistic is that? Most projects are going to have one form or another of a desktop web interface, and some mobile variant thereof in all likelihood. Other projects might have an API available, language bindings, command line tools, fat clients, etc. As soon as you have a second interface, all that logic that you wrote in the client code must be duplicated, and by definition if it's duplicated in another framework or language, there will be behavioral differences. My rule of thumb is any computation/manipulation that you would want to work consistently in multiple interfaces belongs in the back end and should be exposed via an API. In any case, the trade-off here is a business strategy decision that should not be made in isolation by the technical team without clearly explaining the trade-off (harder/slower to build another UI) to the business stakeholders and product managers.
Continuity
He's not clear on exactly what he means but wow, if he's saying over the course of a month he was maintaining several API versions, that seems unnecessary. Support multiple API versions when you absolutely must for business reasons, not as a general rule.
Handle normal and worst cases separately as a rule
Sounds pretty reasonable to me.
Take-aways
The post nicely collects a bunch of common approaches for scalability. A lot of it boils down to "don't do things that don't scale" with a healthy dose of pragmatism thrown in there. The thing I always struggle with is knowing when building for scale from the start will be harder than not worrying about scale at the start and making trade-offs that favor validated learning sooner even if it means recoding later. Ara mentioned something to me last week about building things fast in Rails and then if its wildly successful having enough cash (and time/knowledge/product stability) around to get it rebuilt in Scala (basically the Twitter story). I've even seen Matz himself on video make the argument that if you develop rapidly in Ruby, get market traction, and have so much traffic that you can't handle it in Ruby, you've got a good problem. You can scale your Ruby system while optimizing/rewriting key pieces of it in other languages. But this is better than losing the market to a faster competitor. Both Facebook and Twitter have pulled this move successfully with things like HipHop for PHP and converting Rails code to Java. What we can say for sure is that a lot of projects that were very concerned with handling large scale from the beginning never made it to launch, or launched something nobody wanted, or launched 6 months after a competitor who captured marketshare, etc. I definitely lean toward the agile/lean/YAGNI end of the spectrum.
A general comment about the state of the industry now vs in the first .com bubble is we seem way better at figuring out what the market wants and also our technology stack makes it possible to build web apps that serve hundreds of thousands of users with a small team using readily available tools.
Tweet