I've been doing the consulting/teaching performance thing for a while now. One thing I've noticed is that applications fall into one of two groups, and each group has its own unique performance issues. It's helpful to realize which "group" your application is in, because it may help you to understand what types of performance issues you should focus on, and what types of problems you can generally ignore.
Type A Applications: high volume leads to big problems
The first category is generally what people think about when they think about performance problems: high request volume leads to performance and scaling issues.
The typical business or organization that runs into these problems usually has an application which has a high number of requests per user and a high number of daily active users. Here are some types of applications that fit this model.
- Social networks
- E-Commerce, at higher scales
- B2C applications, at higher scales
These "Type A" applications usually have the follow types of backend performance problems:
- Bad table layout/insufficient denormalization. SQL queries that were fine as the organization grew gradually get worse over time, as excessively complex joins and unions start to take more time as tables grow. Denormalization or a rethink of the table structure would solve this issue, but instead usually a poorly-thought-through caching scheme is implemented, which doesn't solve the problem and just gives DevOps more to do.
- Poor autoscaling, leading to outages. Usuallly, companies at this scale are underscaled and wondering why response times are poor. An autoscaling scheme that actually worked would allow them to easily absorb changes in load.
- Poor database scaling. Around ~50,000 requests-per-minute, single-node SQL databases start to fall apart. The team usually has no experience with sharding or replication, so this process is often painful. Rather than doing that, teams often make the mistake of pursuing alternative datastores which are not suited for their purposes and only temporarily solve the problem, such as NoSQL or Redis, the problem reappearing as soon as _that_ datastore runs out of single-node capacity.
- Scaling with money rather than decreasing response times. Sometimes Type A companies overlook oppportunities for reducing server spending by decreasing response times, and instead opt to just pay large server bills.
This application type, while it gets a lot of headlines and blog ink, is actually far less common than the second type.
Type B Applications: low volume with wide featuresets.
This second category is by far the more common type of application I end up seeing. Generally, the model is of a B2B application at any scale, which is suffering from low response times almost universally across the board.
Typical applications include:
- Almost any B2B SaaS.
- Low-volume consumer applications that are searching for product market fit.
- Legacy applications.
These applications generally don't do a ton of traffic, but have performance problems that are just as serious for the business as Type A applications:
- Low response times caused by excessive amounts of code running per request. The primary burden of Type B applications is the size and complexity of their codebase. This is remedied by a) having more senior engineering talent and b) understanding the use of a profiler.
- Memory bloat. Type B apps deploy so many features and work with large amounts of data, so are primed for memory bloat issues.
- N+1. Type B apps typically work with complex business rules which can easily turn into N+1s without the proper use of ActiveRecord.
- An overstretched developer team, incentivized to focus on velocity and features. This is a hard one to fix, because it's a cultural issue. But I often find that Type B teams have underfunded development by either not having enough senior talent, not having a dev process which incentivizes maintenance, or simply not having enough engineers at all for the size of the codebase.
- Usually overscaled. Type B apps usually get overscaled because the team doesn't have any knowledge around scaling or capacity planning, leading to excessive server bills.
Sometimes, applications can have parts which are Type A and some which are Type B (e.g. a consumer-facing e-commerce section is Type A, but the admin panel is Type B).
What type of application are you running?
|
|
|
|