|
Jeremy presents strategies for identifying common performance problems in Ruby on Rails applications—and shares ways to fix them.
November 04, 2008
URL:http://drdobbs.com/web-development/performance-on-rails/212000386
Jeremy is a software engineer at PatientsLikeMe (www.patientslikeme.com), a social network and health-management tool. He can be contacted at [email protected].
You've built a cool new web application with Ruby on Rails (or similar framework) and released it to the world. Everything works as expected, but users are starting to complain about the app being too slow. And you're starting to regret following that mantra about "premature optimization."
Improving your web application's performance can be a daunting task, but it's an important factor in keeping users happy and productive. In this article, I present strategies for identifying common performance problems in Rails applications and ways to fix them.
The process of tuning the performance of a Rails app is not much different than any other software framework: Identify your biggest bottlenecks, remove them, and repeat until performance is acceptable. If this sounds familiar, you've probably performance tuned other applications built with another framework. Much of your experience will translate from project to project, framework to framework.
The performance characteristics of a Rails app in a development environment can be very different than a production environment. For example, class caching is enabled by default in production, which means that the code is loaded once and kept in memory until the application is restarted. In development, every class is reloaded on every request, which makes for a shorter feedback cycle to enable rapid, iterative development, but adds some noise to your performance metrics.
To remove this and other variations in performance, emulate the production environment as much as possible. The easiest way to do this is to simply run the application in production mode. If you edit your config/database.yml file and point the "production" environment to your development database, you can use your development database in production mode for profiling:
development: &development adapter: sqlite3 database: db/development.sqlite3 production: <<: *development
When you start your server in production mode, Rails will now use your development database:
$ script/server -e production
You'll just need to remember to restart your server to pick up any code changes because class caching is enabled in production mode.
You might also find that your svelte development database doesn't reflect the real world. To simulate a user's experience in the wild, test with a local copy of your production database (sanitizing sensitive data—social security numbers, credit-card information, e-mail addresses, and the like—of course).
There's one exception. I use the QueryTrace plug-in for Rails (github.com/github/query_trace) in development mode, but I don't deploy it to production. This plug-in logs the stack trace when each SQL query is logged, which makes it easy to pinpoint the code that's causing the query to be executed. In production it probably impacts performance (ironically undoing some of the progress it enables), so don't add the plug-in to your project's version-control repository.
A typical midsized Rails application has 10-20 database tables and corresponding model classes, a dozen controllers, and about 100 view templates. That translates into at least a few thousand lines of code. Now where are your bottlenecks?
Consider your application as a whole. What are the most popular page requests? Are there any obviously slow responses? If your app is small enough, you might be able to answer this off the top of your head with confidence. Otherwise, I suggest gathering some statistics about your application's usage and aggregate performance using a service like FiveRuns (www.fiveruns.com) or NewRelic RPM (www.newrelic.com), or build your own tools to mine some basic stats like frequency (how often a request is made) and duration (how long those requests took) from your logs.
Once you've identified your problem areas, prioritize them, starting with the most popular slow pages.
Before making any changes to improve performance, you should have a point of reference, a benchmark, to know how much improvement you've actually made with each change, or if you've caused a regression.
Load your slow page and see how long it takes to complete. The easiest way to do this is to watch your development log:
$ tail -f log/development.log
When you load the page, the last statement that is logged is something like this:
Completed in 3.05327 (0 reqs/sec) | Rendering: 0.87915 (28%) | DB: 1.49075 (48%) | 200 OK [http://localhost/foo]
This request completed in about three seconds, which isn't very good. Your goal is to make that number as small as possible, increasing the number of requests per second. That translates into a more responsive application, but also an application that will be easier to scale because you'll be able to handle more concurrent requests with fewer resources.
It's a good idea to run the same request a few times to make sure the metrics are consistent. There may be some outliers due to garbage collection or other system activity—ignore any outliers and find an average response time.
In the aforementioned example, the rendering time is on the high side, but almost half of the time was spent executing database queries, so I want to focus on that.
Most Rails applications use Active Record, a simple object-relational mapping (ORM) strategy that maps tables to classes, rows to objects, and columns to object attributes. Like any other ORM implementation, Active Record makes a lot of data-related tasks easy, but it's also easy to run into performance problems if you treat the database as little more than a junk drawer. Many common database-related bottlenecks are the result of slow queries or the application executing too many queries.
As fast and powerful as modern relational database engines are, your database won't perform well without monitoring and an occasional tune-up.
Many databases provide a "slow query log," which logs every query that takes more than a specified amount of time—two seconds is a good place to start to find the worst offenders, but you should soon be looking for queries that take as little as 250 milliseconds (0.25s). Another option is to use a profiler or simply tail the log, examining the execution time of each query.
Why is the query slow? There's one way to find out for sure—use your database's "explain plan" feature to see how the query is being processed. Chances are you'll find a full-table scan (sometimes called a "sequential scan") that you could fix by adding an index on some (or all) of the columns that you're filtering or ordering by. You might find that an index is being used, but a better index could be added.
There are too many variables for me to tell you what index would be best, so I encourage you to experiment and research the indexing strategies available in your choice of database. Add an index that you think will help and run the explain plan again. Try modifying the query to obtain the results in a slightly different, more efficient way.
When you've tuned the query and found the right index, apply it to your project by modifying the code that builds the query and create a migration to add the index.
Now consider a blog post, which can have many comments, each comment by a different user. Scanning the log file, you might find a query like this:
User Load (0.000366) SELECT * FROM "users" WHERE ("users"."id"=7)
This log entry shows that the User model was used to load the user with ID 7 from the database, and the query took 0.000366 seconds—pretty quick. However, a red flag should go up if you see a series of similar queries like this:
Comment Load (0.004620) SELECT * FROM "comments" WHERE ("posts"."id" = 43)
User Load (0.000366) SELECT * FROM "users" WHERE ("users"."id"=7)
User Load (0.000306) SELECT * FROM "users" WHERE ("users"."id"=3)
User Load (0.000426) SELECT * FROM "users" WHERE ("users"."id"=9)
User Load (0.000378) SELECT * FROM "users" WHERE ("users"."id"=5)
User Load (0.000452) SELECT * FROM "users" WHERE ("users"."id"=8)
This pattern indicates an n+1 problem. The list of comments for a blog post is fetched in one query, but fetching the user who wrote each comment (presumably in a loop) is n more queries (where n is the number of comments), so a post commented on by 50 readers will require no fewer than 51 queries to fetch all of the data. Even if each of those queries is relatively fast, they can quickly add up. The following code demonstrates the problem:
# posts_controller.rb
def show
@post = Post.find params[:id]
end
# view - show.html.erb
<% for comment in @post.comments %>
<p>
<%=h comment.user.name %> says...<br />
<%=h comment.text %>
</p>
<% end %>
Rails provides an easy way to solve basic n+1 issues: eager fetching with the :include option. When fetching the post, we can fetch all of its comments and each corresponding user in far fewer SQL queries:
def show
@post = Post.find params[:id], :include => { :comments => :user }
end
In versions of Rails prior to 2.1, this statement would fetch the post, its comments, and the users with a single complicated query composed of SQL JOINs. The current version (2.1 at this writing) does not use JOINs, instead favoring one query per model that you specify in your :include parameter:
Post Load (0.000537) SELECT * FROM "posts" WHERE ("posts"."id"=1)
Comment Load (0.001683) SELECT "comments".* FROM "comments" WHERE
("comments".post_id IN (1))
User Load (0.001375) SELECT * FROM "users" WHERE ("users".id IN
('6','1','2','3','4','5'))
In this example, one query fetches the post, a second query fetches the comments for that post, and a third query fetches the users that wrote the comments. This approach makes for a couple of extra queries, but JOINs can be expensive and become a bottleneck in their own right. Either approach is a big improvement over the original n+1 problem.
There are times that the :include option alone isn't sufficient. What happens if a post is wildly popular and has comments by hundreds, even thousands, of users? The IN clause in the users query will have a lot of IDs, and performance will begin to suffer—the query may even fail if the list of IDs is too long. The solution is to fetch even more eagerly using a JOIN in your query. Instead of writing a query for find_by_sql, encourage Active Record to use a JOIN for you:
include => :user, :conditions => 'users.id is not null'
When you include a filter on users.id in the :conditions option, Active Record smartly fetches the users with a JOIN (specifically a LEFT OUTER JOIN) to satisfy the dependency you've introduced in the WHERE clause on the users table. It's probably a good idea to comment code like this to document the intent of the :conditions option.
Posts have a transitive dependency on users: A post has comments, and each comment has a user; therefore, a post has commenters. Use Active Record's has_many :through to declare this dependency:
# post model has_many :commenters, :through => :comments, :source => :user
Now each post provides a #commenters method, which returns the list of users who have commented on the post by executing a SQL query like this:
User Load (0.001009) SELECT "users".* FROM "users" INNER JOIN comments ON users.id = comments.user_id WHERE (("comments".post_id = 1))
To get the user from that list for a particular comment, we could use some simple Ruby in the view:
# this is not optimal!
@post.commenters.detect { |u| u.id == comment.user_id }
Note that I'm calling #detect, not #find. This is because I want to invoke the method provided by the Enumerable module, and ActiveRecord::Base#find overrides Enumerable#find.
This strategy works pretty well for a small number of commenters; however, #detect performs a sequential search, an O(n) operation. This method won't perform well for a large value of n, when there are a lot of commenters. You might want to build a lookup hash for constant-time lookup, or O(1):
# controller @commenter_lookup = post.commenters.inject(Hash.new) do |hash, user| hash[user.id] = user; hash end
Now the view can fetch the user for a given comment from the hash:
# view @commenter_lookup[comment.user_id]
Another example of when the :include option isn't enough is when you are fetching aggregated data. Let's say you'd like to fetch the number of comments each user has created to display that next to the user's name. If you calculate it user by user (via comment.user.comments.count), you'll have another n+1 problem. One approach is to calculate the data for all relevant users in a single SQL query and build a lookup hash.
# controller @comment_count_lookup = @post.comments.all( :select => 'user_id, COUNT(*) as num_comments', :group => 'user_id').group_by(&:user_id)
This code, which could be in a controller or model, gets the number of comments for each user by grouping by user_id and counting the rows in each group with the SQL COUNT function. The #group_by method creates a hash where the key is the user_id (because it's returned by the block), and the value is an array containing all items with that key. The view can use this hash to look up the count for a given user:
# view <%= @comment_count_lookup[comment.user_id].first.num_comments) %>
The view looks up the record for the comment's user_id in the hash, and needs to call #first to pull the one and only record out of the array before getting the count.
Code like this should be well commented, particularly the fact that it addresses a performance issue. There is another option for this kind of problem: denormalization.
"Normalize until it hurts. Denormalize until it works."
In a perfectly normalized database, there is only one representation of any particular fact. Taken to the extreme, this results in a space-efficient database with no chance of duplication or inconsistency. This is a wonderful ideal, but it comes at a cost: time-efficiency. We software developers walk a fine line between idealism and pragmatism.
Active Record makes it very easy to denormalize the number of comments that a user has created. Just enable a counter cache on the association:
class Comment < ActiveRecord::Base belongs_to :post belongs_to :user, :counter_cache => true end
And create a migration to add an integer column named comments_count to the users table. Here's a migration that adds the column and calculates the count for each user because our users have created comments before we added this denormalization:
class AddUsersCommentsCount < ActiveRecord::Migration
def self.up
add_column :users, :comments_count, :integer, :default => 0
User.reset_column_information
User.all.each do |u|
User.update_counters u.id, :comments_count => u.comments.count
end
end
def self.down
remove_column :users, :comments_count
end
end
Every time users create a new comment, their comments counter cache is incremented, and if they delete a comment, the counter cache is decremented. Now the view can display the user's comment count without any additional database queries:
<%= comment.user.comments_count %>
Counter caches are a simple type of denormalization that is built into Active Record, but your requirements might be more complex. Let's say you want to denormalize the date and time of the user's first comment. I would use the before_create and before_destroy lifecycle hooks to keep the data in sync:
# comment.rb def before_save # time can only move forward, so this is pretty simple user.update_attribute (:first_comment_at, Time.now) if user.first_comment_at.nil? end def before_destroy # need to handle case where user deletes their first comment earliest = user.comments.first :conditions => ['id <> ?', self.id], :order => 'created_at' user.update_attribute(:first_comment_at, earliest.created_at) end
Ruby the language and Rails the framework often take a beating from detractors on the question of performance. Ruby's not in the running to be the fastest language, and it's not even the fastest interpreted language. Rails isn't the fastest framework. However, if Ruby or Rails is your bottleneck, consider yourself lucky! Most unexpected performance challenges are related to querying the database and aren't detectable until the app has been in the wild, which is no different than any other software development framework.
Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.