RSpec my authority!

If you've been using RSpec in your Rails apps, you know that it's an excellent tool to help drive the design of your code at the object level. However if you wanted to do integration or acceptance testing, you'd have to use Rails' Test::Unit integration tests or rely on an external framework such as Selenium or Watir. RSpec has bindings for both Selenium and Watir, but both frameworks require a browser to run thus are quite heavy-weight.

A few days ago, David Chelimsky announced that RSpec's Story Runner now works with the rspec_on_rails plugin. Story Runner is the result of Dan North's hard work to merge his rbehave project into RSpec. This gives us the ability to write acceptance tests using RSpec's expressive expectations. In addition, Story Runner provides a structure for your stories (acceptance tests) that helps clearly express the assumptions and expectations of your code.

You might be familiar with the "Given...When...Then..." structure of a story. If not (I wasn't), Dan North has a great explanation of what goes in a story.

Now that you have some background on user stories, I'll show you an example of an integration test I wrote with Test::Unit a while back and then converted to use Story Runner earlier today.

The most fundamental user story we have is a user viewing his home page. A user's home page shows information that's important to him, and only that information that is available to him. So for example the root user should be able to see all the companies in the system, all the videos, etc. A regular user would only see videos that belong to his company. A reseller would see videos belonging to all children companies.

Here's my integration tests written in Test::Unit. I factored out the code into a DSLish¹ library so that the tests read easily.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

class UserStoriesTest < ActionController::IntegrationTest
  def test_login_as_root
    new_user("root")
    
    new_session do |root|
      root.goes_to_login
      root.logs_in_with "root", "test"
      root.is_viewing_page? "main/root_home"
    end
  end

  def test_login_as_publisher_with_no_videos
    new_user("novideos").in_company("No Videos")
    
    new_session do |novideos|
      novideos.goes_to_login
      novideos.logs_in_with "novideos", "test"
      novideos.is_viewing_page? "companies/show_initial"
    end
  end
  
  def test_login_as_publisher
    new_user("publisher").in_company("Publishing Company").with(1).video
    
    new_session do |publisher|
      publisher.goes_to_login
      publisher.logs_in_with "publisher", "test"
      publisher.is_viewing_page? "main/user_home"
    end
  end
  
  def test_login_as_reseller
    new_user("reseller").in_reseller
    
    new_session do |reseller|
      reseller.goes_to_login
      reseller.logs_in_with "reseller", "test"
      reseller.is_viewing_page? "main/reseller_home"
    end
  end
  
  private
  def new_user(login)
    u = User.new  :login => login, :email => "#{login}@twistage.com", :password => "test",
                  :password_confirmation => "test"
    u.role = "root" if "root" == login
    u.save!
    u.activate
    u.extend UserExtensions
    u
  end
  
  module UserExtensions
    def in_reseller(name = "Reseller")
      r = Reseller.create! :name => name
      update_attribute :company, r
      self
    end
    
    def in_company(name = "Company")
      c = Company.create! :name => name
      update_attribute :company, c
      self
    end
    
    def with(num)
      @num_with = num
      self
    end
    
    def videos
      @num_with.times do |i|
        Video.create! :company => company, :site => company.root_site, :title => "Video ##{i}"
      end
      self
    end
    alias_method :video, :videos
  end
  
  module UserActions
    def goes_to_login
      get "/login"
      assert_response :success
      assert_template "sessions/new"
    end
    
    def logs_in_with(username, pass)
      post_via_redirect "/sessions", :login => username, :password => pass
      assert_response :success
    end
    
    def is_viewing_page?(page)
      assert_template page
    end
  end
  
  def new_session
    open_session do |sess|
      sess.extend UserActions
      yield sess if block_given?
    end
  end
end

Personally I like how the stories read. They're tiny (only 6 lines) and read very easily. I factored out some things to read more like English so that our QA guy² could better understand them. Hopefully as time goes on we'll create enough helper methods that he can write some of his own tests.

Despite how sexy the tests are, there were some things that I found a bit unsettling as I wrote them.

• The code reads very well, but it's still code. This can be a mental block for a QA person. My dream of having our QA guy write his own tests is likely a pipe dream, unless we hire a QA person who is also jazzed about Ruby.
• Factoring things out so that it looks like English could easily lead to bugs. The particular example I showed is quite simple, but other stories will contain much more complex setup. You can see that I got a tiny bit clever in writing my "with(1).video" chain. That code is 100% untested. I do not have enough confidence to write tons of untested code for complex situations, particularly when my actual test code relies on it!

After thinking through those two points it seems like my biggest problem is trying to extract a DSL so that a non-programmer can write tests. My code is error-prone, and it's not likely to be used like that anyway, so if I could just abandon that hope then maybe Test::Unit would suffice. But then I'd have assert_template littered throughout my tests. eww.

So I was pretty excited when I heard that Story Runner can be used with Rails apps. Without further ado, here are those same stories converted to Story Runner format.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

Story "View Home Page", %{
  As a user
  I want to view my home page
  So that I can get a birds eye view of the system
}, :type => RailsStory do
  
  Scenario "Publisher with no videos" do
    Given "a company named", "No Videos" do |name|
      @company = Company.create! :name => name
    end
    
    And "a user named", "novideos" do |login|
      @user = create_user login
    end
    
    And "the user belongs to", "company", "No Videos" do |klass, company_name|
      @user.update_attribute :company, klass.classify.constantize.find_by_name(company_name)
    end
    
    And "logged in as", "novideos" do |login|
      post "/sessions", :login => login, :password => "test"
    end
    
    When "visiting", "/" do |page|
      get_via_redirect page
    end
    
    Then "viewer should see", "companies/show_initial" do |template|
      response.should render_template(template)
    end
  end
  
  Scenario "Root user" do
    Given "a user named", "admin"
    And "a company named", "Company1"
    And "a company named", "Company2"
    
    And "the user has the role", "root" do |role|
      @user.update_attribute :role, role
    end
    And "logged in as", "admin"
    
    When "visiting", "/"
    
    Then "viewer should see", "main/root_home"
    And  "page should show company named", "Company1" do |name|
      response.should have_text(/#{name}/)
    end
    And  "page should show company named", "Company2"
  end
  
  Scenario "Publisher with one video" do
    Given "a user named", "publisher"
    And "a company named", "Has Videos"
    And "the user belongs to", "company", "Has Videos"
    And "# published videos belonging to company", "Has Videos", 2 do |company_name, num_videos|
      create_videos_for(Company.find_by_name(company_name), num_videos)
    end
    And "logged in as", "publisher"
    
    When "visiting", "/"
    Then "viewer should see", "main/user_home"
    Then "page should show video titled", "Has Videos 1" do |title|
      response.should have_text(/#{title}/)
    end
    Then "page should show video titled", "Has Videos 2"
  end
  
  Scenario "Reseller user" do
    Given "a user named", "reseller"
    And "a reseller named", "Big Bad Reseller" do |name|
      @company = Reseller.create! :name => name
    end
    And "the user belongs to", "reseller", "Big Bad Reseller"
    And "a company named", "Resold"
    And "the company belongs to reseller", "Big Bad Reseller" do |company_name|
      @company.update_attribute :reseller, Reseller.find_by_name(company_name)
    end
    And "# published videos belonging to company", "Resold", 1
    And "a company named", "Indy"
    And "# published videos belonging to company", "Indy", 1
    And "logged in as", "reseller"
    
    When "visiting", "/"
    
    Then "viewer should see", "main/reseller_home"
    And "page should show company named", "Resold"
    And "page should not show company named", "Indy" do |name|
      response.should_not have_text(/#{name}/)
    end
    And "page should show video titled", "Resold 1"
    Then "page should not show video titled", "Indy 1" do |title|
      response.should_not have_text(/#{title}/)
    end
  end
end

def create_user(login)
  user = User.create! :login => login, :password => "test", :password_confirmation => "test",
                      :email => "#{login}@twistage.com"
  user.activate
  user
end

def create_videos_for(company, num_videos)
  num_videos.times do |i| 
    v = Video.new :company => company, :site => company.root_site, :title => "#{company.name} #{i + 1}"
    v.status = "complete"
    v.publisher_name = "#{company.nickname}-publisher"
    v.save!
  end
end

The first thing that you'll probably notice is that it's a lot more verbose than the corresponding Test::Unit test. You might love it or hate it or not really care, that's going to be a matter of personal preference. Even though it's more verbose, it's not much longer than Test::Unit. TU comes in at 105 lines and Story Runner at 114. But the Story Runner version tests for some things that the TU version doesn't, such as testing that video titles and company names show up on the proper pages. I think if I were to take those expectations out then it would actually be shorter than the TU version.

What does it mean then if Story Runner is shorter but more verbose than Test::Unit? To me it means that Story Runner gets closer to the problem domain. This is evident in how much infrastructure code I had to write. In the 105 line TU file, 60 of those lines just create the DSL I use for the tests. In comparison, the Story Runner version has 14. I had to write lots of code to support my tests so that they could be in a language as close to the problem domain as I want. Code that I'm too lazy to test thus is prone to lots of errors. Story Runner on the other hand gives me the language I want right off the bat, and it's all tested to boot!

Right there we've killed off my biggest objection to integration testing with Test::Unit. Most of my test code uses RSpec's library to test my code. I have very little untested infrastructure code, and the little bit I do have is trivial.

My other objection was that the tests were still too much code for a QA person to handle. Story Runner gets around this by using plain strings everywhere. Ideally your QA people will have the ability to generate sentence fragments, which is basically all it takes to write a story in RSpec.

I don't expect my tester to write actual test code. But with Story Runner he can, or at least get close. He can easily write

1
2
3
4
5
6

Scenario "Basic user" do
  Given "A created user"
  And "two existing videos"
  When "visiting /videos"
  Then "both videos should be shown"
end

When he runs that, one of three things will happen:

1. The test will pass. yay!
2. The test will fail with an error message that he understands
3. The test will blow up with an error message that makes no sense

Story Runner puts the tester in a unique position to handle scenarios 2 and 3. Let's say that the test fails, and because of the error message, the tester determines that some setup code hasn't been written. Perhaps the "two existing videos" given hasn't been implemented anywhere. If he's feeling brave, he can go ahead and attempt it himself. Now we have a fully-functioning acceptance test that the developers can use to complete the feature. If the tester can't figure it out, he can just add a pending command:

1
2
3
4
5

...
  And "two existing videos" do
    pending "Pat get some work done"
  end
  ...

He can spend two minutes trying to complete the test and just move on if he doesn't succeed. Now as a developer I can go in and implement the setup code. I know exactly what to do because he wrote it in plain English. Which brings me to the next benefit of Story Runner-style tests, and it's a huge one.

All assumptions and expectations are clearly documented.

I didn't appreciate this benefit until I had finished writing the stories. I wrestled with how much more verbose the stories were than their Test::Unit cousins. Then I realized just how much more information they actually gave me. Operating under a certain set of assumptions, when a particular action is performed I expect certain outcomes. Or more succinctly, Given...When...Then.

I remember when I was about 8 years old my mother had made one of her amazing pumpkin cheesecakes. I took a big ol' slice and enjoyed it until she yelled at me. She had made the pie for one of those mom clubs, but I assumed it was for our family. This is when I first heard, "You know what happens when you assume...you make an ass out of u and me."

So assumptions are kind of a big deal, especially in programming. Dave Astels even goes so far as to say that if the reader of your code makes ungrounded assumptions, then your code sucks:

If your test stories don't outline their assumptions, then they're merely anecdotes, not stories. The reader can take away whatever the hell he wants from them. Assumptions are what ground the stories to reality and give them communication value. When you have a story structure that requires you to express assumptions, you don't end up with unspoken ones. Those times where a communication error results from an unspoken, ungrounded assumption, it becomes trivial to express the assumption in the test and avoid those communication problems in the future.

I've only played with Story Runner for a little bit, but I'm excited because already I can see how it will help me avoid a number of coding problems and communication problems. And since problems in software development basically fall into either coding or communication, it's nice to have a tool that helps me avoid them.

BDD comes in two flavors, you could say. RSpec has supported BDD at the object level, helping you specify interactions between objects and the outcomes of operations. However up until recently it wasn't practical for specifying the behavior of a system from the user's perspective. With the new Story Runner, RSpec is a complete BDD framework.

Update: Output from integration tests and stories

Another benefit of Story Runner is that it gives you helpful output. You could literally send the output to a customer for verification.

Integration test:

$ ruby test/integration/user_stories_test.rb 
Loaded suite test/integration/user_stories_test
Started
....
Finished in 1.186151 seconds.

4 tests, 16 assertions, 0 failures, 0 errors

Story Runner

$ ruby stories/view_home_page.rb 
Running 4 scenarios:
Story: View Home Page

  As a user
  I want to view my home page
  So that I can get a birds eye view of the system


Scenario: Publisher with no videos

  Given a company named No Videos
  And a user named novideos
  And the user belongs to company,No Videos
  And logged in as novideos

  When visiting /

  Then viewer should see companies/show_initial
.
Scenario: Root user

  Given a user named admin
  And a company named Company1
  And a company named Company2
  And the user has the role root
  And logged in as admin

  When visiting /

  Then viewer should see main/root_home
  And page should show company named Company1
  And page should show company named Company2
.
Scenario: Publisher with one video

  Given a user named publisher
  And a company named Has Videos
  And the user belongs to company,Has Videos
  And # published videos belonging to company Has Videos,2
  And logged in as publisher

  When visiting /

  Then viewer should see main/user_home
  And page should show video titled Has Videos 1
  And page should show video titled Has Videos 2
.
Scenario: Reseller user

  Given a user named reseller
  And a reseller named Big Bad Reseller
  And the user belongs to reseller,Big Bad Reseller
  And a company named Resold
  And the company belongs to reseller Big Bad Reseller
  And # published videos belonging to company Resold,1
  And a company named Indy
  And # published videos belonging to company Indy,1
  And logged in as reseller

  When visiting /

  Then viewer should see main/reseller_home
  And page should show company named Resold
  And page should not show company named Indy
  And page should show video titled Resold 1
  And page should not show video titled Indy 1
.



4 scenarios: 4 succeeded, 0 failed, 0 pending

You'll notice there are a couple funky parts related to passing values to blocks. For example, "Given # of published videos belonging to company Resold,1" is a bit awkward. I have a feeling we'll fix that in the coming weeks. Perhaps something like

Given "%{num_videos} published video belonging to company '%{company_name}'", 1, "Resold" do |num_videos, company_name|

Which would produce a more natural "Given 1 published video belonging to company 'Resold'"

Over all the output is much nicer and uses the ideal language for communicating with a customer. "This story is comprised of four scenarios, and as you can see, they all work."

1. I don't really like the term DSL, since nobody really knows wtf it means
2. I say QA guy, but this all applies the same if you're working directly with a customer