What’s in an app?

That’s a key question when determining what to test and how. We’re looking at rails apps, here, so the basic model is MVC. So is this what’s in an app?

This is the basic model of a Rails app. However, from the perspective of deciding what to specify and test, it’s incomplete. I’m not talking about helper classes which belong in one of those three boxes. I’m not even talking about things like daemons, or mailers, or those kinds of components, because actually they still fit in the MVC model, just one without end-user interaction. What’s missing from this diagram, I believe, is the dependencies.

One of the problems which I (and many people, looking at the rspec-users mailing list and the #rspec channel on freenode) have hit many times and which is very irritating is what to do about those validators and relationships. How do I specify my polymorphic has_many :through, eh?

First, the big stuff

The first point I’d like to make in this article is that with the Story engine, we now have the tools to specify almost the entire system. The mapping is fairly simple:

First, you write Stories (in an outcome-focused way, with the “Given/When/Then” approach), which are essentially a much nicer way to express integration tests in a business-focused way.

Next, you write specification to define how your code will make those stories pass. You write a behaviour-focused specification each for the model, the view, and the controller, because these are the key architectural separations. Please note that this doesn’t imply you should write a specification for each method - specifications should be linked to behaviours, not low-level programming constructs. However, they are inevitably influenced by the architecture of the system.

Finally, once the specifications pass, the stories should pass too. If later, you refactor or change your code, you have your integration tests holding you up and making sure your system is still working as a whole.

You have two scenarios for a change: an internal change (e.g. refactoring) or an external change (change in the user requirements). The first is driven by a change in the code, the second is driven by a change in the Story.

If you change your code, the following things sequence of events should happen:

1. First, the spec for the item you changed will break. Start by fixing that spec to be correct.
2. Simultaneously, if you changed the behaviour, some of your Stories will break. This is not because they are wrong, though, only because your specs are mismatched - ie you’ve updated your spec for your item, but not for its dependencies. Don’t change your Stories at this point.
3. You proceed to update the specs for everything which was dependent on the item you changed. They break. You update the code. They pass. You’ll know that this process is complete because your Stories will pass again.

If you change the Stories, the process is subtly different:

1. If you change the Story, it will start by breaking.
2. You then change some specs to make the Story pass.
3. This may cause other Stories to break. At this point, you should not change those Stories (unless they are directly affected), but instead, fix the specs in the method outlined earlier, when looking at an internal change.

If your Stories are comprehensive enough, these processes will stop you from introducing bugs, while at the same time allowing you to spec the “right way” (at least according to me): by specifying behaviours rather than outcomes.

Issues with speccing validators and relationships

There are still some issues to do with validators and relationships. How should they be specced? The arguments tend to go like this:

Someone writes code like:

##################
describe User, "with neither facebook uid nor phone number set" do
  include UserSpecHelper

  before(:each) do
    @user = User.new
  end

  it "should not be valid" do
    @user.should_not be_valid
  end
end

##################
describe User, "with facebook uid" do
  include UserSpecHelper

  before(:each) do
    @user = User.new(valid_facebook_attributes)
  end

  it "should be valid" do
    @user.save!
    @user.should be_valid
  end

  it "should reject duplicate facebook uid" do
    @user.save
    @user_2 = User.new(valid_facebook_attributes)
    @user_2.should_not be_valid
  end
end

This prompts someone else to ask:

1 - What are you supposed to be testing?

The answer to this is usually “your own code”. Therefore, a test (I refuse to call it a specification) like the above makes no sense. Yet is the most common way that people specify a validator.

What am I testing here? The validator code. That’s really pointless, because Rails already tests that. And it results in a lot of duplication (though technically, it can be argued to be “not duplication” because you’re testing that the business rule of requiring a particular field is there, but I think that’s just rationalising a bad practice). Ultimately, though, this is uncomfortable because you’re testing an outcome, not specifying a behaviour. You may use rspec to do so, but it’s still an outcome. And that’s not what behaviour driven design is about. Ultimately, this is testing, not specification.

2 - But how do a specify one of those weird validator things?

I think a lot of the issue is also caused by the simple language constraint that specifying validators and relationships is just plain fugly at the moment. Here’s the code provided by David Chelimsky (chief maintainer of rspec):

it "should validate_presence_of digits" do
  PhoneNumber.expects(:validates_presence_of).with(:digits)
  load "#{RAILS_ROOT}/app/models/phone_number.rb"
end

Now I love rspec, and very much respect David - I’ve had a few chats with him on freenode#rspec and he’s a very smart guy - but this code is just nasty! That nastiness is required to get around the fact that the validator is loaded when the class loads, so can’t be specified as something that will happen later.

As per the Sapir-Whorf hypothesis that BDDers know and love, language drives behaviour, and nasty language like this doesn’t encourage people to go this way, even though, as I’ll explain a bit further down, I think this is actually the correct way to go, in spirit if not in form.

3 - But if I just specify that “thing”, how do I know that it actually does what it’s supposed to?

A couple of weeks ago I would have replied: This is not a valid concern. It just shows that you really need to write that spec. You need to find out how your code is going to work before you write it. Surely that should be obvious?

But then, like all things, this is not quite so black and white. There are validation and relationship constructs which are complex enough that you want to not only specify them, you also want to unit test that they work, and that they continue working even when things change, for instance when you deploy Rails 2.0. And that’s the key to solving this puzzle. The “specification” in point 1 looks wrong because it’s not a specification, it’s a unit test. It’s not an integration test, so it doesn’t fit in the Story Writer. But it’s also not a specification, so it doesn’t fit in your User spec.

So, the solution?

I put forward that this item belongs in a third category, that we thought eliminated by the arrival of BDD, but which actually should still exist: unit tests. Specifically, unit tests relating to the behaviour of third party code, or of your interpretation of that code. So these are not system integration tests, in the sense that they are not testing your system as a whole. They are more like unit integration tests, there to codify your understanding of the way the rest of the world behaves.

As such, they do not fit in the Story Runner. They have nothing to do with user requirements. What we need is a new directory in the rspec folder, dedicated to what I would like to call Assumption Tests. They are tests of your assumptions about your dependencies. As such, they are both redundant and essential. They are redundant in an ideal world where you know exactly how the external code you use works and you know about any changes to it. They are essential in the ugly world where you have to be just that little bit paranoid about other people’s code, even if it’s specified and tested up the wazoo, because sometimes things change in a way that breaks your assumptions.

And, importantly, these are not specifications. They are unashamedly outcome-driven. This is especially acceptable when you consider that they are not specifying your own code, but testing other people’s code to make sure it fits your assumptions.

Without further ado, here’s an example of one such assumption test:

class ActiveRecordTester < ActiveRecord::Base

  # This makes this model tableless
  def self.columns() @columns ||= []; end
  def self.column(name, sql_type = nil, default = nil, null = true)
    columns << ActiveRecord::ConnectionAdapters::Column.new(name.to_s, default, sql_type.to_s, null)
  end

  column :foo, :string

  validates_presence_of :foo
end

##################
describe "Presence Validator" do

  it "should not be valid without foo" do
    @test = ActiveRecordTester.new
    @test.should_not be_valid
  end

  it "should be valid with foo" do
    @test = ActiveRecordTester.new(:foo => "1")
    @test.should be_valid
  end

end

Once you’ve got this in place, you should never need to test the outcome of validatespresenceof again. So the code in example 1 is now obsolete, and your specifications can remain entirely behaviour-driven, rather than including bits and pieces of outcome testing. All you need to specify is that, say, your User model calls “validatespresenceof” for facebook_uid in the way that’s codified and auto-tested in your assumptions.

So now you have:

• specs for controllers
• specs for models
• specs for views
• stories for the whole system
• assumption tests for things which you didn’t write but want to test anyway

Sorry for the long article, but there were a fair number of ideas to cover. I’d love to hear comments about this, as it’s all still a bit fresh to me. You can put those comments below, or even comment on your own blog (I have trackback enabled). Thanks for reading!