If you build or change software, it needs to be tested. That’s not breathtaking news. Testing can be done in many different ways; automated testing is one way and it might be a very efficient way depending on the individual case.

If you say testing is a requirement then you need to treat is as a requirement like any other feature request. It’s not something that you simply do because you want to be “professional”. Instead, you should choose your appropriate testing strategy depending on aspects of the requested feature like criticality (risk), triviality (trivial things don’t need to be tested extensively) or maintainability. In any case, you would need to make the requirement “stated”.

I also tend to agree with Louis saying that “a passing automated test does not prove anything”. I have seen projects that put hell of a lot of effort in increasing their CodeConverage but finally found out that what they tested was not what was required. I am probably nearby those dirty hackers saying that functionality is the first thing that matters. Having said that, I don’t intent to say that testing doesn’t matter. It matters and it is an important aspect of software in general. Due to its importance, it needs to be made transparent; it needs to become a “Stated Requirement”.

But, the fact remains that in most cases this will never get through to all people and things will still remain the way they are. I’ve tried hard to make my co-devs write automated unit/functional tests, showing them the benefits time and again. Doesn’t work for the most part. But sometimes I do run across people who share these thoughts and it feels great working with them.

I think in many cases it would be hard to understand by somebody being NOT a professional that TPUR is achievable at all.

The idea is to keep the essence of the test (the feature you hope to assert) and the mechanics of the test (the clicking of buttons, selecting drop-downs, clicking links, etc).

So if you’re mixing them both in one test, you’ve got a problem. In our web testing, we have different layers of the test from the “driver” layer (the thing that directly interacts with Selenium) to the fixture/grammar level which is the StoryTeller stuff (which contains the essence of the test expressed in as business-like language as we can get.

So basically, we keep Selenium as far away from our testing code as possible so that we can change how, say, drop-down/selectboxes work in our app without breaking all our tests.

So if you can find a way to isolate how your combo boxes work, then put that in your driver layer. That way your *test* can just say: “Select this value for this field” and your driver will know that “that field” is a combo box and it’ll know how to deal with combo boxes (change values, get the current value, get the list of all the available values, etc).

If that’s simply not possible in silverlight, then that’s a real problem. I find that kind of hard to believe, but I don’t know much about silverlight.

It’s just a pain.

Thanks for your input! I’m really on the fence and still accumulating arguments from both side. I’m glad to hear that you guys are reaping benefits from your automated tests.

Note that I talked about testing in layers (unit, integration, acceptance). “100% coverage” at the acceptance level involves a lot fewer tests than 100% coverage at the unit level because there are more seams (interactions) to test.

If you’re seeing “incredibly high” maintenance cost with tests, then, frankly, you’re doing it wrong. I’m not sure what tech you’re using, but if it’s using WatiN, WatiR, or Selenium directly, then you’re doing it wrong. Those technologies are important, but they should be driven by something higher level (an acceptance testing framework) like StoryTeller, Fitness, or Slim, etc.

“Keep in mind that a passing test doesn’t prove anything” I disagree with this statement categorically unless you say it’s a poorly written test. Tests are not useful simply because they’re tests. I agree that tests must be written correctly (testing the right thing for the right reason with the right expectation). With this established, then tests prove lots of things and are of great value.

We were able to make rather stark and major changes to our infrastructure and all levels of our application (from redoing our data access entirely, to changing out our MVC framework entirely, to changing out our grid technology entirely, etc) with high confidence due to our bevy of StoryTeller/Selenium tests.

This confidence in change and accuracy of delivery is of immeasurable value to us. Any “incredibly high maintenance cost[s]” were entirely justified on the first time we had to make a major readjustment of our infrastructure. We’ve now done it several times and are reaping the dividends.

I see the automated acceptance tests as something akin to seat belts or railings on stairs. Sure they can be a pain and get in your way, but the first time you need them and don’t have them, you will sorely regret it. Likewise if you need them and have them, all pain or cost becomes moot.

“I find that 99% of automated test failures are the result of an intentional change in the behavior of our app, not that something broke” – Us, too (though maybe not 99%, more like 80%). If this is the case, then your testing framework should make it easy to change the guts of the test to match the new reality.

For us, we express our (StoryTeller) tests in business language and not implementation language. The implementation is behind the scenes, in code, close to the actual code of the feature. If the feature changes slightly, so does the underlying implementation of the test. Thus, the essence of the test is still expressed in StoryTeller and only a slight code change need to happen.

If you’re using record/playback mechanics like Selenium, then you’re in for a world of hurt as even a small change to the application can involve long, tedious changes to the web tests. Thus you should separate the essence of the test (what to test) from the implementation of the test (how to test it). This is what, among other things, StoryTeller does for us.

If you couple that with the incredibly high maintenance cost, are they truly useful? I think it was Brian Marick that had observed that automated tests only allow you to find 30% of the defects in your code base, whereas spending that time doing more traditional QA would result in finding 70% of the defects instead. In my own experience, I find that 99% of automated tests failures are the result of an intentional change in the behavior of our app, not that something broke.

Bob Martin says UI testing should be at no more than 20% coverage, with unit tests and integration tests covering the rest.

However, I do agree that we need to prove our features work. The question I’m having is how? Keep in mind that a passing test doesn’t prove anything. (by the way, I might not do TDD or BDD as often as I ought to, but I fully agree with those methodologies and SOLID principles. Feel free to call me out on those :))