2015/03/31

Why developers are so paranoid about failing Integration tests and Why this is not a problem

When it comes to developers talking about Web Integration tests - you may hear something like "Test should not fail if I move this button" or "If the text on the button changes slightly - test should still pass"

But from my point of view it's ok to fail test because of that. And I would even believe failing test more than passing one.

For a long time I cannot understand the reason of this misunderstanding. And today while reading a book on unit testing - I finally got it.

tl;dr changes in Web Page are more sensitive than changes in API


When you talk about unit testing you really don't want your tests to break as a result of refactoring. It's a well-known fact, and I won't argue with that.
It's so because unit tests are working with pretty specific interface. And changes in interface will be caught by compiler - tests wont compile.

Back to Web Integration tests - we don't have such a detailed interface to interact with. Our accessible interface is pretty wide - whole page with all the positions, images, animations and so on.
Also we don't have a compiler to check all the bindings to an interface (Locators for elements) are suitable for us to interact with.


Communication
interface
Entity Properties
(non-functional)
Who is affected Price of unexpected
interface changes
Unit Test API
(closed or open)
Method Method name, Parent service, Method parameters  Developers, Tests Decide yourself
Integration Test Web Page Element on page (field, label, image, etc.) Position on page (in DOM), any style related info (size, color, etc), text content, value. etc End users, Tests


So moving back to the button  - it is in fact a change in communication interface between a Web Page and Web Integration test.

We have two options how to work with it:
  1. Ignore the change
    we can do this by making very narrow Locator for an element - which won't  touch anything except the button itself (most probably by some unique button id)
  2. Fail the test
    we can write a bit broader Locator - which will know some more info about the button (text, siblings, etc.)

Let's take look at advantages:

  1. Ignore the change
    • Advantages
      • We can change anything about the button (refactor) except of it's functionality and unique id
      • Tests will fail only on functional defects
    • Disadvantages
      • Developers need to provide unique id's for all the elements, and maintain it through the development process
      • Button may change it's place (change interface) and we actually won't know about this
  2. Fail the test
    • Advantages or Disadvantages (depending on point of view)
      • Tests will fail on interface changes 
      • Tests will need to be fixed by human in case of changes

In summary 

It's totally up to you to determine the price of unexpected interface changes. 
Since there is no automatic way to check that these changes are valid - I think it's ok to fail the test, and let the human decide whether that change was expected. With a good approach Web Automation you will only need to make changes in one place (update one Locator) which is pretty quick, but in the same time - you become a little more confident in your test results.