The UI Testing Brittleness Problem and How to Solve It

Introduction

UI tests are seen as brittle because small changes to the UI can easily break tests.
Note, though, that it isn’t only UI tests that are brittle: UI tests are brittle because they’re tightly coupled to the UI design; other types of test are brittle because they’re tightly coupled to other things:

  • Unit tests are brittle because they’re tightly coupled to the code (names, arguments and return types of functions)
  • API tests are brittle because they’re tightly coupled to the protocol (HTTP methods, URLs)

UI tests suffer more from brittleness because the UI is generally the most volatile part of the application.
Closely related to brittleness is flakiness, when tests fail sporadically without any changes to the application.
UI tests are also prone to flakiness, mostly because of the vagaries of screen colours.
UI tests tend to be end-to-end (although the concepts are actually orthogonal), and this introduces more potential points of failure, which also makes them flaky.

Kinds of solution to the UI test brittleness problem

Solutions to the UI testing brittleness problem fall broadly into two categories:

  1. Solutions that make tests less likely to fail
    – This means relaxing the success criteria of tests to make them resilient to acceptable variation.
  2. Solutions that make tests easier to fix
    – This means using good software engineering practices to limit the effects of change, and so reduce the effort involved in fixing broken tests.

Solutions that make tests less likely to fail

1. Do smart image comparison

Image matching can cause brittleness problems, as well as flakiness.
Crop images to remove backgrounds whose colour you might change, or might be variable.
Configure Search Type, Tolerance, and Discrepancy in Eggplant Functional.
See Image Capture Best Practices for details.

2. Use optical character recognition (OCR)

Using OCR means that changing the appearance of, for example, a button, but keeping the same caption will not break a test.

3. Don’t search the whole screen, search a rectangle

This focuses attention and avoids duplicate matches.
Set the SearchRectangle in Eggplant Functional.
This will also make your tests run faster.

4. Use robust selectors in Selenium

Ideally, use HTML IDs (add them to your pages if you don’t already have them)
Use relative rather than absolute XPaths.
Consider using multiple selectors, and trying them in sequence, until one works.

5. Wait for, don’t sleep

Sleeps in tests cause a brittleness problem, and also a flakiness problem.
Changes to the application can slow down page loading and cause tests after sleeps that use to work to start failing, but tests after a sleep can also fail randomly when things are running slowly.
Sleeps are by definition too long and will slow down your test runs.
Use a Wait in Selenium.
Use the WaitFor function in Eggplant Functional.

6. Don’t use the UI when you don’t need to

If your tests require a registered user, use an API or access the database to create accounts, etc. instead of going through the UI every time.
But do have one UI test for that workflow.
In Eggplant Functional, you can use API testing, and access ODBC databases.

Solutions that make tests easier to fix

1. Record, refactor, reuse

Don’t record all of your tests in their entirety, because this will result in lots of duplicate, unstructured code, all of which you will need to change when the UI changes.
Instead, record a few tests and then reuse the code, images, etc. to build more tests.
Refactor mercilessly to remove duplication, enhance reusability, and hide implementations (see 2, 3, and 4 below).
Consider using keyword-driven testing to allow non-programmers to build tests.

2. Use the Page Object design pattern

  1. Have a class for each page (or screen)
  2. Have methods in the class that encapsulate the direct manipulation of page elements

This means that you only need to modify the methods if the page changes, rather than having to modify the manipulation code repeated in several places.
An example of Don’t Repeat Yourself (DRY).
It also makes your testing code easier to understand because the structure of the testing code matches the structure of your application.
In Eggplant Functional, this means use one script per page, with handlers in the script to manipulate the UI elements on that page.
The Autosnippet feature will generate code for you using this pattern if you use one Autoscan session per page.
Remember that although it’s called the Page Object pattern, it refers to any kind of application screen.

3. Abstract common workflows into functions

If you have the same steps repeated in several tests, factor them out into a reusable function.
For example, Enter Username, Enter Password, Click Login become a single Login() function.
This way you only need to change one thing if that workflow changes (say you add a “Remember Me” checkbox to the login screen).
This is another example of DRY.
Use in combination with Page Object (the function you add is a member of the Page Object class for the login screen).

4. Keep data separate from code

Including (especially!) selectors and image names.
Use constants to put the data in one place.
This is another example of DRY, and also a Single Source of Truth (SSoT).
Consider also data-driven testing. This allows you to create more tests by just adding more data.
When the data is in an external source, anyone can modify and add to it, not just developers.
Eggplant Functional has support for text files, Excel spreadsheets, and ODBC databases.

5. Take a screenshot when a test fails

This allows you to see what went wrong in an unattended test run and fix it more quickly.
Use TakeScreenshot.getScreenshotAs() in Selenium.
Use the CaptureScreen command in Eggplant Functional.

6. Use the test-fixing abilities of your automated testing software

It’s good if your automated testing software helps you with the test-fixing workflow:
1. Fixing images in Eggplant Functional.
2. Screen Compare in Eggplant Functional

Conclusion

If you apply the solutions above, you should find that your UI tests fail much less often, and when they do, they’re much easier to fix.
Once you’ve fixed your brittleness problem, consider taking your testing the next level with model-based testing (MBT).
Reuse your (now robust and easy to maintain) test steps to create a potentially infinite number of tests.
In Eggplant, this means using Digital Automation Intelligence Suite (DAI).