Front-end automated testing using BackstopJS and KSS style guide

What is front-end testing and why is it helpful

The goal of front-end testing is to test the User Interface of the application — the part that the user directly interacts with. It allows us to spot visual bugs early in the development process.

Types of front-end tests

User interfaces in modern web applications can be very complex, so it’s not surprising that there are many different types of front-end tests, each of them focusing on a different aspect of the website.

Cross browser compatibility testing

There are plenty of browsers on the market, many of them can be installed on different operating systems. Due to differences in browser engines, we can encounter visual bugs only in a specific browser, or even in one particular version of the browser.

The task of cross browser testing is to make sure that all the visual features of your website are working on all major browsers in combination with all popular OS. There are a number of platforms that provide ready environments for cross browser testing, e.g.: BrowserStack, LambdaTest, Browserling.

Accessibility testing

According to WHO, there are over 1 billion people in the world that experience some form of disability. Some of them, like vision impairment or motor disabilities, make navigating and using websites significantly harder or even impossible.

Improving website accessibility will also benefit other users, for example when the website is viewed in direct sunlight or on a low-quality display.

To ensure your website is friendly for all users, you should test the parts of your website that are crucial for accessibility, like text contrast, form labels, alternative texts on the images, or keyboard navigation. You can find more details about making your website accessible in our blog post on that matter: How to make your website accessible for everyone.

Visual regression testing

The goal of visual regression testing is to make sure the changes introduced to one component, will not affect other parts of the website unintentionally. It’s not uncommon that updates to component_A have an influence on how component_B looks (e.g. when it consists of components_A). A developer working on component_A might not realize that and thus introduce visual bugs on the website.

Visual regression testing takes care of this problem, by automatically taking screenshots of specified components, and comparing them with the reference version. I will describe one potential approach for this type of front-end testing in this blog post.

Organize your components in a “living style guide“

If you and your team are working on a website that has a lot of custom elements, it might be a good idea to collect all of them in one place. On one hand, it will make it easier for designers to create new layouts from already defined components, on the other hand, developers have a list of elements that can be plugged in easily during the implementation.

A Living Style Guide is the perfect solution in this case. In comparison with static style guides that were just PDF documents, it doesn’t require manual updates every time something changes in your components. Instead, it is a website – all you need to do is to define a list of elements that you want to have in the style guide. Those elements will be available on that website with the styling coming directly from the source code of your project. This way you don’t have to worry about updating the style guide each time you change some CSS – thus saving time of your team

There are many tools that help you to create a living style guide. In this post, we will focus on kss-node based on KSS documentation syntax for style sheets.

Using kss-node to create a style guide

Installation

Here is the package.json file that contains the kss-node package together with a script that will generate our style guide. It also has node-sass that will compile our SCSS files to CSS.

{
  "scripts": {
    "styles": "node-sass scss/style.scss > style.css",
    "styleguide": "npm run styles && kss --source scss --destination styleguide --css ../style.css --homepage homepage.md",
    "backstop-reference": "npm run styleguide && backstop reference --config=backstop.js --docker",
    "backstop-test": "npm run styleguide && backstop test --config=backstop.js --docker"
  },
  "devDependencies": {
    "kss": "^3.0.1",
    "node-sass": "^6.0.1"
  },
  "dependencies": {}
}

Before installing, have a look at kss command, it takes few required parameters:

• --source --- directory where kss will look for the KSS comments
• --css .css --- file to include in the style guide
• --homepage --- Markdown file that will become your style guide’ homepage

Run npm install to install all dependencies.

Documenting components

Let’s say we have a Button and Card components (in this blog post I will be using SCSS syntax for all stylesheets):

.button {
  font-weight: 700;
  font-size: 18px;
  padding: 8px 20px;
  background-color: #9EBD71;
  color: white;
  text-align: center;

  &.secondary {
    background-color: #116D56;
  }

  &:hover {
    opacity: 0.7;
  }
}

.card {
  position: relative;
  display: block;
  box-shadow: 0 3px 6px 0 rgba(0, 0, 0, 0.15);
  background-color: #F8FAF4;
  padding: 10px;
  width: 300px;

  @media screen and (max-width: 767px){
    width: 220px;
  }

  &__title {
    text-align: center;
    font-weight: 700;
    font-size: 30px;
  }

  &__buttons {
    text-align: center;
  }
}

Alright, so how do we add these components to our style guide? It’s very simple, we just need to add a comment above the element definition, following the KSS syntax:

// Button 

The name of the element. This will be the title of this element’s section in the style guide.

// Standard Kiwee button 

The description of the element.

// Markup:
// <a class="button" href="">Button</a> 

The markup of the element, alternatively you can move the markup to a separate file, and place the file name here.

// Style guide: elements.button 

The path of this element in the style guide navigation.

And similarly for card:

// Card
//
// Card with two buttons
//
// Markup:
// <div class="card">
//   <span class="card__title">Card title</span>
//   <p>
//     Lorem ipsum dolor sit amet, consectetur adipiscing elit,
//     sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
//     Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
//   </p>
//   <div class="card__buttons">
//     <a class="button" href="">Button</a>
//     <a class="button secondary" href="">Read more</a>
//   </div>
// </div>
//
// Style guide: elements.card

Now you can run npm run styleguide. You can notice that the styleguide directory was created. We could just enter styleguide/index.html in our browser now, but not all of the style guide features are working correctly when viewed as a local .html file. Let’s configure a simple local server to serve our style guide quickly.

Create Dockerfile in docker/styleguide-nginx with the following content:

FROM nginx:alpine
COPY style.css /usr/share/nginx/html

Create docker-compose.yml file:

version: '3.7'

services:
 styleguide:
   build:
     context: .
     dockerfile: docker/styleguide-nginx/Dockerfile
   ports:
     - "8081:80"
   volumes:
     - ./styleguide:/usr/share/nginx/html
     - ./style.css:/usr/share/nginx/html/style.css

Note: If you are on Linux, add

extra_hosts:
  host.docker.internal: host-gateway

Below ports declaration. This works only in Docker v20.10+.

Now just run docker-compose up -d, the style guide should be accessible at localhost:8081.

Voilà, our elements are now documented in the style guide. You can see all the information we provided in the comment and a path to the button.scss file where this element is documented.

There is one more thing that can be improved though. If you look at button.scss you will see that we are modifying the look of the button on hover, and we’ve added .secondary class that will change the button background color. However, it’s not visible in the style guide.

Fortunately, there is an easy way to document all states of the element without the need to repeat the same markup with different classes. It’s called a modifier class.

The first step is to define all states that you want to document, in our case that is the .secondary class and the button on :hover. Then you need to add {{modifier_class}} placeholder in the elements class. The resulting comment will look like this:

// Button
//
// Standard Kiwee button
//
// .secondary - Use this class to indicate that the button is the primary
//              feature of this form.
// :hover     - Highlight the button when hovered.
//
// Markup:
// <a class="button {{modifier_class}}" href="">Button</a>
//
// Style guide: elements.button

Let’s run npm run styleguide again, and see how the style guide changed:

Now every state and modifier of our element is well documented and visible for everyone who will be working on the design or implementation.

Use BackstopJS to spot all the visual bugs on your website

BackstopJS is an open-source tool for running visual regression tests. It uses a headless browser to capture screenshots of your website and compare them with previously stored reference files. If these images differ, BackstopJS will show you differences between them as this is a likely indicator of visual regression — an unwanted change in an element’s appearance.

Installation and configuration

Let’s start by installing BackstopJS for our project:
npm install -g backstopjs

Create a configuration file backstop.js in project root.

const styleguideUrl = 'http://host.docker.internal:8081'

module.exports = {
  "id": "backstop_default",
  "viewports": [
    {
      "label": "phone",
      "width": 375,
      "height": 677
    },
    {
      "label": "desktop",
      "width": 1920,
      "height": 1080
    },
  ],
  "scenarios": [
    {
      "label": "Button",
      "url": `${styleguideUrl}/section-elements.html`,
      "referenceUrl": `${styleguideUrl}/section-elements.html`,
      "selectors": ["#kssref-elements-button .kss-modifier__wrapper"],
      "removeSelectors": [".kss-modifier__heading"]
    },
    {
      "label": "Card",
      "url": `${styleguideUrl}/section-elements.html`,
      "referenceUrl": `${styleguideUrl}/section-elements.html`,
      "selectors": ["#kssref-elements-card .kss-modifier__wrapper"],
      "removeSelectors": [".kss-modifier__heading"]
    },
  ],
  "paths": {
    "bitmaps_reference": `backstop_data/bitmaps_reference`,
    "bitmaps_test": `backstop_data/bitmaps_test`,
    "html_report": `backstop_data/html_report`,
  },
  "engine": "puppeteer",
  "engineOptions": {
    "args": ["--no-sandbox"]
  },
}

We start by setting styleguideUrl to locate the previously generated style guide. We used host.docker.internal instead of localhost, otherwise BackstopJS running inside the docker container won’t be able to access the server. I will explain it more later in the post.

“Viewports” object contains all screen sizes that are used for testing. Screenshots will be taken on each defined viewport for each scenario.
When it comes to scenarios, here we have a few fields for each scenario, let’s take a look at some of them:

• url, referenceUrl --- URLs to test and create reference screenshots accordingly. In our case, both variables are set to ${styleguideUrl}/section-elements.html. We will store the references from the base style guide, and run the test on the updated version.
• selectors --- an array of selectors that should be captured. In our case, we want to capture KSS style guide elements: button and card wrappers.
• removeSelectors --- list of selectors to be removed before capturing the screenshot. Here we are getting rid of the KSS heading as it is not part of our component.

Generating reference screenshots

Now that we have all the necessary BackstopJS configuration, we’ll add two scripts in package.json:

"backstop-reference": "npm run styleguide && backstop reference --config=backstop.js --docker",
"backstop-test": "npm run styleguide && backstop test --config=backstop.js --docker"

Notice we are using --docker flag here. What it does is run BackstopJS inside a Docker container, as described in the documentation:

Pass a --docker flag to render your test in a Docker container -- this will help with consistency if you are attempting to compare references across multiple environments.”

Let’s start with generating references from the current style guide.
npm run backstop-reference

If everything went well, you should be able to see the screenshots of our components from the style guide in backstop_data/bitmaps_reference. Those are the references that will be compared with all future BackstopJS tests.

Running visual regression tests

Now that we have our baseline screenshots, we can finally use BackstopJS for testing.

Imagine that we are implementing a new design of the buttons, we want to make their padding bigger. Let’s change padding: 4px 12px; to padding: 8px 20px;.

Okay, we finished the implementation, now it’s time to run the tests:

npm run backstop-test

The test report will automatically open in your browser.

So far so good, the tests obviously failed here, but that was expected since we increased the padding. BackstopJS will highlight any differences between reference and test screenshots in pink. Let’s have a look at other failed tests.

Oops! The card element on mobile devices is now completely broken. Two buttons don't fit in one row with the increased padding. We have few options to fix it now: make the card a little bit wider, decrease the padding of buttons only inside the card etc. However you choose to fix it, the point here is that we were able to spot this visual bug thanks to BackstopJS.

Updating backstop references

Alright, but what if there are tests that failed due to intentional changes in the design? In this case, you should update BackstopJS references in the repository. Do this by running npm run backstop-reference again and commit updated references.

Tracking BackstopJS files in Git

If you go back to the project, you will notice that BackstopJS created two new folders in backstop_data: bitmaps_test and html_report. As the name suggests those directories contain test results. But unlike references, these are our local test results, and they shouldn’t be kept in the repository. Add these two lines to .gitignore:

backstop_data/html_report/
backstop_data/bitmaps_test/

You should add backstop_data/bitmaps_reference to the repository; this is the baseline that should be shared with the whole development team and all tests should be run against them.

Other potential uses for BackstopJS

Test automatically on every branch push

Recently we’ve added BackstopJS tests to the CI pipeline that is triggered on every development branch push. Our workflow is as follows:

1. We store all baseline references in the repository
2. On each branch push, the style guide is generated from that branch source code
3. BackstopJS tests are run, comparing baselines with the style guide generated in the previous step
4. Test results and report are available after each pipeline run

Periodic testing of entire pages

We also have a pipeline that is running every night and testing different aspects of the website. Part of it is running visual regression tests on the live website. In this case, it’s testing whole pages instead of single components. It doesn’t give us precise information about the component that failed, but it helps us cover parts that are not included in the style guide. This is how it works:

1. References are stored on the server
2. In case any test fails, the pipeline will stop and wait for user input. If the failed tests are a result of intended changes, human approval triggers an automatic update of the references. Otherwise, the test report gives us information on what exactly changed, so we can apply fixes quickly.

Summary

Introducing a living style guide to your project is already worth it in itself. It will help to avoid duplicated CSS code and give a nice overview of the components that are available to build with to both Designers and Developers. When combined with BackstopJS regression tests, it becomes a really effective tool to prevent visual bugs early in the development stage.

Obviously, the example presented in this blog post is very simple, but if the project has dozens of components, some of them depending on others, it might be very hard to check all of them and spot unwanted changes or bugs. Instead of manually going through the elements every time, you can as well add visual regression tests to your project. The effort to set up basic BackstopJS configuration is not too big, but it can prevent a lot of mistakes and makes the work of web developers a little bit easier.

You can find the full code of the presented example here: BackstopJS regression tests demo.

• BackstopJS
• front-end-testing
• style-guide