Testing web projects with BackstopJS

Today is time to get to know with testing with BackstopsJS!

Testing with BackstopsJS in the process of web development

In the process of developing the frontend part of a website, it's very likely you'll face a following scenario:

You have just finished styling your first page/view for your new project. Everything is (pixel) perfect. Every element is where it is supposed to be, everything is 1 to 1 with the project design. You’re happy with the results and move on to style the next page/view.

After you finish styling that page, you go back and check the previously finished page just to discover that some elements are misplaced or do not look how they’re supposed to, the way you left them before. Something you added while styling a new page “broke” the previous page. So you go back and fix these issues, making sure that the fixes you make do not “break” the new page/view you’ve just finished, so you go back and forth between these pages until everything is right.

As your project grows and you have more and more pages, this process of making sure that none of the finished pages are broken as you add new code (or fix some bugs during QA), becomes much more arduous and less feasible especially when you do this manually and have to check each page on different screen sizes every time.

Fortunately there are tools that can help you test pages more efficiently. In this article I will take a closer look at testing with BackstopJS.

What exactly BackstopJS does?

BackstopJS “automates visual regression testing of your responsive web UI by comparing DOM screenshots over time.” In other words, it allows you to automatically take snapshots of our web pages at any given time (on many different screen sizes) to create reference files which can later be used to test (compare) the current (new) state of pages with previous (reference) state and generate a report with the results. You can do it by writing specific scenarios for each page which I will cover in more detail later.

Installation and setup

In order to add BackstopJS library to your project via npm, you can use the following command:

$ npm install backstopjs -D

or via yarn:

$ yarn add backstopjs -D

You can also install BackstopJS globally:

$ npm install -g backstopjs -D

but it is best to always add the backstopjs library as a dev dependency to your project.

After installation, you need to initialize the tool. For that there is a command:

$ backstop init

which you run (via CLI) in a root folder of your project.

If you installed the library globally you can just type $ backstop init. If locally (using yarn), then type $ yarn backstop init.

This command will create a configuration file backstop.json and a folder backstop_data.

Next, let’s take a look at a generated default configuration file:

There are quite a few settings there, but in this article I will focus only on these three:

id: It’s a unique name that is added to all the screenshots created by the tool. You can change it to match your project’s name.

viewports: Here, you can specify screen sizes you want the tool to check in the specified scenarios.

scenarios: This is a place to add/remove/edit scenarios for the tests. Usually, you’ll want to create a new separate scenario for each page/element that you want to test.

You can read in detail what other settings do in the official documentation here.

Testing with BackstopJS: Configuration

Before you change anything, let’s see how it works out of the box. First, you need to create reference files / snapshots of your pages. Running a test without these files will not work because you there is nothing to compare.

So, to create reference files, use this command:

$ backstop reference

or

$ yarn backstop reference

This will create screenshot files in a backstop_data\bitmaps_reference folder.

Finally, you can run your first test. To do that, use the following command:

$ backstop test

or

$ yarn backstop test

This command will check your specified scenarios (and viewports) and generate a report in html format and automatically open it in a new window in your default browser. This report (for the default configuration file) will look like this:

In this report, you have a list of specified scenarios (pages/elements) that passed or failed. In our default configuration file, there is a scenario labeled: “BackstopJS Homepage” with page url: https://garris.github.io/BackstopJS/, and viewports labeled: ‘phone’ and ‘tablet’.
Now, you have one reference screenshot and one test screenshot for each viewport (screen size) in the report. You should not expect any of the scenarios to fail because you made the references and test files at around the same time. And indeed, you’ll see in the report that your scenarios have passed the tests.

Customization

Custom scenarios

Let’s now have a look at how you can add your custom scenario, so that you can test a page from your own project. When you initialized backstop, the default configuration file that was created is a JSON file. However, you can also use a JS file for configuration. Using a JS file gives you a lot more possibilities, like using env variables and creating each custom scenario as a separate JS file that is imported into the main backstop configuration file. Similarly you can extract your viewports to a separate file import from there.

When you want to run Backstop with your custom configuration file, you need to tell backstop to use it. This is done via the --config argument. Below, there is an example of running a reference command with custom configuration file named backstop.js that is located in a root folder of your project:

$ backstop reference --config=backstop.js

or

$ yarn backstop reference --config=backstop.js

Creating screnario

For now though, let’s focus on the default JSON configuration file.

In order to add/edit/remove scenarios for your tests, we need to modify “scenarios” in the configuration file:

To create a new scenario, simply add another object to the “scenarios” array. Scenarios can be very simple, and when you want to run a scenario with mostly default setting, your custom scenario might look like this:

Testing scenario

If you want to test your new scenario, you need to create references for it first. You can create all references for all scenarios again via backstop reference or - you can target a specific scenario using --filter argument. So, let’s say you have a scenario labeled “SamplePage”. To create new reference files only for this scenario (assuming that there are no other scenarios that have “SampePage” in their label), run a command like this:

$ backstop reference --filter=SamplePage

or

$ yarn backstop reference --filter=SamplePage

Finally you can run a new test. You will find your new scenario in the generated report :

Testing with BacstopJS

So far we ran our Backstop tests only right after creating reference files and so all tests passed. Let’s now see what the test report would look like if you made some changes to the page you are testing:

As you can see, now you have 2 failed tests and also there is a new column in your report labeled “DIFF” and the screenshot in that column looks differently. Let’s take a closer look by clicking on a diff image:

Here, you can inspect up close the differences between our reference files and new screenshots generated while running a test. The ‘DIFF’ tab shows you a test image and the reference image overlayed on top of each other and the differences between these two images are highlighted in pink. You can switch between each view and also use a “scrubber” to compare parts of the image:

If these differences are unwanted and need to be fixed, then using this view you can identify and then fix errors and run a test again until all of them pass.

On the other hand, if reported differences are intentional, because for instance there were some changes to the page’s design, then you can approve the new screenshots as your new reference files by running approve command:

$ backstop approve

or

$ yarn backstop approve

Similarly to the previous commands, you can use a --filter argument to approve only specific scenarios.

Adding more screen sizes

Using default Backstop configuration we tested our scenarios only on two viewports (screen widths): “tablet” and “mobile”.
But you should also check how your pages/scenarios will look like on desktop for example. In order to do that, you have to add another viewport to the configuration:

You do this by adding another object to the “viewports” array with viewport label and dimensions. In this example I added a “desktop” viewport with dimensions of 1440 by 900.

It is also necessary to create new references for all scenarios.

Now, all scenarios will be tested on your desktop viewport as well. Of course, we can add more viewports as needed.

Advanced scenarios and settings

The custom scenario that we created earlier was very basic, with minimum configuration. But of course testing with BackstopJS allows you to create more complex scenarios. For example, you can delay the execution of your scenario for 5 seconds after loading the url by using the delay parameter which accepts milliseconds as value:

You can also choose to execute the test only after the given selector is ready on your page. That can be done via readySelector where you can select your element.

Furthemore, you can e.g. test only specific elements of a page by using the selectors parameter. So, If in our scenario we wanted to test only page’s footer, if the footer’s class is “main-footer”, then it should look like this:

Now, let’s say that you want to add some CSS styles to your page before the test is executed. You can do that by using onReadyScript which allows you to modify the UI state prior to the screenshot being taken. The default scripting engine used for that is Puppeteer. You can add your own on pre-made handlers in the./backstop_data/engine_scripts/puppet/onReady.js file or create a separate file and provide a path to it:

Check here for more comprehensive description of advanced scenarios.

Project workflow when testing with BackstopJS

As you can see, testing with BackstopJS can be very useful when developing web projects. Additionally it can help you optimize the process. It is a good practice to add new scenarios for each page of your project as soon as you finish developing it. Run backstop tests often to see if previously finished pages were not broken during the development process.

If you want to, you can also incorporate BackstopJS into your build process and e.g. make tests run on each build so that you don’t accidentally deploy your project with broken pages.

It is also a good idea to add the following:

to the project’s .gitignore file, to make sure not to track these files in your repository. This is not necessary and these folders can become very big during development and testing.

To make the best use of BackstopJS there is of course a lot more to learn about its capabilities and how you can tweak it to your specific needs. Hopefully, this article will help you get started with testing BackstopJS in your web project.