Visual Regression Testing with Storybook

Nils Knappmeier

Press "P" to see the speaker notes.

Who am I?

height:60px

Fullstack-Engineer @ Cosee since 2019

Backend testing

Frontend testing

Typical SPA

Utility Functions

it('returns false for 1', () => {
  expect(isPrime(1)).toBe(false);
})

it('returns true for 2', () => {
  expect(isPrime(2)).toBe(true);
})

Business-Logic

it('renders personalized greeting', async () => {
  // Render new instance in every test to prevent leaking state
  const { getByText } = render(<HelloMessage name="Satoshi" />);

  await waitForElement(() => getByText(/hello Satoshi/i));
});

https://react-testing-examples.com/

Backend-Mocks

const nock = require('nock')

const scope = nock('https://api.github.com')
  .get('/repos/atom/atom/license')
  .reply(200, {
    license: {
      key: 'mit', name: 'MIT License',
      spdx_id: 'MIT', url: 'https://api.github.com/licenses/mit', 
      node_id: 'MDc6TGljZW5zZTEz',
    },
  })

What to test?

Image snapshots

Spot the change

jest-image-snapshot

Creating snapshots

import puppeteer from "puppeteer";
import { toMatchImageSnapshot } from "jest-image-snapshot";

expect.extend({ toMatchImageSnapshot });

let browser;
beforeAll(async () => {
  browser = await puppeteer.launch({headless: true});
})

afterAll(() => browser.close())

test("page matches the snapshot", async () => {
  const page = await browser.newPage();
  await page.setViewport({width: 1024, height: 800});

  await page.goto("http://localhost:3000");

  const image = await page.screenshot();

  expect(image).toMatchImageSnapshot();

  await page.close();
});

import { Builder } from "selenium-webdriver";
import { toMatchImageSnapshot } from "jest-image-snapshot";

expect.extend({ toMatchImageSnapshot });

let driver;

beforeAll(() => {
  driver = new Builder()
    .usingServer("http://localhost:4444/wd/hub")
    .withCapabilities({ browserName: "firefox" })
    .build();
}, 20000);

afterAll(() => driver.quit());

test("page matches the selenium-snapshot", async () => {
  await driver.get("http://localhost:3000");
  await driver
    .manage()
    .window()
    .setRect({ x: 0, y: 0, width: 1024, height: 800 });

  const imageBase64 = await driver.takeScreenshot();
  const image = Buffer.from(imageBase64, "base64");

  expect(image).toMatchImageSnapshot();
});

Common pitfalls

How NOT to get frustrated?

width:2000px

  • Mac, Linux, Windows and CI/CD

Isolating UI components

with Storybook

import {action} from "@storybook/addon-actions";
import React from "react";

export default { title: "buttons" };

export const SimpleButton = () => <button onClick={action('click')}>simple button</button>;

height:480px

Screenshot as a Service

  • Chromatic
  • Percy

Self hosted

  • Storyshots

Storyshots

an attempt to self-hosted screenshot comparison

// storyshots.test.js
import initStoryshots from "@storybook/addon-storyshots";

initStoryshots({ framework: "react", suite: "All" });

height:400px

// storyshots-puppeteer.test.js
import initStoryshots from '@storybook/addon-storyshots';
import { imageSnapshot } from '@storybook/addon-storyshots-puppeteer';
 
initStoryshots({ suite: 'Image storyshots', test: imageSnapshot() });


// storyshots-selenium.test.js
import initStoryshots from "@storybook/addon-storyshots";
import {imageSnapshot} from "@knappi/addon-storyshots-selenium";

initStoryshots({
  framework: "react",
  suite: "All",
  test: imageSnapshot({
    browsers: [
      { id: "chrome",  capabilities: { browserName: "chrome" } },
      { id: "firefox", capabilities: {  browserName: "firefox" } },
    ],
    seleniumUrl: "http://localhost:24444/wd/hub",
    storybookUrl: `http://localhost:${REMOTE_TUNNEL_PORT}`,
  }),
});

export const Default: Story = () => <MyComponent />;

Default.parameters= {
  storyshotSelenium: {
    sizes: ["1024x768", "800x600", "360x480"]
  }
}

S l o w ......

  • Wait long enough, but not too long
  • Animations
  • Browser specification ambiguous

Services: Chromatic

by the authors of Storybook...

height:60px
Storybook is uploaded to Chromatic and tested there

height:60px
Free: 5000 Snapshots/month (Chrome only)
Paid plans with IE11 and Firefox and more snapshots

height:60px
Github integration
Command fails if snapshots do not match

Chromatic

Chromatic

Chromatic

Chromatic

Chromatic

Services: Percy

now part of BrowserStack, a Selenium as a Service company.

height:60px
Create a DOM snapshot using puppeteer, re-render in Chrome and Firefox

height:60px
Free: 5000 Snapshots/month (Chrome AND Firefox)
Paid plans with more snapshots (but no IE11)

height:60px
Asynchronous rendering and inspection
Integration for many CI-Tools to mark builds as "failed"

SPIEL.digital

  • Use Percy's free plan
  • Schedule CI-job every morning at 7am
  • Didn't really have time to look at the screenshots
    height:240px

Conclusion (1)

  • Non-Sensitive Data? Supported browsers OK?
  • Use a cloud service
    • Easy setup
    • Prevent false positives
    • Animations
    • Speed
    • Probably cheaper than custom setup

Conclusion (2)

  • Want to help create "storyshots-selenium"?
    height:240px

Further research...

  • Try storybook-puppeteer for functional tests
  • Visual feedback on functional tests

Thank you for listening. Questions?

?

https://github.com/nknapp
https://www.npmjs.com/~knappi
npm@knappi.org

Attribution

Packages

When I started working at i-views in 2007, I was assigned to the "Frontend"- or "Java"-Team. The ironic thing is that "frontend" at that time included the Tomcat-Webserver, Servlets, JSPs to render HTML, which was then enriched with a little javascript and styled with plain CSS . Later we switched to a single-page-app, but I still did a lot of work with Java-Backends.

Now "frontend" means "React", "styled-components", "Typescript". And "Backend" is mostly "Spring Boot" with Rest-Services. So when I applied for a job at Cosee, I applied as Full-Stack-Engineer.

What does this have to do with testing? There are parts of an application that are easy to test automatically, others aren't. For example the business-logic and servlets are easy to test. JSPs give you quite some pain and the generated HTML in combination with styles and injected JS... Let's not talk about it. The only feasible way is to do end-to-end tests, which can be very painful.

With React/Vue and other SPA frameworks, we have moved some parts of the application into the browser. The backend is easier to test now. But why actually?

In the backend, we a working mostly with clear structures that are easy to compare to each other. Also, the backend is designed to be used programmatically, which makes it easy to automate tests. When doing backend development, I WANT to write tests, because it is a much easier way to test then do manual testing. Of course, there are other challenges: Integration with other services, complex algorithms and data-structures. It is not ALWAYS easy.

The closer with get to the human user, the harder it is to write automated tests. We need to simulate a human user. "Wait for this element to show up". "Type this text". "Click here". "Is this element visible now?" It's very tempting to test your code manually, because the code you are writing is meant to be used by humans, not machines.

When we look at the typical Single-Page-Application (for example a React app), we usually also have different parts. And if you look closely you see that the transition from Tomcat/JSP to Spring-Boot/SPA is actually an attempt to reduce the parts of the application that are difficult to test.

We can extract complex logic into utility functions. Those are easy to test as long as they are pure (i.e. stateless)

Even for React components, there are test-helpers. They are a bit more complicated to test, because we need to simulate the user, but it is possible. But: My experience is that this part can be very frustrating. The React testing library uses jsdom, which has no way to give visual feedback of the current component state. We simulate a user and it takes a long time to write such tests, even for very simple components.

We can test the Http-Client by mocking the backend. Use real responses as fixtures and a library like `nock` or `mswjs` to create fake responses for your tests.

So, if we structure our code well, we can test a lot of things, from the backend-access up to (excluding) the presentation layer. We can do this in unit tests, withoug real backend. But what about your CSS. That's the missing link and that's what I want to talk about now.

The talk is named "Visual regression testing" and "regression" means that we want to ensure that something that has previously worked, is still working. This is not Test-Driven-Development. The idea is that we create a screenshot, verify it manually and then commit it as "baseline" screenshot.

Then, as we change our code, it may happen that screenshots do not match the baseline anymore.

jest-image-snapshot is on of many library that compare such snapshots. The reason why I show this as an example is, because we mostly use jest and I've had good experiences with it. When the test is executed the first time, the snapshot is saved into a `__image_snapshots__` directory next to the current test. Afterwards the current snapshot is compared to the baseline the test fails if they differ. The diff is saved into the __image_snapshots__ directory as well. Configuration options include. - How much as a pixel allowed to differ? - How many pixels are allowed to differ? - Structural comparison (SSIM algorithm) After verification, we can commit the new snapshots of fix the bugs.

In order to capture screenshots of a website automatically, we need a remote-controlled browser. Nowadays, there are more or less two ways of capturing image snapshots from the browser - Puppeteer: A remote-controlled Chrome browser. - Selenium: A generic protocol to control a variety of browsers. - There are other ways, like drawing HTML on canvas, but this is more complicated. Short conclusion towards those methods: - Selenium works in a variety of browsers including IE and Safari - Puppeteer is only for Chrome (Firefox experimental), but is faster than Selenium

This is the code to setup snapshots with puppeteer - we need to register the image-snapshot-function for use with expect - run puppeteer in the beginning - close puppeteer in the end

The test just navigates to the page, takes the screenshot and calls the image-snapshot function. Don't forget to set the viewport to a fixed size and yes: We also need to start a web-server.

The selenium code is pretty similar to the puppeteer code. The principles are the same.

While puppeteer is relatively easy to install (it is installed automatically by the npm-package), the selenium-server must be started as well. * There is a "selenium-standalone"-package on npm * There are lots of SaaS solutions that provide selenium as a service * There is a docker-image (Docker-Selenium) that includes chrome, firefox, a VNC-server and video-capturing capabilities

What can possibly go wrong? Of course, we have to make sure that screenshots are not changing. In this case, the "18 days ago" changes every day... So we need to design the test in a way that the image does not change

Common components change. This snapshot shows a different problem: When the logo changes, almost every page in the application will change. This is the cause for a lot of work an frustration. We should have a way to ensure that we can take screenshots of isolated components.

Different environments may have different browser-settings (Anti-Aliasing, Fonts etc). This may cause false-positives. Since we want to run tests locally on different team-members' laptops AND in CI, this is bad.

Storybook is a tool to create design systems. It allows you to create components isolated of other components, without the context of the whole site. It also allows you to run a web-server that shows the components on there own. At i-views, we had built a testing-setup with visual tests based on nightwatch.js, for our frontend. We sometimes had the problem that components were not isolated in that setup. When I heard about storybook, the first thing that came to my mind was: Let's use that to take screenshots of isolated components.

Then I found out that there are already services and projects that do exactly that.

Storyshots is an addon/plugin for storybook that creates snapshots of all stories. The sad thing about the basic version of storyshots is that it uses the React-Snapshot mechanism, to create diffs, so you won't get a visual output and styling changes do not have any effect on test.

However, there is another addon "addon-storyshots-puppeteer". This plugin uses puppeteer to create screenshots. It uses "jest-image-snapshot" to compare the screenshots. A lot of options can be provided in the "imageSnapshot"-function. The problem is, that different plattforms may produce different screenshots.

Using a unique browser environment. There should be a way to take screenshots from the same browser environment. At i-views, we used a dockerized Selenium-Hub with Firefox and Chrome-browsers. In order to enable access from the browser to the local-webserver, we started an SSH-client with a reverse-tunnel configuration. I recently found the project "chisel", which is a small Go program that can do exactly that.

There was a package `addon-storyshots-selenium` on npm, but it was pretty much unmaintained and, as always, was missing some things that I considered essential. So I started writing one myself. And now there are two plugins on npm that are unmaintained and unfinished. That's how it works on npm. The idea was to do the same as "addon-storyshots-puppeteer", but with Selenium. That way, I would be able to use self-hosted selenium docker images as well as cloud services with lots of different browsers.

And I wanted to be able to specify viewport-sizes for each story. Because sometimes you have components that need to be responsive, but for others it doesn't matter.

Of course, when using a service, you will expose your code to a third party. In this case, the whole storybook is uploaded to Chromatic (probably to the US). You get a couple of screenshots for free, but only in Chrome CI-Integration works by invoking the "chromatic" command. It will return a non-zero exit-code when a screenshot differs.

First we create a project

Then we follow the instructions. And receive a project-token that must be used in CI (remember to NOT commit the token to git, but use a secret variable)

Once the command is executed, you can see the screenshots in the UI

Further executions allow you to inspect the diffs and accept them.

Setup similar to Chromatic, this is the screen after creating a project.

Instructions on how to execute percy

After the build

Inspection and acceptance

What I would consider interesting as a next step is: There are ways to use storybook with puppeteer to implement functional tests. This would be a great way to have tests that you can watch running and maybe get a better understanding of why they fail.