Skip to content

Latest commit

 

History

History
79 lines (52 loc) · 9.03 KB

visual_regression_testing_and_the_internal_www_server.md

File metadata and controls

79 lines (52 loc) · 9.03 KB

Visual Regression Testing And The Internal WWW Server

Quick Reference: how to use and extend it

  • to run the visual regression suite : gulp testVisual
  • to run the visual regression suite on a subset of the examples : gulp testVisual --specFilter=<PATTERN>. Only content files matching pattern will be executed.
  • to run the visual regression suite if the web driver binaries are already installed and gulp serve is already running: gulp testVisual_s
  • to view the visual regression results: login to [https://applitools.com] and look at your new tests.
  • to add a new visual regression test that does not require interaction with the widget:
    • create a new file in theSrc/internal_www/content using the content_template, or add an example to an existing content file IF IT FITS (dont mix concerns in your files)
    • add a <div snapshot-name="...."> element that wraps your example(s)
  • to add a new visual regression test that does require interaction with the widget:
  • where does the applitools key go: place it in a file called .keys/applitools.key no formatting, no newlines or spaces, no json, just the key.

Objectives

The objectives of maintaining a set of internal content for each html widget is two fold:

  • Allow visual regression testing : We can compare the content generated by two versions of the code to determine if anything unexpected has changed. This can be done manually via inspection of the content, or automatically using a tool called applitools.
  • Real time feedback during dev : This allows a workflow where I write up a config for a widget that doesn't currently support what I am trying to do, then I start writing the code to support the new feature. Every time I save my work, the widget is redrawn, so I can see in real time how my work is progressing.

How the browser area is built

The internal web server is just hosting all the files in the browser area, which is an auto generated section of the repo. Several gulp steps work in conjunction to build the content in the browser directory and serve it at http://127.0.0.1:9000. The 'important' ones are described below:

  • the compileInternalWeb step compiles ES6 into ES5 for the browser
  • the copy step copies all the html and image files from theSrc/internal_www into the browser area
  • the buildContentManifest step recursively scans the browser/content area and produces a manifest of all the content files in the area. This is used to build the index page that is displayed on http://127.0.0.1. Without this step the author would need to keep this list up to date by manual updates to the index.html file.
  • the connect step starts a static content web server hosting all the files in the browser directory and makes them available on port 9000 of localhost (i.e., http://127.0.0.1:9000)
  • the watch step runs constantly and monitors all the content used to build the browser area. Any time the content is saved, the watch step will rerun one of the other build steps to update the content, and then it sends a signal to the browser to force a page reload

How a widget is rendered in the internal web server area

Using the content file theSrc/internal_www/content/examples/default.html as an example we will now go through how the widget gets drawn. Note this only applies when viewing a widget at http://127.0.0.1 (i.e., the internal web server). For notes on how htmlwidgets work with R, see how the code works.

  • the renderContentPage.js script is a bundled JS file that contains all the widget code, all the dependencies, and some internal web only code that performs the following tasks. The internal web only code, once the page is loaded, scans the HTML content for DOM elements with a class="row". For each row, it then finds all DOM within the row with a class="example". For each example, it takes the content of the DOM, which should be a config for the widget, then calls the widget code with the provided config, passsing the example DOM element as the root element to the widget code. This means the widget code will draw a widget inside the example DOM element.

There are some other features of renderContentPage.js that should be discussed:

  • The default width and height of a widget is 100 x 100. Each row can specify the width and height by using the data-ex-w and data-ex-h attributes. Similarly each exaple can override the width and height using the same attribtues.
  • By default the config will be drawn on the page above the widget. The config can be hidden by adding the hide-config class to the example.
  • By default the resize controls are hidden. The controls can be shown by adding a data-resize-controls="true" attribute to the example DOM element.
  • Any DOM that has a snapshot-name="SNAPSHOT_NAME" attribute will cause the applitools integration to take a snapshot when the visual regression suite is run (more on this below).
    • Note that adding snapshot-name also changes the css display property to inline-block, which affects the layout of the content. This is done so that the width of the container is limited to the content inside, instead of the full availalbe browser width. In other words: Applitools wants a fixed box, display: inline-block provides this. As a result, it's better to wrap your examples with a new <div snapshot-name="NAME"> instead of adding the snapshot-name attribute directly to your examples.

Note that the example of content page features file shows all the options being used.

How protractor works

Protractor is a wrapper on top or a wrapper. At the core you have selenium which provides a programmatic way to interact with a real browser. On top of that we have webdriverJS, which is a node.js wrapper on top of selenium so that we can write selenium code in node.js. On top of that we have protractor, which provides some angular.js specific features and some other niceities. While protractor is not strictly necessary for this project, and an alternative like webdriver.io or other could be considered, the author of the module was familiar with protractor and protractor is well packaged such that the barrier to use was minimal, thus it was chosen. By default (and for this project) protractor uses the jasmine test framework in order to structure tests.

Lets look at protractorInteractionExample.js for a real use of protractor and applitools. The initializeApplitools wrapper name initializes the Applitools Eyes object with our license key and some settings that are best for our project. Next you can see a combination of a describe, beforeEach, and an it block. This is standard convention for structuring tests using jasmine, read this tutorial here for an in depth treatment. Suffice to say for now the it block is the actual test, if there are multiple it blocks that share some setup code then we put that in the beforeEach block, and the describe blocks provide a hierarchical structure to your tests.

In the it block entitled red is selected we start do a few things:

  • tell the browser to load the page /content/examples/default.html
  • wait for the widget to fully render
  • make protractor click on an area of the rendered SVG. This click is done by passing a CSS selector to protractor, and then calling click() : element(by.css('.text.red')).click().
  • make applitools snapshot the state of the widget. This is done by passing a different CSS selector to applitools eyes.checkRegionBy(by.css(svg.rhtmlwidget-0), 'redSelected');

The protractor API docs should be consulted for more information.

How visual snapshots are taken in the content area

The visual snapshots in the content area are taken using similar steps to what is outlined above, however the process is automated such that no new test code is required when we add new content. This automation is performed by the testVisual.js script. These are the steps taken by this script:

  • generate a list of all the content files in the internal_www/content area
  • for each content file:
    • load the file using protractor
    • scan the HTML contents for DOM elements that have a snapshot-name attribute
    • for each DOM element with snapshot-name="NAME":
      • take a snapshot using applitools and name the snapshot NAME

Applitools UI Details

TODO complete this section