Thursday, 28 March 2013

WebVTT reftests

A reftest is a pass or fail test which checks that two web pages look identical. It does this by producing two bitmap images and verifying if they are identical or not. HTML content can be rendered the same way using different methods, such as an image and a specific frame of video. Reftests are useful to ensure that something is rendered correctly by comparing to what is ought to look like. Specifically, this is done by creating two web pages and comparing them.

Further information

Importance

This is tremendously important for WebVTT because it is the only way to ensure consistency to the specification, and therefore consistency between implementations. This means that a WebVTT file will look the same anywhere it is used, and thus enables content developers to use advanced features. WebVTT is a web standard after all.

The goal of web standards is to make implementation interoperable. (W3C) This means "the situation in which all implementations process particular content in a reliable and identical or equivalent way." (W3C) Historically when format implementations were not consistent with each other, developers only used a limited subset of the format features, had multiple versions for each implementation, or used a third-party solution. Recall the browser wars and the push for browsers to comply with web standards. (Web Standards Project)

Reftest Format

A reftest consists of three files:
  • Test page with feature to be tested
  • Reference page that show have the feature should look
  • A reftest.list file that lists the assertions

Example

The following test should pass and serves to demonstrate the basic format.

spaces1.html
<html><body>
X X
</body></html>

spaces2.html
<html><body>
X&nbsp;X
</body></html>

reftests.list
== spaces1.html spaces2.html

Details

There are two main types of assertions, expect pass (==) or expect fail (!=).

The two tests will be processed as soon as the page finished loading. Sometimes the test needs to be delayed for asynchronous content. This can be accomplished by adding the class "reftest-wait" to the HTML element and removing it at the appropriate time. This is required for WebVTT test because the video and text tracks are loaded  asynchronously.

Basic WebVTT Test

basic.html
<html class="reftest-wait">
<head>
 <meta charset="UTF-8">
 
 <style>
  #testVideo {
   position: absolute;
   left: 0px;
   top: 0px;
   width: 640px;
   height: 480px;
   margin: 0px;
   padding: 0px;
  }
 </style>
 
 <title>WebVTT</title>
</head>
<body>
 <video id="testVideo">
  <source src="grey320x240.ogv" type="video/ogg">
  <track src="basic.vtt">
 </video>
 
 <script src="testScript.js"></script>
</body>
</html>

basic-ref.html
<html class="reftest-wait">
<head>
 <meta charset="UTF-8">
 
 <style>
  #testVideo {
   position: absolute;
   left: 0px;
   top: 0px;
   width: 640px;
   height: 480px;
   margin: 0px;
   padding: 0px;
  }
  #testDiv {
   position: absolute;
   left: 0px;
   top: 250px;
   width: 640px;
   margin: 0px;
   padding: 0px;
  }
 </style>
 
 <title>WebVTT</title>
</head>
<body>
 <video id="testVideo">
  <source src="grey320x240.ogv" type="video/ogg">
 </video>
 <div id="testDiv">WebVTT Test</div>

 <script src="testScript.js"></script>
</body>
</html>

testScript.js
/*
   Make sure video is loaded,
   and that it is always at the same frame.
*/
document.getElementById('testVideo');

// Need to play to load video
testVideo.onplay = function() {
 
 // Stop video and seek to 5 seconds
 testVideo.onpause = function() {
 
  // When video is loaded, preform test
  testVideo.oncanplaythrough = function() {
   document.documentElement.className = "";
  };
  testVideo.currentTime = 5;
 };
 testVideo.pause();
};
testVideo.play();

Breakdown

CSS
The style is to make sure margins, padding, and positions are not a factor. We are not testing that, and by knowing the exact properties of the video we can create precisely what the caption should look for. It is important to only test one thing in each test.

Javascript
In order to make sure the test is correct, the bitmap must be created on exactly the same frame in the video. The following steps are performed to ensure that.

  • The video must be loaded and is not loaded until the video has been instructed to play.
  • The test must be performed on the same frame so the video must be paused on that frame.
  • To set the video to the correct frame for the test the video must be set to a predetermined frame.
    • currentTime is a decimal number in seconds
    • 1/24 is a second represents one frame in a 24 frame per second (fps) video
  • It may take time for the video to load the frame so it must wait until the video is loaded
    • A loading overlay may be shown while the video is loading. It must be gone before the test is performed.
  • Remove the "reftest-wait" class to perform the test.

Video

I created the video for the WebVTT tests. To do this I created a completely grey image. I chose grey so that white and black will be easily seen and not blend into the background. I used VirtualDub to create a raw video AVI using an AviSynth file with the ImageSource function and a silent audio file I created. I then converted the AVI to the Theora video format (OGV). Theora was chose because it is a free and open format, is specified by the HTML5 standard, and works in most browsers including Mozilla Firefox and Google Chrome.

File size is 30 kilobytes.

video
Created by Kyle Barnhart (me)
Released to public domain under Unlicense

Running Tests

Test are easily run in a Mozilla build. Just used "./mach reftest" to run all tests. You can also specify a particular set of tests. For WebVTT I used the following.

./mach reftest layout/reftests/webvtt/reftest.list

This also shows the standard location for reftests in mozilla-central.

After you have built the code, running the tests only takes a few seconds or more depending on the number of tests and the time it takes to clear the "reftest-wait" class in each test that must be delayed.

No comments:

Post a Comment