Understanding Travis builds

Understanding Travis builds

There are a few key details to consider when reviewing Travis builds.

Build types

There are two types of Travis builds that will run under certain conditions: pr builds and push builds.

A pr build runs for every opened pull request. This build runs unit tests under a local headless stable Chrome, and then performs a fidelity test.

A push build runs for pull requests opened that compare branches from the same repository (in other words, PRs from contributors working out of the main repository). This build runs the same tests as a pr build, but also includes two additional stages of unit tests: Sauce Labs tests including a battery of modern browsers, and Sauce Labs tests running IE11 and Chrome against the "legacy" bundle of <model-viewer>.

Unit tests

For the most part, reviewing unit tests is very straight forward: Travis will clearly display any fatal error messages encountered during the testing process. However, there are some types of failure that will cause tests to stop running prematurely. This will be represented by some browsers reporting a number of successful test runs that is significantly lower than expected (even if no errors are reported). So, keep an eye out for that.

Fidelity tests

Somewhat unique to <model-viewer> is the inclusion of fidelity tests in CI builds. These builds test the similarity of <model-viewer>'s representation of models to other reference viewers such as iOS Quick Look and Filament.

There is a high likelihood that any proposed change to rendering will cause the fidelity test to announce a change in the relative consistency of <model-viewer> to other renderers. Improvements in consistency are good and are worth noting / applauding. Regressions in consistency should be scrutinized by a reviewer to ensure that there were no critical rendering bugs introduced in the change. All changes will be noted in the CI build, but regressions that exceed 10% relative to the last stable release will be alerted with greater urgency.