Automated Testing Using Travis CI

This page describes the automated testing framework that was setup in November 2017. The objective was to be able to validate behaviour using BDD definitions in Gherkin format, on every pull request, in a low-maintenance testing environment.

To achieve this we decided to use a cloud-based automation tool called Travis-CI. This is a build automation tool for running Continuous Integration tests against committed changes, which creates an environment on a clean VM for each test run.

This Travis project is configured to trigger on changes to the Fixometer-Specs project's Github repository. Note this is a separate repo to the Fixometer; it is intended that these will be merged.

Configuration

Travis is configured using a .travis.yml, which defines stages of preparation (in this order):

• before_install - create an empty fixometer_dev database and user
• install - uses Composer to manage dependencies
• before_script
  • clone the fixometer repo; apply database schema
  • setup Apache and php-fpm
  • configure and start the Fixometer

Then the script section runs Codeception acceptance tests that have been defined in tests/acceptance.

Testing Approach

In general the approach is for each test's feature definition to setup scenarios by stating the contextual data (i.e. "Given there's these records in the database..."), some action (e.g. "When I view the headline stats...") and an expected outcome ("Then the CO2 diverted should be..."). Each scenario should be idempotent, should not affect subsequent scenarios.

Debug Mode

Travis has a Debug Mode that allows you to 'jump into' the running test environment and step through the stages, explore working data etc. The debug session is time limited to 30 minutes. You'll need to generate a API token to enable debug, then get the job ID and call:

curl -s -X POST \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ -H "Travis-API-Version: 3" \ -H "Authorization: token <API-token>" \ -d '{ "quiet": true }' \ https://api.travis-ci.org/job/<job-id>/debug

The log will give you a SSH session ID for TMate that you can connect to.