Adding a New Test

To create a new framework you can start the new test initialization wizard with the command:

tfb --new

This will walk you through the scaffolding process. You will still need to follow the steps in Additional Information below.

Adding a New Test Without the Installation Wizard

When adding a new framework or new test to an existing framework, please follow these steps:

  • Add/Update a test docker file.
  • Add/Update a benchmark_config.json.
  • Implement at least one test (we'd be even happier with six completed tests). When creating a database test, use the table/collection hello_world. Our database setup scripts are stored inside the config/ folder if you need to see the database schema.
  • Add/Update a README for your framework.
  • Add/Update a README for your language.

Additional Information

  • Make sure that any packages/dependencies you are using or any git source repositories you are pulling in are locked down to specific version numbers or releases.
  • Add an entry for the framework test directory in .travis.yml if the framework is new to the benchmarks.
  • Test your framework appropriately.
    • Ensure the framework tests implemented pass in your local environment.
    • Travis CI will test the framework when a pull request is opened. Tip: Setup your own Travis CI account and know the outcome of the tests before you submit a merge request. See Travis CI Tips and Tricks for more details.
  • When all tests are passing, submit a pull request following the pull request procedure.