You can easily list all available tests
$ tfb --list-tests activeweb activeweb-raw aspnet aspnet-jsonnet aspnet-mongodb-raw aspnet-mono aspnet-mono-jsonnet aspnet-mono-mongodb-raw <snip>
There are a number of options that can be specified:
# Run a verification for test beego $ tfb --test beego --mode verify # Run the default benchmark for the beego test $ tfb --test beego # Specify which test types are run during benchmark $ tfb --test beego --type json $ tfb --test beego --type db $ tfb --test beego --type fortune # Specify a number of options for how the load is generated $ tfb --test beego --max-concurrency 24 --max-threads 24 --duration 20 --max-queries 200 # Run a tiny benchmark $ tfb --test beego --max-threads 2 --max-concurrency 2
results directory must be removed after each test has been run
in order to run the test again.
Testing on both Windows and Linux
If your framework and platform can execute on both Windows and Linux, we encourage you to specify tests for both operating systems. This increases the amount of testing you should do before submitting your pull-request, however, so we understand if you start with just one of the two. Travis-CI cannot automatically verify Windows-based tests, and therefore you should verify your code manually.
The steps involved are:
- Assuming you have implemented the Linux test already, add a new test
permutation to your
benchmark_config.jsonfile for the Windows test. When the benchmark script runs on Linux, it skips tests where
Windowsand vice versa.
- Add the necessary tweaks to your setup file to start and stop on the new operating system. See, for example, the script for HttpListener.
- Test on Windows and Linux to make sure everything works as expected.
Enable Travis-CI on your project fork. This is
as simple as going to travis-ci.org, using the
Log in with Github
button, and enabling Travis-CI for your fork. If you're submitting a new
framework, be sure to add it to the
travis.yml file in the root of the
project. When your development is done and your changes pass the Travis-CI
verification, submit a pull request with confidence that it can be merged
quickly. Read more about TFB and Travis-CI.
Important Note About Travis-CI
If you make changes to configuration files, or files outside your framework directory, Travis will run tests on all existing frameworks. For this reason, it may appear that your tests have failed. Be sure to check your Travis build by clicking on the checkmark or red 'X' to dig into your specific test.
Finding output logs
Logs file locations use the format
The general structure is
results/<run name>/<timestamp>/logs/<test name>/<file>.
You can use the
--name flag to change the
If you re-run the same test multiple times, you will get a different folder
<timestamp>, although the
latest folder will be kept up to date.
<test name> is simply the name of the test type you ran, and
<file> is either
err.txt (these are the
logerr arguments passed into each