This is the getting started guide for __benchmarking__.

If you're interested in development please visit the getting started guide for development.

You will need three computers (or virtual machines) networked together to fill the three benchmark roles of database server, load generation server, and web framework server.

Linux Server Setup

When setting up a Linux server for benchmarking, you can follow the same steps as setting up a Linux development environment.

Note: If testing a pull request or doing development, it is usually adequate to only use one computer. In that case, your server, client, and database IPs will be

Benchmark Suite Deployment

Required Host Roles

  • App (aka web framework) Server
    • Runs the web framework containers
    • Operating system: Ubuntu Server 16.04 LTS 64-bit
  • Database Server
    • Runs database containers (e.g. MySQL, PostgreSQL, MongoDB, etc).
    • Operating system: Ubuntu Server 16.04 LTS 64-bit
  • Client (aka load generation) Server
    • Runs the benchmark client container
    • Operating system: Ubuntu Server 16.04 LTS 64-bit

Manual Deployment


Before you get started, the following are the steps you must take on each machine (App, Database, Client) to run the benchmarking suite:

  1. Docker installed (you can verify this via docker run hello-world)
  2. /lib/systemd/system/docker.service needs ExecStart to have an additional flag: -H [machine's ip]:2375
  3. sudo systemctl daemon-reload
  4. sudo service docker restart (you can verify this via sudo service docker status and see the newly added -H [machine's ip]:2375 flag)

Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.

Running the Benchmarking Suite

Let us assume make some assumptions for the example:

  1. You cloned the repository into the home directory of user techempower
  2. Your App machine has the IP
  3. Your Database machine has the IP
  4. Your Client machine has the IP

The following command, given the prerequisites here, would run a benchmark for all the tests in the codebase.

docker run \
  --network=host \
  --mount type=bind,source=/home/techempower/FrameworkBenchmarks,target=/FrameworkBenchmarks \
  techempower/tfb \
  --server-host \
  --database-host \
  --client-host \
  --network-mode host \