Run load test/performance test with Artillery

Introduction

What is performance testing? And why should you do a load test?

Performance testing is software testing that evaluates how well an application performs under different workloads.

This involves measuring various metrics such as response time, bandwidth, and scalability to ensure that the application can handle the expected number of requests from users without degrading performance or crashing. Performance testing can also identify performance issues and bottlenecks, it helps you make necessary changes to optimize performance. The goal of this is to ensure that the application delivers a smooth, fast, and reliable user experience even under heavy load conditions.

About Artillery.io

Artillery.io is an open-source performance testing tool that allows you to test the performance and scalability of your web applications, APIs, and microservices. You can simulate real user behavior, generate load from multiple sources, and test your application flow according to the scenario.

Supported protocols

Artillery.io is written in Node.js and supports HTTP, Websocket, and socket.io protocols.

Also, you can use Artillery’s plugin to test another stack like HLS (HTTP Live Streaming), Amazon Kinesis, Apache Kafka, etc.

Script formatting

Unlike other popular testing tools, Artillery.io provides a flexible YAML-based syntax for defining your load-testing scenarios. It also supports a JSON format.

Install & Basic features

Install

You can globally install Artillery.io via npm or yarn by following this command:

#npm
npm install -g artillery

#yarn
yarn global add artillery

or you can also install it as a dev dependency of a Node.js project with:

npm install -D artillery

Fast test

Use the quick subcommand so that you can run a test without writing a test script. For example, to run 10 virtual users (VUs), each of them makes 20 requests to the address http://localhost:3000/:

artillery quick -c 10 -n 20 http://localhost:3000/api/user

The -c parameter specifies the total number of VUs, and the -n indicates the number of requests per VU.

Run a test script

The run subcommand runs a test script from the local machine. The basic way to run a test is with:

artillery run script.yaml -o report.json

The -o parameter is the option to write a JSON report to a file.

Export report

artillery report report.json -o report.html

The report subcommand allows converting the JSON report generated by the run subcommand into a new HTML report.

Writing a simple load test

In the tutorial below we will use YAML syntax, first we need to create a file name like “script.yml”, then we start by defining the “config” section at the top of a file.

Config Section

We need to set the base URL for your test script with the target option:

config:
  target: "http://localhost:3000/api"

Next, we’ll start to define the load phase, a load phase defines how Artillery generates new VUs in a specified time period:

config:
  ...
  phases:
    - duration: 60
      arrivalRate: 100

Scenarios Section

We can define one or more scenarios. Each scenario definition is an object which requires a flow attribute.

scenarios:
  - name: "Get All Users"
    flow:
      - get:
          url: "/user"
          qs:
            limit: 10
            offset: 20

In the ‘flow’ property, we continue to define the HTTP method as an object attribute, then we need to define the path of the API, we also can set a query string or set form body like URL-encoded forms, Multipart forms or JSON.

Save the scripts and then run your test script.

Realistic User Flows

For example, a user flow could be:

config:
  target: "http://localhost:3000/api"
  phases:
    - duration: 120
      arrivalRate: 10
      name: "Preparing"
    - duration: 240
      arrivalRate: 30
      rampTo: 100
      name: "Increasing"
    - duration: 600
      arrivalRate: 100
      name: "Sustained Load"
  payload:
    path: "login.csv"
    cast: false
    fields:
      - "user"
      - "pass"
  processor: "./processor.js"

before:
  flow:
    - log: "Get auth token"
    - post:
        url: "/auth/login"
        json:
          username: "{{user}}"
          password: "{{pass}}"
        capture:
          - json: $.tokens.accessToken
            as: token
scenarios:
  - name: "Search Users"
    flow:
      - get:
          url: "/user"
          headers:
            authorization: "Bearer {{ token }}"
  - name: "Create New User"
    flow:
      - post:
          url: "/user"
          beforeRequest: "setJsonBody"
          headers:
            authorization: "Bearer {{ token }}"

In the above script, we have three phases in the configuration:

• The first phase creates 10 VUs per second and sends them to the application for 2 minutes.
• In the second phase, the load test will start at 30 users per second and gradually increase to 100 users per second over 4 minutes.
• The final phase simulates a sustained load of 100 users per second for 10 minutes.

By using multiple phases allows you to accurately simulate realistic traffic scenarios and evaluate your application’s ability to handle a sudden barrage of requests.

In the first flow of scenarios, each of the VUs requests GET | “Search user”, then in the second flow they’ll request POST | “Create user”.

To make it easy to generate data. In this way, before executing the request. We use the setJsonBody function on the custom hook beforeRequest, The function setJsonBody is defined in the processor.js file referenced in config.processor.

# processor.js
const { faker } = require("@faker-js/faker");

function setJsonBody(requestParams, context, ee, next) {
  var body = {
    username: faker.internet.userName(),
    password: faker.internet.password(3),
    email: faker.internet.exampleEmail(),
    name: faker.name.fullName(),
    role: faker.datatype.number({ min: 0, max: 1 }),
  };

  requestParams.json = body;
  return next();
}

module.exports = {
  setJsonBody,
};

Compare to K6 load testing

Artillery and k6 are both open-source tools for performance testing and have similar features such as extensibility, integration with CI/CD tools, and the ability to set test patterns. However, there are some differences between them. Artillery is written in Node.js while k6 is written in Go. So, Artillery is slower than k6, because it isn’t multithreaded and uses more memory. Here are the charts comparing memory usage between k6 with Artillery and other tools:

Memory usage

Memory usage per VU level

In conclusion, Artillery and k6 are both powerful tools. However, k6 has more advantages than Artillery. If you want to write a simple load test, Artillery is a good solution that uses YAML syntax.

References

https://www.artillery.io/
https://k6.io/blog/comparing-best-open-source-load-testing-tools/