artillery-engine-playwright

ℹ️ The code has been moved to https://github.com/artilleryio/artillery/tree/main/packages/artillery-engine-playwright

Questions, comments, feedback? ➡️  Artillery Discussion Board

Ever wished you could run load tests with real browsers? Well, now you can.

This Artillery engine lets you combine Playwright with Artillery to be able to launch a whole lot of real browsers to do full browser load testing.

At a glance

• 🤖   Run load tests with real (headless) Chrome instances
• 🛰   Run synthetic checks in CICD with the same Artillery + Playwright scripts
• 📊   See most important front-end metrics (Largest Contentful Paint (LCP), First Contentful Paint (FCP) etc) and how they are affected by high load
• ♻️    Use Playwright for load testing (full access to page API)
• 🏎   Create new load testing scripts 10x faster with playwright codegen
• 🌐   Launch thousands of browsers, with zero infrastructure setup with Artillery Pro

✨ Perfect for testing complex web apps ✨

Read the official launch blog post here: Launching 10,000 browsers for fun and profit

Why load test with browsers?

Load testing complex web apps can be time consuming, cumbersome, and brittle compared to load testing pure APIs and backend services. The main reason is that testing web apps requires a different level of abstraction: whereas APIs work at endpoint level, when testing web apps a page is a much more useful abstraction.

Summarized in the table below:

All of those factors combined make load testing web apps with traditional approaches very frustrating and time consuming. 😞

Usage ⌨️

Installation

Install Artillery and this engine:

npm install -g artillery artillery-engine-playwright

(See Use in Docker/CI if running tests in Docker/CI)

Running a test

Create an Artillery script:

hello-world.yml:

config:
  target: https://www.artillery.io
  # Enable the Playwright engine:
  engines:
    playwright: {}
  processor: "./flows.js"
scenarios:
  - engine: playwright
    flowFunction: "helloFlow"
    flow: []

Use a Playwright script to describe virtual user scenario:

(Note: this script was generated with playwright codegen. page is an instance of Playwright page.)

flows.js:

module.exports = { helloFlow };

async function helloFlow(page) {
  //
  // The code below is just a standard Playwright script:
  //
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
  // Click text=Pricing
  await page.click('text=Pricing');
  // assert.equal(page.url(), 'https://artillery.io/pro/');
  // Click text=Sign up
  await page.click('text=Sign up');
}

Run it:

artillery run hello-world.yml

Artillery runs Playwright-based scenarios, and provides user-centric metrics that measure perceived load speed such as LCP and FCP:

--------------------------------
Summary report @ 11:24:53(+0100)
--------------------------------

vusers.created_by_name.Dev account signup: .................. 1
vusers.created.total: ....................................... 1
vusers.completed: ........................................... 1
vusers.session_length:
  min: ...................................................... 5911.7
  max: ...................................................... 5911.7
  median: ................................................... 5944.6
  p95: ...................................................... 5944.6
  p99: ...................................................... 5944.6
browser.page.domcontentloaded: .............................. 2
browser.page.domcontentloaded.https://artillery.io/: ........ 1
browser.page.domcontentloaded.https://artillery.io/pro/: .... 1
browser.page.FCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.dominteractive:
  min: ...................................................... 162
  max: ...................................................... 1525
  median: ................................................... 162.4
  p95: ...................................................... 162.4
  p99: ...................................................... 162.4
browser.page.dominteractive.https://artillery.io/:
  min: ...................................................... 1525
  max: ...................................................... 1525
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.LCP.https://artillery.io/:
  min: ...................................................... 1521.1
  max: ...................................................... 1521.1
  median: ................................................... 1525.7
  p95: ...................................................... 1525.7
  p99: ...................................................... 1525.7
browser.page.dominteractive.https://artillery.io/pro/:
  min: ...................................................... 162
  max: ...................................................... 162
  median: ................................................... 162.4
  p95: ...................................................... 162.4
  p99: ...................................................... 162.4
browser.page.FCP.https://artillery.io/pro/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5
browser.page.LCP.https://artillery.io/pro/:
  min: ...................................................... 205.3
  max: ...................................................... 205.3
  median: ................................................... 206.5
  p95: ...................................................... 206.5
  p99: ...................................................... 206.5

Configuration

The underlying Playwright instance may be configured through config.engines.playwright.

You can pass the following options in:

• launchOptions - an object containing arguments to browserType.launch() method
• contextOptions - an object containing arguments to browser.newContext() method
• defaultNavigationTimeout and defaultTimeout - a number in seconds. These values are shorthands for browserContext.setDefaultNavigationTimeout() and browserContext.setDefaultTimeout

Example 1: turn off headless mode

You can turn off the default headless mode to see the browser window for local debugging by setting the headless option.

config:
  engines:
    playwright:
      launchOptions:
        headless: false

Example 2: set extra HTTP headers

This example sets the extraHTTPHeaders option for the browser context that is created by the engine.

config:
  engines:
    playwright:
      contextOptions:
        extraHTTPHeaders:
          x-my-header: my-value

Aggregate metrics by scenario name

By default metrics are aggregated separately for each unique URL. When load testing the same endpoint with different/randomized query params, it can be hepful to group metrics by a common name.

To enable the option pass aggregateByName: true to the playwright engine and give a name to your scenarios:

config:
  target: https://artillery.io
  engines:
    playwright: { aggregateByName: true } 
  processor: "./flows.js"
scenarios:
  - name: blog
    engine: playwright
    flowFunction: "helloFlow"
    flow: []

flows.js

module.exports = { helloFlow };

function helloFlow(page) {
  await page.goto(`https://artillery.io/blog/${getRandomSlug()}`);
}

This serves a similar purpose to the useOnlyRequestNames option from the metrics-by-endpoint artillery plugin.

Flow function API

By default, only the page argument (see Playwright's page API) is required for functions that implement Playwright scenarios, e.g.:

module.exports = { helloFlow };

async function helloFlow(page) {
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
}

The functions also have access to virtual user context and events arguments, which can be used to access scenario variables for different virtual users, or to track custom metrics.

module.exports = { helloFlow };

async function helloFlow(page, vuContext, events) {
  // Increment custom counter:
  events.emit('counter', 'user.page_loads', 1);
  // Go to https://artillery.io/
  await page.goto('https://artillery.io/');
}

More examples

See Artillery + Playwright examples in the main artillery repo.

Use in Docker/CI

Use the Dockerfile which bundles Chrome, Playwright and Artillery to run your tests in CI.

Note: To keep the Docker image small, browsers other than Chromium are removed (the saving is ~500MB)

Performance (Does it scale?)

When you run load tests with browsers your main bottleneck is going to be memory. Memory usage is what will limit how many Chrome instances can run in parallel in each VM/container. (Remember that every virtual user (VUs) will run its own copy of Chrome - just like in the real world.)

Two factors will determine how many VUs will be able to run in parallel:

1. Memory usage of the pages that the VU navigates to and uses. Lightweight pages will mean each VU uses less memory while it's running.
2. How long each VU runs for - the longer each session, the more VUs will be running at the same time, the higher memory usage will be.

The engine tracks a browser.memory_used_mb metric which shows the distribution of memory usage across all pages in each scenario (in megabytes). The value is read from window.performance.memory.usedJSHeapSize API. The metric is recorded for every load event.

Ultimately, there is no formula to determine how many VUs can be supported on a given amount of memory unfortunately as it will depend on the application and VU scenario. The rule of thumb is to scale out horizontally and run with as much memory as possible and track VU session length and memory usage of Chrome to be able to tweak the number of containers/VMs running your tests.

Questions, comments, feedback?

➡️   Let us know via Artillery Discussion board

License 📃

MPL 2.0