# Load testing 

Load testing sends a controlled amount of traffic at your app to find out how it behaves at the request rates you expect from real users. This guide covers which environment to test, how to pick a tool, how to write a test that stays inside your app's limits, and how to read the results. The examples use [Grafana k6](https://grafana.com/docs/k6/latest/), an open source tool that runs test scripts written in JavaScript.

## Test your production environment 

development environments have much lower rate limits than production environments, and they run on shared development infrastructure. A load test against a development environment reaches its limits long before your production app would, so its numbers do not tell you how production behaves. Run load tests against your production environment.

Load test traffic counts the same as real traffic. It consumes your production environment's [rate limits](https://docs.gadget.dev/guides/development-tools/rate-limits), it consumes platform credits for every request and action, and requests above the surge threshold are billed at the [surge compute](https://docs.gadget.dev/guides/account-and-billing#surge-compute) rate. Run your tests at a quiet time of day so the test and your real users are not competing for the same limits, and stop a test as soon as it starts returning `429` errors.

Your production environment's limits depend on your plan. The [pricing](https://gadget.dev/pricing) page lists the included requests per 10-second window and the included database time for each plan, and each [execution capacity](https://docs.gadget.dev/guides/account-and-billing#execution-capacity) unit adds `+500` requests and `+20,000ms` of database time per 10 seconds. The included request count is the surge threshold, not the point of rejection. Above it, requests keep running on [surge compute](https://docs.gadget.dev/guides/account-and-billing#surge-compute), which is billed at a higher rate, until usage reaches the environment's hard limit, where requests are rejected with `429` errors. Read the numbers for your plan before you pick a target rate. For example, a production environment on the Pro plan includes 500 requests per 10 seconds. A test that sends 60 requests per second fills that bucket in under a minute. From then on, about 50 requests per second keep passing through the included bucket as it drains, and the other 10 per second spill over into surge compute and are billed at the surge rate.

Rate limits exist to protect your app and the platform. You do not need to tell Gadget before running a load test. A test that reaches the hard limit is rejected with `429` errors in the same way as any other burst of traffic.

## Choose a tool 

Any HTTP load testing tool can test a Gadget app, because your app's Public API, Internal API, and HTTP routes are plain HTTPS endpoints. Pick a tool that can:

*   send a fixed number of requests per second, so a slow response does not quietly lower the load
*   stop the test on its own when an error rate or a latency percentile crosses a line you set
*   read a response body, so it can tell a rate limit rejection apart from a real failure
*   keep its scripts in your repository next to your app code

These tools meet those needs:

*   **[Grafana k6](https://grafana.com/docs/k6/latest/)**: JavaScript scripts run by a single binary, with arrival rate executors, thresholds that can abort the test, and a hosted runner for tests larger than one machine can generate
*   **[Artillery](https://www.artillery.io/docs)**: YAML scenarios with JavaScript hooks and a hosted runner
*   **[Locust](https://docs.locust.io/)**: Python scripts with a web UI for steering the test while it runs
*   **[Gatling](https://docs.gatling.io/)**: Java, Kotlin, or Scala scripts with detailed HTML reports

For a quick check of one endpoint, a command line tool like [oha](https://github.com/hatoo/oha) or [autocannon](https://github.com/mcollina/autocannon) is enough. The rest of this guide uses k6, but the same steps apply to any of these tools.

## Choose what to test 

Test the requests that real users make, in the same proportions. A test that only reads records tells you little about an app whose busiest moment is a checkout that runs three actions. Each kind of request exercises a different part of your app:

*   **Public API reads**, such as `findMany` queries, are served without a worker. They consume the request rate limit and the database rate limit, but not worker CPU time.
*   **Actions**, called through the Public API, run your code on a worker inside a database transaction. They consume several units of the request rate limit per call, plus database time and worker CPU time.
*   **HTTP routes** run your code on a worker. They consume one unit of the request rate limit per request, plus whatever the route handler does.
*   **Frontend assets** are served from Gadget's CDN and are [exempt from the request rate limit](https://docs.gadget.dev/guides/development-tools/rate-limits#request-rate-limit-exemptions). Leave them out of your test, because they add load to the CDN and not to your app.

An action consumes more than one unit of the request rate limit per call. A `create` action that does nothing but save its record consumes 4 units: 1 to start the action, 2 to open and commit the transaction, and 1 for the `save` call. With 500 included requests per 10 seconds, that is 125 creates per 10 seconds, or 12 per second, before the environment starts surging and before any other traffic. See [request rate limit](https://docs.gadget.dev/guides/development-tools/rate-limits#request-rate-limit) for how each call is counted.

## Set up k6 

Install k6 with a package manager, or run it from the Docker image:

```bash
brew install k6
```

Keep your test scripts in a `load-tests` folder at the root of your project. Create or update the `.ignore` file at the root of your project so the scripts are not synced to Gadget or bundled into your app:

```markdown
// in .ignore
load-tests/
```

Add the entry before you create the folder. If `ggt dev` is running, files that exist before the `.ignore` entry is pushed are synced to your development environment.

Create an API key for the test from the **Settings** > **API keys** page of the Gadget editor, and turn **Access control** off for it. The cleanup step in this guide deletes records through the Internal API, which only accepts a key with access control turned off. A key like this can read and write every record in the environment, so treat it as a secret and delete it when you finish testing.

API keys belong to one environment. Create the key in your production environment, because that is the environment you are testing.

You can split this into two keys: one with a role that grants only the permissions the requests under test need, and one with access control off that the script uses only in `teardown`. See [API key roles](https://docs.gadget.dev/guides/access-control#api-key-roles) for how roles apply to API keys.

Pass the app URL and the API key into k6 as environment variables rather than writing them into the script:

```bash
export APP_URL=https://your-app.gadget.app
export GADGET_API_KEY=gsk-a1z1z1z1z1z1z1z1z11z
```

The scripts read them from `__ENV`, which k6 fills from the `--env` flag and from the shell environment.

## Write a smoke test 

A smoke test sends a handful of requests to prove that the script works and the app answers. Write and run it before any test that adds real load.

The Public API is a GraphQL endpoint at `/api/graphql` that accepts an API key as a bearer token. This script reads the first 50 products, checks the response, and counts `429` responses on their own metric so a rate limit rejection is not mistaken for a broken app:

```javascript
// in load-tests/smoke.js
import http from "k6/http";
import { check } from "k6";
import { Rate } from "k6/metrics";

const APP_URL = __ENV.APP_URL;
const API_KEY = __ENV.GADGET_API_KEY;

// a 429 is a rate limit rejection, not a failed request
http.setResponseCallback(http.expectedStatuses(200, 429));
const rateLimited = new Rate("rate_limited");

export const options = {
  thresholds: {
    // abort the run if more than 5% of requests are rate limited
    rate_limited: [{ threshold: "rate<0.05", abortOnFail: true, delayAbortEval: "10s" }],
    // anything other than a 200 or a 429 is a real failure
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<500"],
  },
};

const productsQuery = `
  query LoadTestProducts {
    products(first: 50) {
      edges {
        node {
          id
          title
          price
        }
      }
    }
  }
`;

export default function () {
  const res = http.post(`${APP_URL}/api/graphql`, JSON.stringify({ query: productsQuery }), {
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${API_KEY}`,
    },
    // group every request to this query under one name in the results
    tags: { name: "product.findMany" },
  });

  rateLimited.add(res.status === 429);
  check(res, {
    "status is 200": (r) => r.status === 200,
    "no GraphQL errors": (r) => r.status === 200 && !r.json("errors"),
    "returned products": (r) => r.status === 200 && r.json("data.products.edges").length > 0,
  });
}
```

Run it with a fixed number of iterations:

```bash
k6 run --iterations 10 load-tests/smoke.js
```

k6 prints a summary when the run ends. The `checks` line shows how many assertions passed, `http_req_duration` shows the latency percentiles, and each threshold is marked as passed or failed. If any check fails, fix the script before adding load. A `403` response with the code `GGT_PERMISSION_DENIED` means the API key has access control on and its role is missing a permission the query needs. The `returned products` check needs at least one record in the model, so run it after the model has data.

The first requests to an environment that has been quiet are slower than the rest while Gadget warms up the caches and workers that serve it, and one of them can time out. Expect the latency threshold to fail on the first smoke run and pass on the second. If the second run is still slow, the environment is not warming up, and the result is real.

The `rate_limited` threshold is what makes the test safe to point at production. When more than 5% of requests are rejected, k6 stops sending traffic instead of holding your app at its limit for the rest of the run. The `delayAbortEval` option gives the first 10 seconds a chance to settle before the threshold is evaluated.

## Test actions and routes 

Actions are where your code runs, so they are the most useful thing to load test. They also need the most care. Every iteration creates real records in your production database, and a rate limit reached inside an action comes back as a `200` response with the error in the action's `errors` list rather than as a `429`.

Give every record the test creates a value you can filter on, and delete them in `teardown` when the run ends. This script calls a `create` action on a `product` model and cleans up after itself:

```javascript
// in load-tests/create-product.js
import http from "k6/http";
import { check, sleep } from "k6";
import { Rate } from "k6/metrics";
import exec from "k6/execution";

const APP_URL = __ENV.APP_URL;
const API_KEY = __ENV.GADGET_API_KEY;

http.setResponseCallback(http.expectedStatuses(200, 429));
const rateLimited = new Rate("rate_limited");

export const options = {
  // cleanup may have to wait out the rate limit that stopped the test, so give it longer than the 60-second default
  teardownTimeout: "5m",
  thresholds: {
    rate_limited: [{ threshold: "rate<0.05", abortOnFail: true, delayAbortEval: "10s" }],
    // an action that fails validation or permissions still returns a 200, so abort on failed checks too
    checks: [{ threshold: "rate>0.99", abortOnFail: true, delayAbortEval: "10s" }],
    http_req_failed: ["rate<0.01"],
    // scope latency to the request under test, so the cleanup request does not skew it
    "http_req_duration{name:product.create}": ["p(95)<1000"],
  },
};

function graphql(query, variables, name) {
  return http.post(`${APP_URL}/api/graphql`, JSON.stringify({ query, variables }), {
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${API_KEY}`,
    },
    tags: { name },
  });
}

// null for a non-200 response or a body that is not JSON, such as a gateway timeout
function resultAt(res, path) {
  if (res.status !== 200) return null;
  try {
    return res.json(path);
  } catch (error) {
    return null;
  }
}

const createProductMutation = `
  mutation LoadTestCreateProduct($product: CreateProductInput) {
    createProduct(product: $product) {
      success
      errors {
        code
        message
      }
      product {
        id
      }
    }
  }
`;

export function setup() {
  // one marker per run, shared with every virtual user and with teardown
  return { runMarker: `load-test-${__ENV.RUN_ID || Date.now()}` };
}

export default function ({ runMarker }) {
  const res = graphql(
    createProductMutation,
    // idInTest is unique across load generators, where __VU is not
    { product: { title: runMarker, price: 10, sku: `${runMarker}-${exec.vu.idInTest}-${__ITER}` } },
    "product.create"
  );

  const result = resultAt(res, "data.createProduct");
  // a limit reached inside the action arrives as a 200 with this code, and errors is null on success
  const rejectedInsideAction = result && (result.errors || []).some((error) => error.code === "GGT_TOO_MANY_REQUESTS");
  rateLimited.add(res.status === 429 || rejectedInsideAction);

  check(res, {
    "product created": () => result && result.success === true,
  });
}

const deleteManyMutation = `
  mutation LoadTestCleanupProducts($title: String!) {
    internal {
      deleteManyProduct(filter: { title: { equals: $title } }) {
        success
        errors {
          message
        }
      }
    }
  }
`;

export function teardown({ runMarker }) {
  // the Internal API deletes by filter without running the delete action, so cleanup is one request
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = graphql(deleteManyMutation, { title: runMarker }, "product.cleanup");
    if (resultAt(res, "data.internal.deleteManyProduct.success")) return;
    // cleanup runs right after the load, so wait out a rate limit rather than leave records behind
    sleep(5);
  }
  throw new Error(`cleanup did not finish for ${runMarker}`);
}
```

The `setup` function runs once before the first iteration and its return value is passed to every iteration and to `teardown`. Code outside these functions runs again for each virtual user and again for `teardown`, so a marker built there would take a different value in each place. The `teardown` function runs once after the last iteration, and the cleanup request is tagged `product.cleanup` so it shows up separately from the requests under test in the summary.

Cleanup goes through the Internal API because it deletes every record that matches a filter in one request and runs no action code. The [Internal API](https://docs.gadget.dev/guides/data-access/api#public-api-vs-internal-api) skips access control, which is why the API key for the test has access control turned off.

If your `delete` action has code that must run for each record, call the `bulkDeleteProducts` mutation on the Public API instead. That runs the `delete` action once per record, so deleting what the test created costs about as much as creating it, and it happens all at once. For example, a 7-minute run that created 2,700 products at up to 10 per second and deleted them with bulk calls at the end ran 2,700 `delete` actions inside one minute, and used more database time in that minute than the creates did in any minute of the run. Expect that spike at the end of the run on the operations dashboard, and read the charts for the test window without it.

Only load test actions whose side effects you can undo. An action that sends an email, charges a card, or calls a third-party API does those things for every iteration. For actions like these, point the integration at its sandbox for the duration of the test, or skip the side effect only when the action can verify that the caller is the test, for example by comparing a parameter against a secret stored in an environment variable. Do not add a parameter that any caller can pass, because that opens the same bypass to every user of your production API.

HTTP routes need no GraphQL wrapper. Use the `name` tag so requests with different path segments are grouped under one name:

```javascript
// in load-tests/inventory-route.js
import http from "k6/http";
import { check } from "k6";

const APP_URL = __ENV.APP_URL;
const skuCodes = ["SKU-100", "SKU-200", "SKU-300"];

export default function () {
  const sku = skuCodes[__ITER % skuCodes.length];
  const res = http.get(`${APP_URL}/inventory/${sku}`, {
    // without this tag, k6 reports each sku as a separate URL
    tags: { name: "GET /inventory/:sku" },
  });

  check(res, {
    "status is 200": (r) => r.status === 200,
  });
}
```

This calls a route defined in `api/routes/inventory/GET-[sku].ts`. See [route structure](https://docs.gadget.dev/guides/http-routes/route-structure#dynamic-segments) for how path segments map to route files.

The route script and the page script that follows leave out the `rate_limited` metric and the thresholds to stay short, while the signed-in script later in this guide includes them. A failed check on its own does not stop a run. Copy the response callback, the metric, and the thresholds from the smoke test into any script before you give it a scenario that adds real load.

## Test web pages 

In a React Router app in framework mode with server rendering on, which is how Gadget sets up new apps, a page is rendered on a worker when a request arrives, and the route's `loader` functions run there too. A page request is an HTTP route request plus whatever its loaders do, and it consumes the request rate limit like any other route. The JavaScript, CSS, and images the page references are served from the CDN and are exempt, so a test that fetches only the page HTML measures your server rendering and loaders without the assets.

If your app sets `ssr: false` in `react-router.config.ts`, or uses declarative mode, the page HTML is a static shell and the page's work happens in the API requests the browser makes after it loads. Fetching the HTML then measures very little. Test the actions and queries the page calls instead, as a signed-in user where the page needs one.

Request the page as a browser would, with redirects turned off, and check for something the server render must produce. A page behind sign-in answers with a redirect, which then shows up as a failed status check rather than as a successful load of the sign-in page:

```javascript
// in load-tests/pages.js
import http from "k6/http";
import { check } from "k6";

const APP_URL = __ENV.APP_URL;

export default function () {
  const res = http.get(`${APP_URL}/`, {
    headers: { Accept: "text/html" },
    // a protected page redirects to sign-in, and following it would measure the wrong page
    redirects: 0,
    tags: { name: "GET /" },
  });

  check(res, {
    "status is 200": (r) => r.status === 200,
    "rendered the page": (r) => r.body.includes("<title>"),
  });
}
```

Gadget scales frontends to zero when they are idle, so the first page request after a quiet period pays a cold boot. For example, on a quiet app the first request for the root page took 2 seconds and the next ones took 150ms. Run the page test twice and read the second run, because the smoke test only reaches the Public API and does not start a frontend worker. See [cold boot times for SSR](https://docs.gadget.dev/guides/frontend/react-router-in-gadget#framework) for more details.

To measure what a visitor sees, such as the time to the largest contentful paint, k6 can also drive a real browser through its [`k6/browser` module](https://grafana.com/docs/k6/latest/using-k6-browser/). A browser virtual user starts a Chromium instance and fetches the page's assets as well as its HTML, so it is a way to observe a page while an HTTP scenario supplies the load, not a way to generate load.

Embedded Shopify app pages only render with data inside the Shopify admin, where App Bridge supplies the session token that your loaders and API calls authenticate with. A request from outside the admin returns the page shell without a shop. The HTTP page test still measures the render, and how you authenticate a test as a Shopify user beyond that is up to you.

## Create test user accounts 

If your app uses [email/password authentication](https://docs.gadget.dev/guides/plugins/authentication), the requests that matter most run as a signed-in user, and each virtual user needs an account of its own so that sessions and per-user data do not collide. Create the accounts once in `setup`, sign each virtual user in once, and reuse its session token for the rest of the run.

The Internal API creates users in bulk without running the `signUp` action. Gadget hashes the `password` field when the record is saved, so pass the plain password, and set `emailVerified` to `true` so the accounts can sign in without a verification email. Set `roles` as well, because the `signed-in` role is normally assigned when a user verifies their email, and the Internal API skips that step. Use the default auth role from your app's authentication settings if you changed it. Sign-in requests must carry no API key, because a session belongs to the request that created it. The session token comes back in the `x-set-authorization` response header and goes into the `Authorization` header of every request made as that user:

```javascript
// in load-tests/signed-in.js
import http from "k6/http";
import { check, fail, sleep } from "k6";
import { Rate } from "k6/metrics";
import exec from "k6/execution";

const APP_URL = __ENV.APP_URL;
const API_KEY = __ENV.GADGET_API_KEY;
const USER_COUNT = 50;
// the accounts exist in production for the length of the run, so the password must be a secret
const PASSWORD = __ENV.LOAD_TEST_PASSWORD;
if (!PASSWORD) {
  throw new Error("set LOAD_TEST_PASSWORD to a generated secret before running");
}

http.setResponseCallback(http.expectedStatuses(200, 429));
const rateLimited = new Rate("rate_limited");

// cleanup deletes sessions one user at a time, and each delete may retry 5 times with a 5-second sleep, so
// give setup and teardown enough time to finish the whole retry budget for every user
const CLEANUP_TIMEOUT = `${(USER_COUNT + 2) * 30}s`;

export const options = {
  setupTimeout: CLEANUP_TIMEOUT,
  teardownTimeout: CLEANUP_TIMEOUT,
  thresholds: {
    rate_limited: [{ threshold: "rate<0.05", abortOnFail: true, delayAbortEval: "10s" }],
    checks: [{ threshold: "rate>0.99", abortOnFail: true, delayAbortEval: "10s" }],
    http_req_failed: ["rate<0.01"],
    // scope latency to the request under test, so setup, sign-in, and cleanup requests do not skew it
    "http_req_duration{name:order.findMany}": ["p(95)<1000"],
  },
};

// authorization is a bearer API key by default, a session token for a signed-in user, or null for no header
function graphql(query, variables, name, authorization = `Bearer ${API_KEY}`) {
  const headers = { "Content-Type": "application/json" };
  if (authorization) headers.Authorization = authorization;
  return http.post(`${APP_URL}/api/graphql`, JSON.stringify({ query, variables }), { headers, tags: { name } });
}

// null for a non-200 response or a body that is not JSON, such as a gateway timeout
function resultAt(res, path) {
  if (res.status !== 200) return null;
  try {
    return res.json(path);
  } catch (error) {
    return null;
  }
}

const bulkCreateUsersMutation = `
  mutation LoadTestCreateUsers($users: [InternalUserInput]!) {
    internal {
      bulkCreateUsers(users: $users) {
        success
        errors {
          message
        }
      }
    }
  }
`;

export function setup() {
  const runMarker = `load-test-${__ENV.RUN_ID || Date.now()}`;
  const emails = [];
  for (let index = 0; index < USER_COUNT; index++) {
    emails.push(`${runMarker}-${index}@example.com`);
  }

  const users = emails.map((email) => ({ email, password: PASSWORD, emailVerified: true, roles: ["signed-in"] }));
  const res = graphql(bulkCreateUsersMutation, { users }, "user.setup");
  if (!resultAt(res, "data.internal.bulkCreateUsers.success")) {
    // k6 does not run teardown when setup throws, and the request may have succeeded before the response was lost
    cleanup(runMarker);
    throw new Error(`could not create test users: ${res.status} ${res.body}`);
  }
  return { runMarker, emails };
}

const signInMutation = `
  mutation LoadTestSignIn($email: String!, $password: String!) {
    signInUser(email: $email, password: $password) {
      success
      errors {
        message
      }
    }
  }
`;

const ordersQuery = `
  query LoadTestOrders {
    orders(first: 20) {
      edges {
        node {
          id
          total
        }
      }
    }
  }
`;

// each virtual user keeps its own copy of this variable, so it signs in once and reuses the token
let sessionToken = null;

export default function ({ emails }) {
  if (!sessionToken) {
    const email = emails[(exec.vu.idInTest - 1) % emails.length];
    const res = graphql(signInMutation, { email, password: PASSWORD }, "user.signIn", null);
    rateLimited.add(res.status === 429);
    // a failed sign-in can still carry a token for an anonymous session, so only keep it when the action succeeded
    const signedIn = resultAt(res, "data.signInUser.success") === true;
    sessionToken = signedIn ? res.headers["X-Set-Authorization"] : null;
    // a failed check counts against the checks threshold, and fail() ends the iteration before the query
    // would fall back to the API key and measure the wrong thing
    if (!check(res, { "signed in": () => Boolean(sessionToken) })) {
      fail(`sign-in did not return a session token: ${res.body}`);
    }
  }

  const res = graphql(ordersQuery, {}, "order.findMany", sessionToken);
  rateLimited.add(res.status === 429);
  check(res, {
    "status is 200": (r) => r.status === 200,
    "no GraphQL errors": (r) => r.status === 200 && !r.json("errors"),
    // a new account has no orders yet, so check the shape rather than the count
    "returned an orders list": (r) => r.status === 200 && Array.isArray(r.json("data.orders.edges")),
  });
}

const findUsersQuery = `
  query LoadTestFindUsers($prefix: String!, $after: String) {
    users(first: 250, after: $after, filter: { email: { startsWith: $prefix } }) {
      pageInfo {
        hasNextPage
        endCursor
      }
      edges {
        node {
          id
        }
      }
    }
  }
`;

const deleteSessionsMutation = `
  mutation LoadTestCleanupSessions($userId: GadgetID!) {
    internal {
      deleteManySession(filter: { user: { equals: $userId } }) {
        success
        errors {
          message
        }
      }
    }
  }
`;

const deleteUsersMutation = `
  mutation LoadTestCleanupUsers($prefix: String!) {
    internal {
      deleteManyUser(filter: { email: { startsWith: $prefix } }) {
        success
        errors {
          message
        }
      }
    }
  }
`;

// cleanup runs right after the load, so wait out a rate limit rather than leave records behind
function withRetry(query, variables, name, resultPath) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const result = resultAt(graphql(query, variables, name), resultPath);
    if (result) return result;
    sleep(5);
  }
  throw new Error(`${name} did not finish: ${JSON.stringify(variables)}`);
}

function cleanup(runMarker) {
  // the trailing dash keeps run 1 from matching the accounts of run 10
  const prefix = `${runMarker}-`;

  // deleting a user does not delete its sessions, so remove each user's sessions first
  let after = null;
  while (true) {
    const page = withRetry(findUsersQuery, { prefix, after }, "user.cleanup", "data.users");
    for (const edge of page.edges) {
      withRetry(deleteSessionsMutation, { userId: edge.node.id }, "session.cleanup", "data.internal.deleteManySession.success");
    }
    if (!page.pageInfo.hasNextPage) break;
    after = page.pageInfo.endCursor;
  }

  withRetry(deleteUsersMutation, { prefix }, "user.cleanup", "data.internal.deleteManyUser.success");
}

export function teardown({ runMarker }) {
  cleanup(runMarker);
}
```

Each sign-in runs the `signIn` action once, so with 50 virtual users the run makes 50 sign-ins and then only the requests under test. The sign-ins also create one record each in the `session` model. Those are not deleted with their user, so the teardown deletes each user's sessions before it deletes the users. Size `USER_COUNT` to the number of virtual users you expect to run at once, because two virtual users that share an account also share its session and its per-user records.

Generate the password once per run and keep it out of the script and out of source control, because the accounts can sign in to your production app while the test runs:

```bash
export LOAD_TEST_PASSWORD=$(openssl rand -base64 24)
```

The `orders` query runs with the signed-in user's role, so it sees exactly what that user would see in your app. This is the point of signing in rather than using the API key for everything: an access control filter that is slow, or a permission that is missing, shows up in the test the way it would for a real user.

## Shape the load 

By default, k6 runs a fixed number of virtual users in a loop, and each one sends its next request as soon as the last one returns. When your app slows down, the users send fewer requests, and the test measures a lighter load than you asked for. Use an arrival rate executor instead, which sends a fixed number of iterations per second no matter how long each one takes.

Set the rate from your app's limits and the cost of one iteration. For example, if each iteration runs one `create` action at 4 units and the environment includes 500 units per 10 seconds, a peak of 10 iterations per second consumes 400 units per 10 seconds and leaves 100 for the rest of your traffic before surge compute starts. Replace the `options` block in `create-product.js` with one that adds a scenario:

```javascript
// in load-tests/create-product.js
export const options = {
  teardownTimeout: "5m",
  scenarios: {
    createProducts: {
      executor: "ramping-arrival-rate",
      startRate: 2,
      timeUnit: "1s",
      // spare virtual users keep the rate up when responses slow down
      preAllocatedVUs: 20,
      maxVUs: 100,
      stages: [
        // hold at the expected average
        { duration: "2m", target: 2 },
        // ramp to the expected peak
        { duration: "1m", target: 10 },
        // hold at the peak
        { duration: "3m", target: 10 },
        // ramp down
        { duration: "1m", target: 0 },
      ],
    },
  },
  thresholds: {
    rate_limited: [{ threshold: "rate<0.05", abortOnFail: true, delayAbortEval: "10s" }],
    // an action that fails validation or permissions still returns a 200, so abort on failed checks too
    checks: [{ threshold: "rate>0.99", abortOnFail: true, delayAbortEval: "10s" }],
    http_req_failed: ["rate<0.01"],
    // scope latency to the request under test, so the cleanup request does not skew it
    "http_req_duration{name:product.create}": ["p(95)<1000"],
  },
};
```

If k6 reports that it ran out of virtual users, responses are taking longer than the rate allows. Raise `maxVUs`, or treat it as a result, because your app could not keep up with that rate.

Run the shapes in this order, and move to the next one only when the previous one passes:

*   **Smoke**: a few iterations, to prove the script and the app work
*   **Average load**: your expected everyday rate, held for 5 to 10 minutes
*   **Stress**: above your expected peak, to find the rate at which latency or `429` errors appear
*   **Spike**: a jump from low to very high in a few seconds and back, to see how the app recovers from a burst, such as a flash sale or a product launch
*   **Soak**: the average rate held for an hour or more, to find slow leaks in worker memory or growth in query time as records accumulate

See [load test types](https://grafana.com/docs/k6/latest/testing-guides/test-types/) for k6's guidance on each shape.

## Watch the operations dashboard 

k6 tells you what the app returned. The [operations dashboard](https://docs.gadget.dev/guides/development-tools/operations-dashboard) tells you why. Keep it open while the test runs, and set its time range to the test window:

*   **[HTTP status codes](https://docs.gadget.dev/guides/development-tools/operations-dashboard#http-status-codes)** on the **Overview** tab confirms how many requests were rejected with `429` and how many failed with `5xx` errors
*   **[Request rate limit usage](https://docs.gadget.dev/guides/development-tools/operations-dashboard#request-rate-limit-usage)** on the **Overview** tab shows how close the test pushed the environment to the surge threshold and the hard limit. Its **Request type** breakdown separates the test's external requests from the internal requests your actions make
*   **[Errors](https://docs.gadget.dev/guides/development-tools/operations-dashboard#errors)** on the **Overview** tab, broken down by **Message**, tells you which limit was reached if there was more than one
*   **[Database rate limit usage](https://docs.gadget.dev/guides/development-tools/operations-dashboard#database-rate-limit-usage)** on the **Database** tab shows whether the test was bounded by database time rather than by request count
*   **[Worker event loop utilization](https://docs.gadget.dev/guides/development-tools/operations-dashboard#worker-event-loop-utilization)** and **[Worker count](https://docs.gadget.dev/guides/development-tools/operations-dashboard#worker-count)** on the **Worker health** tab show whether your action and route code kept up, and how many workers Gadget started to handle the load

The request rate limit usage chart shows how full the bucket is, not how many requests arrived. Capacity drains out of the bucket continuously, so a steady rate well below the limit keeps the chart near zero. For example, 10 `create` actions per second against a limit of 10,000 per 10 seconds showed as a fill of about 10. The line only climbs when requests arrive faster than the bucket drains.

Compare the request rate limit usage chart with the `rate_limited` metric in the k6 summary. If the chart shows usage at the hard limit at the same time as the metric rises, the test found your environment's capacity. If the metric rises while the chart shows headroom, look at the errors chart for a different limit, such as the database rate limit or the computed view limit.

## Act on the results 

Each result points to a different fix:

*   **`429` errors or a rising `rate_limited` metric**: the test reached a rate limit. Reduce the requests each iteration makes by following [optimizing your rate limit usage](https://docs.gadget.dev/guides/development-tools/rate-limits/optimizing-rate-limit-usage), or add [execution capacity](https://docs.gadget.dev/guides/account-and-billing#execution-capacity) if the rate is one you need to serve.
*   **Database rate limit usage at the limit**: the test is bounded by query time. Start with [fix your slowest queries first](https://docs.gadget.dev/guides/development-tools/rate-limits/optimizing-rate-limit-usage#fix-your-slowest-queries-first).
*   **High `p(95)` latency with headroom on both limits**: your action or route code is slow. Profile it by following [debugging and profiling](https://docs.gadget.dev/guides/development-tools/debugging-and-profiling#profiling), and check worker event loop utilization for a worker that is saturated.
*   **`5xx` errors**: open the [logs](https://docs.gadget.dev/guides/development-tools/logger) for the test window and filter by error level to find the failing action or route.

## Run larger tests 

One machine can generate a few hundred requests per second before the test runner itself becomes the bottleneck. To test above that, run the same script from a hosted runner such as [Grafana Cloud k6](https://grafana.com/docs/grafana-cloud/testing/k6/), which distributes the virtual users across several load generators:

```bash
k6 cloud run --env APP_URL=$APP_URL --env GADGET_API_KEY=$GADGET_API_KEY load-tests/create-product.js
```

Cloud runs do not see your shell environment, so pass each variable with `--env`. The thresholds, the `rate_limited` metric, and the cleanup in `teardown` work the same way when the test runs in the cloud, but a virtual user number is only unique within one load generator, which is why the scripts number virtual users with `exec.vu.idInTest` rather than `__VU`.