Load Testing Harness

Framework M provides a starter load-testing harness so app developers can benchmark standard DocType CRUD, custom Litestar controllers, and Framework RPC routes without writing boilerplate authentication or payload logic for every test.

The harness is intentionally a thin layer over Locust. It gives teams a quick path for common scenarios and an escape hatch to plain Python/Locust when a scenario outgrows configuration.

---

Usage

Once framework-m-studio is installed, the benchmark command is available:

m benchmark --config examples/benchmarks/config.yaml

For CI or environments where you prefer a direct Python entry point:

python -m framework_m_studio.benchmarks.load_test --config examples/benchmarks/config.yaml

Why a Starter Harness?

Most Framework M applications expose the same kinds of endpoints:

• Resource routes for DocTypes (/api/resource/{doctype}).
• Custom controllers for business workflows.
• RPC methods under /api/method/<dotted.path>.

Writing a Locustfile for each app repeats the same concerns: logging in, discovering fields, generating valid payloads, and cleaning up. The harness centralizes those concerns so developers can focus on what to test.

---

Architecture

The harness ships as part of framework-m-studio and is exposed through the m benchmark CLI command. The source code lives under apps/studio/src/framework_m_studio/benchmarks/.

apps/studio/src/framework_m_studio/benchmarks/
├── load_test.py                  # CLI entry point
├── locustfile.py                 # Dispatches scenarios to Locust User classes
└── ...

examples/benchmarks/
└── config.yaml                   # Example configuration

Scenario Types

A scenario is the smallest unit of load. The harness ships three built-in scenario types.

DocType Scenario

A doctype scenario discovers the target DocType from the Meta API and runs a configurable mix of create, read, update, and delete operations. Payloads are generated automatically from field metadata, with overrides available through templates.

- type: doctype
  name: Invoice
  mix:
    read: 0.7
    create: 0.2
    update: 0.08
    delete: 0.02

Endpoint Scenario

An endpoint scenario targets a custom Litestar controller. The developer specifies the HTTP method, path, headers, body, and expected status codes.

- type: endpoint
  name: bulk_import_invoices
  method: POST
  path: /api/invoices/bulk-import
  headers:
    X-Import-Mode: async
  body:
    source: csv
    file_url: "{{ faker.url }}"
  expect_status: [202, 200]

RPC Scenario

An rpc scenario invokes a Framework RPC method. The harness maps the dotted method name to the correct HTTP request shape.

- type: rpc
  name: reconcile_stock
  method: framework.m.wms.reconcile_stock
  args:
    warehouse_id: "{{ seeded.warehouse_id }}"
    item_codes:
      - "{{ faker.uuid4 }}"
  expect_status: [200]

---

Authentication

Auth is provided through a pluggable provider interface. The global config selects a default provider, and individual scenarios can override it.

Provider	Use Case
api_key	Recommended default. Stable headless tests that avoid login overhead and backend session bloat.
bearer	OIDC/federated or machine-to-machine tokens.
session	Simulate real browser users. Creates one backend session per virtual user, so use it only when session behavior itself must be load-tested.
none	Public endpoints, health checks, webhooks.
custom	Signature-based webhooks or other domain-specific schemes.

auth:
  mode: api_key
  header_name: X-API-Key
  key: "${LOAD_TEST_API_KEY}"

---

Payload Generation

Static payloads are rarely sufficient. The harness supports two mechanisms:

1. Templating: Jinja2-style expressions with built-in generators.

   • {{ faker.email }}
   • {{ faker.uuid4 }}
   • {{ seeded.user_id }}
   • {{ random.choice(['a', 'b']) }}

2. Python hooks: For complex or stateful payloads, reference a factory function.

   body_factory: myapp.benchmarks.allocate_payload
   validate_factory: myapp.benchmarks.validate_allocate_response

DocType scenarios use a built-in faker_utils mapper that converts Framework M field types into sensible fake values.

---

Escape Hatch: Plain Locust

The harness does not replace Locust. It exposes a base FrameworkMUser class that teams can import and extend for scenarios that need custom state machines, multi-step flows, or advanced Locust features.

from framework_m_benchmarks.locust_user import FrameworkMUser
from locust import task

class ProcurementUser(FrameworkMUser):
    @task(3)
    def create_purchase_order(self):
        ...

    @task(1)
    def approve_po_workflow(self):
        ...

This keeps the starter suite approachable while preserving the full Python ecosystem for demanding cases.

---

Reporting

The harness produces two outputs:

1. Locust’s built-in web UI and CSV stats for interactive runs.
2. A JSON summary suitable for CI gates, containing total requests, RPS, error rate, and percentile latencies.

{
  "duration_seconds": 300,
  "total_requests": 48200,
  "requests_per_second": 160.7,
  "error_rate": 0.002,
  "p50_ms": 45,
  "p95_ms": 189,
  "p99_ms": 312
}

---

Relationship to Other Guides

For practical examples of writing a Locustfile from scratch, see the Load Testing Guide. For the exact configuration schema and reference benchmark targets, see Load Testing Reference.