Load Testing

Reference configuration schema, benchmark targets, and metrics for the Framework M load-testing harness.

---

Configuration Schema

The harness reads a single YAML file passed to the CLI.

m benchmark --config config.yaml

Or, for a direct Python entry point:

python -m framework_m_studio.benchmarks.load_test --config config.yaml

Top-Level Keys

Key	Type	Required	Description
target_url	string	Yes	Base URL of the application under test.
auth	object	Yes	Default authentication provider for all scenarios.
users	object	No	Spawn settings: peak, ramp_up_seconds, host.
duration_seconds	integer	No	Total benchmark duration. Default: 300.
seed	object	No	Shared seed data (IDs, tenants, warehouses) referenced by scenarios.
targets	list[object]	Yes	List of scenarios to execute.

Auth Providers

Session

auth:
  mode: session
  login_url: /api/auth/login
  username: "${LOAD_TEST_USER}"
  password: "${LOAD_TEST_PASSWORD}"

API Key

auth:
  mode: api_key
  header_name: X-API-Key
  key: "${LOAD_TEST_API_KEY}"

Bearer

auth:
  mode: bearer
  token: "${LOAD_TEST_TOKEN}"

No Auth

auth:
  mode: none

Custom

auth:
  mode: custom
  factory: myapp.benchmarks.auth.create_auth_header

Scenario Types

DocType Scenario

Key	Type	Required	Description
type	string	Yes	doctype
name	string	Yes	DocType name.
mix	object	No	Ratios for read, create, update, delete. Default: {read: 1.0}.
filters	object	No	Default query filters for list operations.
payload_overrides	object	No	Static or templated field overrides.
weight	integer	No	Relative weight among scenarios. Default: 1.
auth	object	No	Per-scenario auth override.

Endpoint Scenario

Key	Type	Required	Description
type	string	Yes	endpoint
name	string	Yes	Scenario identifier.
method	string	Yes	HTTP method.
path	string	Yes	Request path. Supports templating.
headers	object	No	Static or templated headers.
body	object	No	Static or templated JSON body.
body_factory	string	No	Python callable returning a body dict.
expect_status	list[int]	No	Allowed HTTP status codes. Default: [200].
weight	integer	No	Relative weight. Default: 1.
auth	object	No	Per-scenario auth override.

RPC Scenario

Key	Type	Required	Description
type	string	Yes	rpc
name	string	Yes	Scenario identifier.
method	string	Yes	Dotted RPC method name.
args	list	No	Positional arguments.
kwargs	object	No	Keyword arguments.
expect_status	list[int]	No	Allowed HTTP status codes. Default: [200].
weight	integer	No	Relative weight. Default: 1.
auth	object	No	Per-scenario auth override.

---

Example Configuration

target_url: https://staging.myapp.example.com

auth:
  mode: api_key
  header_name: X-API-Key
  key: "${LOAD_TEST_API_KEY}"

users:
  peak: 100
  ramp_up_seconds: 60
  spawn_rate: 5

duration_seconds: 300

seed:
  warehouse_id: "wh-001"
  tenant_id: "tenant-001"

targets:
  - type: doctype
    name: Invoice
    weight: 10
    mix:
      read: 0.7
      create: 0.2
      update: 0.1

  - type: endpoint
    name: bulk_import_invoices
    method: POST
    path: /api/invoices/bulk-import
    weight: 2
    headers:
      X-Import-Mode: async
    body:
      source: csv
      file_url: "{{ faker.url }}"
    expect_status: [202, 200]

  - type: rpc
    name: reconcile_stock
    method: framework.m.wms.reconcile_stock
    weight: 1
    kwargs:
      warehouse_id: "{{ seed.warehouse_id }}"
      item_codes:
        - "{{ faker.uuid4 }}"

---

Reference Benchmark Targets

These targets are starting points for production-like staging environments. Adjust them to match your deployment size and SLA.

Target	Value	Notes
Concurrent users	1,000	Simulated active users at peak.
Request rate	10,000 requests / minute	Aggregate across all scenarios.
p95 latency	< 200 ms	For standard CRUD and RPC operations.
Error rate	< 1%	Excluding expected 429 / 503 from rate limiting or readiness gates.
CPU utilization	< 70% sustained	Headroom for traffic spikes.
Memory growth	Stable	No visible leak over a 1-hour run.

Sustained Baseline Run

Before releasing to production, run a sustained 1-hour baseline and capture:

1. Application container CPU and memory.
2. Database CPU, connections, and slow query count.
3. Cache hit rate (Redis or in-memory).
4. Background job queue depth and worker throughput.
5. Error rate and percentile latency over time.

Store the results in version control or a monitoring dashboard so regressions are easy to spot.

---

Metrics Glossary

Metric	Description
RPS	Requests per second observed by Locust.
p50	50th percentile response time.
p95	95th percentile response time.
p99	99th percentile response time.
Error rate	Failed requests divided by total requests.
Failures	Requests whose HTTP status or assertion did not match expect_status.