Skip to main content

Automated Screenshots and E2E Testing

Framework M includes a built-in screenshot engine based on Playwright. This tool serves two purposes:

  1. Content Generation: Automatically capturing images for documentation (located in website/static/img/screenshots).
  2. E2E Verification: Ensuring that the application boots, renders correctly, and authenticates without errors across different domains (Desk and Studio).

The Screenshot Manifest​

The engine is driven by a configuration file located at docs/screenshots.yaml. Each entry defines a target to capture:

  - id: "desk-dashboard"
    url: "/app/dashboard"
    base_url: "http://localhost:5173"
    selector: "text=Welcome to Framework M"  # Job fails if this isn't found
    scale: 2                                 # High-DPI for marketing
    theme: "light"                           # light or dark

Running the Engine​

You can run the capture tool locally using the m CLI:

uv run m screenshots \
  --target-dir website/static/img/screenshots \
  --username admin@example.com \
  --password Secret@1234

Options:​

  • --verbose: Shows every step and network request.
  • --trace: Generates a Playwright trace file (trace-2x.zip) for debugging failures.

E2E Verification in CI​

In the CI pipeline, the update-screenshots job acts as a full-stack E2E smoke test. It:

  1. Provisions a fresh database.
  2. Starts the production API (m prod) and the Studio service (m studio).
  3. Attempts to log in and capture all pages defined in the manifest.
  4. Fails the PR if any selector is missing or if any page hits a 429 or 500 error.

Troubleshooting CI Failures​

If a screenshot job fails in CI:

  1. Check the Log: Look for [Network Error] or [Browser Console] log entries.
  2. Download Artifacts: Download the trace-2x.zip and open it with Trace Viewer to see exactly what the browser saw at the moment of failure.
  3. Selector Timeouts: If you see a timeout, the page might be loading too slowly for the CI runner. You may need to increase the timeout in screenshots.py or use a more robust selector.