Phase 12: LLM-Ready Documentation & Automation

Objective: Create comprehensive, machine-readable documentation with automated screenshot generation and LLM-assisted content workflows.

[!IMPORTANT] This phase addresses the documentation debt accumulated during rapid development sprints. Documentation is both human-readable AND machine-readable (RAG/LLM source).

[!NOTE] Tooling Location: All documentation generation tools (m docs:*) reside in framework-m-studio. The runtime framework-m package remains lightweight.

---

1. Documentation Architecture

1.1 Source-of-Truth Hierarchy

[!IMPORTANT] Single Source of Truth: Code generates BOTH human docs AND machine docs. They are always in sync because they come from the same source.

┌─────────────────────────────────────────────────────────────────┐
│                    SINGLE SOURCE OF TRUTH                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Python Source Code (DocTypes, Protocols, Controllers)         │
│                           │                                     │
│                           ▼                                     │
│                   `m docs:generate`                             │
│                     /           \                               │
│                    ▼             ▼                              │
│     ┌──────────────────┐ ┌──────────────────┐                   │
│     │   HUMAN DOCS     │ │   MACHINE DOCS   │                   │
│     │   (website/docs) │ │   (JSONL/YAML)   │                   │
│     │                  │ │                  │                   │
│     │ • API Reference  │ │ • RAG Corpus     │                   │
│     │ • User Guides    │ │ • .agent/        │                   │
│     │ • Examples       │ │ • Embeddings     │                   │
│     └──────────────────┘ └──────────────────┘                   │
│              │                    │                             │
│              ▼                    ▼                             │
│     Humans read docs      LLMs query corpus                     │
│     in browser            for accurate code                     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

• Define documentation sources:

  • ADRs (docs/adr/) → Design decisions, rationale
  • Checklists (checklists/) → Feature completion status
  • DocType Source (*.py) → API reference (auto-generated)
  • User Docs (docs/user/) → How-to guides
  • Developer Docs (docs/developer/) → Architecture, patterns

• Create documentation discovery logic (Dynamic manifest):

  • Implemented in m docs:generate and m docs:export
  • Auto-scans docs/, checklists/, and src/doctypes/

1.2 Documentation Site Setup

Source: todo.py

Fields

Field	Type	Required	Default	Description
title	str	✓	—	Task title
is_completed	bool		false	Completion status

Permissions

Role	Read	Write	Create	Delete
Employee	✓
Manager	✓	✓	✓	✓

### 2.2 Website Setup (Docusaurus)

- [x] Initialize Docusaurus in `website/` directory:
```bash
pnpm create docusaurus website classic --typescript

• Configure website/docusaurus.config.ts:

  • Set docs.path to ../docs (reuse existing markdown)
  • Set docs.routeBasePath to /docs
  • Site metadata (title, tagline, URL)

• Setup website/ folder structure:

  website/
  ├── blog/           # Release announcements
  ├── src/            # Custom React components
  ├── static/         # Static assets
  └── docusaurus.config.ts

• Reuse existing docs/ for content:

  • docs/user/ -> Documentation content
  • docs/developer/ -> Documentation content
  • docs/adr/ -> Documentation content

---

2. Auto-Generated Reference Documentation

2.1 DocType API Reference

• Implement m docs:generate CLI command:

  • Parse all DocTypes from MetaRegistry
  • Extract JSON Schema from Pydantic models
  • Scope: Includes Core, Standard, and any installed 3rd-party Apps.
  • Generate markdown for each DocType:
    • Field table (name, type, required, default, description)
    • Permission matrix (roles × actions)
    • Controller hooks (before_create, validate, etc.)
    • Link to source file

• Output format matching standard template:

  # Todo
  
  > Represents a task or action item.
  
  **Source**: [`todo.py`](file:///libs/framework-m/src/framework_m/doctypes/todo.py)
  
  ## Fields
  
  ...

2.2 Protocol Reference

• Auto-generate Protocol documentation:
  • List all Protocols in framework_m_core
  • Extract method signatures and docstrings
  • List available adapters for each Protocol
  • Generate comparison table (Indie vs Enterprise adapters)

2.3 OpenAPI Documentation

• Export OpenAPI spec: m docs:openapi --output docs/api/openapi.json
• Generate human-readable API docs from OpenAPI
• Include request/response examples
• Group by DocType and module

---

3.1 ADR/RFC Structure Standardization

• Create ADR/RFC template (docs/adr/template.md):

  # ADR-NNNN: [Title]
  
  ## Status
  
  [Proposed | Accepted | Deprecated | Superseded by ADR-XXXX]
  
  ## Context
  
  [Problem description]
  
  ## Decision
  
  [What we decided]

• Implement incremental numbering logic:

  • Start with 4 digits (0001-0999).
  • Automatically expand to more digits if needed (10000+).
  • CI check to ensure no duplicate or skipped numbers in PRs.

• Add CLI: m docs:adr create "Title" → Creates numbered ADR from template.

• Add CLI: m docs:rfc create "Title" → Creates numbered RFC from template.

3.2 Automated Chronological Indexing

• Implement m docs:adr index:
  • Scans docs/adr/ and docs/rfcs/.
  • Generates index.md for both with a chronological list.
  • Anchored "At-a-glance" view for quick reference.
  • Support lazy-loading behavior in Docusaurus (similar to blog).

---

4. Checklist-to-Feature Documentation

4.1 Feature Extraction

• Parse checklists for completed items ([x])
• Group by phase and category
• Generate "Features" page with completion status
• Link features to relevant documentation sections

4.2 Release Notes Generation

• Track checklist changes between versions (git diff)
• Auto-generate release notes from newly completed items
• Format: CHANGELOG.md or GitHub Release notes

---

5. Automated Screenshot Pipeline

5.1 Screenshot Manifest

• Create docs/screenshots.yaml:

  screenshots:
    - id: studio-doctype-list
      description: DocType list in Studio
      url: /studio/doctypes
      selector: .doctype-table
      viewport: { width: 1280, height: 720 }
      output: docs/images/studio-doctype-list.png
      docs: [docs/user/studio.md]
  
    - id: form-view-todo
      description: Todo form view
      url: /app/Todo/new
      selector: .form-container
      viewport: { width: 1280, height: 800 }
      output: docs/images/form-view-todo.png
      docs: [docs/user/forms.md]
  
    - id: workflow-actions
      description: Workflow action buttons
      url: /app/PurchaseOrder/detail/PO-001
      selector: .workflow-actions
      wait_for: .workflow-button
      output: docs/images/workflow-actions.png
      docs: [docs/user/workflows.md]

5.2 Playwright Screenshot Automation

• Create scripts/generate-screenshots.ts:

  • Read screenshots.yaml
  • Start dev server (m start)
  • For each screenshot:
    • Navigate to URL
    • Wait for selector
    • Capture screenshot
    • Save to output path
  • Generate diff report (new vs old screenshots)

• Install dependencies:

  pnpm add -D playwright @playwright/test yaml

5.3 Screenshot CI Workflow

• Add to .gitlab-ci.yml:

  update-screenshots:
    stage: docs
    image: mcr.microsoft.com/playwright:v1.40.0-jammy
    services:
      - docker:dind
    script:
      - docker compose up -d
      - sleep 10
      - npx ts-node scripts/generate-screenshots.ts
    artifacts:
      paths:
        - docs/images/*.png
      expire_in: 1 week
    rules:
      - if: $CI_COMMIT_BRANCH == "main"
        changes:
          - frontend/**
          - docs/screenshots.yaml
      - when: manual
        allow_failure: true

• Auto-commit screenshots (optional):

  • Use git push in CI with deploy token
  • Or: Create MR with updated screenshots for review

5.4 Screenshot Staleness Detection

• Add timestamp to screenshot manifest
• Compare file modification time vs manifest timestamp
• Warn in CI if screenshots are stale (>30 days)
• Option to regenerate on schedule (weekly)

---

6. Machine-Readable Documentation (RAG/LLM)

6.1 Structured DocType Export

• Implement m docs:export --format yaml:

  # docs/machine/doctypes.yaml
  doctypes:
    - name: Todo
      module: framework_m.doctypes.todo
      description: Represents a task or action item
      source_file: libs/framework-m/src/framework_m/doctypes/todo.py
      fields:
        - name: title
          type: str
          required: true
          description: Task title
        - name: is_completed
          type: bool
          default: false
      permissions:
        read: [Employee, Manager]
        write: [Manager]
      hooks:
        before_create: validate_due_date
        after_create: send_notification

6.2 RAG Corpus Generation

• Implement m docs:export --format jsonl --output docs/machine/corpus.jsonl:

  {"id": "doctype-todo", "type": "doctype", "title": "Todo DocType", "content": "...", "source": "todo.py", "tokens": 450}
  {"id": "adr-0001", "type": "adr", "title": "Taskiq + NATS for Jobs", "content": "...", "source": "0001-taskiq-nats.md", "tokens": 320}
  {"id": "protocol-repository", "type": "protocol", "title": "RepositoryProtocol", "content": "...", "source": "protocols.py", "tokens": 280}

• Chunk content appropriately for embeddings (500-1000 tokens)

• Include metadata for filtering (type, module, version)

6.3 Agent Knowledge Base

• Create .agent/knowledge/ directory structure:

  .agent/
  ├── workflows/           # Existing workflow definitions
  └── knowledge/
      ├── doctypes.yaml    # All DocTypes (machine-readable)
      ├── protocols.yaml   # All Protocols
      ├── patterns.md      # Code patterns and conventions
      ├── anti-patterns.md # Things to avoid
      └── corpus.jsonl     # Full RAG corpus

• Auto-update on each build/deploy

6.4 Inline Code Documentation

• Ensure all Protocols have comprehensive docstrings:

  class RepositoryProtocol(Protocol[T]):
      """
      Generic repository interface for CRUD operations.
  
      Usage:
          repo = container.get(RepositoryProtocol[Todo])
          todo = await repo.get(session, todo_id)
  
      Implementations:
          - SQLAlchemyRepository (default, PostgreSQL)
          - MongoRepository (framework-mx-mongo)
  
      See Also:
          - UnitOfWorkProtocol for transaction management
          - PermissionProtocol for authorization
      """

• Add Examples sections to all public APIs

• Use Google-style docstrings for consistency

6.5 LLM Docs Distribution & Sync (Federated Knowledge)

• Publication Strategy:

  • Include machine-readable assets in documentation build artifacts.
  • Path: docs/static/llm/v{version}/corpus.jsonl.

• Configuration (framework_config.toml):

  [docs.llm]
  upstream_url = "https://docs.framework-m.org/llm/v1" # Configurable source
  include_upstream = true

• Sync Command (m docs:sync):

  • Fetches upstream corpus from upstream_url.
  • Generates local project corpus via m docs:export.
  • Merges upstream + local into .agent/knowledge/combined.jsonl.
  • Updates .agent/context/project.md with summary of available knowledge.

---

7. LLM-Assisted Documentation Workflow

7.1 Documentation Gap Detection

• Create m docs:audit command:
  • Scan all DocTypes for missing descriptions
  • Check all Protocols for missing docstrings
  • Identify ADRs without implementation links
  • Report undocumented controller hooks
  • Output: docs/audit-report.md

7.2 LLM Documentation Generator

• Create m docs:draft command:

  • Input: Source file or DocType name
  • Process: Send code + context to LLM
  • Output: Draft documentation in markdown
  • Human reviews and edits before commit

• Prompt template:

  You are documenting Framework M, a Python business application framework.
  
  Given this DocType definition:
  {source_code}
  
  And these related ADRs:
  {related_adrs}
  
  Generate user documentation including:
  1. Overview (what this DocType represents)
  2. Field descriptions
  3. Common use cases
  4. Example code snippets

7.3 Diff-Based Documentation Updates

• Create m docs:update command:
  • Input: Git diff of code changes
  • Process: Identify affected documentation sections
  • Output: Suggested documentation updates
  • Human reviews and applies changes

---

8. Documentation Deployment

8.1 Build Pipeline

• Add docs build to .gitlab-ci.yml:

  pages:
    stage: docs
    image: node:20
    script:
      - cd docs
      - npm ci
      - npm run build
      - mv build ../public
    artifacts:
      paths:
        - public
    rules:
      - if: $CI_COMMIT_BRANCH == "main"

• Deploy to GitLab Pages:

  • Docs available at https://\<namespace\>.gitlab.io/\<project\>/
  • Custom domain: Configure in Settings → Pages

8.2 Versioned Documentation

[!NOTE] Documentation versions should align with release versions and branching strategy defined in Phase 10: Production - Section 6.3 & 6.4.

• Use Docusaurus for documentation (update Section 1.2):

  npx create-docusaurus@latest docs classic --typescript

• Enable Docusaurus versioning:

  # Create version snapshot when releasing
  npm run docusaurus docs:version 1.0

• Version strategy (aligned with git branching):

  • Next → Current main branch (unstable, latest features)
  • 1.0.x, 2.0.x → Tagged releases (stable)
  • Archive versions when LTS support ends

• Configure docusaurus.config.js:

  docs: {
    lastVersion: 'current',
    versions: {
      current: { label: 'Next (Unreleased)', path: 'next' },
      '1.0': { label: '1.0.x (LTS)', path: '1.0' },
    },
  },

• Sync with release workflow:

  • On git tag vX.Y.0 (minor/major): Create new docs version
  • On git tag vX.Y.Z (patch): Update existing version docs
  • Archive docs when release/vX branch is EOL

• Version banner for old docs:

  • Docusaurus built-in: "This is documentation for version X.0."

• Per-version API reference:

  • Generate OpenAPI spec per version
  • Store in docs/static/api/v{version}/openapi.json
  • Embed Swagger UI or Redoc in versioned docs

• Per-version LLM Corpus:

  • Generate corpus.jsonl and doctypes.yaml for each version.
  • Store in docs/static/llm/v{version}/.
  • Allows agents to sync docs matching their installed framework version.

8.3 Search Integration

• Add search integration (Searchable ADRs, RFCs, and Docs):
  • Use docusaurus-plugin-search-local or similar
  • Ensure docs/adr/ and docs/rfcs/ are indexed
  • Include code examples in search

---

9. Testing Documentation

9.1 Documentation Tests

• Test code examples compile/run:

  • Extract code blocks from markdown
  • Run against test environment
  • Fail CI if examples are broken

• Test internal links:

  • Check all markdown links resolve
  • Check all file references exist

9.2 Screenshot Regression Tests

• Compare new screenshots to baseline
• Alert on unexpected visual changes
• Require manual approval for UI changes

---

10. Documentation Contribution Guide

10.1 For Contributors

• Create docs/CONTRIBUTING.md:
  • How to write documentation
  • Style guide (tone, formatting)
  • How to add new screenshots
  • How to update ADRs

10.2 For AI Agents

• Create .agent/knowledge/documentation-patterns.md:
  • How Framework M documentation is structured
  • Where to find different types of information
  • How to suggest documentation improvements

---

11. LLM-Assisted Development Support

[!IMPORTANT] TDD is the core enabler for LLM-assisted coding. Without tests, an LLM cannot verify its generated code is correct. Framework M's architecture + TDD = effective AI pair programming.

11.1 Why TDD Enables LLM Coding

┌─────────────────────────────────────────────────────────────────┐
│                    LLM CODING FEEDBACK LOOP                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   1. LLM reads: Docs + Tests + Code                             │
│                    ↓                                            │
│   2. LLM generates: New code / modifications                    │
│                    ↓                                            │
│   3. LLM runs: `pytest` / `mypy` / `ruff`                       │
│                    ↓                                            │
│   4. Tests PASS? → Done ✓                                       │
│      Tests FAIL? → LLM reads error, returns to step 2           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

• Document the TDD-LLM workflow:
  • Tests are executable specifications — LLM knows what "correct" means
  • Test failures are feedback — LLM can self-correct
  • Type hints + mypy are compile-time checks — catch errors before runtime
  • Protocol interfaces are contracts (Documented in developer guides)

11.2 Agent-Ready Repository Structure

• Create .agent/ directory for AI coding assistants:

  .agent/
  ├── workflows/           # Existing workflow definitions
  └── knowledge/           # Machine-readable knowledge base (Done via machine export)

• .agent/workflows/ populated with key tasks

11.3 Test-First Development for LLM

• Provide test templates for common patterns:

  # tests/doctypes/test_todo.py
  import pytest
  from framework_m.testing import DocTypeTestCase
  
  class TestTodoDocType(DocTypeTestCase):
      doctype = "Todo"
  
      async def test_create_todo(self, repo, uow):
          """LLM can use this as a template for new DocTypes."""
          todo = Todo(title="Test Task")
          async with uow:
              created = await repo.save(uow.session, todo)
              await uow.commit()
          assert created.id is not None
          assert created.title == "Test Task"

• Document test commands in .agent/workflows/run-tests.md:

  # Running Tests
  
  ## Quick test (single file)
  
  uv run pytest tests/doctypes/test_todo.py -v
  
  ## Full suite
  
  uv run pytest
  
  ## With coverage
  
  uv run pytest --cov=framework_m --cov-report=html
  
  ## Type checking
  
  uv run mypy --strict .

11.4 LLM-Friendly Error Messages

• Ensure error messages are actionable:

  • Include expected vs actual values
  • Reference relevant documentation
  • Suggest fixes when possible

• Custom exceptions with context:

  class PermissionDeniedError(FrameworkMError):
      def __init__(self, user: str, action: str, doctype: str):
          super().__init__(
              f"User '{user}' lacks '{action}' permission on '{doctype}'. "
              f"See: docs/permissions.md#granting-permissions"
          )

11.5 Agentic Coding Workflow

• Document the agentic workflow for Framework M (See Architecture and Quality guides)
• Create .agent/workflows/ with development best practices

11.6 Why Framework M Enables LLM-Assisted Coding

[!NOTE] Framework M's design choices make it exceptionally LLM-friendly:

Design Choice	LLM Benefit
Hexagonal Architecture	Clear boundaries — LLM knows where to make changes
Protocol Interfaces	Explicit contracts — LLM knows method signatures
Pydantic Models	Type safety — LLM generates typed code
Code-First DocTypes	No hidden DB state — all logic is in readable Python
100% Type Hints	mypy catches LLM errors at "compile time"
No Global State	Explicit DI — LLM doesn't miss hidden dependencies
TDD Culture	Tests verify LLM output — self-correcting loop

11.7 RAG Corpus for Code Completion

• Include code examples in RAG corpus:

  • Annotated examples of each pattern
  • Common mistakes and their fixes
  • Before/after refactoring examples

• Export tests as documentation:

  • Tests show "how to use" each component
  • m docs:export --include-tests adds test patterns to corpus

Validation Checklist

Before completing this phase:

• m docs:generate produces complete API reference
• m docs:export --format jsonl exports RAG corpus
• Screenshot pipeline runs in CI
• All DocTypes have descriptions
• All Protocols have docstrings (in code)
• Documentation site builds without errors
• Search works across all documentation
• Code examples in docs are tested
• .agent/ directory structure in place
• Test templates provided for LLM coding
• Agentic workflow documented (in intro/developer docs)
• LLM can run tests and self-correct (verified manually)

---

Anti-Patterns to Avoid

❌ Don't: Write documentation separately from code ✅ Do: Generate documentation from source code

❌ Don't: Use static screenshots that become stale ✅ Do: Automate screenshot generation in CI

❌ Don't: Write documentation only for humans ✅ Do: Create machine-readable formats for LLM/RAG

❌ Don't: Document at the end when details are forgotten ✅ Do: Document incrementally as features are built

❌ Don't: Ignore documentation in code review ✅ Do: Require documentation updates with code changes

---

Timeline Estimate

Task	Duration
Documentation architecture setup	1 day
Auto-generated reference docs	2 days
Screenshot pipeline	2 days
Machine-readable export	1 day
LLM workflow integration	2 days
Testing and polish	1 day
Total	~9 days