Document: checklists/phase-15-frontend-plugin-architecture.md

# Phase 15: Frontend Plugin Architecture

**Objective**: Implement multi-app plugin system for Frontend M, enabling modular business applications with auto-discovered plugins and composed UI.

**Related Documents**:

- [ADR-0008: Frontend Plugin Architecture](pathname:///docs/adr/0008-frontend-plugin-architecture)
- [Framework M Sidebar Architecture](pathname:///docs/rfcs/rfc-0007-sidebar-architecture)
- [Plugin Architecture Implementation Guide](pathname:///docs/developer/plugin-architecture-implementation)
- [Plugin Package Architecture](pathname:///docs/developer/plugin-package-architecture)
- [Plugin Menu Code Examples](pathname:///docs/developer/plugin-menu-code-examples)

**Prerequisites**:

- Phase 09A (Frontend - Desk) must be complete
- Current sidebar implementation with favorites, recent, search, keyboard navigation

**Architecture Principles**:

- Keep `@framework-m/desk` unchanged (no breaking changes)
- Plugin SDK is opt-in (single-app users don't need it)
- Build-time composition (not Module Federation)
- Workspace-local plugins (no npm publishing required for apps)

---

## Stage 1: Package Infrastructure

### 1.1 Create @framework-m/plugin-sdk Package

**Purpose**: Core plugin runtime - registry, types, hooks, context providers

- [ ] Create package structure:

  ```bash
  mkdir -p libs/framework-m-plugin-sdk/src/{types,core,hooks,context}
  cd libs/framework-m-plugin-sdk
  ```

- [ ] Create `libs/framework-m-plugin-sdk/package.json`:
  - [ ] Set `name: "@framework-m/plugin-sdk"`
  - [ ] Set `version: "0.1.0"`
  - [ ] Add peer dependencies: `react@^19.0.0`
  - [ ] Add peer dependency: `@framework-m/desk@^0.1.0`
  - [ ] Add dependency: `react-router-dom@^7.0.0`
  - [ ] Set `main: "dist/index.js"`
  - [ ] Set `types: "dist/index.d.ts"`
  - [ ] Set `exports` with ESM/CJS support
  - [ ] Add build script: `"build": "tsc && vite build"`

- [ ] Create `libs/framework-m-plugin-sdk/tsconfig.json`:
  - [ ] Extend from root tsconfig
  - [ ] Set `declaration: true`
  - [ ] Set `outDir: "dist"`
  - [ ] Set `include: ["src"]`

- [ ] Create `libs/framework-m-plugin-sdk/vite.config.ts`:
  - [ ] Configure library build mode
  - [ ] Set external deps: `react`, `react-dom`, `react-router-dom`
  - [ ] Configure rollup output formats (ESM, CJS)

### 1.2 Define Plugin Types

- [ ] Create `src/types/plugin.ts`:
  - [ ] Define `MenuItem` interface:
    - [ ] `name: string` - Unique ID (e.g., "sales.invoice")
    - [ ] `label: string` - Display name
    - [ ] `route: string` - React Router path
    - [ ] `icon?: string` - Icon name (lucide-react)
    - [ ] `module?: string` - Group name (e.g., "Sales")
    - [ ] `category?: string` - Subgroup (e.g., "Transactions")
    - [ ] `order?: number` - Sort order
    - [ ] `children?: MenuItem[]` - Nested items
    - [ ] `hidden?: boolean` - Visibility flag
  - [ ] Define `ServiceFactory<T>` type:
    - [ ] Type: `() => Promise<T> | T`
  - [ ] Define `Provider` interface:
    - [ ] `component: () => Promise<{ default: ComponentType }>`
    - [ ] `props?: Record<string, any>`
  - [ ] Define `DocTypeExtension` interface:
    - [ ] `doctype: string` - DocType to extend
    - [ ] `fields?: Record<string, any>` - Additional fields
    - [ ] `actions?: Action[]` - Custom actions
    - [ ] `components?: Record<string, ComponentType>` - Custom renderers
  - [ ] Define `Widget` interface:
    - [ ] `id: string`
    - [ ] `title: string`
    - [ ] `component: () => Promise<{ default: ComponentType }>`
    - [ ] `size?: 'small' | 'medium' | 'large'`
    - [ ] `permissions?: string[]`
  - [ ] Define `FrameworkMPlugin` interface (main plugin contract):
    - [ ] `name: string` - Plugin identifier
    - [ ] `version: string` - Semantic version
    - [ ] `menu?: MenuItem[]` - Menu contributions
    - [ ] `routes?: RouteObject[]` - Route definitions
    - [ ] `services?: Record<string, ServiceFactory>` - DI services
    - [ ] `providers?: Provider[]` - Context providers
    - [ ] `doctypes?: DocTypeExtension[]` - DocType extensions
    - [ ] `widgets?: Widget[]` - Dashboard widgets
    - [ ] `onInit?: () => Promise<void> | void` - Initialization hook
    - [ ] `onDestroy?: () => Promise<void> | void` - Cleanup hook

### 1.3 Implement PluginRegistry

- [ ] Create `src/core/PluginRegistry.ts`:
  - [ ] Create `PluginRegistry` class:
    - [ ] Private field: `plugins: Map<string, FrameworkMPlugin>`
    - [ ] Private field: `menuCache: MenuItem[] | null`
    - [ ] Private field: `serviceContainer: Map<string, any>`
  - [ ] Implement `register(plugin: FrameworkMPlugin): Promise<void>`:
    - [ ] Validate plugin has `name` and `version`
    - [ ] Check for duplicate plugin names (warn if exists)
    - [ ] Add plugin to `plugins` map
    - [ ] Call `plugin.onInit()` if defined
    - [ ] Invalidate menu cache
  - [ ] Implement `getMenu(): MenuItem[]`:
    - [ ] Return cached menu if available
    - [ ] Collect menu items from all plugins
    - [ ] Call `mergeMenus()` to organize by module/category
    - [ ] Cache and return merged menu tree
  - [ ] Implement private `mergeMenus(items: MenuItem[]): MenuItem[]`:
    - [ ] Create `moduleMap: Map<string, MenuItem>`
    - [ ] Group items by `module` field
    - [ ] Within each module, group by `category`
    - [ ] Create hierarchy: Module → Category → Items
    - [ ] Sort by `order` field (default 999)
    - [ ] Return sorted module array
  - [ ] Implement private `getModuleIcon(moduleName: string): string`:
    - [ ] Map common module names to icons:
      - [ ] Sales → "shopping-cart"
      - [ ] Inventory → "package"
      - [ ] HR → "users"
      - [ ] Finance → "dollar-sign"
      - [ ] Core → "settings"
    - [ ] Default: "folder"
  - [ ] Implement `getRoutes(): RouteObject[]`:
    - [ ] Collect routes from all plugins
    - [ ] Flatten into single array
    - [ ] Return merged routes
  - [ ] Implement `getPlugin(name: string): FrameworkMPlugin | undefined`:
    - [ ] Return plugin from map by name
  - [ ] Implement `getAllPlugins(): FrameworkMPlugin[]`:
    - [ ] Convert plugins map to array
    - [ ] Return all registered plugins
  - [ ] Implement `getService<T>(name: string): T | undefined`:
    - [ ] Check service container cache
    - [ ] If not cached, find service factory in plugins
    - [ ] Call factory and cache result
    - [ ] Return service instance

### 1.4 Implement ServiceContainer

- [ ] Create `src/core/ServiceContainer.ts`:
  - [ ] Create `ServiceContainer` class:
    - [ ] Private field: `services: Map<string, any>`
    - [ ] Private field: `factories: Map<string, ServiceFactory>`
  - [ ] Implement `register<T>(name: string, factory: ServiceFactory<T>): void`:
    - [ ] Store factory in factories map
  - [ ] Implement `get<T>(name: string): T | undefined`:
    - [ ] Check services cache
    - [ ] If not cached and factory exists:
      - [ ] Call factory
      - [ ] Cache result
    - [ ] Return service instance
  - [ ] Implement `clear(): void`:
    - [ ] Clear services cache
    - [ ] Clear factories

### 1.5 Create React Context Provider

- [ ] Create `src/context/PluginRegistryContext.tsx`:
  - [ ] Create context: `PluginRegistryContext = createContext<PluginRegistry | null>(null)`
  - [ ] Define `PluginRegistryProviderProps` interface:
    - [ ] `children: ReactNode`
    - [ ] `registry?: PluginRegistry`
  - [ ] Create `PluginRegistryProvider` component:
    - [ ] Accept `registry` prop or create new instance
    - [ ] Use `useState` to hold registry instance
    - [ ] Use `useEffect` to mark ready state
    - [ ] Show loading state while initializing
    - [ ] Render children wrapped in context provider

### 1.6 Create React Hooks

- [ ] Create `src/hooks/usePluginMenu.ts`:
  - [ ] Implement `usePluginMenu(): MenuItem[]`:
    - [ ] Get registry from context
    - [ ] Throw error if used outside provider
    - [ ] Use `useMemo` to call `registry.getMenu()`
    - [ ] Return memoized menu tree

- [ ] Create `src/hooks/usePlugin.ts`:
  - [ ] Implement `usePlugin(name: string): FrameworkMPlugin | undefined`:
    - [ ] Get registry from context
    - [ ] Throw error if used outside provider
    - [ ] Use `useMemo` to call `registry.getPlugin(name)`
    - [ ] Return memoized plugin

- [ ] Create `src/hooks/useService.ts`:
  - [ ] Implement `useService<T>(name: string): T | undefined`:
    - [ ] Get registry from context
    - [ ] Throw error if used outside provider
    - [ ] Use `useMemo` to call `registry.getService<T>(name)`
    - [ ] Return memoized service instance

### 1.7 Create Public API Exports

- [ ] Create `src/index.ts`:
  - [ ] Export all types from `./types/plugin`
  - [ ] Export `PluginRegistry` from `./core/PluginRegistry`
  - [ ] Export `ServiceContainer` from `./core/ServiceContainer`
  - [ ] Export context and provider from `./context/PluginRegistryContext`
  - [ ] Export hooks: `usePluginMenu`, `usePlugin`, `useService`

### 1.8 Build and Test Plugin SDK

- [ ] Run build:

  ```bash
  cd libs/framework-m-plugin-sdk
  pnpm install
  pnpm build
  ```

- [ ] Verify output:
  - [ ] Check `dist/index.js` exists
  - [ ] Check `dist/index.d.ts` exists
  - [ ] Verify TypeScript types are exported

- [ ] Create test file `tests/PluginRegistry.test.ts`:
  - [ ] Test plugin registration
  - [ ] Test menu merging
  - [ ] Test duplicate plugin warning
  - [ ] Test route aggregation
  - [ ] Test service container

---

## Stage 2: Build-Time Plugin Discovery

### 2.1 Create @framework-m/vite-plugin Package

**Purpose**: Auto-discover plugins at build time, generate virtual module

- [ ] Create package structure:

  ```bash
  mkdir -p libs/framework-m-vite-plugin/src
  cd libs/framework-m-vite-plugin
  ```

- [ ] Create `libs/framework-m-vite-plugin/package.json`:
  - [ ] Set `name: "@framework-m/vite-plugin"`
  - [ ] Set `version: "0.1.0"`
  - [ ] Add dependency: `vite@^6.0.0`
  - [ ] Add dependency: `@framework-m/plugin-sdk@workspace:^0.1.0`
  - [ ] Add dependency: `glob@^10.0.0`
  - [ ] Set `main: "dist/index.js"`
  - [ ] Set `types: "dist/index.d.ts"`
  - [ ] Add build script

- [ ] Create `libs/framework-m-vite-plugin/tsconfig.json`:
  - [ ] Extend from root
  - [ ] Set `declaration: true`
  - [ ] Set `moduleResolution: "bundler"`

### 2.2 Implement Plugin Discovery Logic

- [ ] Create `src/index.ts`:
  - [ ] Import `Plugin` from `vite`
  - [ ] Import `globSync` from `glob`
  - [ ] Import `fs` and `path`
  - [ ] Create `frameworkMPlugin(): Plugin` function:
    - [ ] Define `pluginPaths: string[]` variable
    - [ ] Implement `buildStart()` hook:
      - [ ] Use `globSync` to find all `package.json` files
      - [ ] Exclude `node_modules` and `dist`
      - [ ] For each package.json:
        - [ ] Parse JSON
        - [ ] Check for `"framework-m"` metadata field
        - [ ] If `pkg["framework-m"]?.plugin` exists:
          - [ ] Resolve plugin config path
          - [ ] Add to `pluginPaths` array
      - [ ] Log: `✓ Discovered ${count} Framework M plugins`
    - [ ] Implement `resolveId(id)` hook:
      - [ ] If `id === "virtual:framework-m-plugins"`:
        - [ ] Return `"\0virtual:framework-m-plugins"`
    - [ ] Implement `load(id)` hook:
      - [ ] If `id === "\0virtual:framework-m-plugins"`:
        - [ ] Generate import statements for each plugin
        - [ ] Generate default export array
        - [ ] Return generated code
    - [ ] Return plugin object with hooks

### 2.3 Configure Code Splitting

- [ ] Update `src/index.ts`:
  - [ ] Add `config()` hook to plugin:
    - [ ] Configure `build.rollupOptions.output.manualChunks`
    - [ ] Create chunk per plugin (e.g., `wms-[hash].js`)
    - [ ] Keep framework code in main chunk

### 2.4 Build Vite Plugin

- [ ] Run build:

  ```bash
  cd libs/framework-m-vite-plugin
  pnpm install
  pnpm build
  ```

- [ ] Verify output exists

---

## Stage 3: Shell Application Integration

### 3.1 Update Frontend Package Dependencies

- [ ] Update `frontend/package.json`:
  - [ ] Add dependency: `"@framework-m/plugin-sdk": "workspace:^0.1.0"`
  - [ ] Add devDependency: `"@framework-m/vite-plugin": "workspace:^0.1.0"`

- [ ] Install dependencies:
  ```bash
  cd frontend
  pnpm install
  ```

### 3.2 Configure Vite Plugin

- [ ] Update `frontend/vite.config.ts`:
  - [ ] Import `frameworkMPlugin` from `@framework-m/vite-plugin`
  - [ ] Add `frameworkMPlugin()` to plugins array
  - [ ] Verify HMR configuration

### 3.3 Bootstrap PluginRegistry in App

- [ ] Update `frontend/src/App.tsx`:
  - [ ] Import `PluginRegistry`, `PluginRegistryProvider` from plugin-sdk
  - [ ] Import virtual module: `import plugins from 'virtual:framework-m-plugins'`
  - [ ] Create `bootstrap()` async function:
    - [ ] Create new `PluginRegistry` instance
    - [ ] Loop through discovered plugins
    - [ ] Call `registry.register(plugin)` for each
    - [ ] Return registry
  - [ ] Add state: `useState<PluginRegistry | null>(null)`
  - [ ] Add `useEffect` to call `bootstrap()` on mount:
    - [ ] Call `bootstrap()`
    - [ ] Set registry state
  - [ ] Show loading state while registry is null
  - [ ] Wrap app in `PluginRegistryProvider`:
    - [ ] Pass registry prop
    - [ ] Wrap existing `<Refine>` component

### 3.4 Update Sidebar to Use Plugin Menu

- [ ] Update `frontend/src/layout/Sidebar.tsx`:
  - [ ] Import `usePluginMenu` hook
  - [ ] Call hook: `const pluginMenu = usePluginMenu()`
  - [ ] Merge plugin menu with favorites/recent:
    - [ ] Keep existing favorites section (top)
    - [ ] Keep existing recent section (after favorites)
    - [ ] Add plugin menu groups (after recent)
  - [ ] Update menu rendering:
    - [ ] Map over `pluginMenu` array
    - [ ] Each top-level item is a module group
    - [ ] Render module header with icon
    - [ ] Render children (categories and items)
    - [ ] Preserve existing collapse/expand logic
  - [ ] Preserve all existing Phase 4 features:
    - [ ] Search highlighting
    - [ ] Breadcrumbs
    - [ ] Keyboard navigation
    - [ ] Mobile drawer
    - [ ] Touch gestures

### 3.5 Integrate Plugin Routes

- [ ] Update `frontend/src/App.tsx`:
  - [ ] After bootstrap, get routes: `const pluginRoutes = registry.getRoutes()`
  - [ ] In `<Routes>` component:
    - [ ] Map over `pluginRoutes`
    - [ ] Render `<Route>` for each plugin route
    - [ ] Preserve existing routes (dashboard, list, form, login)

---

## Stage 4: Create Example Plugin (WMS)

### 4.1 Scaffold WMS Plugin Package

- [ ] Create plugin structure:

  ```bash
  mkdir -p apps/wms/frontend/src/{pages,components,services}
  cd apps/wms/frontend
  ```

- [ ] Create `apps/wms/frontend/package.json`:
  - [ ] Set `name: "@my-company/wms"`
  - [ ] Set `version: "1.0.0"`
  - [ ] Set `private: true`
  - [ ] Add `framework-m` metadata:
    ```json
    "framework-m": {
      "plugin": "./dist/plugin.config.js",
      "type": "frontend-module"
    }
    ```
  - [ ] Add dependencies:
    - [ ] `@framework-m/desk@workspace:^0.1.0`
    - [ ] `@framework-m/plugin-sdk@workspace:^0.1.0`
    - [ ] `react@^19.0.0`
    - [ ] `react-router-dom@^7.0.0`

- [ ] Create `apps/wms/frontend/tsconfig.json`:
  - [ ] Extend from root
  - [ ] Set `jsx: "react-jsx"`

- [ ] Create `apps/wms/frontend/vite.config.ts`:
  - [ ] Configure library build mode
  - [ ] Set entry to `plugin.config.ts`
  - [ ] Configure external deps

### 4.2 Define WMS Plugin Manifest

- [ ] Create `apps/wms/frontend/plugin.config.ts`:
  - [ ] Import `FrameworkMPlugin` type
  - [ ] Export default plugin object:
    - [ ] Set `name: "wms"`
    - [ ] Set `version: "1.0.0"`
  - [ ] Define `menu` array:
    - [ ] Warehouse item:
      - [ ] name: "wms.warehouse"
      - [ ] label: "Warehouse"
      - [ ] route: "/doctypes/wms.warehouse"
      - [ ] icon: "warehouse"
      - [ ] module: "Inventory"
      - [ ] category: "Masters"
      - [ ] order: 1
    - [ ] Bin Location item:
      - [ ] name: "wms.bin_location"
      - [ ] label: "Bin Location"
      - [ ] route: "/doctypes/wms.bin_location"
      - [ ] icon: "map-pin"
      - [ ] module: "Inventory"
      - [ ] category: "Masters"
      - [ ] order: 2
    - [ ] Stock Entry item:
      - [ ] name: "wms.stock_entry"
      - [ ] label: "Stock Entry"
      - [ ] route: "/doctypes/wms.stock_entry"
      - [ ] icon: "package"
      - [ ] module: "Inventory"
      - [ ] category: "Transactions"
      - [ ] order: 3

  - [ ] Add `satisfies FrameworkMPlugin` type assertion

### 4.3 Create WMS Pages (Optional)

- [ ] Create `apps/wms/frontend/src/pages/Dashboard.tsx`:
  - [ ] Simple WMS dashboard component
  - [ ] Show warehouse stats
  - [ ] Export as default

- [ ] Create `apps/wms/frontend/src/pages/Receiving.tsx`:
  - [ ] Receiving workflow page
  - [ ] Export as default

- [ ] Add routes to plugin config:
  - [ ] Dashboard route: `/wms/dashboard`
  - [ ] Receiving route: `/wms/receiving`
  - [ ] Use lazy imports: `() => import('./pages/Dashboard')`

### 4.4 Build WMS Plugin

- [ ] Run build:

  ```bash
  cd apps/wms/frontend
  pnpm install
  pnpm build
  ```

- [ ] Verify `dist/plugin.config.js` exists

---

## Stage 5: Testing and Validation

### 5.1 Test Plugin Discovery

- [ ] Run shell app in dev mode:

  ```bash
  cd frontend
  pnpm dev
  ```

- [ ] Check browser console:
  - [ ] Verify: "✓ Discovered 1 Framework M plugins" message
  - [ ] Verify: "Registered plugin: wms@1.0.0" message
  - [ ] No errors during bootstrap

### 5.2 Test Menu Integration

- [ ] Open app in browser
- [ ] Check sidebar:
  - [ ] Favorites section visible (if any)
  - [ ] Recent section visible (if any)
  - [ ] **Inventory** module group visible
  - [ ] **Masters** category under Inventory
    - [ ] "Warehouse" item visible
    - [ ] "Bin Location" item visible
  - [ ] **Transactions** category under Inventory
    - [ ] "Stock Entry" item visible

- [ ] Test menu interactions:
  - [ ] Click module group to expand/collapse
  - [ ] Click menu items to navigate
  - [ ] Verify routes work (even if pages are empty)

### 5.3 Test Search with Plugin Menu

- [ ] Type in sidebar search box:
  - [ ] Search: "warehouse"
    - [ ] Verify "Warehouse" item highlighted
    - [ ] Verify breadcrumb shows "Inventory"
  - [ ] Search: "stock"
    - [ ] Verify "Stock Entry" highlighted
    - [ ] Verify breadcrumb shows "Inventory"
  - [ ] Clear search
    - [ ] Verify menu returns to normal grouped view

### 5.4 Test Keyboard Navigation

- [ ] Use Tab to focus sidebar
- [ ] Use Arrow keys to navigate menu
  - [ ] Verify focus moves through plugin menu items
  - [ ] Verify focus indicator visible
- [ ] Press Enter on focused item
  - [ ] Verify navigation works

### 5.5 Test Mobile Responsive

- [ ] Open browser DevTools
- [ ] Toggle device toolbar (mobile view)
- [ ] Test mobile drawer:
  - [ ] Click hamburger menu button
  - [ ] Verify sidebar slides in
  - [ ] Verify plugin menu items visible
  - [ ] Click backdrop
  - [ ] Verify sidebar closes
  - [ ] Swipe left on sidebar
  - [ ] Verify sidebar closes

### 5.6 Test Production Build

- [ ] Build shell app:

  ```bash
  cd frontend
  pnpm build
  ```

- [ ] Check `dist/` output:
  - [ ] Verify `dist/assets/main-[hash].js` (shell + framework)
  - [ ] Verify `dist/assets/wms-[hash].js` (WMS plugin chunk)
  - [ ] Check bundle sizes reasonable

- [ ] Preview production build:

  ```bash
  pnpm preview
  ```

- [ ] Test in browser:
  - [ ] Verify app loads
  - [ ] Verify plugin menu works
  - [ ] Check Network tab for lazy-loaded chunks

---

## Stage 6: Documentation and Polish

### 6.1 Update ERP Sidebar RFC Status

- [ ] Update `docs/rfcs/erp-sidebar-architecture.md`:
  - [ ] Mark Phase 3 "Plugin menu integration" as ✅ COMPLETE
  - [ ] Update integration section with actual implementation
  - [ ] Add note about plugin discovery process

### 6.2 Create Plugin Developer Guide

- [ ] Create `docs/tutorials/creating-plugins.md`:
  - [ ] Quickstart: Create your first plugin
  - [ ] Plugin manifest structure
  - [ ] Menu item configuration
  - [ ] Route definition
  - [ ] Service registration
  - [ ] Testing plugins locally
  - [ ] Building and deploying plugins

### 6.3 Create Migration Guide

- [ ] Create `docs/guides/migrating-to-plugins.md`:
  - [ ] For single-app users (no action needed)
  - [ ] For multi-app users:
    - [ ] Install plugin-sdk and vite-plugin
    - [ ] Create plugin manifests
    - [ ] Update App.tsx to bootstrap registry
    - [ ] Update Sidebar to use usePluginMenu
  - [ ] Breaking changes (none expected)

### 6.4 Update Architecture Documentation

- [ ] Update `ARCHITECTURE.md`:
  - [ ] Add Frontend Plugin System section
  - [ ] Explain relationship to backend plugins
  - [ ] Document build-time vs runtime composition

---

## Stage 7: Advanced Features (Optional)

### 7.1 Add Plugin Lifecycle Events

- [ ] Update `PluginRegistry`:
  - [ ] Add event emitter support
  - [ ] Emit events: `plugin:registered`, `plugin:error`
  - [ ] Allow plugins to subscribe to events

### 7.2 Add Plugin Hot Reload (Dev Mode)

- [ ] Update Vite plugin:
  - [ ] Watch for plugin config changes
  - [ ] Trigger HMR when plugin updates
  - [ ] Re-register updated plugin

### 7.3 Add Plugin Permissions

- [ ] Update `MenuItem` interface:
  - [ ] Add `permissions?: string[]` field
- [ ] Update `usePluginMenu` hook:
  - [ ] Filter menu items by user permissions
  - [ ] Call `authProvider.can()` for each item

### 7.4 Add Widget System

- [ ] Create `useWidgets()` hook:
  - [ ] Get widgets from all plugins
  - [ ] Filter by permissions
  - [ ] Return widget components

- [ ] Update Dashboard page:
  - [ ] Render widgets from registry
  - [ ] Support widget sizing
  - [ ] Support drag-to-reorder

---

## Validation Checklist

After completing all stages, verify:

- [ ] **SDK Package**:
  - [ ] `@framework-m/plugin-sdk` builds successfully
  - [ ] TypeScript types exported correctly
  - [ ] No runtime dependencies except React

- [ ] **Vite Plugin**:
  - [ ] `@framework-m/vite-plugin` builds successfully
  - [ ] Discovers plugins correctly
  - [ ] Generates virtual module without errors

- [ ] **Shell App**:
  - [ ] Bootstraps registry without errors
  - [ ] Renders plugin menus in sidebar
  - [ ] Integrates plugin routes
  - [ ] All Phase 4 features still work

- [ ] **Example Plugin**:
  - [ ] WMS plugin builds successfully
  - [ ] Menu items appear in sidebar
  - [ ] Navigation works
  - [ ] Shows in correct module/category

- [ ] **Developer Experience**:
  - [ ] Clear error messages
  - [ ] TypeScript autocomplete works
  - [ ] HMR works in dev mode
  - [ ] Production build optimized

- [ ] **Documentation**:
  - [ ] ADR-0008 matches implementation
  - [ ] Plugin guide complete
  - [ ] Code examples accurate
  - [ ] Migration path clear

---

## Rollback Plan

If critical issues arise:

1. **Single-app users**: No impact (they don't use plugin-sdk)
2. **Multi-app users**: Can temporarily:
   - [ ] Remove `@framework-m/vite-plugin` from vite.config
   - [ ] Manually import and register plugins
   - [ ] Keep using plugin-sdk for types/hooks

3. **Complete rollback**:
   - [ ] Remove plugin-sdk dependency from frontend
   - [ ] Revert Sidebar to use `useMenu` hook
   - [ ] Remove PluginRegistry bootstrap from App.tsx
   - [ ] Plugin packages remain usable as regular workspace packages

---

## Success Criteria

✅ Phase 15 is complete when:

1. **Infrastructure**:
   - [ ] `@framework-m/plugin-sdk` package published to workspace
   - [ ] `@framework-m/vite-plugin` package published to workspace
   - [ ] Both packages build without errors
   - [ ] TypeScript types work correctly

2. **Integration**:
   - [ ] Shell app bootstraps PluginRegistry successfully
   - [ ] Sidebar renders plugin menus
   - [ ] Plugin routes integrate with React Router
   - [ ] All existing sidebar features preserved

3. **Example**:
   - [ ] WMS plugin demonstrates full workflow
   - [ ] Menu items appear correctly grouped
   - [ ] Navigation works end-to-end
   - [ ] Plugin can be built independently

4. **Quality**:
   - [ ] No console errors
   - [ ] No TypeScript errors
   - [ ] Production build under 500KB (main chunk)
   - [ ] Plugin chunks lazy-load correctly

5. **Documentation**:
   - [ ] Plugin developer guide published
   - [ ] Architecture diagrams updated
   - [ ] Code examples tested and accurate
   - [ ] Migration guide available

---

## Notes

- **Package Publishing**: Initial implementation uses workspace protocol (`workspace:^0.1.0`). Before external use, publish to GitLab npm registry.

- **Backward Compatibility**: This phase maintains 100% backward compatibility. Single-app users see no changes.

- **Future Work**: Module Federation can be added later without changing the plugin contract.

- **Testing Strategy**: Unit test PluginRegistry, integration test shell app, E2E test full workflow.

- **Performance**: Plugin discovery happens once at build time. Runtime overhead minimal (just map lookups).


Document: checklists/phase-08-workflows-advanced.md

# Phase 08: Workflows & Advanced Features

**Objective**: Implement pluggable workflow engine, DocType overrides, app-defined ports, and advanced extensibility features.

---

## 1. Workflow System

### 1.1 Workflow Protocol

- [x] Create `src/framework_m/core/interfaces/workflow.py`
- [x] Define `WorkflowProtocol`:
  - [x] `async def start_workflow(doctype: str, doc_id: str, workflow_name: str)`
  - [x] `async def get_workflow_state(doc_id: str) -> str`
  - [x] `async def transition(doc_id: str, action: str, user: UserContext)`
  - [x] `async def get_available_actions(doc_id: str, user: UserContext) -> list[str]`

### 1.2 Internal Workflow Adapter

- [x] Create `src/framework_m/adapters/workflow/internal_workflow.py`
- [x] Create `WorkflowState` DocType:

  - [x] `name: str`
  - [x] `workflow: str` - Workflow name
  - [x] `doctype: str`
  - [x] `document_name: str`
  - [x] `current_state: str`
  - [x] `updated_at: datetime`

- [x] Create `WorkflowTransition` DocType:

  - [x] `name: str`
  - [x] `workflow: str`
  - [x] `from_state: str`
  - [x] `to_state: str`
  - [x] `action: str` - Action name
  - [x] `allowed_roles: list[str]`
  - [x] `condition: str | None` - Python expression

- [x] Create `Workflow` DocType:

  - [x] `name: str` - Workflow name
  - [x] `doctype: str` - Target DocType
  - [x] `initial_state: str`
  - [x] `states: list[dict]` - State definitions
  - [x] `transitions: list[WorkflowTransition]`

- [x] Implement `InternalWorkflowAdapter`:
  - [x] Load workflow definition
  - [x] Validate transitions
  - [x] Check permissions
  - [x] Update state
  - [x] Trigger hooks / **Emit Event** (Side effects via Event Bus only)

### 1.3 Temporal Workflow Adapter (Optional)

- [x] Create `src/framework_m/adapters/workflow/temporal_adapter.py`
- [x] Implement `TemporalWorkflowAdapter`:
  - [x] Connect to Temporal server
  - [x] Start workflows
  - [x] Query workflow state
  - [x] Signal workflows for transitions

---

## 2. DocType Overrides

### 2.1 Override Registry

- [x] Update `MetaRegistry`:
  - [x] Add `register_override(base_doctype: str, override_class: Type[BaseDocType])`
  - [x] When loading DocType, check for overrides
  - [x] Use override class if registered

### 2.2 Schema Extension

- [x] Implement schema merging:
  - [x] Combine fields from base and override
  - [x] Override can add new fields
  - [x] Override can modify field properties
  - [x] Base fields cannot be removed

### 2.3 Table Alteration

- [x] Update `SchemaMapper`:
  - [x] Detect schema changes from overrides
  - [x] Generate ALTER TABLE migrations
  - [x] Add new columns
  - [x] Modify column types (with caution)

### 2.4 Example Usage

```python
# In custom app
from framework_m.public import User

class ExtendedUser(User):
    department: str = Field(description="Department")
    employee_id: str | None = None

# Register override
registry.register_override("User", ExtendedUser)
```

---

## 3. App-Defined Ports

### 3.1 Custom Protocol Registration

- [x] Update Container to support app-defined protocols:
  - [x] Allow apps to define custom protocols
  - [x] Register via entrypoints
  - [x] Other apps can override using `container.override()`

### 3.2 Example: Payment Gateway

```python
# In e-commerce app - define protocol
class PaymentGatewayProtocol(Protocol):
    async def charge(amount: Decimal, token: str) -> str: ...
    async def refund(charge_id: str) -> bool: ...

# Default implementation
class StripeAdapter:
    async def charge(amount, token):
        # Stripe API call
        pass

# In e-commerce app's container
from dependency_injector import containers, providers

class EcommerceContainer(containers.DeclarativeContainer):
    payment_gateway = providers.Singleton(StripeAdapter)

# Register in pyproject.toml
[project.entry-points."framework_m.containers"]
ecommerce = "ecommerce.container:EcommerceContainer"

# Another app can override
# In custom_app
class PayPalAdapter:
    async def charge(amount, token):
        # PayPal API call
        pass

# Override in app initialization
ecommerce_container.payment_gateway.override(providers.Singleton(PayPalAdapter))
```

---

## 4. Child Tables (Nested DocTypes)

### 4.1 Child Table Support

- [x] Add `is_child` flag to DocType Config
- [x] Child tables don't have independent routes
- [x] Stored as JSON or separate table with parent reference

### 4.2 Implementation

```python
class OrderItem(DocType):
    item: str
    quantity: int
    rate: Decimal

    class Config:
        is_child = True

class Order(DocType):
    customer: str
    items: list[OrderItem]  # Child table
```

- [x] Update `SchemaMapper`:

  - [x] Create separate table for child
  - [x] Add `parent` and `parenttype` columns
  - [x] Add `idx` for ordering

- [x] Update `GenericRepository`:
  - [x] Save child records when saving parent
  - [x] Delete old children and insert new ones
  - [x] Load children when loading parent

---

## 5. Virtual Fields

### 5.1 Computed Fields

- [x] Add `@computed_field` decorator:

  ```python
  class Invoice(DocType):
      items: list[InvoiceItem]

      @computed_field
      @property
      def total(self) -> Decimal:
          return sum(item.amount for item in self.items)
  ```

- [x] Virtual fields not stored in database
- [x] Computed on load
- [x] Included in API responses

---

## 6. Link Fields (Foreign Keys)

### 6.1 Link Field Type

- [x] Add Link field type using json_schema_extra:

  ```python
  from uuid import UUID
  from framework_m import Field

  class Order(DocType):
      customer: UUID | None = Field(
          description="Customer link",
          json_schema_extra={"link": "Customer"}
      )
  ```

### 6.2 Implementation

- [x] Update `SchemaMapper`:

  - [x] Detect link fields via json_schema_extra
  - [x] Create foreign key constraint to target table
  - [x] Reference target table's id column

- [x] Add database enforcement:
  - [x] Foreign key constraints enforce referential integrity
  - [x] Works with SQLite and PostgreSQL (database-agnostic)
  - [x] SQLite foreign keys enabled via PRAGMA

**Implementation Notes:**

- Link fields use UUID type (not string names)
- Metadata stored in `json_schema_extra={"link": "TargetDocType"}`
- SchemaMapper creates ForeignKey constraint: `ForeignKey(f"{target_table}.id")`
- SQLite requires `PRAGMA foreign_keys=ON` for constraint enforcement
- 14 comprehensive tests covering detection, constraints, validation, queries

### 6.3 Link Fetching

- [x] Add `fetch_from` option:

  ```python
  from uuid import UUID
  from framework_m import Field

  class Order(DocType):
      customer: UUID | None = Field(json_schema_extra={"link": "Customer"})
      customer_name: str | None = Field(
          default=None,
          json_schema_extra={"fetch_from": "customer.customer_name"}
      )
  ```

- [x] Auto-populate field from linked document
- [x] Update GenericRepository to fetch values before save
- [x] Fetch values automatically on insert and update
- [x] Handle null links gracefully
- [x] Support multiple fetch_from fields

**Implementation Notes:**

- Fetch fields use `json_schema_extra={"fetch_from": "link_field.target_field"}`
- Values are fetched from linked document before insert/update
- Automatically syncs with link changes (prevents data inconsistency)
- Null links result in null fetch values
- Supports fetching from multiple different links
- 12 comprehensive tests covering detection, fetching, updates, edge cases

---

## 7. Naming Series (Human-Readable Names)

> [!NOTE] > **`id` vs `name`**: The `id` field (UUID, primary key) is always auto-generated with no contention.
> The `name` field is an optional human-readable identifier (e.g., "INV-2024-0001").
> Naming series applies to `name`, NOT the primary key.

### 7.1 Auto-Naming Configuration

- [x] Add naming configuration to DocType:

  ```python
  class Invoice(DocType):
      class Meta:
          name_pattern = "INV-.YYYY.-.####"  # INV-2024-0001
  ```

- [x] Implement naming patterns:
  - [x] `.YYYY.` - Year (2026)
  - [x] `.MM.` - Month (01-12)
  - [x] `.DD.` - Day (01-31)
  - [x] `.####` - Sequential number with padding (0001, 0002...)
  - [x] `{field}` - Field value from entity

**Implementation Notes:**

- Dots (.) in patterns are separators and removed in output
- Pattern "INV-.YYYY.-.####" generates "INV-2026-0001"
- Pattern `"TASK-.{priority}.-.####"` with `priority="HIGH"` generates "TASK-HIGH-0001"

### 7.2 Implementation (Optimistic Approach)

- [x] Name generation happens AFTER insert (id already assigned):

  ```python
  # 1. Insert with UUID id (no contention)
  # 2. Generate name using current counter
  # 3. If duplicate (unique constraint), retry with next number
  ```

- [x] For high-volume DocTypes, use PostgreSQL sequences:
  ```python
  class HighVolumeInvoice(DocType):
      class Meta:
          name_pattern = "sequence:invoice_seq"  # Uses DB sequence
  ```

### 7.3 Counter Storage

- [x] Create `NamingCounter` DocType:
  - [x] `prefix: str` - e.g., "INV-2024-"
  - [x] `current: int` - Current counter value
  - [x] **Note**: No row-level locking needed. Use optimistic update with retry.

> [!IMPORTANT] > **No Frappe Anti-Pattern**: We do NOT use `SELECT ... FOR UPDATE` on a shared counter table.
> The `id` (UUID) is the primary key. The `name` field uses optimistic generation with retry on conflict.

---

## 8. Validation Rules

### 8.1 Field Validators

- [x] Add Pydantic validators:

  ```python
  from pydantic import field_validator

  class Invoice(DocType):
      total: Decimal

      @field_validator("total")
      def validate_total(cls, v):
          if v < 0:
              raise ValueError("Total cannot be negative")
          return v
  ```

### 8.2 Document Validators

- [x] Use controller `validate()` hook:
  ```python
  class InvoiceController(BaseController[Invoice]):
      async def validate(self):
          if self.doc.total != sum(item.amount for item in self.doc.items):
              raise ValidationError("Total mismatch")
  ```

---

## 9. Testing

### 9.1 Unit Tests

- [x] Test workflow transitions (15 tests in `test_workflow.py`)
- [x] Test DocType overrides (16 tests in `test_meta_registry_overrides.py`)
- [x] Test child table operations (38 tests in `test_child_tables.py`)
- [x] Test link field validation (16 tests in `test_link_fields.py`)
- [x] Test naming series (29 tests in `test_naming_series.py` + validation tests)

### 9.2 Integration Tests

- [x] Test full workflow lifecycle (6 tests in `test_workflow_lifecycle.py`)
- [x] Test override schema migration (10 tests in `test_override_migration.py`)
- [x] Test child table CRUD (12 tests in `test_child_table_integration.py`)
- [x] Test app-defined ports (9 tests in `test_app_ports.py`)

---

## Validation Checklist

Before moving to Phase 09, verify:

- [x] Workflows can be defined and executed (15 unit + 6 integration tests)
- [x] DocTypes can be extended via overrides (16 unit + 10 integration tests)
- [x] Child tables work correctly (38 unit + 12 integration tests)
- [x] Link fields create proper foreign keys (16 tests in `test_link_fields.py`)
- [x] Naming series generates unique names (29 tests in `test_naming_series.py`)
- [x] App-defined ports can be registered (9 tests in `test_app_ports.py`)

---

## Anti-Patterns to Avoid

❌ **Don't**: Hardcode workflow logic in controllers
✅ **Do**: Use pluggable workflow adapter

❌ **Don't**: Modify core DocTypes directly
✅ **Do**: Use override mechanism

❌ **Don't**: Store child records as JSON only
✅ **Do**: Use proper relational tables

❌ **Don't**: Use row-level locking for naming (Frappe anti-pattern)
✅ **Do**: Use optimistic naming with retry on conflict


Document: checklists/phase-12-llm-documentation.md

# Phase 12: LLM-Ready Documentation & Automation

**Objective**: Create comprehensive, machine-readable documentation with automated screenshot generation and LLM-assisted content workflows.

> [!IMPORTANT]
> This phase addresses the documentation debt accumulated during rapid development sprints.
> Documentation is both human-readable AND machine-readable (RAG/LLM source).

> [!NOTE]
> **Tooling Location**: All documentation generation tools (`m docs:*`) reside in `framework-m-studio`.
> The runtime `framework-m` package remains lightweight.

---

## 1. Documentation Architecture

### 1.1 Source-of-Truth Hierarchy

> [!IMPORTANT]
> **Single Source of Truth**: Code generates BOTH human docs AND machine docs.
> They are always in sync because they come from the same source.

```
┌─────────────────────────────────────────────────────────────────┐
│                    SINGLE SOURCE OF TRUTH                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Python Source Code (DocTypes, Protocols, Controllers)         │
│                           │                                     │
│                           ▼                                     │
│                   `m docs:generate`                             │
│                     /           \                               │
│                    ▼             ▼                              │
│     ┌──────────────────┐ ┌──────────────────┐                   │
│     │   HUMAN DOCS     │ │   MACHINE DOCS   │                   │
│     │   (website/docs) │ │   (JSONL/YAML)   │                   │
│     │                  │ │                  │                   │
│     │ • API Reference  │ │ • RAG Corpus     │                   │
│     │ • User Guides    │ │ • .agent/        │                   │
│     │ • Examples       │ │ • Embeddings     │                   │
│     └──────────────────┘ └──────────────────┘                   │
│              │                    │                             │
│              ▼                    ▼                             │
│     Humans read docs      LLMs query corpus                     │
│     in browser            for accurate code                     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

- [x] Define documentation sources:
  - [x] **ADRs** (`docs/adr/`) → Design decisions, rationale
  - [x] **Checklists** (`checklists/`) → Feature completion status
  - [x] **DocType Source** (`*.py`) → API reference (auto-generated)
  - [x] **User Docs** (`docs/user/`) → How-to guides
  - [x] **Developer Docs** (`docs/developer/`) → Architecture, patterns

- [x] Create documentation discovery logic (Dynamic manifest):
  - [x] Implemented in `m docs:generate` and `m docs:export`
  - [x] Auto-scans `docs/`, `checklists/`, and `src/doctypes/`

### 1.2 Documentation Site Setup

**Source**: [`todo.py`](file:///libs/framework-m/src/framework_m/doctypes/todo.py)

## Fields

| Field        | Type | Required | Default | Description       |
| ------------ | ---- | -------- | ------- | ----------------- |
| title        | str  | ✓        | —       | Task title        |
| is_completed | bool |          | false   | Completion status |

## Permissions

| Role     | Read | Write | Create | Delete |
| -------- | ---- | ----- | ------ | ------ |
| Employee | ✓    |       |        |        |
| Manager  | ✓    | ✓     | ✓      | ✓      |

````

### 2.2 Website Setup (Docusaurus)

- [x] Initialize Docusaurus in `website/` directory:
```bash
pnpm create docusaurus website classic --typescript
````

- [x] Configure `website/docusaurus.config.ts`:
  - [x] Set `docs.path` to `../docs` (reuse existing markdown)
  - [x] Set `docs.routeBasePath` to `/docs`
  - [x] Site metadata (title, tagline, URL)

- [x] Setup `website/` folder structure:

  ```
  website/
  ├── blog/           # Release announcements
  ├── src/            # Custom React components
  ├── static/         # Static assets
  └── docusaurus.config.ts
  ```

- [x] Reuse existing `docs/` for content:
  - [x] `docs/user/` -> Documentation content
  - [x] `docs/developer/` -> Documentation content
  - [x] `docs/adr/` -> Documentation content

---

## 2. Auto-Generated Reference Documentation

### 2.1 DocType API Reference

- [x] Implement `m docs:generate` CLI command:
  - [x] Parse all DocTypes from MetaRegistry
  - [x] Extract JSON Schema from Pydantic models
  - [x] **Scope**: Includes Core, Standard, and any installed 3rd-party Apps.
  - [x] Generate markdown for each DocType:
    - [x] Field table (name, type, required, default, description)
    - [x] Permission matrix (roles × actions)
    - [x] Controller hooks (before_create, validate, etc.)
    - [x] Link to source file

- [x] Output format matching standard template:

  ```markdown
  # Todo

  > Represents a task or action item.

  **Source**: [`todo.py`](file:///libs/framework-m/src/framework_m/doctypes/todo.py)

  ## Fields

  ...
  ```

### 2.2 Protocol Reference

- [x] Auto-generate Protocol documentation:
  - [x] List all Protocols in `framework_m_core`
  - [x] Extract method signatures and docstrings
  - [x] List available adapters for each Protocol
  - [x] Generate comparison table (Indie vs Enterprise adapters)

### 2.3 OpenAPI Documentation

- [x] Export OpenAPI spec: `m docs:openapi --output docs/api/openapi.json`
- [x] Generate human-readable API docs from OpenAPI
- [x] Include request/response examples
- [x] Group by DocType and module

---

### 3.1 ADR/RFC Structure Standardization

- [x] Create ADR/RFC template (`docs/adr/template.md`):

  ```markdown
  # ADR-NNNN: [Title]

  ## Status

  [Proposed | Accepted | Deprecated | Superseded by ADR-XXXX]

  ## Context

  [Problem description]

  ## Decision

  [What we decided]
  ```

- [x] Implement incremental numbering logic:
  - [x] Start with 4 digits (`0001-0999`).
  - [x] Automatically expand to more digits if needed (`10000+`).
  - [x] CI check to ensure no duplicate or skipped numbers in PRs.

- [x] Add CLI: `m docs:adr create "Title"` → Creates numbered ADR from template.
- [x] Add CLI: `m docs:rfc create "Title"` → Creates numbered RFC from template.

### 3.2 Automated Chronological Indexing

- [x] Implement `m docs:adr index`:
  - [x] Scans `docs/adr/` and `docs/rfcs/`.
  - [x] Generates `index.md` for both with a chronological list.
  - [x] Anchored "At-a-glance" view for quick reference.
  - [x] Support lazy-loading behavior in Docusaurus (similar to blog).

---

## 4. Checklist-to-Feature Documentation

### 4.1 Feature Extraction

- [x] Parse checklists for completed items (`[x]`)
- [x] Group by phase and category
- [x] Generate "Features" page with completion status
- [x] Link features to relevant documentation sections

### 4.2 Release Notes Generation

- [x] Track checklist changes between versions (git diff)
- [x] Auto-generate release notes from newly completed items
- [x] Format: `CHANGELOG.md` or GitHub Release notes

---

## 5. Automated Screenshot Pipeline

### 5.1 Screenshot Manifest

- [ ] Create `docs/screenshots.yaml`:

  ```yaml
  screenshots:
    - id: studio-doctype-list
      description: DocType list in Studio
      url: /studio/doctypes
      selector: .doctype-table
      viewport: { width: 1280, height: 720 }
      output: docs/images/studio-doctype-list.png
      docs: [docs/user/studio.md]

    - id: form-view-todo
      description: Todo form view
      url: /app/Todo/new
      selector: .form-container
      viewport: { width: 1280, height: 800 }
      output: docs/images/form-view-todo.png
      docs: [docs/user/forms.md]

    - id: workflow-actions
      description: Workflow action buttons
      url: /app/PurchaseOrder/detail/PO-001
      selector: .workflow-actions
      wait_for: .workflow-button
      output: docs/images/workflow-actions.png
      docs: [docs/user/workflows.md]
  ```

### 5.2 Playwright Screenshot Automation

- [ ] Create `scripts/generate-screenshots.ts`:
  - [ ] Read `screenshots.yaml`
  - [ ] Start dev server (`m prod`)
  - [ ] For each screenshot:
    - [ ] Navigate to URL
    - [ ] Wait for selector
    - [ ] Capture screenshot
    - [ ] Save to output path
  - [ ] Generate diff report (new vs old screenshots)

- [ ] Install dependencies:
  ```bash
  pnpm add -D playwright @playwright/test yaml
  ```

### 5.3 Screenshot CI Workflow

- [ ] Add to `.gitlab-ci.yml`:

  ```yaml
  update-screenshots:
    stage: docs
    image: mcr.microsoft.com/playwright:v1.40.0-jammy
    services:
      - docker:dind
    script:
      - docker compose up -d
      - sleep 10
      - npx ts-node scripts/generate-screenshots.ts
    artifacts:
      paths:
        - docs/images/*.png
      expire_in: 1 week
    rules:
      - if: $CI_COMMIT_BRANCH == "main"
        changes:
          - frontend/**
          - docs/screenshots.yaml
      - when: manual
        allow_failure: true
  ```

- [ ] Auto-commit screenshots (optional):
  - [ ] Use `git push` in CI with deploy token
  - [ ] Or: Create MR with updated screenshots for review

### 5.4 Screenshot Staleness Detection

- [ ] Add timestamp to screenshot manifest
- [ ] Compare file modification time vs manifest timestamp
- [ ] Warn in CI if screenshots are stale (>30 days)
- [ ] Option to regenerate on schedule (weekly)

---

## 6. Machine-Readable Documentation (RAG/LLM)

### 6.1 Structured DocType Export

- [ ] Implement `m docs:export --format yaml`:
  ```yaml
  # docs/machine/doctypes.yaml
  doctypes:
    - name: Todo
      module: framework_m.doctypes.todo
      description: Represents a task or action item
      source_file: libs/framework-m/src/framework_m/doctypes/todo.py
      fields:
        - name: title
          type: str
          required: true
          description: Task title
        - name: is_completed
          type: bool
          default: false
      permissions:
        read: [Employee, Manager]
        write: [Manager]
      hooks:
        before_create: validate_due_date
        after_create: send_notification
  ```

### 6.2 RAG Corpus Generation

- [x] Implement `m docs:export --format jsonl --output docs/machine/corpus.jsonl`:

  ```jsonl
  {"id": "doctype-todo", "type": "doctype", "title": "Todo DocType", "content": "...", "source": "todo.py", "tokens": 450}
  {"id": "adr-0001", "type": "adr", "title": "Taskiq + NATS for Jobs", "content": "...", "source": "0001-taskiq-nats.md", "tokens": 320}
  {"id": "protocol-repository", "type": "protocol", "title": "RepositoryProtocol", "content": "...", "source": "protocols.py", "tokens": 280}
  ```

- [x] Chunk content appropriately for embeddings (500-1000 tokens)
- [x] Include metadata for filtering (type, module, version)

### 6.3 Agent Knowledge Base

- [ ] Create `.agent/knowledge/` directory structure:

  ```
  .agent/
  ├── workflows/           # Existing workflow definitions
  └── knowledge/
      ├── doctypes.yaml    # All DocTypes (machine-readable)
      ├── protocols.yaml   # All Protocols
      ├── patterns.md      # Code patterns and conventions
      ├── anti-patterns.md # Things to avoid
      └── corpus.jsonl     # Full RAG corpus
  ```

- [ ] Auto-update on each build/deploy

### 6.4 Inline Code Documentation

- [ ] Ensure all Protocols have comprehensive docstrings:

  ```python
  class RepositoryProtocol(Protocol[T]):
      """
      Generic repository interface for CRUD operations.

      Usage:
          repo = container.get(RepositoryProtocol[Todo])
          todo = await repo.get(session, todo_id)

      Implementations:
          - SQLAlchemyRepository (default, PostgreSQL)
          - MongoRepository (framework-mx-mongo)

      See Also:
          - UnitOfWorkProtocol for transaction management
          - PermissionProtocol for authorization
      """
  ```

- [ ] Add `Examples` sections to all public APIs
- [ ] Use Google-style docstrings for consistency

### 6.5 LLM Docs Distribution & Sync (Federated Knowledge)

- [ ] **Publication Strategy**:
  - [ ] Include machine-readable assets in documentation build artifacts.
  - [ ] Path: `docs/static/llm/v{version}/corpus.jsonl`.

- [ ] **Configuration** (`framework_config.toml`):

  ```toml
  [docs.llm]
  upstream_url = "https://docs.framework-m.org/llm/v1" # Configurable source
  include_upstream = true
  ```

- [ ] **Sync Command** (`m docs:sync`):
  - [ ] Fetches upstream corpus from `upstream_url`.
  - [ ] Generates local project corpus via `m docs:export`.
  - [ ] Merges upstream + local into `.agent/knowledge/combined.jsonl`.
  - [ ] Updates `.agent/context/project.md` with summary of available knowledge.

---

## 7. LLM-Assisted Documentation Workflow

### 7.1 Documentation Gap Detection

- [ ] Create `m docs:audit` command:
  - [ ] Scan all DocTypes for missing descriptions
  - [ ] Check all Protocols for missing docstrings
  - [ ] Identify ADRs without implementation links
  - [ ] Report undocumented controller hooks
  - [ ] Output: `docs/audit-report.md`

### 7.2 LLM Documentation Generator

- [ ] Create `m docs:draft` command:
  - [ ] Input: Source file or DocType name
  - [ ] Process: Send code + context to LLM
  - [ ] Output: Draft documentation in markdown
  - [ ] Human reviews and edits before commit

- [ ] Prompt template:

  ```
  You are documenting Framework M, a Python business application framework.

  Given this DocType definition:
  {source_code}

  And these related ADRs:
  {related_adrs}

  Generate user documentation including:
  1. Overview (what this DocType represents)
  2. Field descriptions
  3. Common use cases
  4. Example code snippets
  ```

### 7.3 Diff-Based Documentation Updates

- [ ] Create `m docs:update` command:
  - [ ] Input: Git diff of code changes
  - [ ] Process: Identify affected documentation sections
  - [ ] Output: Suggested documentation updates
  - [ ] Human reviews and applies changes

---

## 8. Documentation Deployment

### 8.1 Build Pipeline

- [ ] Add docs build to `.gitlab-ci.yml`:

  ```yaml
  pages:
    stage: docs
    image: node:20
    script:
      - cd docs
      - npm ci
      - npm run build
      - mv build ../public
    artifacts:
      paths:
        - public
    rules:
      - if: $CI_COMMIT_BRANCH == "main"
  ```

- [ ] Deploy to GitLab Pages:
  - [ ] Docs available at `https://\<namespace\>.gitlab.io/\<project\>/`
  - [ ] Custom domain: Configure in Settings → Pages

### 8.2 Versioned Documentation

> [!NOTE]
> Documentation versions should align with release versions and branching strategy defined in
> **[Phase 10: Production - Section 6.3 & 6.4](./phase-10-production.md#63-release-versioning--publishing)**.

- [ ] Use **Docusaurus** for documentation (update Section 1.2):

  ```bash
  npx create-docusaurus@latest docs classic --typescript
  ```

- [ ] Enable Docusaurus versioning:

  ```bash
  # Create version snapshot when releasing
  npm run docusaurus docs:version 1.0
  ```

- [ ] Version strategy (aligned with git branching):
  - [ ] `Next` → Current `main` branch (unstable, latest features)
  - [ ] `1.0.x`, `2.0.x` → Tagged releases (stable)
  - [ ] Archive versions when LTS support ends

- [ ] Configure `docusaurus.config.js`:

  ```javascript
  docs: {
    lastVersion: 'current',
    versions: {
      current: { label: 'Next (Unreleased)', path: 'next' },
      '1.0': { label: '1.0.x (LTS)', path: '1.0' },
    },
  },
  ```

- [ ] Sync with release workflow:
  - [ ] On git tag `vX.Y.0` (minor/major): Create new docs version
  - [ ] On git tag `vX.Y.Z` (patch): Update existing version docs
  - [ ] Archive docs when `release/vX` branch is EOL

- [ ] Version banner for old docs:
  - [ ] Docusaurus built-in: "This is documentation for version X.0."

- [ ] Per-version API reference:
  - [ ] Generate OpenAPI spec per version
  - [ ] Store in `docs/static/api/v{version}/openapi.json`
  - [ ] Embed Swagger UI or Redoc in versioned docs

- [ ] Per-version LLM Corpus:
  - [ ] Generate `corpus.jsonl` and `doctypes.yaml` for each version.
  - [ ] Store in `docs/static/llm/v{version}/`.
  - [ ] Allows agents to sync docs matching their installed framework version.

### 8.3 Search Integration

- [x] Add search integration (Searchable ADRs, RFCs, and Docs):
  - [x] Use `docusaurus-plugin-search-local` or similar
  - [x] Ensure `docs/adr/` and `docs/rfcs/` are indexed
  - [x] Include code examples in search

---

## 9. Testing Documentation

### 9.1 Documentation Tests

- [ ] Test code examples compile/run:
  - [ ] Extract code blocks from markdown
  - [ ] Run against test environment
  - [ ] Fail CI if examples are broken

- [ ] Test internal links:
  - [ ] Check all markdown links resolve
  - [ ] Check all file references exist

### 9.2 Screenshot Regression Tests

- [ ] Compare new screenshots to baseline
- [ ] Alert on unexpected visual changes
- [ ] Require manual approval for UI changes

---

## 10. Documentation Contribution Guide

### 10.1 For Contributors

- [ ] Create `docs/CONTRIBUTING.md`:
  - [ ] How to write documentation
  - [ ] Style guide (tone, formatting)
  - [ ] How to add new screenshots
  - [ ] How to update ADRs

### 10.2 For AI Agents

- [ ] Create `.agent/knowledge/documentation-patterns.md`:
  - [ ] How Framework M documentation is structured
  - [ ] Where to find different types of information
  - [ ] How to suggest documentation improvements

---

## 11. LLM-Assisted Development Support

> [!IMPORTANT]
> **TDD is the core enabler for LLM-assisted coding.**
> Without tests, an LLM cannot verify its generated code is correct.
> Framework M's architecture + TDD = effective AI pair programming.

### 11.1 Why TDD Enables LLM Coding

```
┌─────────────────────────────────────────────────────────────────┐
│                    LLM CODING FEEDBACK LOOP                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   1. LLM reads: Docs + Tests + Code                             │
│                    ↓                                            │
│   2. LLM generates: New code / modifications                    │
│                    ↓                                            │
│   3. LLM runs: `pytest` / `mypy` / `ruff`                       │
│                    ↓                                            │
│   4. Tests PASS? → Done ✓                                       │
│      Tests FAIL? → LLM reads error, returns to step 2           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

- [x] Document the TDD-LLM workflow:
  - [x] Tests are **executable specifications** — LLM knows what "correct" means
  - [x] Test failures are **feedback** — LLM can self-correct
  - [x] Type hints + mypy are **compile-time checks** — catch errors before runtime
  - [x] Protocol interfaces are **contracts** (Documented in developer guides)

### 11.2 Agent-Ready Repository Structure

- [x] Create `.agent/` directory for AI coding assistants:
  ```
  .agent/
  ├── workflows/           # Existing workflow definitions
  └── knowledge/           # Machine-readable knowledge base (Done via machine export)
  ```
- [x] `.agent/workflows/` populated with key tasks

### 11.3 Test-First Development for LLM

- [ ] Provide test templates for common patterns:

  ```python
  # tests/doctypes/test_todo.py
  import pytest
  from framework_m.testing import DocTypeTestCase

  class TestTodoDocType(DocTypeTestCase):
      doctype = "Todo"

      async def test_create_todo(self, repo, uow):
          """LLM can use this as a template for new DocTypes."""
          todo = Todo(title="Test Task")
          async with uow:
              created = await repo.save(uow.session, todo)
              await uow.commit()
          assert created.id is not None
          assert created.title == "Test Task"
  ```

- [ ] Document test commands in `.agent/workflows/run-tests.md`:

  ```markdown
  # Running Tests

  ## Quick test (single file)

  uv run pytest tests/doctypes/test_todo.py -v

  ## Full suite

  uv run pytest

  ## With coverage

  uv run pytest --cov=framework_m --cov-report=html

  ## Type checking

  uv run mypy --strict .
  ```

### 11.4 LLM-Friendly Error Messages

- [ ] Ensure error messages are actionable:
  - [ ] Include expected vs actual values
  - [ ] Reference relevant documentation
  - [ ] Suggest fixes when possible

- [ ] Custom exceptions with context:
  ```python
  class PermissionDeniedError(FrameworkMError):
      def __init__(self, user: str, action: str, doctype: str):
          super().__init__(
              f"User '{user}' lacks '{action}' permission on '{doctype}'. "
              f"See: docs/permissions.md#granting-permissions"
          )
  ```

### 11.5 Agentic Coding Workflow

- [x] Document the agentic workflow for Framework M (See `Architecture` and `Quality` guides)
- [x] Create `.agent/workflows/` with development best practices

### 11.6 Why Framework M Enables LLM-Assisted Coding

> [!NOTE]
> Framework M's design choices make it exceptionally LLM-friendly:

| Design Choice              | LLM Benefit                                          |
| -------------------------- | ---------------------------------------------------- |
| **Hexagonal Architecture** | Clear boundaries — LLM knows where to make changes   |
| **Protocol Interfaces**    | Explicit contracts — LLM knows method signatures     |
| **Pydantic Models**        | Type safety — LLM generates typed code               |
| **Code-First DocTypes**    | No hidden DB state — all logic is in readable Python |
| **100% Type Hints**        | mypy catches LLM errors at "compile time"            |
| **No Global State**        | Explicit DI — LLM doesn't miss hidden dependencies   |
| **TDD Culture**            | Tests verify LLM output — self-correcting loop       |

### 11.7 RAG Corpus for Code Completion

- [ ] Include code examples in RAG corpus:
  - [ ] Annotated examples of each pattern
  - [ ] Common mistakes and their fixes
  - [ ] Before/after refactoring examples

- [ ] Export tests as documentation:
  - [ ] Tests show "how to use" each component
  - [ ] `m docs:export --include-tests` adds test patterns to corpus

## Validation Checklist

Before completing this phase:

- [x] `m docs:generate` produces complete API reference
- [x] `m docs:export --format jsonl` exports RAG corpus
- [ ] Screenshot pipeline runs in CI
- [x] All DocTypes have descriptions
- [x] All Protocols have docstrings (in code)
- [x] Documentation site builds without errors
- [x] Search works across all documentation
- [ ] Code examples in docs are tested
- [x] `.agent/` directory structure in place
- [ ] Test templates provided for LLM coding
- [x] Agentic workflow documented (in intro/developer docs)
- [x] LLM can run tests and self-correct (verified manually)

---

## Anti-Patterns to Avoid

❌ **Don't**: Write documentation separately from code
✅ **Do**: Generate documentation from source code

❌ **Don't**: Use static screenshots that become stale
✅ **Do**: Automate screenshot generation in CI

❌ **Don't**: Write documentation only for humans
✅ **Do**: Create machine-readable formats for LLM/RAG

❌ **Don't**: Document at the end when details are forgotten
✅ **Do**: Document incrementally as features are built

❌ **Don't**: Ignore documentation in code review
✅ **Do**: Require documentation updates with code changes

---

## Timeline Estimate

| Task                             | Duration    |
| -------------------------------- | ----------- |
| Documentation architecture setup | 1 day       |
| Auto-generated reference docs    | 2 days      |
| Screenshot pipeline              | 2 days      |
| Machine-readable export          | 1 day       |
| LLM workflow integration         | 2 days      |
| Testing and polish               | 1 day       |
| **Total**                        | **~9 days** |


Document: checklists/phase-04-jobs-events.md

# Phase 04: Background Jobs & Events

**Objective**: Implement asynchronous job processing, event-driven architecture, and webhook system with **pluggable backends** (Redis, NATS, In-Memory).

---

## 0. Backend Selection (Cross-Platform)

### 0.1 Decision
| Component | Backend | Library | Notes |
|-----------|---------|---------|-------|
| **Jobs** | NATS JetStream | `taskiq` + `taskiq-nats` | Cross-platform, persistent |
| **Events** | NATS JetStream | `nats-py` | Pub/Sub + Replay |
| **WS Backplane** | NATS | `nats-py` | Distributed WebSockets |
| **Cache** | Redis | `redis-py` | Windows: Memurai/Garnet |

### 0.2 Dev Mode (No External Deps)
- [x] Implement `InMemoryJobQueue` for local dev (asyncio.Queue).
- [x] Implement `InMemoryEventBus` for local dev.
- [x] Auto-detect: If `NATS_URL` not set, use in-memory.

## 1. Taskiq + NATS Integration (Primary)

### 1.1 Taskiq Setup

- [x] Add dependencies: `taskiq`, `taskiq-nats`.
- [x] Create `src/framework_m/adapters/jobs/taskiq_adapter.py`
- [x] Configure Taskiq with NATS JetStream:
  - [x] Set NATS connection URL from environment (`NATS_URL`)
  - [x] Configure job timeout (default: 300s)
  - [x] Configure max retries (default: 3)
  - [x] Use `PullBasedJetStreamBroker` for reliable delivery

- [x] Create broker instance:
  ```python
  from taskiq_nats import PullBasedJetStreamBroker

  broker = PullBasedJetStreamBroker(
      servers=["nats://localhost:4222"],
      stream="framework_m_jobs",
  )
  ```

### 1.2 Job Queue Protocol Implementation

- [x] Implement `TaskiqJobQueueAdapter`:
  - [x] `async def enqueue(job_name: str, **kwargs) -> str`:
    - [x] Get registered task
    - [x] Call `.kiq()` to enqueue
    - [x] Return job ID
  - [x] `async def schedule(job_name: str, cron: str, **kwargs)`:
    - [x] Use `TaskiqScheduler` with cron triggers

### 1.3 Job Registration

- [x] Create job decorator `@job`:
  ```python
  @broker.task(name="send_email", retry=3)
  async def send_email(to: str, subject: str, body: str):
      # Job logic
      pass
  ```

- [x] Implement job registry:
  - [x] Store all `@broker.task` decorated functions
  - [x] Auto-discover on startup

---

## 2. Worker Process

### 2.1 Worker Implementation

- [x] Create `src/framework_m/cli/worker.py`
- [x] Implement worker command:
  ```bash
  m worker
  ```

- [x] Worker startup:
  - [x] Initialize database connection
  - [x] Discover and register all jobs
  - [x] Start Taskiq worker
  - [x] Listen for jobs from NATS JetStream

### 2.2 Job Context

- [x] Create job context for dependency injection:
  - [x] Provide database session
  - [x] Provide repository factory
  - [x] Provide event bus
  - [x] Pass to job functions

- [x] Add job metadata:
  - [x] Job ID
  - [x] Enqueued time
  - [x] Started time
  - [x] User context (Must re-hydrate from job arguments if applicable)

---

## 3. Scheduler

### 3.1 Cron Jobs

- [x] Support Taskiq's native cron syntax:
  ```python
  cron_jobs = [
      cron("send_daily_report", hour=9, minute=0),
      cron("cleanup_old_files", hour=2, minute=0),
  ]
  ```

- [x] Create `ScheduledJob` DocType:
  - [x] `name: str` - Job identifier
  - [x] `function: str` - Dotted path to function
  - [x] `cron_expression: str` - Cron syntax
  - [x] `enabled: bool` - Active/inactive
  - [x] `last_run: datetime | None`
  - [x] `next_run: datetime | None`

### 3.2 Dynamic Scheduler

- [x] Implement dynamic job scheduling:
  - [x] Load `ScheduledJob` DocTypes from database
  - [x] Register with Taskiq at startup
  - [ ] Support adding/removing jobs without restart (via signal) **NOT NEEDED**: Worker restart is acceptable for schedule changes; k8s rolling updates handle this gracefully. If needed in future, can use NATS pub/sub to signal workers in Phase 06+ (Ops/Deployment).

---

## 4. Event Bus

### 4.1 NATS Event Bus Implementation

- [x] Create `src/framework_m/adapters/events/nats_event_bus.py`
- [x] Implement `NatsEventBusAdapter`:
  - [x] `async def publish(topic: str, event: BaseModel)`:
    - [x] Serialize event to JSON
    - [x] Publish to NATS JetStream subject
  - [x] `async def subscribe(topic: str, handler: Callable)`:
    - [x] Subscribe to NATS JetStream subject
    - [x] Deserialize event
    - [x] Call handler function

### 4.2 Event Types

- [x] Create base event class:
  ```python
  class BaseEvent(BaseModel):
      event_type: str
      timestamp: datetime
      user_id: str | None
      data: dict
  ```

- [x] Define standard events:
  - [x] `DocCreated`
  - [x] `DocUpdated`
  - [x] `DocDeleted`
  - [x] `DocSubmitted`
  - [x] `DocCancelled`

### 4.3 Event Publishing from Lifecycle Hooks

- [x] Update `GenericRepository`:
  - [x] Inject event bus
  - [x] Publish `DocCreated` after insert
  - [x] Publish `DocUpdated` after update
  - [x] Publish `DocDeleted` after delete

- [x] Add event publishing to controller hooks:
  ```python
  async def after_insert(self):
      await self.event_bus.publish(
          "doc.created",
          DocCreated(doctype="Todo", doc_id=self.doc.name)
      )
  ```

---

## 5. Webhook System

### 5.1 Webhook DocType

- [x] Create `Webhook` DocType:
  - [x] `name: str`
  - [x] `event: str` - Event to listen to (e.g., "doc.created")
  - [x] `doctype_filter: str | None` - Filter by DocType
  - [x] `url: str` - Webhook endpoint URL
  - [x] `method: str` - HTTP method (POST, PUT)
  - [x] `headers: dict` - Custom headers
  - [x] `enabled: bool`
  - [x] `secret: str` - For signature verification

### 5.2 Webhook Listener

- [x] Create `src/framework_m/adapters/webhooks/listener.py`
- [x] Implement webhook listener:
  - [x] Subscribe to all events
  - [x] Load active webhooks from database
  - [x] Filter events by webhook configuration
  - [x] Trigger HTTP call to webhook URL

### 5.3 Webhook Delivery

- [x] Implement webhook delivery:
  - [x] Use `httpx` for async HTTP calls
  - [x] Add signature header (HMAC-SHA256 with secret)
  - [x] Set timeout (default: 30s)
  - [x] Handle errors and retries

- [x] Add retry logic:
  - [x] Retry on failure (exponential backoff)
  - [x] Max retries: 3
  - [x] Log failed deliveries

### 5.4 Webhook Logs

- [x] Create `WebhookLog` DocType:
  - [x] `webhook: str` - Reference to Webhook
  - [x] `event: str` - Event that triggered
  - [x] `status: str` - Success/Failed
  - [x] `response_code: int`
  - [x] `response_body: str`
  - [x] `error: str | None`
  - [x] `timestamp: datetime`

---

## 6. Job Monitoring

### 6.1 Job Status Tracking

- [x] Create `JobLog` DocType:
  - [x] `job_id: str`
  - [x] `job_name: str`
  - [x] `status: str` - Queued/Running/Success/Failed
  - [x] `enqueued_at: datetime`
  - [x] `started_at: datetime | None`
  - [x] `completed_at: datetime | None`
  - [x] `error: str | None`
  - [x] `result: dict | None`

- [x] Update job execution to log status:
  - [x] Create log entry on enqueue
  - [x] Update on start
  - [x] Update on completion/failure

### 6.2 Job Management API
- [x] Create endpoints:
  - [x] `GET /api/v1/jobs` - List jobs
  - [x] `GET /api/v1/jobs/{job_id}` - Get job status
  - [x] `POST /api/v1/jobs/{job_id}/cancel` - Cancel job
  - [x] `POST /api/v1/jobs/{job_id}/retry` - Retry failed job

### 6.3 Job Monitor (Admin UI)
- [x] **Option A: Native Desk UI** (IMPLEMENTED):
  - [x] `Job Log` List View (auto-generated via `api_resource=True`).
  - [x] Shows Status, Error, Duration (`duration_seconds` property).
- [ ] ~~**Option B: Taskiq Dashboard (External)**~~ - **NOT NEEDED for MVP**: Native JobLog UI is sufficient for debugging. External dashboards can be mounted in Phase 06+ (Ops/Deployment) if production monitoring requires deeper worker introspection.
  - [ ] Mount `taskiq-dashboard` (if available) or similar tool.
  - [ ] Useful for sysadmins to debug worker health.

---

## 4. Real-time Events (WebSockets)

### 4.1 WebSocket Protocol (The Port)
**Rationale**: Abstraction allows switching backplanes (Redis/NATS/Memory).

- [x] Create `src/framework_m/core/interfaces/socket.py`
- [x] Define `SocketProtocol`:
  - [x] `async def broadcast(topic: str, message: dict)`
  - [x] `async def send_to_user(user_id: str, message: dict)`

### 4.2 NATS Backplane (The Adapter)
**Target**: Kubernetes support (Distributed).

- [x] Create `tests/adapters/socket/test_nats_socket.py`
- [x] Write tests using testcontainers-nats to verify Pub/Sub.
- [x] Create `src/framework_m/adapters/socket/nats_socket.py`
- [x] Implement `NatsSocketAdapter`:
  - [x] On Boot: Subscribe to `framework.events.*`.
  - [x] On Event: `await nc.publish(subject, json.dumps(msg).encode())`.
  - [x] On Receive (Sub): Forward to local connected Websocket clients (Connection Manager).

### 4.3 WebSocket Endpoint (Litestar)

- [x] Create `src/framework_m/adapters/web/socket.py`
- [x] Implement `ConnectionManager`:
  - [x] Store active connections `Dict[UserId, List[WebSocket]]`.
- [x] Add route `WS /api/v1/stream`:
  - [x] Authenticate user (Query param token).
  - [x] Upgrade connection.
  - [x] Register with `ConnectionManager`.
  - [x] Listen for client disconnect.

---

## 7. Testing

### 7.1 Unit Tests

- [x] Test job registration:
  - [x] Test `@job` decorator
  - [x] Test job registry

- [x] Test event bus:
  - [x] Test publish/subscribe
  - [x] Test event serialization

### 7.2 Integration Tests

- [x] Test job execution:
  - [x] Enqueue job
  - [x] Verify job runs
  - [x] Check job result

- [x] Test scheduled jobs:
  - [x] Register cron job
  - [x] Verify execution at scheduled time (use time mocking)

- [x] Test webhooks:
  - [x] Create webhook
  - [x] Trigger event
  - [x] Verify HTTP call made
  - [x] Check webhook log

- [x] Use testcontainers for NATS

---

## 8. CLI Commands

- [x] Add job management commands:
  - [x] `m worker` - Start worker process
  - [x] `m job:list` - List registered jobs
  - [x] `m job:run <name>` - Run job immediately
  - [x] `m job:status <id>` - Check job status

---

## Validation Checklist

Before moving to Phase 05, verify:

- [x] Jobs can be enqueued and executed
- [x] Scheduled jobs run at correct times
- [x] Events are published and received
- [x] Webhooks are triggered on events
- [x] Job logs are created and updated
- [x] Worker process starts and processes jobs

---

## Anti-Patterns to Avoid

❌ **Don't**: Run long-running tasks in HTTP handlers
✅ **Do**: Enqueue jobs for async processing

❌ **Don't**: Use polling for event detection
✅ **Do**: Use NATS Pub/Sub for real-time events

❌ **Don't**: Hardcode webhook URLs
✅ **Do**: Store webhooks in database for dynamic configuration

❌ **Don't**: Ignore failed jobs
✅ **Do**: Implement retry logic and logging


Document: checklists/phase-03-api-authorization.md

# Phase 03: API Layer & Authorization

**Objective**: Expose DocTypes via HTTP with auto-generated CRUD endpoints, implement permission system with row-level security, and add RPC support.

---

## 1. Litestar Application Setup

- [x] Create `src/framework_m/adapters/web/app.py`
- [x] Initialize Litestar app:
  - [x] Configure CORS for development
  - [x] Add exception handlers
  - [x] Configure OpenAPI documentation
  - [x] Integrate with `dependency-injector` Container (wire modules)
  - [x] **Middleware Discovery**:
    - [x] Scan `MiddlewareRegistry` for registered App Middlewares.
    - [x] Allow Apps to inject ASGI Middleware (e.g. `RateLimit`, `cors`, `Compression`).
    - [x] **APM & Tracing**: Support OpenTelemetry, DataDog, Sentry via standard ASGI middleware.
    - [x] Support ordering (priority).

- [x] Add application lifecycle:
  - [x] `on_startup`: Initialize Container, database, discover DocTypes
  - [x] `on_shutdown`: Close database connections, unwire Container

---

## 2. Authentication Middleware

- [x] Create `src/framework_m/adapters/web/middleware.py`
- [x] Implement `AuthMiddleware`:
  - [x] Extract `x-user-id` header
  - [x] Extract `x-roles` header (comma-separated)
  - [x] Extract `x-tenants` header (optional, comma-separated)
  - [x] Create `UserContext` object
  - [x] Store in `request.state.user`
  - [x] Return 401 if headers missing (configurable)

- [x] Add middleware to Litestar app *(available via `create_auth_middleware()`)*

---

## 3. Permission System

### 3.1 Permission Protocol Implementation

- [x] Create `src/framework_m/adapters/auth/rbac_permission.py`
- [x] Implement `RbacPermissionAdapter`:
  - [x] Implement `evaluate(request: PolicyEvaluateRequest) -> PolicyEvaluateResult`
  - [x] Read permissions from `DocType.Meta.permissions`
  - [x] Match `request.principal_attributes["roles"]` against allowed roles
  - [x] `CustomPermission` DocType created *(integration pending)*
  - [x] Return `PolicyEvaluateResult(authorized=True/False, decision_source=DecisionSource.RBAC)`

### 3.2 Permission Configuration

- [x] Define permission structure in DocType:
  ```python
  class Meta:
      # Access Control Flags
      requires_auth: bool = True    # Does user need to be logged in?
      apply_rls: bool = True        # Is Row-Level Security applied?

      # RBAC Permissions (only used by RbacPermissionAdapter)
      permissions = {
          "read": ["Employee", "Manager"],
          "write": ["Manager"],
          "create": ["Manager"],
          "delete": ["Admin"],
          "submit": ["Manager"]
      }
  ```

**Access Control Combinations:**

| `requires_auth` | `apply_rls` | Use Case |
|---|---|---|
| `True` | `True` | Default. Alice only sees her own invoices. |
| `True` | `False` | Lookup tables. All users see all countries. |
| `False` | `False` | Public data. App config, announcements. |
| `False` | `True` | Public profiles. Only owner's data exposed. |

- [x] Implement `has_permission()`:
  - [x] Check `requires_auth` — if False and no user, allow read
  - [x] Check if user has required role
  - [x] Check database overrides via `PermissionLookupService` (CustomPermission DocType)
  - [x] For object-level: check ownership or custom rules
  - [x] Return boolean

### 3.3 System Context (Elevated Operations)

> [!IMPORTANT]
> **No `ignore_permissions` flag**. For system operations, use explicit `SystemContext`.

For background jobs, migrations, or service-to-service calls that need elevated access:

- [x] Create `src/framework_m/core/system_context.py`
- [x] Implement `SystemContext`:
  ```python
  @asynccontextmanager
  async def system_context():
      """Context manager for system-level operations.

      All operations are logged as 'system' principal.
      Use only in background jobs, migrations, webhooks.
      """
      yield SystemPrincipal(
          id="system",
          roles=["System"],
          is_system=True,
      )
  ```
- [x] Usage in background jobs:
  ```python
  async def sync_external_data():
      async with system_context() as ctx:
          # Runs with system principal, bypasses RLS
          await repo.save(ctx.session, data)
  ```
- [x] **Audit**: All system operations logged with `principal="system"`

### 3.4 Row-Level Security (RLS)

- [x] Implement `get_permitted_filters()`:
  - [x] For standard users: add `WHERE owner = :user_id`
  - [x] For admins: no filter (see all)
  - [x] For custom rules: via `Meta.rls_field` (completed by Section 3.5 Team-Based Access)
  - [x] Return dict of SQLAlchemy filters

- [x] Update `GenericRepository.list()`:
  - [x] `list_for_user()` method with RLS filtering
  - [x] `apply_rls_filters()` helper available
  - [x] Merge with user-provided filters
  - [x] Apply to SQL query

- [x] Update `GenericRepository.get()`:
  - [x] `get_for_user()` method with permission check
  - [x] Raise `PermissionDeniedError` if denied

### 3.5 Team-Based Access (Indie)

For team/department-level access without full ReBAC:

- [x] Add `Meta.rls_field` option:
  ```python
  class Invoice(BaseDocType):
      owner: str
      team: str  # e.g., "sales"

      class Meta:
          apply_rls = True
          rls_field = "team"  # RLS: WHERE team IN :user_teams
  ```
- [x] Update `get_permitted_filters()`:
  - [x] If `rls_field = "owner"` (default): `WHERE owner = :user_id`
  - [x] If `rls_field = "team"`: `WHERE team IN :user_teams`
  - [x] Support custom fields

> [!NOTE]
> **Where does `user_teams` come from?**
> - From `PolicyEvaluateRequest.principal_attributes["teams"]`
> - The auth layer (middleware) populates this from headers, JWT claims, or external IAM
> - Framework M core does NOT assume a built-in User model
> - **Indie**: Phase 06 adds optional `User` + `UserAttribute` DocTypes for self-contained apps

### 3.6 Explicit Sharing (`DocumentShare`)

For sharing specific documents with specific users/roles:

- [x] Create built-in `DocumentShare` DocType:
  ```python
  class DocumentShare(BaseDocType):
      doctype_name: str      # e.g., "Invoice"
      doc_id: str            # e.g., "INV-001"
      shared_with: str       # User ID or Role name
      share_type: ShareType  # USER or ROLE
      granted_permissions: list[str]  # ["read"] or ["read", "write"]

      class Meta:
          requires_auth = True
          apply_rls = True  # Users only see shares they created
  ```
- [x] Update `get_permitted_filters()` to include shares via `get_rls_filters_with_shares()`:
  ```python
  # Returns filters including shared doc IDs:
  # {"owner": "user-123", "_shared_doc_ids": ["INV-001", "INV-002"]}
  filters = await get_rls_filters_with_shares(user, "Invoice", shares, "read")
  ```
- [x] Add share management API:
  - [x] `POST /api/v1/share` — Create share
  - [x] `DELETE /api/v1/share/{id}` — Remove share
  - [x] `GET /api/v1/{doctype}/{id}/shares` — List shares for a doc

> [!NOTE]
> **Enterprise Alternative**: For complex sharing graphs, swap to SpiceDB adapter.
> Relationships like "team member", "org hierarchy", "inherited access" are better in ReBAC.

### 3.7 Indie Mode: Permission Conveniences

> [!TIP]
> **0-Cliff Principle**: These helpers internally construct `PolicyEvaluateRequest`.
> Indie devs get simplicity; enterprise devs can use the full API when needed.

#### `@requires_permission` Decorator

- [x] Create `src/framework_m/core/decorators.py` (if not exists)
- [x] Implement `@requires_permission(action: PermissionAction)`:
  ```python
  @requires_permission(PermissionAction.WRITE)
  async def update_invoice(self, invoice_id: str, data: InvoiceUpdate):
      # Permission already checked, just do business logic
      ...
  ```
  - [x] Internally builds `PolicyEvaluateRequest` from `self.user`, `self.doctype_name`
  - [x] Calls `permission.evaluate(request)`
  - [x] Raises `PermissionDeniedError` if not authorized
  - [x] Works on Controller methods

#### `check_permission()` Helper

- [x] Add to `BaseController`:
  ```python
  async def check_permission(self, action: str, doc_id: str | None = None) -> bool:
      """Simple helper that builds PolicyEvaluateRequest internally."""
      # Returns bool - use require_permission() to raise on failure
  ```

#### `require_permission()` Helper

- [x] Add to `BaseController`:
  ```python
  async def require_permission(self, action: str, doc_id: str | None = None) -> None:
      """Raises PermissionDeniedError if not authorized."""
  ```

#### Auto-Permission in CRUD Routes

- [x] Verify: All auto-generated CRUD routes (Section 4) automatically call `permission.evaluate()`:
  - [x] `POST /api/v1/{doctype}` → checks `CREATE` permission *(verified via test)*
  - [x] `GET /api/v1/{doctype}/{id}` → checks `READ` permission *(verified via test)*
  - [x] `PUT /api/v1/{doctype}/{id}` → checks `WRITE` permission *(verified via test)*
  - [x] `DELETE /api/v1/{doctype}/{id}` → checks `DELETE` permission *(verified via test)*
- [x] **No manual code required** for indie devs using auto-CRUD *(verified via test)*

---

## 4. Auto-CRUD Router

### 4.1 Meta Router Implementation

- [x] Create `src/framework_m/adapters/web/meta_router.py`
- [x] Implement `create_crud_routes(doctype_class: type[BaseDocType])`:
  - [x] **Check Opt-in**: Verify `DocType.Meta.api_resource is True`. Skip if False.
  - [x] Generate 5 routes per DocType
  - [x] Inject repository and permission service
  - [x] Parse filters from JSON string via `_parse_filters()`

### 4.2 List Endpoint

- [x] Create `GET /api/v1/{doctype}`:
  - [x] Accept query parameters: `limit`, `offset`, `filters`, `order_by`
  - [x] Parse filters from JSON string
  - [x] Call repository `list_entities()` with RLS filters via `apply_rls_filters()`
  - [x] Return paginated response with metadata:
    ```json
    {
      "data": [...],
      "total": 100,
      "limit": 20,
      "offset": 0
    }
    ```

### 4.3 Create Endpoint

- [x] Create `POST /api/v1/{doctype}`:
  - [x] Validate request body against Pydantic model
  - [x] Check `CREATE` permission via `_check_permission()`
  - [x] Set `owner` to current user (`data["owner"] = user_id`)
  - [x] Call repository `save()`
  - [x] Return 201 with created document

### 4.4 Read Endpoint

- [x] Create `GET /api/v1/{doctype}/{id}`:
  - [x] Call repository `get(id)`
  - [x] Check `READ` permission
  - [x] Return 404 if not found (via `NotFoundException`)
  - [x] Return 403 if permission denied
  - [x] Return document

### 4.5 Update Endpoint

- [x] Create `PUT /api/v1/{doctype}/{id}`:
  - [x] Load existing document via `repo.get()`
  - [x] Check `WRITE` permission
  - [x] Check if submitted (deny if immutable) via `_check_submitted()`
  - [x] Merge changes via `entity.model_copy(update=data)`
  - [x] Call repository `save()`
  - [x] Return updated document

### 4.6 Delete Endpoint

- [x] Create `DELETE /api/v1/{doctype}/{id}`:
  - [x] Load document
  - [x] Check `DELETE` permission
  - [x] Check if submitted (deny if immutable) via `_check_submitted()`
  - [x] Call repository `delete()`
  - [x] Return 204 No Content

### 4.7 Router Registration

- [x] Implement `create_meta_router()`:
  - [x] Get all DocTypes from MetaRegistry
  - [x] For each DocType with `api_resource=True`, call `create_crud_routes()`
  - [x] Register all routes with Litestar
  - [x] Return Router instance

---

## 5. RPC System

### 5.1 Whitelisted Controller Methods

- [x] Create `@whitelist` decorator:
  - [x] Mark controller methods as publicly callable
  - [x] Store metadata on method (`WHITELIST_ATTR`)
  - [x] `is_whitelisted()` helper function
  - [x] `get_whitelist_options()` helper function

- [x] Create `POST /api/v1/rpc/{doctype}/{method}`:
  - [x] Load DocType controller
  - [x] Check if method has `@whitelist` decorator
  - [x] Validate method exists
  - [x] Parse request body as method arguments
  - [x] Instantiate controller with document (if doc_id provided)
  - [x] Call method
  - [x] Return result

### 5.2 Arbitrary RPC Functions

- [x] Create `@rpc` decorator for standalone functions:
  - [x] Register function in RPC registry (`RpcRegistry` singleton)
  - [x] Store function path (e.g., `my_app.api.send_email`)
  - [x] `is_rpc_function()` helper
  - [x] `get_rpc_options()` helper

- [x] Create `POST /api/v1/rpc/fn/{dotted.path}`:
  - [x] Parse dotted path (e.g., `my_app.api.send_email`)
  - [x] Look up function in registry
  - [x] Validate function has `@rpc` decorator
  - [x] Parse request body as function arguments
  - [x] Call function
  - [x] Return result

- [x] Add permission check for RPC:
  - [x] Support `@rpc(permission="custom_permission")`
  - [x] Check permission before calling function
  - [x] Support `@rpc(allow_guest=True)` for public endpoints

---

## 6. Metadata API

- [x] Create `GET /api/meta/{doctype}`:
  - [x] Get DocType class from registry
  - [x] Generate JSON Schema from Pydantic model:
    ```python
    schema = DocType.model_json_schema()
    ```
  - [x] Extract layout from `Meta.layout`
  - [x] Extract permissions from `Meta.permissions`
  - [x] Return:
    ```json
    {
      "doctype": "...",
      "schema": {...},
      "layout": {...},
      "permissions": {...},
      "metadata": {...}
    }
    ```

- [x] Add field metadata:
  - [x] Field labels (from `Field(title=...)`) - included in JSON Schema
  - [x] Help text (from `Field(description=...)`) - included in JSON Schema
  - [x] Validation rules (from Pydantic validators) - included in JSON Schema
  - [x] Field types and options - included in JSON Schema

---

## 7. OpenAPI Documentation

- [x] Configure Litestar OpenAPI:
  - [x] Set title, version, description
  - [x] Add authentication scheme (header-based: `x-user-id`, `x-roles`)
  - [x] Enable Swagger UI at `/schema/swagger`
  - [x] Enable ReDoc at `/schema/redoc`

- [x] Enhance auto-generated docs:
  - [x] Add descriptions to CRUD endpoints (via docstrings)
  - [x] Add examples for request/response (via Pydantic models)
  - [x] Document RPC endpoints (auto-generated from route handlers)
  - [x] Add permission requirements to docs (via security schemes)

---

## 8. Error Handling

- [x] Create exception handlers:
  - [x] `ValidationError` → 400 Bad Request (`validation_error_handler`)
  - [x] `PermissionDeniedError` → 403 Forbidden (`permission_denied_handler`)
  - [x] `DocTypeNotFoundError` → 404 Not Found (`not_found_handler`)
  - [x] `DuplicateNameError` → 409 Conflict (`duplicate_name_handler`)
  - [x] Generic exceptions → 500 Internal Server Error (`framework_error_handler`)

- [x] Return consistent error format:
  ```json
  {
    "error": "PermissionDenied",
    "message": "You don't have permission to delete this document",
    "details": {...}
  }
  ```

---

## 9. Request/Response DTOs

- [x] Use Litestar DTOs for validation:
  - [x] Auto-generate from Pydantic models (via Litestar)
  - [x] Add DTO for list response (pagination): `PaginatedResponse[T]`
  - [x] Add DTO for error responses: `ErrorResponse`

- [x] Implement response filtering:
  - [x] Exclude sensitive fields based on permissions (via field selection)
  - [x] Support field selection via query param: `?fields=name,title`

---

## 10. Testing

### 10.1 Unit Tests

- [x] Test permission system:
  - [x] Test role-based access (`test_rbac_permission.py`)
  - [x] Test RLS filter generation (`test_rls.py`)
  - [x] Test object-level permissions (`test_object_level_permissions.py`)

- [x] Test RPC system:
  - [x] Test `@whitelist` decorator (`test_whitelist_decorator.py`)
  - [x] Test `@rpc` decorator (`test_rpc_decorator.py`)
  - [x] Test dotted path resolution (`test_rpc_routes.py`)

### 10.2 Integration Tests

- [x] Test CRUD endpoints (`test_api_endpoints.py`):
  - [x] Test create with valid data
  - [x] Test create with invalid data (validation)
  - [x] Test create without permission (403)
  - [x] Test list with RLS (only see own docs)
  - [x] Test update immutable doc (covered by existing `test_meta_router.py`)
  - [x] Test delete with permission (covered by existing permission tests)

- [x] Test RPC endpoints (`test_api_endpoints.py`, `test_rpc_routes.py`):
  - [x] Test whitelisted controller method
  - [x] Test arbitrary RPC function
  - [x] Test RPC with permissions

- [x] Test with real HTTP client (`test_real_http_client.py`):
  - [x] Uses `httpx.AsyncClient` against real uvicorn server
  - [x] Tests: list endpoint, create endpoint, health check
  - [x] Manual testing scripts available in `examples/` folder
      ```python
      async with httpx.AsyncClient() as client:
          response = await client.post(
              "http://localhost:8000/api/v1/todo",
              json={"title": "Test"},
              headers={"x-user-id": "user123", "x-roles": "Employee"}
          )
          assert response.status_code == 201
      ```

---

## Validation Checklist

Before moving to Phase 04, verify:

- [x] All CRUD endpoints work with permissions (verified via `test_meta_router.py`, `test_api_endpoints.py`)
- [x] RLS filters are applied correctly (verified via `test_rls.py`, `test_object_level_permissions.py`)
- [x] RPC system works for controller methods and functions (verified via `test_rpc_routes.py`)
- [x] OpenAPI docs are generated correctly (SwaggerUI + ReDoc enabled)
- [x] Error handling returns proper status codes (verified via `test_app.py`)
- [x] Integration tests pass with real HTTP requests (verified via `test_real_http_client.py`)

---

## Anti-Patterns to Avoid

❌ **Don't**: Hardcode permission checks in endpoints
✅ **Do**: Use `PermissionProtocol` for all checks

❌ **Don't**: Return all data without RLS
✅ **Do**: Always apply permission filters in repository

❌ **Don't**: Use global state for current user
✅ **Do**: Pass `UserContext` via dependency injection

❌ **Don't**: Mix authorization logic with business logic
✅ **Do**: Keep permissions in separate adapter

❌ **Don't**: Add `ignore_permissions=True` parameter (Frappe anti-pattern)
✅ **Do**: Use `SystemContext` for elevated operations, or mark DocType as `requires_auth=False`

❌ **Don't**: Use `sudo()` or implicit elevation (Odoo anti-pattern)
✅ **Do**: Explicit `system_context()` with audit logging

❌ **Don't**: Return linked documents without permission check (data leakage)
✅ **Do**: Check permission on Link fields before including related data in response

❌ **Don't**: Allow bulk delete/update to bypass RLS
✅ **Do**: Apply RLS filters to all bulk operations (`DELETE WHERE ...`, `UPDATE WHERE ...`)

---

## Edge Cases to Handle

### Child Table Permissions
- [x] Define: Child tables inherit parent's RLS by default (`Meta.is_child_table = True`)
- [x] Child rows are not independently permission-checked (`api_resource = False`)
- [x] Loading parent loads all its children (no separate RLS query)
- [x] Document: If independent child access needed, make it a separate DocType (tests in `test_child_table_permissions.py`)

### Link Field Data Leakage
- [x] When serializing a doc with Link fields, do NOT auto-include linked doc data (Link fields store UUID, not embedded objects)
- [x] If linked doc data is needed, make a separate API call (permission checked) - documented pattern
- [x] Optional: `?expand=customer` param design documented (checks permission before expanding)

### Bulk Operations & RLS
- [x] `GenericRepository.delete_many_for_user(filters)` applies RLS
- [x] `GenericRepository.update_many_for_user(filters, data)` applies RLS
- [x] User can only bulk-modify docs they have access to (tests in `test_bulk_operations_rls.py`)