Concepts

This section provides a deep dive into the fundamental concepts and recurring patterns used across Framework M.

8.1 The DocType Pattern

At the core of Framework M is the DocType (Document Type), a metadata-driven approach to defining business entities.

• Pydantic Metadata: Schemas are defined as Pydantic models, providing runtime validation and IDE type-safety.
• Metadata Endpoints: Global metadata registries allow generic UI components (like the Desk) to automatically render forms and lists without manual frontend coding.

8.2 Dependency Injection (DI) as Universal Composer

Framework M uses the dependency-injector library as its primary orchestration mechanism. This provides a unified composition layer across all execution contexts, ensuring that the same services and adapters are available regardless of how the system is invoked:

• API Layer: Services are wired into request handlers via the DI container.
• CLI Framework (Cyclopts): Services are provided to terminal commands.
• Background Jobs (Taskiq): Workers bootstrap the same DI container to process jobs.
• Interactive REPL: Developers can access the full service layer in a shell.
• Testing (Pytest): Providers are overridden via fixtures for mocking.

8.3 Hexagonal Adapters & Repository Pattern

Business logic is decoupled from infrastructure through formal Protocols (Interfaces).

• Repositories: Standardized CRUD interfaces for data access.
• Controllers: Where DocType-specific business logic and lifecycle hooks (validate, before_save, after_save) live.
• Adapters: Concrete implementations for specific databases (SQLAlchemy), message brokers (NATS), or caches (Redis).

8.4 Plugin Discovery via Python Entrypoints

Framework M discovers adapters and plugins dynamically through Python entrypoints in pyproject.toml. This allows for a "Zero-Cliff" future where any part of the system can be extended or replaced by a third-party package without modifying the core framework code.

8.5 Unit of Work (UoW)

The Unit of Work (UoW) is a behavioral pattern used to maintain a list of objects affected by a business transaction and coordinate the writing out of changes and the resolution of concurrency problems.

In Framework M, the UoW acts as the transactional orchestrator:

• Atomic Operations: It ensures that changes across multiple repositories (e.g., updating an Order and decreasing Stock) are committed atomically within a single database transaction.
• Adapter Coordination: It manages the lifecycle of database sessions and message broker connections, ensuring that events are only emitted to NATS JetStream after the database transaction has successfully committed.
• Rollback Safety: If any part of the unit fails, the UoW guarantees a clean rollback across all participating adapters to prevent data corruption.

8.6 Frontend Modular Architecture (MFE)

The frontend is a modular, high-performance architecture centered around the Desktop Workspace Shell and external Plugins. Depending on the deployment needs, it supports two distinct orchestration modes:

• Distributed MFE Mode: Utilizing a Micro-Frontend approach, the framework-m-desk acts as a host that dynamically loads "Plugin MFEs" via Vite-powered module federation. This is ideal for large, independently-deployable macroservices.
• Monolith Single Bundle Mode: For smaller projects or unified deployments, the framework-m-vite-plugin can automatically collapse all discovered plugins into a single, optimized production bundle. This simplifies deployment by serving everything from a single static directory.
• Tamagui Styling Engine: The framework uses Tamagui as its primary styling and layout engine, providing performance-optimized CSS-in-JS that remains compatible with React Native.
• MComponents Abstraction: To avoid vendor lock-in, all UI primitives (Tabs, Modals, Tables) are wrapped and exposed via MComponents. This ensures that the underlying engine can be swapped or upgraded without modifying plugin code.
• Token-Driven Themes: Architecture is strictly token-based (Themes, Spacing, Typography), enabling unified customization ("White Labeling") across the entire system by simply updating the framework-m-ui theme configuration.
• Host-Plugin Bridge: Communication between the shell and plugins is handled through the Plugin SDK, which provides formal JS protocols for sidebar registration, menu items, and global state access.

8.7 Virtual DocTypes (Data Abstraction)

"Virtual DocTypes" allow DocTypes to be backed by any data source, not just the primary SQL database.

• SQL Binds: For other SQL databases (e.g., Legacy MySQL), we use SQLAlchemy Binds (Meta.db_bind).
• Non-SQL Backends: For external APIs, NoSQL (MongoDB), or System resources (RAM/Files), we use Custom Repositories.
  • Mechanism: The App registers a custom implementation of RepositoryProtocol for that DocType.
• Consistency: For Virtual DocTypes backed by non-SQL stores, we recommend the Outbox Pattern or Saga Pattern to guarantee eventual consistency across primary and virtual data sources.

8.8 Optimistic Concurrency Control (OCC)

Framework M supports opt-in OCC to prevent lost updates in valid-first-wins scenarios.

Opting In

A DocType participates by setting Meta.concurrency = "optimistic":

class Invoice(DocType):
    title: str

    class Meta:
        concurrency: ClassVar[str] = "optimistic"

For participating DocTypes, SchemaMapper adds a _version integer column (NOT NULL DEFAULT 0). DocTypes that do not opt in are completely unaffected: no column, no token required, no CAS check.

The Reviewed-Snapshot Token

_version is a public token, not a hidden implementation detail:

• Reads expose it. GET /{doctype}/{id}, list responses, and repository get()/list_entities() all surface _version, and EntityRowMapper stashes it on the entity (entity.__dict__["_version"]).
• Writes must present it. A single-entity update must supply the token the caller last reviewed, either as the API _version field or via the repository save(..., version=n) argument. The router strips _version from the payload before assigning fields, so it is never treated as a settable document field.
• The server advances it atomically on a successful write and returns the new value to the caller.

If a caller omits the token on a participating DocType, the repository falls back to a token carried on a prior read of that entity. If neither is available, it raises MissingVersionError rather than silently overwriting.

Compare-And-Swap

The repository performs a single conditional update:

UPDATE invoice SET ..., _version = :expected + 1
WHERE id = :id AND _version = :expected

• Row affected → the write succeeded; the entity's _version is advanced.
• Zero rows affected → a competing writer won. The repository distinguishes the two failure modes:
  • StaleEntityError — the row still exists but _version moved. Carries expected_version and current_version. The caller should re-read and retry.
  • VersionConflictError — the row is gone (deleted mid-update).

FOR UPDATE row locks are not used for this. They do not provide a reviewed-snapshot token to the client, and SQLite compiles the flag without taking a real row lock. The conditional UPDATE is the mechanism; it is portable across SQLite, PostgreSQL, MySQL/MariaDB, Oracle, and MSSQL.

HTTP Mapping

Condition	Error	Status
_version does not match stored value	StaleEntityError	409 Conflict
Participating DocType updated with no token	MissingVersionError	428 Precondition Required

Both are subclasses of VersionConflictError → FrameworkError, and both are registered explicitly on the app so they produce typed responses instead of a generic 500. A 409 response includes the current stored version so the client can refresh and re-issue.

Participation Boundaries

Path	Participates?	Notes
GET / list	Exposes token	Yes — the token source.
Single-entity PUT / save()	Yes	Requires a token; CAS enforced.
bulk_set_value()	No (intentional)	Bulk field updates bypass OCC.
bulk_create()	No	Inserts cannot conflict.
Child-table rows	Follows the child model's own Meta.concurrency

The bulk_set_value exception is deliberate: bulk operations are used for administrative or system-wide updates that intentionally broadcast a value to many rows and do not carry per-row validated snapshots. Code that must not lose concurrent updates should use the single-entity path. This exception is part of the contract, not an oversight.

Enabling OCC on an Existing Table

Framework M does not write migrations for consumer applications — the migration is app-owned. For tables that already contain rows, add the column with a server default so existing rows are backfilled to 0:

-- PostgreSQL / MySQL / MariaDB / SQL Server
ALTER TABLE invoice ADD COLUMN _version INTEGER NOT NULL DEFAULT 0;

-- SQLite (ADD COLUMN NOT NULL requires a non-NULL default)
ALTER TABLE invoice ADD COLUMN _version INTEGER NOT NULL DEFAULT 0;

Or via Alembic in the app's own migration tree:

def upgrade() -> None:
    op.add_column(
        "invoice",
        sa.Column("_version", sa.Integer(), nullable=False, server_default="0"),
    )

warning

Adding the column as NOT NULL without a default fails on non-empty tables, and a nullable column leaves existing rows as NULL, which permanently breaks compare-and-swap (WHERE _version = :expected never matches NULL). Always use NOT NULL DEFAULT 0, then optionally drop the server default once all rows are backfilled.

The framework's own m migrate sync path applies the same DEFAULT 0 rule automatically when it detects the column is missing.

8.9 Domain Events & Audit Log

Every state change in the system emits a Domain Event via the EventBusProtocol.

• Standard Events: doc.create, doc.update, doc.delete, doc.submit.
• Audit Trail: The "Audit Log" is a built-in service that subscribes to doc.* events and persists them to a SystemLog table. This decouples logging from the business transaction.

8.10 Controller Lifecycle (Hooks)

In a Hexagonal architecture, lifecycle hooks are placed on the Controller (Service) rather than the Model.

• Hooks: validate, before_save, after_save, on_submit.
• Execution: The GenericRepository invokes these hooks on the registered Controller during the transaction lifecycle.

8.11 The Studio Architecture (Twin-Mode)

The Studio provides a visual development environment that maintains a single source of truth in Source Code.

• Mode 1: Local Bridge: Runs locally and writes directly to *.doctype.py files using LibCST to preserve manual code changes and comments.
• Mode 2: Cloud Studio: A hosted environment that interacts with the project via a Git Adapter, committing and pushing changes directly to the repository.

8.12 Extensibility & Overrides

Framework M avoids monkey-patching in favor of Dependency Injection and Python Entrypoints.

• Overrides: Any framework Protocol (Auth, Storage, etc.) can be overridden by a custom implementation registered via [project.entry-points."framework_m.overrides"] in a package's pyproject.toml.
• Boot Phase: During startup, the DI container checks these entrypoints and binds custom implementation to the kernel.

8.13 Governed Output & Observability

Framework M enforces a Zero-Print Policy to ensure that all system output is either human-optimized (CLI) or machine-optimized (Logs), and always governed by the security redaction layer.

• CLIConsole: A centralized Rich-based utility for terminal output. It handles consistent styling (Success, Error, Warning) and ensures that human-facing diagnostics are clean and readable.
• LoggingProtocol & Standard Logging: A structured logging interface (backed by structlog) used for all system events. Standard Python logging.getLogger(__name__) is also fully supported as it is intercepted by our Unified Bridge and routed through the same redaction pipeline.
• Unified Bridge: During bootstrap, the framework intercepts standard Python logging and Uvicorn logs, routing them through the unified structlog pipeline for consistent observability across the entire stack.

8.14 Architectural Isolation & API Prefixing

To support a seamless transition from monolithic to distributed Macroservice architectures, Framework M implements a dynamic API prefixing strategy.

• Dynamic Prefixing: Micro-frontends (MFEs) are aware of their "service context" via build-time injection. They dynamically construct API endpoints (e.g., /api/{service}/v1) at runtime.
• Service Namespacing: The frameworkMDataProvider supports namespaced resource resolution (e.g., finance/PurchaseInvoice). This allows a single frontend shell to coordinate with multiple backend macroservices by routing requests to the correct service-scoped endpoint.
• Zero-Cliff Migration: This pattern ensures that the same MFE code can operate in both a monolithic deployment (where all services share an /api/v1 prefix) and a distributed deployment (where each service has its own namespace), without requiring code modifications.
• Protocol-Agnostic Resolution: The system handles API, Metadata, and WebSocket endpoints through a unified normalization layer, ensuring consistent prefixing even when using absolute base URLs or CDN-hosted assets.

8.15 Transactional Outbox Pattern

To achieve reliable data propagation without tight database coupling or distributed transactions, Framework M implements a Transactional Outbox Pattern.

• Atomicity: Changes to primary tables and the staging of external event messages are committed within the same database transaction.
• Outbox Table: Pending operations are saved as OutboxEntry records with target, payload, status (pending, processed, failed), and retry metadata.
• Async Dispatching: A dedicated process_outbox worker polls the outbox database table periodically, dispatches events to the primary EventBus, and marks entries accordingly, guaranteeing at-least-once delivery semantics.

8.16 Background Workers & Stateless Scheduling

Periodic background tasks and scheduled cron jobs are integrated directly into the CLI worker environment.

• Embedded & Stateless Scheduling: Instead of a standalone stateful daemon, scheduling is driven by a pluggable TickerProtocol (e.g., InProcessTickerAdapter running inside m worker). The ticker wakes up on time boundaries, formats a unique time-bucketed deduplication_id, and enqueues jobs dynamically.
• Broker-Level Deduplication: Multiple worker processes run concurrent ticker loops safely. The backing queue/broker (like NATS JetStream) uses the deduplication_id to discard duplicate triggers at the transport layer, ensuring exactly-once execution without database or application locks.
• Immediate Async Tasks: High-throughput operations, such as sending emails, are dispatched immediately as single background tasks (e.g., send_email) via the JobQueueProtocol, avoiding latency and database-polling overhead.

8.17 Nested Set Tree Management and Tree Scope

For hierarchical data structures (like Account charts, Warehouse trees, and Categories), Framework M provides built-in nested set management.

• Nested Set (lft/rgt): Optimized tree manipulation via bounds shifting. Root and descendant insertions/updates shift bounds in database transactions rather than rebuilding whole trees.
• Tree Scope Fields: To support multi-tenant or domain-isolated tree structures, tree_scope_fields can be declared in the DocType Meta configuration. This forces tree boundary shifts and parent lookups to be isolated dynamically within specific scopes (e.g., tenant_id).

8.18 User Self-Service Federation Lock

When an application is configured in Federated Mode, the external Identity Provider (such as Keycloak or Azure AD) serves as the Single Source of Truth (SSoT) for user profiles.

To prevent data drift and JIT synchronization conflicts, Framework M enforces a federation lock:

• Local self-service profile and password update endpoints are disabled.
• The user is redirected to the upstream IdP's configured profile_url for all profile edits.
• Standard forms render in read-only mode in the frontend shell, providing a link to redirect users upstream.

8.19 Enterprise Data Handover

During employee offboarding or administrative restructuring, document and asset ownership must be transferred between users. To prevent breaking business rules:

• Framework M rejects raw database SQL updates (e.g., blanket UPDATE owner_id = ... statements).
• Instead, the OwnershipTransferService delegates ownership changes to each document type's repository and controller lifecycle hooks (e.g. before_transfer_ownership and after_transfer_ownership).
• Historical logs (like AuditLog and ActivityLog) are marked immutable and explicitly excluded from transfers to preserve compliance history.

8.20 OIDC Frontchannel Logout

In multi-application federated environments, logging out from one application should invalidate the user's session across all other active client applications.

• Iframe Invalidation: The central IdP invalidates the user's central session and loads a hidden page rendering logout iframes pointing to each client's frontchannel logout URI.
• BFF Session Destruction: Framework M's web adapter intercepts the browser-level frontchannel iframe call (which includes the user's cookies), destroys the corresponding local session record in the SessionStore, and sends headers to clear browser session cookies.

8.21 Row-Level Security (RLS) & Multi-Tenant Filtering

In single-database multi-tenant applications, both tenant isolation and application-level access control (like user or team restrictions) are enforced via Row-Level Security (RLS) query filters. Framework M divides the management of these filters:

• Framework-Managed Filtering (Tenants): The framework automatically injects tenant-isolation filters (e.g., WHERE tenant_id = :tenant_id) globally at the repository boundary to prevent cross-tenant leaks.
• Developer-Managed Filtering (Users/Teams): Developers can configure additional, granular RLS filters using DocType metadata:
  • rls_field specifies the column in the database (e.g., owner, team_id) to filter on.
  • rls_scope specifies the attribute name on the UserContext object (e.g., id, teams) to match.

8.22 Application-Layer Payload Encryption (ALPE)

Framework M protects sensitive request bodies from inspection by reverse proxies, WAFs, APM loggers, and ingress controllers through Application-Layer Payload Encryption (ALPE). The payload is encrypted in the browser using the server's public key and decrypted by PayloadDecryptionMiddleware before it reaches route handlers.

Four Security Modes

Mode	Behavior	Use Case
disabled	No encryption. Standard JSON bodies.	Local development, CI, internal mTLS services.
selective	Encrypts a configured list of routes (auth endpoints by default).	Standard SaaS compliance.
metadata	Encrypts DocType fields flagged with encrypted=True or pii=True in their schema.	Granular PII protection for Desk forms.
strict	Encrypts all mutating (POST, PUT, PATCH) requests.	Fintech, zero-trust enterprise.

Key Exchange

• A single GET /api/v1/security/jwks endpoint serves the active public encryption key (use: "enc") alongside signature keys.
• The frontend fetches and caches the key once per app load, avoiding per-request handshakes.
• Backend decryption uses the corresponding private key via the PayloadCryptoProvider protocol.

Field-Level Flags

DocType fields are flagged declaratively with standard Pydantic Field(..., json_schema_extra={"encrypted": True, "pii": True}). The metadata router exposes these flags in GET /meta/{doctype}, and the Desk form controller uses them to decide whether to encrypt a form submission.

caution

These flags must be applied to DocType models only. Litestar request/response DTOs (such as LoginRequest) cannot use unknown json_schema_extra keys because Litestar's OpenAPI generator validates them against its internal Schema object. Auth routes rely on selective mode route configuration instead.