Skip to main content

RFC-0008: Compliance, Versioning, and Corrections

  • Status: Proposed
  • Author: @revant.one
  • Created: 2026-02-24
  • Updated: 2026-03-03
  • TSC Decision: Pending

Summary​

This RFC formalizes how Framework M handles complex auditing, document versioning, and corrections for both Submittable (immutable) and Master (mutable) DocTypes. It ensures high-integrity proof of change, temporal querying (Time Travel), and user-friendly correction flows while maintaining strict ledger integrity.

Motivation​

In business systems, documents have different requirements for immutability and record-keeping. Financial documents (Invoices) must be strictly immutable once submitted, yet operational realities (typos, changed transporters) require a compliant way to record updates.

This RFC aims to answer critical business and compliance questions:

  • Current State (99% use case): What does this document look like right now?
  • Audit: Who changed what, when, and most importantly—why?
  • Temporal Query: What did this document look like at time T? (e.g., "What were the payment terms on Jan 1st?").
  • Compliance Reporting: Show changes to documents after specific milestones (e.g., "Invoices edited after return filing").
  • Recovery & Restore: Revert to a previous state via a new append-only version.
  • Traceability: What automation or job (Job ID) triggered this specific update?

Detailed Design​

0. The Principle of Modularity & Sensible Defaults​

Framework M is designed to handle both simple CRUD apps and highly regulated enterprise flows. Therefore, compliance tracking is progressive, relying on the better, lightweight pattern as the sensible default.

  1. The Default (Lightweight Audit): By default, AuditMixin (Global JSON ActivityLog) provides out-of-the-box auditability ("Who changed what, when, and why") for Submittable DocTypes. This guarantees perfect trace-ability with minimal database footprint for small/mid-sized businesses.
  2. Deployment Configuration (Runtime Level): Even if a developer includes VersionedMixin, the ORM will not provision these shadow tables unless the specific Deployment has enabled strict temporal versioning via environment configuration (TOML/Env Vars).

This dual-layer guarantees that:

  • The lightweight, efficient pattern (AuditMixin) protects every installation by default.
  • A large Enterprise using the exact same app can escalate to strict temporal SQL ledgers (VersionedMixin) with zero code changes, while Indie deployments face no database bloat.

1. The Audit Timeline (AuditMixin)​

For most "Who changed what?" requirements, we rely on a centralized audit log.

  • Global Storage: All audit events across the system are stored in a single global table (driven by the ActivityLog DocType).
  • JSON Diffs: When a document with AuditMixin is updated, a JSON patch of the changed fields is appended to the global log.
  • Append-only: Entries cannot be edited or deleted once saved.
  • Contextual Meta: Captures the user_id, timestamp, and job_id (to distinguish system-driven automation from human edits).

Because the ActivityLog stores changes as JSON in a single massive table, it is optimized for rendering the UI timeline (the "History" tab on a specific document form), not for running complex analytical SQL queries.

2. Audit Versioning (Shadow Tables via VersionedMixin)​

For high-compliance environments requiring temporal SQL querying (e.g., "Sum all invoice totals for Q1 as they existed on April 1st"), parsing JSON from a global ActivityLog is too slow and un-indexable. For these scenarios, Framework M introduces Versioned Storage.

  • Shadow Table Pattern: When a DocType opts-in via VersionedMixin, the ORM provisions a dedicated shadow table (e.g., Invoice_history) with the exact same schema as the primary table, plus a sequential version_no.
  • Querying:
    • Standard queries hit the primary table (Invoice).
    • Temporal/Time-Travel queries use the dedicated history table.
  • Configurability:
    • Submittable Docs: Versioned on Submit, Cancel, and Amend transitions.
    • Master Data (Items/Parties): Configurable per field (e.g., track Price change history, but ignore minor description edits).
  • Data Retention & Pruning: Version storage is configurable. Organizations can define pruning policies (e.g., "Prune after 7 years"). To manage scale, history tables can be routed to a separate high-capacity database (e.g., MongoDB for logs via the MX Pattern). (Note: Archiving, rehydrating, and cluster management of this secondary database are considered standard DBA operational tasks and fall outside the scope of Framework M's core ORM).

3. Sibling Extension Pattern (Metadata Overlay)​

For "volatile" metadata that must remain mutable after document submission (e.g., Driver Name, Vehicle Number, ETA) but should not lived on the immutable core document:

  • Sibling DocType: A separate model that explicitly declares itself as an extension of a parent. To differentiate a "Sibling" from a standard "Related Doc" (like a StockEntry merely linking to an Invoice), the Sibling's DocType Meta must include an extends or is_sibling attribute.
    • Example: LogisticsMeta links to Invoice AND has Meta(extends="Invoice"). This tells the UI to treat it as an editable overlay, not just a related list entry.
  • Overlay UI: Surfaced in the parent document view via the is_metadata_overlay meta tag in Studio, allowing the side-pane or a specific UI tab to remain editable even while the primary document is locked (docstatus=1).
  • Sync & Integration: Since siblings represent non-ledger, volatile metadata, they do not trigger standard ledger syncs post-submission. However, for specific enterprise needs (e.g., WMS sync), developers can implement on_metadata_update hooks in the controller.

🟢 Sibling Usage Guidelines:​

  1. Cross-Departmental Isolation: Use separate siblings when different teams (e.g., Logistics vs. Finance) need to attach distinct metadata without race conditions or shared mutable state.
  2. Third-Party Extensions: Sibling entities are the primary way for plugins to extend core DocTypes without modifying the base schema.

🔴 Anti-Patterns to Avoid:​

  1. Fragmented Core Data: Never move core logic (Price, Quantity, Tax) out of the parent into a sibling. Siblings are for metadata, not ledger data.
  2. Sibling Proliferation: Avoid creating more than 1-2 sibling DocTypes per parent. If you need 3+, it often indicates that your "Metadata" is actually a first-class Entity (like a DispatchNote or Payment) and should be modeled as such.
  3. Expensive Reporting: Beware that over-using siblings increases the complexity of standard SQL reports due to additional JOINs.

4. Transition State Machine​

Framework M defines the following standard transitions for complex documents:

TransitionActionOutcome
Submit → Cancel → AmendTraditional UXCreates a new Document ID with copied data; old doc stays cancelled.
Submit → Cancel → RestoreRollbackReverts state to SUBMITTED; creates a new audit trail entry linking back to the previous snapshot.
Submit → Edit → Re-SubmitDirect Versioned EditInserts a new row version in history but retains the primary Document ID. Used when external integrations mandate ID persistence.

5. The Correction Wizard (UX)​

To automate these patterns and make them frictionless, the UI provides a unified flow:

  1. Trigger: User clicks "Correct" on a submitted document.
  2. Context Check: Controller hooks (before_edit_after_submit) decide if a full "Amend" is required OR if a "Direct Versioned Edit" is allowed.
  3. Mandatory Reason: captured and saved to the Audit Timeline.
  4. Execution: System handles the cloning/archiving and redirects the user to the active revision.

6. Implementation Architecture Details​

To successfully implement these patterns, the framework enforces the following low-level mechanisms:

Deployment-Level Configurability (TOML / Env Vars)​

To prevent database table bloat for simpler deployments, VersionedMixin shadow tables are gated by environment-level configurations, not database settings tables (which would create a chicken-and-egg schema generation loop).

  • Configuration Source: The Database Adapter checks the Deployment's config.toml or environment variables (e.g., FRAMEWORK_M_STRONGSYNC_VERSIONING=true or [compliance]\nversioning_enabled = true).
  • Runtime Behavior: During install_doctype or schema_sync, if this config is false, the ORM skips creating the _history table entirely, gracefully degrading to standard AuditMixin JSON tracking.

Database Schema Lifecycle (Shadow Tables)​

When a DocType includes the VersionedMixin, the Database Adapter's install_doctype or schema_sync method automatically provisions the shadow table.

  • DDL Strategy: The ORM executes CREATE TABLE {table_name}_history (LIKE {table_name}) followed by ALTER TABLE {table_name}_history ADD COLUMN _version_no INT.
  • Database Routing (Anti-Bloat Strategy): To prevent primary relational database bloat, the Database Adapter must support connection routing for history tables. If the deployment config.toml specifies an audit_db_url, the Adapter must run the DDL and route all subsequent _history inserts to that separate database node.
  • Schema Evolution: Any subsequent schema alterations (adding/dropping columns) to the primary table must dynamically cascade to the _history shadow table, utilizing the routed connection if configured.

Controller Hooks (Audit & Version DML)​

Logging and versioning must occur seamlessly within the core BaseController lifecycle, bound to the same UnitOfWork transaction (unless routing strictly dictates asynchronous logging for performance).

  • AuditMixin & JSON Diffs: During before_save, the controller captures doc.model_dump(). During after_save, it diffs this against existing_doc. Non-empty diffs are automatically pushed to the ActivityLog repository (routed to audit_db_url if defined).
  • VersionedMixin: During after_save (and after_insert), an exact copy of the row data is inserted into {table_name}_history, incrementing the _version_no (routed to audit_db_url if defined).

State Machine Enforcement (The Correction Flow)​

Since BaseController._validate_submitted_changes hard-blocks raw edits to docstatus=1 documents via ImmutableDocumentError, there is no bypass flag. Instead, the controller strictly only permits changing docstatus from 1 (SUBMITTED) to 2 (CANCELLED). The 3 primary UI flows orchestrated by the CorrectionService are explicitly governed by the DocType's Meta configuration (specifically a correction_strategy option), ensuring developers have absolute control over which rules apply:

  1. Standard Amend (correction_strategy = "amend_only"): The universal default for financial documents. Cancels the root document and creates a new Draft (docstatus=0) under a new incremented human-readable identifier (name, e.g., INV-0001-1), generating a new machine primary key (id).
  2. Restore: Cancels the root document and reverts back to the previous snapshot, adding an audit trail.
  3. In-Place Edit (PK Retention) (correction_strategy = "in_place") (The Ecommerce / Ops Middle-Ground): Reserved for operational workflows where external systems crash if the machine id or human name changes.
    • The Acrobatics: Instead of inserting a new row, the CorrectionService executes a strict transaction: It deep-copies the current submitted row into the immutable {table}_history shadow table. Then, it explicitly updates the existing primary row back to docstatus=0 (Draft), retaining the exact same id and name. After the user edits and re-submits, another history row is added. The primary table mutates to satisfy operational APIs, but the cryptographically complete ledger is preserved in the append-only _history shadow table.

Temporal Querying (As-Of-Date)​

The standard repository must support querying the shadow table transparently.

  • Repository Interface: repo.get(doc_id, as_of=datetime) or repo.get_list(as_of=datetime).
  • Query Strategy: The Database Adapter builds queries targeting the _history table, selecting the row with the MAX(_version_no) where the modified timestamp is <= as_of.
  • Missing Config Fallback: If the Developer attempts a Temporal Query (as_of=...), but the Deployment Configuration has disabled versioning_enabled, the Repository must raise a TemporalQueryNotSupportedError (or similar framework-level exception). It must not silently return the current state, as this violates the API contract and creates silent business logic failures.

Drawbacks​

  • Storage Overhead: Versions and shadow rows will significantly increase database size for high-velocity documents.
  • Query Complexity: Reconstructing temporal states across linked documents (e.g., a versioned Invoice linked to a versioned Customer) requires sophisticated join/filter logic.
  • Performance: Generating JSON diffs on every save adds minor overhead to the database controller.

Alternatives​

AlternativeProsConsWhy Not
allow_on_submitSimple to implementNo audit/integrityBreaks cryptographic and legal guarantees of the ledger.
Manual Amend OnlyNo new database schemasVery tedious/unfriendlyHigh friction for operators; leads to users hacking the DB.
DB-Level TriggersFramework neutralLacks business "Why"Cannot capture the correction reason or trigger app-level notifications.

Migration​

  • Existing Docs: Existing documents without AuditMixin will need a migration to create history tables if versioning is retrospectively enabled.
  • TimelineMixin: Existing comment-heavy DocTypes will need to be refactored to the stricter AuditMixin.
  • Breaking Changes: This RFC enforces ImmutableDocumentError on any save attempt to SUBMITTED docs without going through the Correction Flow or Sibling pattern.

Implementation Plan​

  • Phase 1: Refactor TimelineMixin to AuditMixin with JSON diff support.
  • Phase 2: Add VersionedMixin and @history table support to the database adapters.
  • Phase 3: Implement the "Time Travel" utility in the SDK and Frontend.
  • Phase 4: Deliver the "Correction Wizard" UI component and Controller hooks.

References​