Database Migrations

Framework M uses Alembic to manage database migrations. It supports both automatic discovery of DocTypes and selective migration based on "installed apps".

Migration Anatomy

Migrations are stored in the host application's directory or within modular library paths.

• alembic/env.py: Orchestrates how DocTypes are discovered and how MetaData is built.
• alembic/versions/ or migrations/: Contains the generated migration scripts.
• alembic.ini: Configuration file for Alembic.

Discovery Strategies

Framework M supports two modes of DocType discovery for migrations:

1. Selective Discovery (Recommended for Production)

This mode mimics Frappe's behavior. Migrations ONLY manage DocTypes from apps specified in the INSTALLED_APPS list.

How to trigger:

• Use the --apps flag in the CLI:

  m migrate create "added finance" --apps "finance,m_imprest"

• Or set the INSTALLED_APPS environment variable:

  export INSTALLED_APPS="finance,wms"
  m migrate status

2. Automated Entry-Point Discovery

In modular deployments, Framework M scans for apps registered via the framework_m.apps entry point.

Example pyproject.toml configuration:

[project.entry-points."framework_m.apps"]
wms = "wms.plugin:app"

The system will automatically find these apps and their associated DocTypes during the migration scan.

MIGRATIONS_DIR Discovery Sequence

When scanning an app, the engine looks for migration files in the following order:

1. Custom Directory: A migrations/ directory at the package root.
2. Alembic Default: The standard alembic/versions/ path.
3. Fallback: Skips if no migration directory is found.

Sync vs. Migrate (The Dual-Track Approach)

To provide a Frappe-like "it just works" experience while maintaining production-grade control, Framework M uses a dual-track strategy:

1. The "Sync" Track (Declarative / Declarative DDL)

Use m migrate sync when you want the database to exactly match your current DocType definitions.

• Behavior: Compares current DocTypes with the DB schema and applies CREATE TABLE / ALTER TABLE immediately.
• Safety First: The sync engine is additive-only. It will NOT drop tables or columns found in the database but missing from code, as it lacks the physical code to execute destructive DDL.
• Verification: Use m migrate verify to identify "ghost" tables or columns that are safe to prune manually via Alembic.
• Self-Contained: No migration files are created. It is entirely driven by code.
• Use Case: Rapid development and "Syncing" schema stability.

Smart Sync (Deterministic Optimization)

The sync command uses a deterministic hashing strategy to avoid expensive database reflection. On every run, it compares a checksum of your local DocType definition against a signature stored in the __schema_versions__ system table.

If the definitions haven't changed, the command skips the database scan entirely, resulting in near-instant execution. Use the --force flag to bypass this cache and force a full reflection. For more details on how this works, see the Deterministic Schema Migrations explanation.

Migration Hooks

Each app can implement before_sync and after_sync hooks in its migrations/__init__.py file to handle custom setup or cleanup logic. See the Migration Hooks Reference for more details.

2. The "Migrate" Track (Imperative / Versioned Patches)

Use the standard commands (create, run) when you need to ship ordered patches or logic-heavy transitions (data migrations, complex index changes).

• Behavior: Uses version files inside each app's migration folder.
• Traceability: Provides a clear audit log of every change applied to the database.
• Use Case: Production rollouts and complex data transformations.

Comparisons with Frappe

Concept	Frappe	Framework M
Auto-Sync	bench migrate (sync phase)	m migrate sync
Manual Patches	patches.txt + Python scripts	m migrate run + Alembic scripts
Modularity	Distributed patches.txt in each app	Distributed alembic/versions or migrations/ in each app
Dependencies	Handled via after_install or global order	Handled via Alembic's depends_on

Virtual DocTypes (Schema-Less Support)

If you need a DocType for API exposure or SDUI modeling but do NOT want a corresponding database table, use the create_schema flag:

class MyVirtualDocType(DocType):
    class Meta:
        create_schema = False  # The Migration Engine will skip this DocType

Virtual DocTypes are typically used with custom repositories that implement the RepositoryProtocol for alternative storage (e.g., Redis, YAML, or external APIs).

Safe-by-Default Deployment

The Migration Engine implements a Positive Table Identification model to prevent accidental data loss.

• Managed Tables: Only tables corresponding to registered DocTypes are considered "managed".
• External Tables: Tables like alembic_version or third-party artifacts are automatically ignored and will never be suggested for dropping.
• Explicit Drops: Destructive operations are gated behind the --allow-drop flag:

  m migrate sync --allow-drop

Commands

Command	Description
m migrate all	(Recommended) Runs both versioned patches (run) and declarative sync.
m migrate sync [--force]	Direct DocType -> DB synchronization (no files).
m migrate verify	Performs a dry-run comparison between code and DB to assess risk.
m migrate init	Initialize Alembic for this project.
m migrate create "msg" [--app ...]	Generate a versioned patch for a specific app.
m migrate status	Show current database revision and pending status.

Dry-Run Verification

Before executing a sync or a migration, use m migrate verify. This command categorizes schema changes based on risk:

• 🟢 SAFE: Non-destructive changes (e.g., adding a nullable column, creating a new table).
• 🟡 WARNING: Changes with potential for data loss (e.g., shrinking a VARCHAR length).
• 🔴 BREAKING: Destructive changes requiring a phased approach (e.g., dropping a column).

Production Migration Strategy

To ensure database stability, we classify all schema changes based on their risk level:

Change Classification

Category	Examples	Auto-Apply (Dev)?	Production Strategy
Safe	Add nullable column, add table, add index	✅ Yes	Apply directly (m migrate)
Review	Add non-nullable column, change type (compatible)	⚠️ Generate only	Review → Staging → Prod
Dangerous	Drop column, rename column, change type (incompatible)	❌ No	Manual migration with data handling

Zero-Downtime Deployment Strategy (DevOps)

By decoupling sync and run, Framework M enables advanced zero-downtime workflows:

1. Stage 1: Safe Sync - Run m migrate sync (declarative) before updating application code. This adds new nullable columns or tables that don't break existing code.
2. Stage 2: Code Rollout - Update application instances to the new version.
3. Stage 3: Data Patches - Run m migrate run (imperative) to perform data transformations or breaking schema changes once the new code is fully operational.

Tips for Monorepos

In a monorepo, it is best to run migrations from the main app (apps/business-m) but specify which libs should be included in the schema using the --apps flag.