Phase 08: Workflows & Advanced Features

Objective: Implement pluggable workflow engine, DocType overrides, app-defined ports, and advanced extensibility features.

---

1. Workflow System

1.1 Workflow Protocol

• Create src/framework_m/core/interfaces/workflow.py
• Define WorkflowProtocol:
  • async def start_workflow(doctype: str, doc_id: str, workflow_name: str)
  • async def get_workflow_state(doc_id: str) -> str
  • async def transition(doc_id: str, action: str, user: UserContext)
  • async def get_available_actions(doc_id: str, user: UserContext) -> list[str]

1.2 Internal Workflow Adapter

• Create src/framework_m/adapters/workflow/internal_workflow.py

• Create WorkflowState DocType:

  • name: str
  • workflow: str - Workflow name
  • doctype: str
  • document_name: str
  • current_state: str
  • updated_at: datetime

• Create WorkflowTransition DocType:

  • name: str
  • workflow: str
  • from_state: str
  • to_state: str
  • action: str - Action name
  • allowed_roles: list[str]
  • condition: str | None - Python expression

• Create Workflow DocType:

  • name: str - Workflow name
  • doctype: str - Target DocType
  • initial_state: str
  • states: list[dict] - State definitions
  • transitions: list[WorkflowTransition]

• Implement InternalWorkflowAdapter:

  • Load workflow definition
  • Validate transitions
  • Check permissions
  • Update state
  • Trigger hooks / Emit Event (Side effects via Event Bus only)

1.3 Temporal Workflow Adapter (Optional)

• Create src/framework_m/adapters/workflow/temporal_adapter.py
• Implement TemporalWorkflowAdapter:
  • Connect to Temporal server
  • Start workflows
  • Query workflow state
  • Signal workflows for transitions

---

2. DocType Overrides

2.1 Override Registry

• Update MetaRegistry:
  • Add register_override(base_doctype: str, override_class: Type[BaseDocType])
  • When loading DocType, check for overrides
  • Use override class if registered

2.2 Schema Extension

• Implement schema merging:
  • Combine fields from base and override
  • Override can add new fields
  • Override can modify field properties
  • Base fields cannot be removed

2.3 Table Alteration

• Update SchemaMapper:
  • Detect schema changes from overrides
  • Generate ALTER TABLE migrations
  • Add new columns
  • Modify column types (with caution)

2.4 Example Usage

# In custom app
from framework_m.public import User

class ExtendedUser(User):
    department: str = Field(description="Department")
    employee_id: str | None = None

# Register override
registry.register_override("User", ExtendedUser)

---

3. App-Defined Ports

3.1 Custom Protocol Registration

• Update Container to support app-defined protocols:
  • Allow apps to define custom protocols
  • Register via entrypoints
  • Other apps can override using container.override()

3.2 Example: Payment Gateway

# In e-commerce app - define protocol
class PaymentGatewayProtocol(Protocol):
    async def charge(amount: Decimal, token: str) -> str: ...
    async def refund(charge_id: str) -> bool: ...

# Default implementation
class StripeAdapter:
    async def charge(amount, token):
        # Stripe API call
        pass

# In e-commerce app's container
from dependency_injector import containers, providers

class EcommerceContainer(containers.DeclarativeContainer):
    payment_gateway = providers.Singleton(StripeAdapter)

# Register in pyproject.toml
[project.entry-points."framework_m.containers"]
ecommerce = "ecommerce.container:EcommerceContainer"

# Another app can override
# In custom_app
class PayPalAdapter:
    async def charge(amount, token):
        # PayPal API call
        pass

# Override in app initialization
ecommerce_container.payment_gateway.override(providers.Singleton(PayPalAdapter))

---

4. Child Tables (Nested DocTypes)

4.1 Child Table Support

• Add is_child flag to DocType Config
• Child tables don't have independent routes
• Stored as JSON or separate table with parent reference

4.2 Implementation

class OrderItem(DocType):
    item: str
    quantity: int
    rate: Decimal

    class Config:
        is_child = True

class Order(DocType):
    customer: str
    items: list[OrderItem]  # Child table

• Update SchemaMapper:

  • Create separate table for child
  • Add parent and parenttype columns
  • Add idx for ordering

• Update GenericRepository:

  • Save child records when saving parent
  • Delete old children and insert new ones
  • Load children when loading parent

---

5. Virtual Fields

5.1 Computed Fields

• Add @computed_field decorator:

  class Invoice(DocType):
      items: list[InvoiceItem]
  
      @computed_field
      @property
      def total(self) -> Decimal:
          return sum(item.amount for item in self.items)

• Virtual fields not stored in database

• Computed on load

• Included in API responses

---

6. Link Fields (Foreign Keys)

6.1 Link Field Type

• Add Link field type using json_schema_extra:

  from uuid import UUID
  from framework_m import Field
  
  class Order(DocType):
      customer: UUID | None = Field(
          description="Customer link",
          json_schema_extra={"link": "Customer"}
      )

6.2 Implementation

• Update SchemaMapper:

  • Detect link fields via json_schema_extra
  • Create foreign key constraint to target table
  • Reference target table's id column

• Add database enforcement:

  • Foreign key constraints enforce referential integrity
  • Works with SQLite and PostgreSQL (database-agnostic)
  • SQLite foreign keys enabled via PRAGMA

Implementation Notes:

• Link fields use UUID type (not string names)
• Metadata stored in json_schema_extra={"link": "TargetDocType"}
• SchemaMapper creates ForeignKey constraint: ForeignKey(f"{target_table}.id")
• SQLite requires PRAGMA foreign_keys=ON for constraint enforcement
• 14 comprehensive tests covering detection, constraints, validation, queries

6.3 Link Fetching

• Add fetch_from option:

  from uuid import UUID
  from framework_m import Field
  
  class Order(DocType):
      customer: UUID | None = Field(json_schema_extra={"link": "Customer"})
      customer_name: str | None = Field(
          default=None,
          json_schema_extra={"fetch_from": "customer.customer_name"}
      )

• Auto-populate field from linked document

• Update GenericRepository to fetch values before save

• Fetch values automatically on insert and update

• Handle null links gracefully

• Support multiple fetch_from fields

Implementation Notes:

• Fetch fields use json_schema_extra={"fetch_from": "link_field.target_field"}
• Values are fetched from linked document before insert/update
• Automatically syncs with link changes (prevents data inconsistency)
• Null links result in null fetch values
• Supports fetching from multiple different links
• 12 comprehensive tests covering detection, fetching, updates, edge cases

---

7. Naming Series (Human-Readable Names)

[!NOTE] > id vs name: The id field (UUID, primary key) is always auto-generated with no contention. The name field is an optional human-readable identifier (e.g., "INV-2024-0001"). Naming series applies to name, NOT the primary key.

7.1 Auto-Naming Configuration

• Add naming configuration to DocType:

  class Invoice(DocType):
      class Meta:
          name_pattern = "INV-.YYYY.-.####"  # INV-2024-0001

• Implement naming patterns:

  • .YYYY. - Year (2026)
  • .MM. - Month (01-12)
  • .DD. - Day (01-31)
  • .#### - Sequential number with padding (0001, 0002...)
  • {field} - Field value from entity

Implementation Notes:

• Dots (.) in patterns are separators and removed in output
• Pattern "INV-.YYYY.-.####" generates "INV-2026-0001"
• Pattern "TASK-.{priority}.-.####" with priority="HIGH" generates "TASK-HIGH-0001"

7.2 Implementation (Optimistic Approach)

• Name generation happens AFTER insert (id already assigned):

  # 1. Insert with UUID id (no contention)
  # 2. Generate name using current counter
  # 3. If duplicate (unique constraint), retry with next number

• For high-volume DocTypes, use PostgreSQL sequences:

  class HighVolumeInvoice(DocType):
      class Meta:
          name_pattern = "sequence:invoice_seq"  # Uses DB sequence

7.3 Counter Storage

• Create NamingCounter DocType:
  • prefix: str - e.g., "INV-2024-"
  • current: int - Current counter value
  • Note: No row-level locking needed. Use optimistic update with retry.

[!IMPORTANT] > No Frappe Anti-Pattern: We do NOT use SELECT ... FOR UPDATE on a shared counter table. The id (UUID) is the primary key. The name field uses optimistic generation with retry on conflict.

---

8. Validation Rules

8.1 Field Validators

• Add Pydantic validators:

  from pydantic import field_validator
  
  class Invoice(DocType):
      total: Decimal
  
      @field_validator("total")
      def validate_total(cls, v):
          if v < 0:
              raise ValueError("Total cannot be negative")
          return v

8.2 Document Validators

• Use controller validate() hook:

  class InvoiceController(BaseController[Invoice]):
      async def validate(self):
          if self.doc.total != sum(item.amount for item in self.doc.items):
              raise ValidationError("Total mismatch")

---

9. Testing

9.1 Unit Tests

• Test workflow transitions (15 tests in test_workflow.py)
• Test DocType overrides (16 tests in test_meta_registry_overrides.py)
• Test child table operations (38 tests in test_child_tables.py)
• Test link field validation (16 tests in test_link_fields.py)
• Test naming series (29 tests in test_naming_series.py + validation tests)

9.2 Integration Tests

• Test full workflow lifecycle (6 tests in test_workflow_lifecycle.py)
• Test override schema migration (10 tests in test_override_migration.py)
• Test child table CRUD (12 tests in test_child_table_integration.py)
• Test app-defined ports (9 tests in test_app_ports.py)

---

Validation Checklist

Before moving to Phase 09, verify:

• Workflows can be defined and executed (15 unit + 6 integration tests)
• DocTypes can be extended via overrides (16 unit + 10 integration tests)
• Child tables work correctly (38 unit + 12 integration tests)
• Link fields create proper foreign keys (16 tests in test_link_fields.py)
• Naming series generates unique names (29 tests in test_naming_series.py)
• App-defined ports can be registered (9 tests in test_app_ports.py)

---

Anti-Patterns to Avoid

❌ Don't: Hardcode workflow logic in controllers ✅ Do: Use pluggable workflow adapter

❌ Don't: Modify core DocTypes directly ✅ Do: Use override mechanism

❌ Don't: Store child records as JSON only ✅ Do: Use proper relational tables

❌ Don't: Use row-level locking for naming (Frappe anti-pattern) ✅ Do: Use optimistic naming with retry on conflict