Skip to main content

Phase 06: Built-in DocTypes & Core Features

Objective: Implement essential built-in DocTypes (User, Role, Permission) and core features like file attachments, audit logging, and printing.

1. Identity & Access Management (PIM)​

1.1 Identity Provider Protocol​

  • Create IdentityProtocol (Interface):
    • get_user(id: str) -> User
    • authenticate(credentials) -> Token
    • get_attributes(user) -> dict[str, Any] (ABAC replacement for get_roles)

1.2 User DocTypes (Pluggable)​

  • Mode A: Local (Indie):

    • LocalUser (SQL Table): Stores email, password_hash, full_name.
    • LocalIdentityAdapter: argon2 password hashing, JWT generation.

  • Mode B: Federated (Enterprise):

    • FederatedIdentityAdapter (Proxy):
      • No SQL Table required.
      • Hydrates from Auth Header (X-User-ID, X-Email) via hydrate_from_headers().
      • UserPreferences DocType stores settings locally.

1.3 User Controller (Adapter)​

  • Implement UserManager:
    • Delegates to configured IdentityProvider via IdentityProtocol.
    • Abstraction layer: user = await user_manager.get(id).
    • Additional create() method for Indie mode with password hashing.

1.4 Auth API (Indie Mode)​

  • Create GET /api/v1/auth/me:
    • Returns current user context (or 401).
  • Create POST /api/v1/auth/login (Indie Only):
    • Accepts username, password.
    • Validates against LocalUser via UserManager.authenticate().
    • Returns JWT token.
  • Create POST /api/v1/auth/logout:
    • Returns success message (JWT auth is client-side logout).

1.5 Authentication Strategies (Multi-Method)​

[!NOTE] Same user can authenticate via multiple methods (browser, API, CLI). All strategies resolve to the same UserContext.

  • Create AuthenticationProtocol (Interface):

    • supports(headers: Mapping) -> bool — Can this strategy handle this request?
    • authenticate(headers: Mapping) -> UserContext | None — Extract user from request.

  • Built-in Strategies:

    • SessionCookieAuth — Browser sessions via cookies.
    • BearerTokenAuth — JWT/OAuth2 access tokens.
    • ApiKeyAuth — X-API-Key header for scripts/integrations.
    • HeaderAuth — Federated mode gateway header hydration.
    • BasicAuth — HTTP Basic for CLI tools.

  • Strategy Chain (AuthChain):

    • Try each strategy in priority order.
    • First successful match wins.
    • Configurable order via framework_config.toml (create_auth_chain_from_config).

  • API Key DocType (Indie Mode):

    • ApiKey DocType:
      • key_hash: str — Hashed API key (never store plaintext, excluded from serialization).
      • user_id: str — Owner of the key.
      • name: str — Human-readable label.
      • scopes: list[str] — Optional permission scopes.
      • expires_at: datetime | None — Expiration.
      • last_used_at: datetime | None — Usage tracking.
    • POST /api/v1/auth/api-keys — Create new key.
    • DELETE /api/v1/auth/api-keys/{id} — Revoke key.
    • GET /api/v1/auth/api-keys — List user's keys.

1.6 Social Login (OAuth2/OIDC)​

[!IMPORTANT] Social login helps Indies avoid password storage and PII liability.

  • Option A: Built-in OAuth2 (Indie):

    • Use authlib for OAuth2 flows. (Full implementation deferred)
    • Supported Providers (config structure ready):
      • Google (well-known URLs configured)
      • GitHub (well-known URLs configured)
      • Microsoft (well-known URLs configured)
      • Generic OIDC (any compliant provider via get_oidc_well_known)
    • Configuration via framework_config.toml: 
      [auth.oauth]
      enabled = true
      providers = ["google", "github"]

      [auth.oauth.google]
      client_id = "${GOOGLE_CLIENT_ID}"
      client_secret = "${GOOGLE_CLIENT_SECRET}"

      # Generic OIDC provider example
      [auth.oauth.keycloak]
      client_id = "${KEYCLOAK_CLIENT_ID}"
      client_secret = "${KEYCLOAK_CLIENT_SECRET}"
      authorization_url = "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/auth"
      token_url = "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/token"
      userinfo_url = "https://keycloak.example.com/auth/realms/myrealm/protocol/openid-connect/userinfo"
      scope = "openid email profile"
    • Endpoints:
      • GET /api/v1/auth/oauth/{provider}/start — Redirect to provider.
      • GET /api/v1/auth/oauth/{provider}/callback — Handle callback. (Placeholder)

  • Option B: Federated (Enterprise):

    • Delegate to Keycloak / Authelia / Auth0 via HeaderAuth strategy.
    • Framework receives headers only (X-User-ID, X-Email).
    • Zero PII stored locally.

  • SocialAccount DocType:

    • provider: str — OAuth provider name.
    • provider_user_id: str — Unique ID from provider.
    • user_id: str — Links to local user.
    • email: str | None — For lookup (optional).
    • display_name: str — For UI display.
    • One User can have multiple SocialAccounts.

  • Passwordless Option (Requires Section 10: Email):

    • POST /api/v1/auth/magic-link — Send email with login link.
    • GET /api/v1/auth/magic-link/{token} — Verify and create session.
    • MagicLinkCredentials added to identity protocol
    • Token generation with HMAC-SHA256 signing

1.7 Session Management​

  • Session Storage:

    • Redis (Default): RedisSessionAdapter with JSON storage and TTL.
    • Database (Fallback): DatabaseSessionAdapter for indie apps.
    • Configurable via framework_config.toml: 
      [auth.session]
      backend = "redis"  # or "database"
      ttl_seconds = 86400  # 24 hours
      cookie_name = "session_id"
      cookie_secure = true
      cookie_httponly = true
      cookie_samesite = "lax"

  • Session DocType (database backend):

    • session_id: str — Unique session identifier.
    • user_id: str — Links to user.
    • expires_at: datetime
    • ip_address: str | None
    • user_agent: str | None

  • SessionProtocol Interface:

    • create() — Create new session
    • get() — Retrieve by ID
    • delete() — Remove session
    • delete_all_for_user() — Logout all
    • list_for_user() — List active sessions

  • Session API:

    • GET /api/v1/auth/sessions — List active sessions.
    • DELETE /api/v1/auth/sessions/{id} — Revoke session.
    • DELETE /api/v1/auth/sessions — Logout all.

1.8 PII Handling (Indie Guidance)​

[!TIP] Store less PII = Less GDPR/compliance burden.

  • Auth Mode Presets (AuthMode enum in core/pii.py):

    [auth]
    mode = "social_only"  # No local passwords, minimal PII
    # OR
    mode = "passwordless"  # Email magic links only
    # OR
    mode = "local"  # Traditional username/password
    # OR
    mode = "federated"  # External IdP, zero local PII

  • Minimal PII Pattern:

    • LocalUser stores only:
      • email (for login/notification)
      • display_name (for UI)
      • password_hash (only if mode = "local")
    • Do NOT store by default (PII_SENSITIVE_FIELDS):
      • Full legal name
      • Phone number
      • Address
      • Date of birth
    • Helper: is_sensitive_pii(field_name) to detect sensitive fields.

  • Data Deletion (DeletionMode enum):

    • DELETE /api/v1/auth/me — Delete user account (GDPR right to erasure).
    • GET /api/v1/auth/me/data — Export user data (GDPR data portability).
    • Configurable: Hard delete vs anonymize. 
      [auth.deletion]
      mode = "hard_delete"  # or "anonymize"

2. Tenancy & Attributes (Multi-Tenant Core)​

2.1 Tenant Protocol​

  • Create TenantProtocol (Interface in core/interfaces/tenant.py):
    • get_current_tenant() -> str
    • get_tenant_attributes(tenant_id) -> dict
    • TenantContext model for request context
    • Configuration helpers: is_multi_tenant(), get_default_tenant_id()

2.2 Tenancy Adapters​

  • Mode A: Single Tenant (Indie):

    • ImplicitTenantAdapter (adapters/tenant.py):
      • tenant_id="default" (configurable).
      • attributes={"plan": "unlimited", "features": "*"}.
      • get_context() returns TenantContext with is_default=True.
    • Factory: create_tenant_adapter_from_headers() auto-selects.

  • Mode B: Multi-Tenant (Enterprise):

    • HeaderTenantAdapter (adapters/tenant.py):
      • Extracts X-Tenant-ID from Gateway headers.
      • Extracts X-Tenant-Attributes (JSON) for feature flags, plan.
      • get_context() parses headers into TenantContext.
      • Benefit: Feature toggling per tenant without DB lookup.

2.3 Attributes (The New Roles)​

  • Goal: Replace "Roles" with refined "Attributes" (ABAC).
  • Implementation (in UserContext):
    • user.attributes (dict): Stores {"department": "sales", "level": 5, "roles": ["admin"]}.
    • user.get_attribute(key, default) — Get attribute value.
    • user.has_attribute(key, value) — Check attribute equality.
    • Legacy Support: user.has_role("x") checks both roles list AND attributes.roles.
    • is_system_user uses has_role("System") for consistency.

3. Permission Management (Pluggable)​

3.1 Permission Protocol​

  • Create PermissionProtocol (core/interfaces/permission.py):
    • PolicyEvaluateRequest - Stateless authorization request
    • PolicyEvaluateResult - Result with authorized, decision_source, reason
    • async def evaluate(request) -> PolicyEvaluateResult
    • async def get_permitted_filters(...) -> dict - RLS support
    • PermissionAction enum (read, write, create, delete, submit, cancel, amend)
    • DecisionSource enum (rbac, abac, rebac, combo, custom)

3.2 Permission Adapters​

  • Mode A: Standard (Indie) - RbacPermissionAdapter:

    • Reads permissions from DocType.Meta.permissions.
    • Checks CustomPermission rules from database.
    • Role matching against principal_attributes["roles"].
    • Admin role bypass (configurable).
    • RLS via get_permitted_filters() using Meta.rls_field.

  • Mode B: Citadel Policy (Enterprise) - CitadelPolicyAdapter:

    • AuthorizationRequest model (Cedar-compatible format):
      • Principal: user.id
      • Action: doctype:action (e.g., Invoice:create)
      • Resource: doctype:name (e.g., Invoice:INV-001)
      • TenantID: user.tenant_id
      • Context: doc.as_dict()
      • PrincipalAttributes: user.attributes (Diet Claims)
    • AuthorizationResponse model with allowed, reason, policy_id
    • Calls configurable policy endpoint (Cedar/OPA)
    • Fallback behavior on error (configurable)
    • Configuration via [permissions.citadel] in framework_config.toml

4. File Management​

4.1 File DocType​

  • Create src/framework_m/core/doctypes/file.py
  • Define File DocType:
    • name: str - File ID (inherited from BaseDocType)
    • file_name: str - Original filename
    • file_url: str - Storage URL
    • file_size: int - Size in bytes
    • content_type: str - MIME type
    • attached_to_doctype: str | None
    • attached_to_name: str | None
    • is_private: bool - Private/public file (default: True)

4.2 File Upload API​

  • Create POST /api/v1/file/upload (adapters/web/file_routes.py):
    • Accept multipart/form-data via UploadFile
    • Validate file size (max 10MB default, configurable)
    • Validate file type (configurable whitelist)
    • Generate unique storage path (YYYY/MM/DD/random_filename)
    • Create File DocType record
    • Return file URL and metadata
  • Configuration via [files] in framework_config.toml
  • Response models: FileUploadResponse, FileDeleteResponse

4.3 File Download API​

  • Create GET /api/v1/file/{file_id}:
    • Check permissions for private files (RLS on owner)
    • Prepare streaming response from storage
    • Set correct Content-Type header
    • Set Content-Disposition header (inline/attachment)
    • Support ?inline=true for browser display
  • Create GET /api/v1/file/{file_id}/info:
    • Return metadata without downloading content

4.4 Storage Adapters​

  • Implement LocalStorageAdapter (adapters/storage/local.py):

    • Save files to local directory with atomic writes
    • Organize by date (YYYY/MM/DD) via path parameter
    • Generate unique filenames (handled by file_routes)
    • Path traversal protection
    • Async operations via aiofiles

  • Implement S3StorageAdapter (adapters/storage/s3.py):

    • Use aioboto3 for async S3 operations
    • Presigned URLs for direct client uploads (PUT method)
    • Multipart upload for large files (>25MB)
    • Configure bucket and region from framework_config.toml
    • Support S3-compatible services (MinIO, DigitalOcean Spaces)

  • InMemoryStorageAdapter for testing (adapters/storage/memory.py)

5. Audit Logging (Pluggable)​

5.1 Audit Log Protocol​

  • Create AuditLogProtocol (core/interfaces/audit.py):
    • AuditEntry model (user, action, doctype, document_id, changes, metadata)
    • async def log(...) - Record audit entry
    • async def query(...) - Query with filters/pagination
    • InMemoryAuditAdapter for testing

5.2 Audit Adapters​

  • Mode A: Database (Indie):

    • DatabaseAuditAdapter: Writes to ActivityLog records.
    • Uses ActivityLog DocType from 5.3

  • Mode B: External (Enterprise):

    • FileAuditAdapter: Writes to audit.log (JSONL) for Splunk/Filebeat.
    • ElasticAuditAdapter: Writes directly to Elasticsearch (Phase 10).
    • Prevents main DB bloat.

5.3 Activity Log DocType (Indie Only)​

  • Create src/framework_m/core/doctypes/activity_log.py
  • Define ActivityLog DocType:
    • user_id: str
    • action: str (create, read, update, delete)
    • doctype: str
    • document_id: str
    • timestamp: datetime
    • changes: dict | None
    • metadata: dict | None
    • Immutable (no write/delete permissions)

5.4 Activity Feed API​

  • Create GET /api/v1/activity:
    • List recent activities
    • Filter by user, doctype, document, action
    • Paginate results (limit, offset)

6. Error Logging​

6.1 Error Log DocType​

  • Create src/framework_m/core/doctypes/error_log.py
  • Define ErrorLog DocType:
    • title: str - Short error description
    • error_type: str - Exception class name
    • error_message: str - Full error message
    • traceback: str | None - Full stack trace
    • request_url: str | None
    • user_id: str | None
    • request_id: str | None - For log correlation
    • timestamp: datetime
    • context: dict | None - Additional context
    • Admin-only permissions (immutable)

6.2 Error Logging Integration​

  • Add global exception handler (adapters/web/error_handler.py):
    • Catch all unhandled exceptions
    • Create ErrorLog entry with full context
    • Return user-friendly ErrorResponse (no sensitive data)
    • Log to console in dev mode (configurable)
    • create_error_handler() factory for Litestar

7. Printing & PDF Generation​

7.1 Print Format DocType​

  • Create src/framework_m/core/doctypes/print_format.py
  • Define PrintFormat DocType:
    • name: str - Display name
    • doctype: str - Target DocType
    • template: str - Jinja2 template path
    • is_default: bool
    • css: str | None - Custom CSS
    • header_html: str | None, footer_html: str | None
    • page_size: str (A4, Letter, etc.)
    • orientation: str (portrait, landscape)

7.2 Jinja Print Adapter​

  • Create src/framework_m/adapters/print/jinja_adapter.py
  • Implement JinjaPrintAdapter:
    • async def render_html(doc, template) -> str
      • Load Jinja2 template
      • Render with document context
      • Return HTML string
    • async def render_html_string(template_string, doc) -> str
    • Custom filters: currency, date, datetime

7.3 Gotenberg PDF Adapter​

  • Create src/framework_m/adapters/print/gotenberg_adapter.py
  • Implement GotenbergPrintAdapter:
    • async def html_to_pdf(html, page_size, orientation, ...) -> bytes
      • Send HTML to Gotenberg service
      • Return PDF bytes
    • async def is_available() -> bool - Health check
  • GotenbergConfig with URL, timeout, page size, margins
  • Page sizes: A4, Letter, Legal, A3, A5

7.4 Print API​

  • Create GET /api/v1/print/{doctype}/{id}:
    • Query parameters:
      • format - pdf/html (default: pdf)
      • print_format - Custom format name
    • Load document (TODO: integrate with repository - phase 10)
    • Check read permission (TODO: integrate with permission - phase 10)
    • Render using JinjaPrintAdapter → GotenbergPrintAdapter
    • Return PDF or HTML with appropriate headers
    • Fallback to HTML if Gotenberg unavailable

8. System Settings​

8.1 System Settings DocType​

  • Create src/framework_m/core/doctypes/system_settings.py
  • Define SystemSettings DocType (singleton):
    • name: str - Always "System Settings" (frozen)
    • app_name: str - Application display name
    • timezone: str - Default timezone
    • date_format: str - Date format string
    • time_format: str - Time format string
    • language: str - Default language code
    • enable_signup: bool - Allow registration
    • session_expiry: int - Session timeout in minutes
    • maintenance_mode: bool - Admin-only access

8.2 Settings API​

  • Create GET /api/v1/settings:
    • Return system settings via SettingsResponse
    • Cache aggressively (5 min Cache-Control header)
    • invalidate_settings_cache() helper
    • set_settings_cache() helper

9. Notification System​

9.1 Notification DocType​

  • Create src/framework_m/core/doctypes/notification.py
  • Define Notification DocType:
    • user_id: str - Recipient
    • subject: str
    • message: str
    • notification_type: str (info, success, warning, error, etc.)
    • read: bool
    • doctype: str | None - Related DocType
    • document_id: str | None
    • timestamp: datetime
    • from_user: str | None
    • metadata: dict | None
    • RLS on user_id field

9.2 Notification API​

  • Create GET /api/v1/notifications:

    • List user's notifications
    • Filter by read/unread

  • Create PATCH /api/v1/notifications/{id}/read - Mark as read

  • Create DELETE /api/v1/notifications/{id} - Delete notification

  • Create WebSocket endpoint for real-time notifications:

    • WS /api/v1/notifications/stream - Real-time notification stream
    • push_notification(user_id, notification) - Push to connected clients
    • create_notification_websocket_router() factory

10. Email Integration​

10.1 Email Queue DocType​

  • Create src/framework_m/core/doctypes/email_queue.py

  • Define EmailQueue DocType:

    • to: list[str] - Recipients
    • cc/bcc: list[str] | None
    • subject: str
    • body: str - HTML body
    • text_body: str | None - Plain text alt
    • status: str - Queued/Sending/Sent/Failed/Cancelled
    • priority: str - low/normal/high
    • error: str | None
    • retry_count/max_retries: int
    • queued_at/sent_at: datetime
    • reference_doctype/reference_id - for tracking

  • Port-Adapter Pattern:

    • EmailQueueProtocol (interface/port)
    • DatabaseEmailQueueAdapter (default - uses EmailQueue DocType)
    • InMemoryEmailQueueAdapter (testing)
    • queue_email() helper function
    • configure_email_queue() for swapping adapters
    • Ready for NotificationServiceAdapter integration

10.2 Email Sender​

  • Create email sender protocol and adapters:

    • EmailSenderProtocol (interface/port)
    • SMTPEmailSender - SMTP delivery adapter
    • LogEmailSender - Console logging (development)
    • SMTPConfig - Configuration dataclass

  • Create background job EmailProcessor:

    • process_queue() - Process pending emails
    • process_single(queue_id) - Process single email
    • Update EmailQueue status
    • Retry logic (placeholder)

  • Add helper function:

    • queue_email() - Queue for sending
    • send_queued_email() - Process single email

11. Testing​

11.1 Unit Tests​

  • Test User DocType validation (core/doctypes/test_user.py, core/services/test_user_manager.py)
  • Test Role assignment (adapters/auth/test_rbac_permission.py)
  • Test Permission checking (8 permission test files)
  • Test File upload/download (core/test_file.py, adapters/web/test_file_routes.py)
  • Test Print rendering (core/test_print_format.py, adapters/print/test_*.py)

11.2 Integration Tests​

  • Test user creation and login (integration/test_api_endpoints.py)
  • Test permission enforcement (core/test_object_level_permissions.py)
  • Test file storage (local and S3) (adapters/storage/test_local.py, adapters/storage/test_s3.py)
  • Test PDF generation with Gotenberg (adapters/print/test_gotenberg_adapter.py)
  • Test email sending (adapters/email/test_sender_processor.py)

Validation Checklist​

Before moving to Phase 07, verify:

  • User and Role management works
  • Permissions are enforced correctly
  • Files can be uploaded and downloaded
  • Audit logs are created
  • PDFs can be generated
  • Emails can be sent

Anti-Patterns to Avoid​

❌ Don't: Store files in database ✅ Do: Use storage adapter (local/S3)

❌ Don't: Generate PDFs in main process ✅ Do: Use microservice (Gotenberg)

❌ Don't: Send emails synchronously ✅ Do: Queue emails as background jobs

❌ Don't: Hardcode system settings ✅ Do: Store in SystemSettings DocType