File Storage Architecture

Framework M uses a pluggable, protocol-based file storage system. This page explains the design decisions, upload flows, and how the backend verifies S3 uploads without relying on event notifications.

---

Storage Protocol

All storage adapters implement StorageProtocol from framework_m_core.interfaces.storage:

class StorageProtocol:
    async def save_file(path, content, content_type) -> str
    async def get_file(path) -> bytes
    async def delete_file(path) -> None
    async def exists(path) -> bool
    async def get_metadata(path) -> FileMetadata | None
    async def get_url(path, expires) -> str
    async def copy(src, dest) -> str
    async def move(src, dest) -> str

This abstraction allows the application to run identically against local disk, in-memory (tests), or S3 without changing any business logic.

---

Upload Modes

Framework M provides three upload modes, selected by the client based on file size:

1. Streaming Single-Shot (POST /api/v1/file/upload)

Used for files below max_upload_size (default 10 MB). The server reads the multipart body in 4 MB chunks, writing each to a temp file before committing to storage. This avoids buffering the entire file in RAM.

Browser  ──multipart POST──►  Server  ──save_file()──►  Storage

2. Server-Side Chunked (/upload/init → /upload/chunk → /upload/complete)

Used for large files. The client slices the file into 4 MB parts and sends each as a separate PUT request. The server assembles them in order:

POST /upload/init          → { upload_id, chunk_size, total_chunks }
PUT  /upload/chunk?...     (repeated N times)
POST /upload/complete      → assembles parts, saves to storage, cleans up temp files

This approach does not require S3 — it works identically against the local and memory backends and is useful when presigned S3 uploads are not desired.

3. Presigned S3 Upload (/upload/presign → browser → S3 → /upload/presign/complete)

Used when storage_backend = "s3". The browser uploads directly to S3, bypassing the server entirely. This eliminates server bandwidth costs for large binary uploads.

Browser                      Server                        S3
  │                             │                           │
  ├─ POST /upload/presign ──────►                           │
  │◄─ { presigned URLs, key } ──┤                           │
  │                             │                           │
  ├─ PUT <presigned_url> ───────┼──────────────────────────►│
  │◄─ 200 OK (ETag) ───────────┼───────────────────────────┤
  │                             │                           │
  ├─ POST /upload/presign/complete ─►                       │
  │                             ├─ head_object / complete ─►│  ← verification
  │◄─ { key, file_url } ────────┤                           │

---

How the Backend Verifies Presigned Uploads

S3-compatible APIs (MinIO, Backblaze B2, DigitalOcean Spaces, Cloudflare R2, etc.) do not universally support SNS event notifications or webhooks. The backend cannot passively observe a presigned PUT.

Framework M uses a "client-confirms + server-verifies" pattern in POST /upload/presign/complete:

Upload type	Verification method
Multipart (upload_id + parts supplied)	CompleteMultipartUpload is called on S3 — the API atomically finalises the object and fails if any part is missing or mismatched.
Simple PUT (parts empty)	HeadObject is called after the client signals completion. Returns HTTP 422 if the object is not present.

HeadObject is part of the S3 core specification and is supported by every S3-compatible service. It requires no event configuration and adds only one lightweight HTTP round-trip.

Extending with Stronger Guarantees

If an application needs to validate file size or checksum, it can use the FileMetadata returned by storage.get_metadata(key):

metadata = await storage.get_metadata(key)
if metadata.size != expected_size:
    raise HTTPException(422, "File size mismatch")

The framework intentionally leaves this as an open extension point for application-level logic.

---

S3 Configuration Loading

S3 settings are resolved by load_s3_config() in framework_m_standard.adapters.storage.s3, which merges two sources:

1. [files] section of framework_config.toml
2. FRAMEWORK_M_S3_* environment variables (environment variables take precedence)

# Simplified merge logic
toml_config = S3Config(**toml_data.get("files", {}))
env_config  = S3Config()                               # reads FRAMEWORK_M_S3_* vars
merged      = {**toml_config.model_dump(), **env_config.model_dump(exclude_unset=True)}
return S3Config(**merged)

Field	TOML key	Environment variable
Bucket name	s3_bucket	FRAMEWORK_M_S3_BUCKET
Region	s3_region	FRAMEWORK_M_S3_REGION
Custom endpoint	s3_endpoint	FRAMEWORK_M_S3_ENDPOINT
Access key	s3_access_key	FRAMEWORK_M_S3_ACCESS_KEY
Secret key	s3_secret_key	FRAMEWORK_M_S3_SECRET_KEY

---

Temp File Lifecycle

During chunked and streaming uploads, Framework M writes to <storage_path>/.tmp_chunks/. The cleanup strategy is:

• Streaming upload: temp file is deleted in a finally block whether the upload succeeds or fails.
• Chunked upload /complete: all .part* files and the .manifest file are deleted in a finally block after assembly, regardless of outcome.
• Abandoned sessions: .manifest and .part* files for sessions where /complete is never called are not automatically purged. Applications that require cleanup should implement a periodic job to remove stale entries older than a configurable TTL.

tip

In production, configure a lifecycle rule on the .tmp_chunks/ S3 prefix (or a cron job for the local backend) to expire stale incomplete uploads after 24 hours.