ADR-0001: Use NATS JetStream with Taskiq for Background Jobs
- Status: Accepted
- Date: 2025-12-29
- Deciders: @anshpansuriya14
- Supersedes: N/A
- Superseded by: N/A
Context
Framework M needs a reliable background job system for:
- Sending emails/notifications asynchronously
- Processing webhooks
- Running scheduled tasks (cron jobs)
- Handling long-running operations outside HTTP request lifecycle
Forces at play:
- Cross-platform: Must work on Windows, macOS, Linux (unlike Celery which has Windows issues)
- Persistence: Jobs must survive restarts (at-least-once delivery)
- Observability: Need job status tracking and retry logic
- Python Ecosystem: Integration with async/await patterns
- Dev Experience: Simple decorator-based job definition
Decision
We will use Taskiq with NATS JetStream as the job queue backend.
from taskiq_nats import PullBasedJetStreamBroker
broker = PullBasedJetStreamBroker(
servers=["nats://localhost:4222"],
stream="framework_m_jobs",
)
@broker.task(name="send_email", retry=3)
async def send_email(to: str, subject: str, body: str):
# Job logic
pass
Consequences
Positive
- Cross-platform: NATS runs on Windows, macOS, Linux natively
- Persistent: JetStream provides durable message storage and replay
- Modern Architecture: Native async/await, works well with Litestar
- Single Infrastructure: NATS handles both jobs AND events (Section 4.1)
- Taskiq Philosophy: Clean decorator-based API similar to Celery but modern
Negative
- Learning Curve: NATS is less known than Redis/Celery
- Operational Complexity: Requires running NATS server
- Ecosystem Size: Smaller than Celery ecosystem
Neutral
- Uses gRPC-style pull-based consumers (efficient for high throughput)
- Supports scheduled jobs via
TaskiqScheduler
Alternatives Considered
| Option | Pros | Cons |
|---|---|---|
| Chosen: Taskiq + NATS | Cross-platform, persistent, modern async | Smaller ecosystem |
| ARQ + Redis | Python native, simple | No Windows support, less features |
| Celery + Redis | Most popular, mature | Windows issues, sync-first design |
| Dramatiq + Redis | Modern API | Limited scheduling, Redis dependency |
| Temporal | Enterprise-grade workflows | Heavy, separate system |