Private Video Streaming Service

A private video streaming service built with Rust (Axum) and SvelteKit. Video bytes never flow through the API, the browser uploads directly to object storage via presigned multipart URLs, and playback begins the moment the upload is complete.

Architecture

The API is a thin, stateless control plane. Bytes flow directly between the browser and S3-compatible object storage; the API only orchestrates.

flowchart LR
    subgraph Client
        Browser
    end
    subgraph Control["Control Plane"]
        API["API (Rust/Axum)"]
    end
    subgraph Data
        PG[(PostgreSQL)]
        S3[(Object Storage\nGarage / S3)]
    end

    Browser -- "metadata & presign requests" --> API
    API -- "state & metadata" --> PG
    API -- "create/complete multipart, stream" --> S3
    Browser -- "PUT parts directly" --> S3

Design decisions

Decision	Rationale
Direct browser-to-S3 multipart upload	Keeps the API lightweight and horizontally scalable. Upload bandwidth never touches the control plane. 8 MiB parts, up to 4 concurrent uploads.
Source-first playback	The original file is available for playback immediately after multipart completion, before any transcoding. Optional background HLS processing runs asynchronously and is disabled by default.
Guarded finite state machine	`UPLOADING → PROCESSING → READY` (or `→ FAILED`) with a `video_events` audit trail. Invalid transitions are rejected at the application layer; the database is the source of truth.
HTTP polling instead of WebSocket	Simpler implementation for V1. Frontend polls `GET /api/videos/{share_id}` every 3 s (pauses when the tab is hidden).
Unlisted share links	32-character UUID-derived IDs. Possession implies access. No authentication required. Per-IP rate limiting guards against abuse.
Stateless API replicas	No in-memory session state; multiple replicas run behind a load balancer without coordination.

Video lifecycle

stateDiagram-v2
    [*] --> UPLOADING : POST /api/uploads
    UPLOADING --> PROCESSING : multipart complete
    PROCESSING --> READY : source available
    UPLOADING --> FAILED : error
    PROCESSING --> FAILED : error
    READY --> [*]
    FAILED --> [*]

Tech Stack

Layer	Technology	Notes
API	Rust, Axum 0.8	Async HTTP, streaming, rate limiting
Domain	`crates/domain` (Rust)	Ports, types, FSM - no framework dependency
Web	SvelteKit 2, Svelte 5, Tailwind CSS 4	Upload UI, share page, visibility-aware polling
Database	PostgreSQL 18 via SQLx 0.8	Migrations embedded at compile time
Object storage	Garage v2.2.0 (S3-compatible)	AWS SDK for S3 on the API side
HLS	FFmpeg (optional, bundled in Docker)	Background HLS generation; disabled by default
Container build	cargo-chef + Debian bookworm-slim	Layer-cached dependency build; small runtime image
Shared types	`packages/contracts` (TypeScript)	API request/response types shared between API docs and web

Quick Start

Important

The only prerequisite is Docker with Compose v2.

docker compose up --build

This starts PostgreSQL, Garage (S3-compatible storage), the Garage bootstrap init container, and the API. The web app is not included in Docker Compose; run it separately for local development (see Development).

Verify

# Liveness
curl http://localhost:3000/health

# Readiness (DB + S3 connectivity)
curl http://localhost:3000/ready

Expected responses: 200 ok and 200 ready.

Note

If ready returns 503, Garage initialization is still running. Check docker compose logs garage-init. If it failed, run docker compose down -v && docker compose up --build to start fresh.

Project Structure

.
├── apps/
│   ├── api/                   # Rust API (Axum)
│   │   ├── src/
│   │   │   ├── application/   # Use-case services (upload, video)
│   │   │   ├── domain/        # App-layer domain re-exports
│   │   │   ├── http/          # Handlers, DTOs, router, rate-limit middleware
│   │   │   └── infrastructure/# Postgres repos, S3 adapter, video processor
│   │   ├── config/            # Environment-specific config files
│   │   └── tests/             # Integration tests (real Postgres, test doubles for S3)
│   └── web/                   # SvelteKit frontend
│       └── src/
│           ├── lib/
│           │   ├── api/       # Typed fetch wrappers (client, uploads, videos)
│           │   ├── components/# UploadForm, VideoPlayer, ShareActions
│           │   └── stores/    # videoPoller.svelte.ts, upload.svelte.ts
│           └── routes/        # SvelteKit file-based routes
├── crates/
│   └── domain/                # Pure domain crate: ports (traits), types, FSM states
├── db/
│   ├── migrations/            # SQLx migration files (embedded in the binary)
│   └── seeds/                 # Optional smoke-test seed
├── docker/                    # Garage config, bootstrap script, API entrypoint
├── packages/
│   └── contracts/             # Shared TypeScript API types (request/response)
├── docker-compose.yml
├── Cargo.toml                 # Workspace root
└── package.json               # Yarn workspaces root

Backend Configuration

Configuration is layered: apps/api/config/default.toml supplies defaults, an optional development.toml overrides them locally, and environment variables with the APP__ prefix (double-underscore as separator) override everything. Docker Compose injects these variables directly, so no .env file is needed when running via Docker.

cp .env.example .env   # for local (non-Docker) development

API settings (`APPAPI*`)

Variable	Default	Description
`APP__API__BIND_ADDRESS`	`0.0.0.0:3000`	Host and port the HTTP server listens on
`APP__API__THROTTLE_UPLOADS_PER_MINUTE`	`300`	Maximum upload-related requests per minute per IP
`APP__API__THROTTLE_READS_PER_MINUTE`	`1200`	Maximum read requests per minute per IP
`APP__API__THROTTLE_MAX_BUCKETS`	`50000`	Maximum number of distinct IPs tracked by the rate limiter
`APP__API__TRUSTED_PROXY_IPS`	(empty)	Comma-separated list of trusted proxy IPs; when set, `X-Forwarded-For` is used for client IP detection
`APP__API__MAX_UPLOAD_BYTES`	`1073741824` (1 GiB)	Maximum allowed total upload size in bytes
`APP__API__UPLOAD_SESSION_TTL_SECONDS`	`86400` (24 h)	How long an incomplete upload session remains valid before being expired

Background worker / HLS settings (`APPAPIBGWORKER__*`)

The background worker polls for videos in PROCESSING state and optionally transcodes them to HLS using FFmpeg. It is disabled by default and must be explicitly enabled.

Variable	Default	Description
`APP__API__BGWORKER__ENABLED`	`false`	Set to `true` to enable HLS background processing
`APP__API__BGWORKER__POLL_INTERVAL_SECONDS`	`5`	How often (in seconds) the worker polls for videos to process
`APP__API__BGWORKER__MAX_CONCURRENCY`	`2`	Maximum number of HLS jobs that run in parallel
`APP__API__BGWORKER__MAX_HLS_ATTEMPTS`	`3`	Maximum retry attempts before a video is moved to `FAILED`
`APP__API__BGWORKER__FFMPEG_BINARY`	`ffmpeg`	Path (or name) of the FFmpeg executable
`APP__API__BGWORKER__PRESIGN_TTL_SECONDS`	`3600`	Lifetime of the presigned S3 URL passed to FFmpeg as the input source
`APP__API__BGWORKER__FFMPEG_PRESET`	`veryfast`	FFmpeg `libx264` encoding preset (tradeoff between speed and compression; options: `ultrafast` → `veryslow`)
`APP__API__BGWORKER__FFMPEG_CRF`	`23`	Constant Rate Factor for video quality (0 = lossless, 51 = worst; 23 is a sensible default)
`APP__API__BGWORKER__FFMPEG_MAXRATE`	`4M`	Peak video bitrate cap (e.g. `4M` = 4 Mbit/s); prevents runaway bandwidth usage
`APP__API__BGWORKER__FFMPEG_BUFSIZE`	`8M`	Size of the rate-control buffer; typically 2× `maxrate`
`APP__API__BGWORKER__FFMPEG_AUDIO_BITRATE`	`128k`	AAC audio bitrate (e.g. `128k` = 128 kbit/s)
`APP__API__BGWORKER__HLS_SEGMENT_SECONDS`	`10`	Duration of each HLS `.ts` segment in seconds

S3 / Object storage settings (`APPS3*`)

Variable	Default	Description
`APP__S3__ENDPOINT`	(required)	Full URL of the S3-compatible endpoint (e.g. `http://localhost:3900` for local Garage)
`APP__S3__PUBLIC_ENDPOINT`	(optional)	Browser-facing endpoint for presigned GET redirects. Use when API talks to storage via an internal Docker host (e.g. `ENDPOINT=http://garage:3900`, `PUBLIC_ENDPOINT=http://localhost:3900`)
`APP__S3__REGION`	`us-east-1`	S3 region name; set to `garage` when using Garage
`APP__S3__BUCKET`	(required)	Name of the bucket used for video storage
`APP__S3__ACCESS_KEY_ID`	(required)	S3 access key ID; generated by the `garage-init` bootstrap container
`APP__S3__SECRET_ACCESS_KEY`	(required)	S3 secret access key; generated by the `garage-init` bootstrap container
`APP__S3__UPLOAD_PRESIGN_TTL_SECONDS`	`900` (15 min)	Lifetime of presigned PUT URLs issued to the browser for multipart uploads

When HLS playback uses presigned redirect URLs from /v1/play/..., Garage must return CORS headers for the web origin (for local dev: http://localhost:5173). The Docker bootstrap config applies this automatically; for non-Docker setups, configure bucket CORS with S3 API (put-bucket-cors). Use a single allowed origin (or *) for Access-Control-Allow-Origin, not a comma-separated list.

Database settings (`APPDATABASE*`)

Variable	Default	Description
`APP__DATABASE__URL`	(required)	PostgreSQL connection string for the application database
`APP__DATABASE__TEST_URL`	(required for tests)	PostgreSQL connection string used by the test harness; the harness creates and drops per-test databases against this server
`APP__DATABASE__MAX_CONNECTIONS`	`5`	Size of the SQLx connection pool

Docker Setup

The full stack is defined in docker-compose.yml. Running docker compose up --build starts four services:

`postgres`

Runs PostgreSQL 18 (Alpine image). The API connects to it via APP__DATABASE__URL. A named volume (postgres_data) persists data between restarts. A health check (pg_isready) ensures the database is accepting connections before dependent services start.

`garage`

Runs Garage v2.2.0, a self-hosted S3-compatible object store. Garage is configured via docker/garage.toml, which is bind-mounted into the container at /etc/garage.toml. Four ports are exposed:

Port	Purpose
`3900`	S3 API (used by the API service and presigned browser uploads)
`3901`	Internal Garage RPC (cluster coordination)
`3902`	Garage web endpoint (static site hosting, unused by this project)
`3903`	Garage admin API (used by the bootstrap script)

Named volumes (garage_data, garage_meta) persist object data and cluster metadata.

docker/garage.toml settings:

Setting	Value	Description
`metadata_dir`	`/var/lib/garage/meta`	Directory for cluster metadata (bucket list, layout, keys)
`data_dir`	`/var/lib/garage/data`	Directory for object data blocks
`db_engine`	`lmdb`	Embedded database engine used for metadata storage
`replication_factor`	`1`	Number of copies of each object (1 = no replication; suitable for single-node dev)
`rpc_bind_addr`	`[::]:3901`	Address Garage listens on for internal RPC traffic
`rpc_public_addr`	`garage:3901`	Advertised RPC address; other nodes (or the init container) connect to this
`rpc_secret`	(fixed dev value)	Shared secret authenticating RPC messages between Garage nodes; change in production
`s3_api.s3_region`	`garage`	S3 region name returned in API responses
`s3_api.api_bind_addr`	`[::]:3900`	Address the S3 API server listens on
`admin.api_bind_addr`	`[::]:3903`	Address the admin API listens on
`admin.admin_token`	(dev value)	Bearer token required for admin API calls; change in production

`garage-init`

A one-shot init container built from docker/garage-init.Dockerfile. It copies the garage CLI binary from the official Garage image into a Debian slim base, then runs docker/garage-bootstrap.sh to perform first-time cluster setup:

Waits for the Garage RPC to become reachable (up to 90 seconds).
Discovers the Garage node ID from garage status.
Assigns the node to a layout zone (dc1) with a 1 GiB capacity quota.
Applies the layout so the cluster can accept requests.
Creates the bucket (name from GARAGE_BUCKET, default video-streaming).
Creates an API key (name from GARAGE_KEY_NAME, default api-key) and writes the credentials to a shared volume at GARAGE_CREDS_FILE (default /run/garage-creds/credentials).
Grants the key read/write/owner permissions on the bucket.

The garage_creds named volume is shared with the api service so the API container can read the generated credentials at startup.

`api`

Runs the compiled Rust API binary. At startup, docker/api-entrypoint.sh sources the credentials file written by garage-init (exporting APP__S3__ACCESS_KEY_ID and APP__S3__SECRET_ACCESS_KEY), then executes the API binary.

The service waits for both postgres (healthy) and garage-init (completed successfully) before starting. The background HLS worker is enabled in the Docker Compose configuration (APP__API__BGWORKER__ENABLED=true) with FFmpeg pre-installed in the runtime image.

Volumes

Volume	Used by	Purpose
`postgres_data`	`postgres`	Persistent PostgreSQL data directory
`garage_data`	`garage`, `garage-init`	Persistent Garage object data
`garage_meta`	`garage`, `garage-init`	Persistent Garage cluster metadata
`garage_creds`	`garage-init`, `api`	Credentials file shared from init container to API

API Dockerfile (`apps/api/Dockerfile`)

The API image is built in three stages:

chef (planner) - Uses cargo-chef to compute a dependency recipe from Cargo.toml and Cargo.lock, enabling Docker layer caching of compiled dependencies.
builder - Restores the cached dependencies, then compiles the full api binary in release mode.
runtime - A minimal debian:bookworm-slim image. Installs only ca-certificates, libssl3, and ffmpeg (for HLS processing). Copies the compiled binary, the default config files, and the entrypoint script.

FFmpeg Pipeline

When the background worker is enabled and a video enters PROCESSING state, the worker:

Generates a short-lived presigned S3 URL for the source file (TTL controlled by APP__API__BGWORKER__PRESIGN_TTL_SECONDS).
Runs FFmpeg with that URL as the input, writing HLS output to a temporary directory.
Uploads the resulting .m3u8 playlist and .ts segments back to S3.
Marks the video as hls_ready in the database.

The FFmpeg command issued is equivalent to:

ffmpeg \
  -y \
  -i "<presigned-source-url>" \
  -c:v libx264 \
  -preset veryfast \
  -crf 23 \
  -maxrate 4M \
  -bufsize 8M \
  -c:a aac \
  -b:a 128k \
  -f hls \
  -hls_time 10 \
  -hls_list_size 0 \
  -hls_base_url hls/ \
  -hls_segment_filename /tmp/<job>/segment_%03d.ts \
  /tmp/<job>/playlist.m3u8

Flag-by-flag explanation

Flag	Value	Description
`-y`	-	Overwrite output files without prompting
`-i <url>`	presigned S3 URL	Input source; FFmpeg streams the video directly from S3 over HTTPS
`-c:v libx264`	-	Encode video with H.264, the most widely compatible codec for HLS
`-preset`	`veryfast`	`libx264` encoding speed/compression tradeoff. `veryfast` prioritizes speed over file size; configurable via `APP__API__BGWORKER__FFMPEG_PRESET`
`-crf`	`23`	Constant Rate Factor: quality target for variable bitrate encoding. Lower = better quality, larger file. Configurable via `APP__API__BGWORKER__FFMPEG_CRF`
`-maxrate`	`4M`	Caps the peak bitrate to prevent very high-bitrate scenes from spiking; configurable via `APP__API__BGWORKER__FFMPEG_MAXRATE`
`-bufsize`	`8M`	Rate-control buffer size; set to 2× `maxrate` so the encoder has room to absorb bitrate spikes. Configurable via `APP__API__BGWORKER__FFMPEG_BUFSIZE`
`-c:a aac`	-	Encode audio with AAC, the standard codec for HLS
`-b:a`	`128k`	Target audio bitrate; configurable via `APP__API__BGWORKER__FFMPEG_AUDIO_BITRATE`
`-f hls`	-	Output format: HTTP Live Streaming (HLS)
`-hls_time`	`10`	Target duration of each `.ts` segment in seconds; configurable via `APP__API__BGWORKER__HLS_SEGMENT_SECONDS`
`-hls_list_size`	`0`	Keep all segments in the playlist (0 = unlimited); required for VOD playback
`-hls_base_url`	`hls/`	Prefix prepended to segment filenames in the playlist, matching the API's `/hls/` route
`-hls_segment_filename`	`segment_%03d.ts`	Output path pattern for segment files (`%03d` = zero-padded three-digit index)
(last positional)	`playlist.m3u8`	Output path for the HLS master playlist

API Reference

All API routes are versioned under /v1/.

Method	Path	Description
`GET`	`/health`	Liveness probe
`GET`	`/ready`	Readiness probe (DB + S3)
`POST`	`/v1/uploads`	Create upload session (requires `Idempotency-Key` header) → `upload_id`, `share_id`, `part_size`
`GET`	`/v1/uploads/{id}/parts/{num}`	Mint presigned PUT URL for one part (idempotent)
`PATCH`	`/v1/uploads/{id}`	Finalise upload with `{ "status": "completed", "parts": [...] }`
`GET`	`/v1/videos/{share_id}`	Video metadata (poll until `status === "READY"`)
`GET`	`/v1/videos/{share_id}/source`	`307` redirect to a presigned storage URL; Range requests (seeking, buffering) are handled natively by object storage
`GET`	`/v1/videos/{share_id}/playlist.m3u8`	HLS master playlist with segment URLs rewritten to `/v1/play/` (404 until `hls_ready`)
`GET`	`/v1/videos/{share_id}/hls/{asset}`	HLS segments or sub-playlists streamed directly from storage (404 until `hls_ready`)
`GET`	`/v1/play/{share_id}/{*asset_path}`	`307` redirect to a short-lived presigned S3 URL for HLS segment playback

OpenAPI Specification

A machine-readable OpenAPI 3.1 spec is committed to the repository root as openapi.json and kept up-to-date automatically.

Regenerate locally

# Rebuild the spec + refresh TypeScript types in one step
yarn openapi

# Or individually
make openapi                                        # writes openapi.json
yarn workspace @project/contracts generate          # writes packages/contracts/src/generated/openapi.ts

Implementation details

Concern	Approach
Generator	`utoipa` v5 - OpenAPI 3.1.0 is the default output format
Schema derivation	`ToSchema` is derived on all DTO structs in `apps/api/src/http/dto/`. Domain structs (`VideoStatus`, `ShareId`, `VideoId`, `Video`) derive `ToSchema` only when the optional `openapi` feature is enabled on `crates/domain`, keeping the domain crate free of API-framework concerns
Aggregation	`apps/api/src/openapi.rs` - a single `ApiDoc` struct decorated with `#[derive(OpenApi)]` lists all paths and component schemas
Static export	`cargo run -p api --bin export-openapi` writes `openapi.json` to the working directory
CI	`.github/workflows/openapi.yml` regenerates the spec on every push to `master` that touches Rust sources, then opens a pull request so the diff can be reviewed before merging
TypeScript types	`packages/contracts/src/generated/openapi.ts` is produced by `openapi-typescript` v7 from `openapi.json`. Import via `@project/contracts/generated`

Using the generated TypeScript types

import type { components, operations } from "@project/contracts/generated";

// Strongly-typed response for GET /v1/videos/{share_id}
type VideoMeta = components["schemas"]["VideoMetadataResponse"];

// Full request/response pair for POST /v1/uploads
type InitReq  = operations["initUpload"]["requestBody"]["content"]["application/json"];
type InitResp = operations["initUpload"]["responses"]["200"]["content"]["application/json"];

// Finalise upload - PATCH /v1/uploads/{id}
type FinalizeReq = operations["finalizeUpload"]["requestBody"]["content"]["application/json"];

Development

Prerequisites

Rust (stable toolchain - pinned in rust-toolchain.toml)
Node.js ≥ 20 + Yarn
PostgreSQL 18
Garage v2.2.0 (or any S3-compatible object store)
sqlx-cli for migrations: cargo install sqlx-cli --no-default-features --features rustls,postgres

Running locally

# Start API (port 3000) + web dev server (port 5173) together
yarn dev

# Or individually
cargo run -p api
yarn workspace web dev

The Vite dev server proxies /api and /__storage (presigned S3 uploads) to avoid CORS issues.

Database migrations

# Apply all pending migrations
yarn db:migrate:run

# Revert the latest migration
yarn db:migrate:revert

# Show migration status
yarn db:migrate:info

# Create a new migration
yarn db:migrate:add <name>

Workspace commands

yarn build     # Build Rust workspace + web
yarn test      # cargo test --workspace + web tests
yarn lint      # rustfmt check + clippy (warnings as errors) + svelte-check
yarn openapi   # Regenerate openapi.json and TypeScript types

Database Schema Design

The schema is split across six migration files under db/migrations/. Every decision below is derived directly from those files.

Enum types (`20260306024252`)

CREATE TYPE video_status  AS ENUM ('UPLOADING', 'PROCESSING', 'READY', 'FAILED');
CREATE TYPE upload_status AS ENUM ('ACTIVE', 'COMPLETED', 'ABORTED');

Why native PostgreSQL enums instead of varchar?

Storage: A PostgreSQL enum value is stored as a 4-byte OID reference to pg_enum, whereas varchar requires at least 8 bytes of header plus the string length. Every row in video_states references both columns, so two enums save 8-16 bytes per row.
Validity enforcement at zero cost: The database rejects any unknown status string at INSERT time without a CHECK constraint or trigger. Invalid states cannot silently enter the system.
Index compactness: Enum OIDs sort numerically, so every B-Tree index entry on these columns is 4 bytes instead of a variable-length string, keeping indexes smaller and more cache-friendly.

`videos` table - the "Vault" (`20260306024317`)

CREATE TABLE videos (
    id                   bigint PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    external_id          uuid        UNIQUE NOT NULL DEFAULT uuidv7(),
    share_id             varchar(12) UNIQUE NOT NULL,
    bytes                bigint NOT NULL DEFAULT 0,
    mime                 varchar(64) NOT NULL,
    storage_key_original text   NOT NULL,
    created_at           timestamptz NOT NULL DEFAULT now()
) WITH (fillfactor = 100);

Primary key: bigint identity, not UUID

A sequential bigint identity column (8 bytes) is used as the internal primary key instead of UUID (16 bytes). Every foreign-key reference (video_states.video_id, video_events.video_id) is 8 bytes instead of 16. With multiple secondary indexes referencing the PK, the total savings in index pages kept warm in shared_buffers can be substantial (Percona benchmark: sequential integer PKs sustain ~3× the write throughput of random UUID v4 PKs on InnoDB; the PostgreSQL heap story is analogous).

external_id: UUID v7 (time-sortable)

external_id is the UUID exposed in API URLs. Using PostgreSQL 18's native uuidv7() function means every new row appends to the end of the external_id B-Tree index - the same access pattern as a sequential integer. UUID v4 inserts at random positions, causing page splits and up to 50% free space waste in the index (documented in the 2023 AWS Aurora paper on UUID primary keys). UUID v7 eliminates that fragmentation.

share_id: 12-character base62

12 characters over a 62-symbol alphabet ([0-9A-Za-z]) with no collisions.

fillfactor = 100

This table is written once (at upload init) and then read for every API call. No UPDATE ever touches it. Setting fillfactor = 100 packs every heap page to 100% occupancy.

`video_states` table - the "Hot" table (`20260306024403`)

CREATE TABLE video_states (
    video_id            bigint PRIMARY KEY REFERENCES videos(id) ON DELETE CASCADE,
    upload_status       upload_status NOT NULL DEFAULT 'ACTIVE',
    multipart_upload_id text,
    upload_expires_at   timestamptz,
    status              video_status NOT NULL DEFAULT 'UPLOADING',
    hls_ready           boolean NOT NULL DEFAULT false,
    hls_playlist        text,
    hls_attempts        smallint NOT NULL DEFAULT 0,
    last_error          text,
    updated_at          timestamptz NOT NULL DEFAULT now(),
    state_version       integer NOT NULL DEFAULT 0
) WITH (fillfactor = 70);

videos is 99% reads after insert so mostly it never changes after init_upload. video_states is 90% writes (status transitions, expiry updates, HLS progress).

fillfactor = 70

PostgreSQL's HOT (Heap-Only Tuple) update mechanism lets an UPDATE reuse an existing heap slot instead of writing a new tuple and a new index entry, of course only if the updated columns are not covered by any index and there is free space on the same page. fillfactor = 70 reserves 30% of each page for in-place updates, enabling HOT. For a column like status (not indexed directly) that changes with every FSM transition, HOT eliminates the secondary index rewrite entirely.

Autovacuum tuning

ALTER TABLE video_states SET (
    autovacuum_vacuum_scale_factor  = 0.01,
    autovacuum_analyze_scale_factor = 0.005,
    autovacuum_vacuum_cost_limit    = 1000,
    autovacuum_vacuum_cost_delay    = 2
);

PostgreSQL's default autovacuum_vacuum_scale_factor is 0.2 (20%). Tightening to 1% keeps dead-tuple accumulation low. autovacuum_vacuum_cost_limit = 1000 gives the autovacuum worker 5× the I/O budget compared to the default (200), so it finishes each pass faster. These settings can be changed or improved based on future metrics but right now they are applied only to video_states to avoid over-stressing the autovacuum system for the rest of the database.

state_version - the atomic FSM counter

Each FSM transition executes:

UPDATE video_states
SET status = '...', state_version = state_version + 1, ...
WHERE video_id = $1 AND status = '<expected>'
RETURNING state_version

The UPDATE holds a row lock on video_states, serialising concurrent transitions for the same video. The returned state_version is immediately used as the version stamp in the video_events INSERT, making (video_id, state_version) globally unique by construction.

`video_events` table - the audit log (`20260306024528`)

CREATE TABLE video_events (
    video_id      bigint       NOT NULL,
    to_status     video_status NOT NULL,
    from_status   video_status,
    reason        text,
    state_version integer      NOT NULL,
    created_at    timestamptz  NOT NULL DEFAULT now()
) PARTITION BY RANGE (created_at);

CREATE TABLE video_events_default PARTITION OF video_events DEFAULT;

CREATE UNIQUE INDEX idx_video_events_default_unique_version
    ON video_events_default (video_id, state_version);

Why PARTITION BY RANGE (created_at)?

As the table grows to hundreds of millions of rows, partition pruning means queries filtered to a recent time window (e.g. "last 30 days of state changes for video X") scan only the relevant partition instead of the full table. Old partitions can be archived and dropped in O(1) and a DROP TABLE on a partition is instantaneous, whereas DELETE FROM video_events WHERE created_at < ... on an unpartitioned table requires a full sequential scan and blocks vacuuming for hours.

No foreign key to videos(id)

PostgreSQL cannot enforce cross-partition foreign keys efficiently (it would need to check every partition on every insert). Referential integrity is maintained by the application: events are written only inside transactions that have already fetched the video_id from video_states, which has a proper REFERENCES videos(id) constraint.

from_status: NULL for the first event

Without from_status, reconstructing the full state history requires diffing consecutive rows, and that is error-prone across partition boundaries or in production incidents. Storing the previous state explicitly makes each row self-contained.

reason

A short free-text string (upload_initiated, multipart_upload_completed, hls_processing_complete, etc.) which lets me distinguish expected transitions from unexpected ones during debugging or in the future during incident response.

Performance indexes (`20260306024641`)

A. API lookup covering index

CREATE INDEX idx_videos_api_lookup
    ON videos (share_id)
    INCLUDE (external_id, id, bytes);

INCLUDE columns are stored in the index leaf but not in the index B-Tree key. A request for GET /v1/videos/{share_id} can be satisfied entirely from this index (Index-Only Scan) without touching the heap.

B. Session cleanup partial index

CREATE INDEX idx_active_sessions_cleanup
    ON video_states (upload_expires_at)
    WHERE upload_status = 'ACTIVE';

The session-cleanup background job only cares about ACTIVE sessions with a past expiry date. At steady state, the vast majority of rows are COMPLETED or ABORTED and are invisible to this index.

C. HLS worker queue partial index

CREATE INDEX idx_worker_queue
    ON video_states (updated_at)
    WHERE status = 'PROCESSING' OR status = 'UPLOADING';

The HLS background worker polls for videos awaiting processing. At steady state, 99%+ of rows are READY and are excluded by the partial index predicate.

D. Event timeline composite index

CREATE INDEX idx_events_timeline
    ON video_events (video_id, created_at DESC);

Pre-sorts events by video and descending time, I added this because I wanted the most common query pattern: "show the last N state changes for this video." we need this index, because every query would require a full partition scan plus a sor without it.

`client_key` idempotency column (`20260306024755`)

ALTER TABLE video_states
    ADD COLUMN client_key TEXT UNIQUE;

The client sends a deterministic Idempotency-Key header with every POST /v1/uploads request. Storing it on video_states with a UNIQUE constraint lets PostgreSQL atomically block duplicate multipart sessions at the database level: the ON CONFLICT (client_key) DO UPDATE SET updated_at = NOW() clause in the INSERT turns a duplicate into a no-op, and the UNIQUE index makes the check lock-free compared to a SELECT … FOR UPDATE / INSERT pair.

Testing

API tests use real PostgreSQL via an isolated per-test database harness. Each test creates a fresh database, runs migrations, executes, then drops the database. S3 is replaced with an in-memory test double.

# Ensure APP__DATABASE__TEST_URL is set (already in config/development.toml for local dev)
cargo test -p api

Test layout inside apps/api/tests/:

File	Coverage
`db_smoke.rs`	Test harness + migrations work
`domain_models.rs`	Domain types and FSM (no DB)
`repositories.rs`	Repository integration tests against Postgres
`services.rs`	Service-layer tests with Postgres backends
`api_db.rs`	Full HTTP integration tests against the Axum router

Name		Name	Last commit message	Last commit date
Latest commit History 32 Commits
apps		apps
crates/domain		crates/domain
db		db
docker		docker
packages/contracts		packages/contracts
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml
openapi.json		openapi.json
package.json		package.json
rust-toolchain.toml		rust-toolchain.toml
rustfmt.toml		rustfmt.toml
yarn.lock		yarn.lock

Folders and files

Latest commit

History

Repository files navigation

Private Video Streaming Service

Table of Contents

Architecture

Design decisions

Video lifecycle

Tech Stack

Quick Start

Verify

Project Structure

Backend Configuration

API settings (APP__API__*)

Background worker / HLS settings (APP__API__BGWORKER__*)

S3 / Object storage settings (APP__S3__*)

Database settings (APP__DATABASE__*)

Docker Setup

postgres

garage

garage-init

api

Volumes

API Dockerfile (apps/api/Dockerfile)

FFmpeg Pipeline

Flag-by-flag explanation

API Reference

OpenAPI Specification

Regenerate locally

Implementation details

Using the generated TypeScript types

Development

Prerequisites

Running locally

Database migrations

Workspace commands

Database Schema Design

Enum types (20260306024252)

videos table - the "Vault" (20260306024317)

video_states table - the "Hot" table (20260306024403)

video_events table - the audit log (20260306024528)

Performance indexes (20260306024641)

client_key idempotency column (20260306024755)

Testing

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

API settings (`APPAPI*`)

Background worker / HLS settings (`APPAPIBGWORKER__*`)

S3 / Object storage settings (`APPS3*`)

Database settings (`APPDATABASE*`)

`postgres`

`garage`

`garage-init`

`api`

API Dockerfile (`apps/api/Dockerfile`)

Enum types (`20260306024252`)

`videos` table - the "Vault" (`20260306024317`)

`video_states` table - the "Hot" table (`20260306024403`)

`video_events` table - the audit log (`20260306024528`)

Performance indexes (`20260306024641`)

`client_key` idempotency column (`20260306024755`)

Packages