Uniflow CDP

Open-source Customer Data Platform — self-hosted on your own AWS account.

Quick Start · Architecture · Structure · Local Dev · Connectors · Contributing

---

Uniflow gives you the core features of a modern CDP — event collection, identity resolution, unified profiles, audience segmentation, and destination connectors — packaged as a single CLI + AWS CDK library you can deploy in minutes to your own cloud account.

Status: Early development · MVP in progress · Contributions welcome

✨ Features

Feature	Description
Event Collection	Segment-compatible HTTP API (track, identify, page, group)
Identity Resolution	Anonymous ID → known user ID linking with a persistent identity graph
Unified Profiles	Merged profile with traits and full event history in DynamoDB
Segmentation	Rule-based audiences evaluated on a schedule via AWS Glue PySpark
Destinations	Webhook and S3 export built-in · plugin SDK for community connectors
Admin UI	Next.js dashboard: Sources, Destinations, Profile Explorer, Segments
CLI	uniflow init / deploy / status / upgrade / destroy
Client SDKs	@uniflow/js (browser + Node) · uniflow-python

🏗 Architecture

Client SDK
  └─▶ API Gateway (HTTP API)
        └─▶ Lambda (validate + enrich)
              ├─▶ Kinesis Data Stream ──▶ Firehose ──▶ S3 (raw, GZIP)
              └─▶ Kinesis Data Stream
                    └─▶ Lambda (processor)
                          ├─▶ DynamoDB  (identity graph + profile upsert)
                          └─▶ SQS       (destination fan-out)
                                └─▶ Lambda (connector) ──▶ External system

EventBridge Scheduler (hourly)
  └─▶ Glue PySpark Job (audience-builder)
        └─▶ S3 Parquet + DynamoDB (segment membership)

Cognito ──▶ API Gateway ──▶ Lambda (management API)
Next.js static export ──▶ S3 ──▶ CloudFront

Compute decisions:

Component	Compute	Why
Ingest API	Lambda	Zero idle cost, auto-scales
Stream processor	Lambda	Short-lived, stateless
Audience builder	AWS Glue (PySpark)	Serverless Spark for complex segment queries, no VPC needed
Destination connectors	Lambda	Event-driven, short-lived
Management API	Lambda	Low-traffic CRUD

🚀 Quick Start

Prerequisites

• Node.js ≥ 20
• pnpm ≥ 9
• AWS CLI configured (aws configure)
• AWS CDK bootstrapped (npx cdk bootstrap)

# Install the CLI
npm install -g uniflow

# Interactive setup — generates uniflow.config.yaml
uniflow init

# Deploy to your AWS account
uniflow deploy

# Check deployment health
uniflow status

After deploy you'll get:

• Ingest endpoint — send events from your apps
• Admin UI URL — manage sources, destinations, segments
• Write key — authenticate your SDK calls

📦 Repo Structure

uniflow/
├── infra/                    # @uniflow/cdk — CDK constructs
│   └── src/
│       ├── stacks/UnifowStack.ts
│       └── constructs/
│           ├── StorageConstruct.ts      # DynamoDB · S3 · Kinesis · Firehose
│           ├── IngestionConstruct.ts    # API Gateway · Ingest Lambda
│           ├── ProcessingConstruct.ts   # Kinesis consumer · SQS fan-out
│           ├── AudienceConstruct.ts     # Glue PySpark Job · EventBridge Scheduler
│           └── AdminConstruct.ts        # Cognito · CloudFront · Management API
│
├── services/
│   ├── ingest/               # Validate events → Kinesis
│   ├── processor/            # Kinesis consumer → DynamoDB + SQS
│   ├── audience-builder/     # Glue PySpark: segment evaluation → S3 + DynamoDB
│   └── management-api/       # CRUD: sources · destinations · segments · profiles
│
├── connectors/
│   ├── sdk/                  # BaseConnector abstract class
│   ├── webhook/              # HTTP webhook (HMAC signing)
│   └── s3-export/            # S3 NDJSON export
│
├── libs/
│   ├── event-schema/         # Zod schemas for all event types
│   ├── identity/             # Identity resolution logic
│   └── logger/               # Structured JSON logging
│
├── cli/                      # uniflow CLI (npm: uniflow)
├── sdk/
│   ├── js/                   # @uniflow/js — browser + Node tracking SDK
│   └── python/               # uniflow-python tracking SDK
│
├── ui/                       # Next.js 15 admin dashboard
├── docker/                   # docker-compose + LocalStack for local dev
└── examples/                 # CDK app example · tracking scripts

📐 Data Model

Multi-table DynamoDB design — each entity type has a dedicated table:

Table	PK	SK	GSI
profilesTable	userId	sortKey (META or EVENT#ts#id)	—
identityTable	anonymousId	—	—
sourcesTable	id	—	writeKeyHashIndex on writeKeyHash
destinationsTable	id	—	—
segmentsTable	id	—	—
segmentMembersTable	segmentId	userId	—

💻 Local Dev

# Prerequisites: Docker, Node.js 20+, pnpm 9+

git clone https://github.com/maroil/uniflow
cd uniflow
pnpm install

# Start LocalStack (DynamoDB, S3, Kinesis, SQS)
docker compose -f docker/docker-compose.yml up -d localstack

# Send test events
npx ts-node examples/send-events.ts

# Run all tests
pnpm test

# Build all packages
pnpm build

🔌 Connector SDK

Build your own destination connector and publish it as @uniflow/connector-<name>:

import { BaseConnector, type ConnectorEvent, type ConnectorResult } from '@uniflow/connector-sdk';
import { z } from 'zod';

const ConfigSchema = z.object({ apiKey: z.string() });

export class MyConnector extends BaseConnector<z.infer<typeof ConfigSchema>> {
  readonly metadata = {
    id: 'my-connector',
    name: 'My Service',
    description: 'Send events to My Service',
    configSchema: ConfigSchema,
  };

  async handle(event: ConnectorEvent, config: z.infer<typeof ConfigSchema>): Promise<ConnectorResult> {
    await fetch('https://api.myservice.com/events', {
      method: 'POST',
      headers: { Authorization: `Bearer ${config.apiKey}` },
      body: JSON.stringify(event),
    });
    return { success: true };
  }
}

⚙️ Configuration

uniflow.config.yaml is generated by uniflow init and should be version-controlled:

version: "0.1"
region: us-east-1
adminEmail: admin@acme.com
retentionDays: 90
connectors:
  - webhook
  - s3-export
stackName: UnifowStack

🛠 Tech Stack

Layer	Technology
IaC	AWS CDK (TypeScript)
Ingest	Lambda + API Gateway HTTP API
Audience builder	AWS Glue (PySpark)
Event buffer	Kinesis Data Streams (7-day retention)
Hot store	DynamoDB (multi-table, PAY_PER_REQUEST)
Analytics	S3 + Athena + Glue Catalog
Auth	Cognito User Pool
Reliability	SQS + Dead-letter queues
Admin UI	Next.js 15 + Tailwind CSS v4
Monorepo	pnpm + Turborepo
Local dev	docker-compose + LocalStack

🗺 Roadmap

• Lambda authorizer for write-key validation
• Cognito JWT on management API
• Visual segment rule builder in Admin UI
• Docusaurus documentation site
• uniflow dev command (local hot-reload)
• CDK snapshot tests
• Community connector: Braze, Mixpanel, BigQuery

🤝 Contributing

Contributions are very welcome! Please read CONTRIBUTING.md before opening a PR.

📄 License

MIT — see LICENSE.