LiveAvatar × Hyperframes

A voice-driven HeyGen LiveAvatar that listens to you and renders real-time motion-graphic overlays by prompting Claude Code with the Hyperframes skill.

Talk to the avatar. When you ask for something visual, a background Claude Code session writes a composition.html and it overlays onto the avatar's video stream — live, transparent, looping.

How it works

1. LiveAvatar FULL mode handles speech recognition, LLM reply, TTS, and WebRTC video. The avatar holds normal small-talk.
2. Each user transcript is appended to a turn buffer, flushed on USER_SPEAK_ENDED or after a 3s debounce.
3. The flushed turn is sent to a Haiku classifier (claude -p --model haiku) that returns either {"intent":"chat"} or {"intent":"generate","prompt":"..."}. ASR transcripts are noisy, so the classifier is told to phonetically reconstruct mangled subjects.
4. A generate decision spawns Claude Code (claude -p --output-format stream-json --verbose) inside a per-job workspace. The hyperframes skill writes composition.html.
5. A chokidar watcher picks up the new file and the backend pushes a hyperframes_ready event over SSE. The frontend swaps the URL on a <hyperframes-player> overlay that loops on top of the avatar video.
6. Single-flight job control: a new generation cancels the previous one.

Setup

Prereqs:

• Node 20+
• Claude Code installed and authenticated (uses your subscription — no Anthropic API key)
• The Hyperframes skill installed: npx add-skill heygen-com/hyperframes
• A LiveAvatar API key from https://app.liveavatar.com

git clone https://github.com/heygen-com/liveavatar-hyperframes-demo
cd liveavatar-hyperframes-demo
npm install
cp .env.example .env
# fill in LIVEAVATAR_API_KEY (+ optional avatar/voice IDs)
npm run dev

Open http://localhost:3000, click Start session, allow mic access, then say something like "animate three orange cats" or "show me a glowing sunset". The status pill turns purple while Claude Code works; the overlay fades in when the composition is ready and loops until the next one.

Env

Var	Notes
LIVEAVATAR_API_KEY	required — from https://app.liveavatar.com
LIVEAVATAR_AVATAR_ID	required if SANDBOX=false; pick a VIDEO avatar from /v1/avatars
LIVEAVATAR_VOICE_ID	optional
LIVEAVATAR_SANDBOX	true = free 1-min sessions on the fixed sandbox avatar
BACKEND_PORT	default 4000

Architecture

                 voice                       SSE events
  user  ──────────────────►  LiveAvatar  ──────────────►  frontend
                              (FULL)          ▲              │
                                              │              │ <hyperframes-player>
                                              │              ▼
                                          backend  ◄──── overlay on avatar video
                                              │
                              turn buffer ────┤
                                              ▼
                                    Haiku classifier  (intent: chat | generate)
                                              │
                                              ▼ (generate)
                                    Claude Code (-p, stream-json)
                                              │
                                              ▼
                                  workspace/<jobId>/composition.html
                                              │
                                       chokidar watcher
                                              │
                                              ▼
                                       hyperframes_ready

• backend/server.js — Express on :4000. Session tokens, classifier, generator spawn, SSE broadcast, file watcher.
• frontend/src/main.js — LiveAvatar SDK wiring, status pill, log panel, <hyperframes-player> overlay.
• workspace/<jobId>/composition.html — generated artifacts. Served at /workspace/....
• frontend/test-player.html — standalone player harness for debugging compositions outside the avatar.

Notes

• Generation isn't instant. Even with --model haiku, a hyperframes job takes ~30s–2min because Claude Code reads the skill's reference files before writing. The avatar acknowledges briefly so you're not staring at silence.
• LiveAvatar context "Hyperframes Studio v2" instructs the avatar to react to ideas but never claim it's generating — only the real pipeline calls session.repeat("On it…") when a generate decision actually fires.
• 5-min session timeout. Frontend keep-alive pings every 2 min.
• session.repeat() interrupts the avatar's current speech — useful for "On it" / "Here it is!" cues.
• Workspace dirs are gitignored. Generated compositions don't ship with the repo.

License

MIT