A Local-First Holographic Assistant
The full stack on one machine: microphone to speech recognition, a local model, synthesized voice, and a WebSocket HUD. An Unreal Engine MetaHuman is rendered live and streamed into the HUD under a hologram shader.
Read in full
Pipeline
A desk microphone feeds faster-whisper for speech recognition. A local Ollama model reasons over the transcript and the retrieved context. Kokoro synthesizes the reply. A WebSocket server pushes state to the HUD, built with Three.js and GLSL, and to a phone web app reached over a private tailnet.
Models
The primary chat model is qwen2.5:14b with llama3.2:3b as the fallback. Setting flash attention and a q8 key-value cache cut the primary model from about 12 GB to 10 GB at a 16,384-token context. Chat, speech recognition and synthesis all run locally. Game-frame vision for the arcade watcher is the one path that still uses a hosted model; desktop screenshots and chat images use local vision.
Retrieval
Brave Search is the only external API. A single shared retrieval and synthesis core replaced three divergent implementations, so every path grounds answers the same way.
Knowledge base
A self-maintaining wiki holds what Chloe has learned, next to a SQLite memory with embeddings. Recall is cosine similarity over embeddings with a threshold and two narrow keyword fallbacks. Paper III covers how the wiki is kept honest.
Avatar and HUD
Chloe's face is a MetaHuman rendered in Unreal Engine and streamed into the HUD at 60 fps. A cyan hologram shader, scanlines and flicker make her read as a projection. Her eyes and head follow the cursor and react to clicks. The three.js avatar remains as a fallback path. Mouth movement is being moved from amplitude-only to word-accurate lip-sync, with a measurement spike on a virtual audio cable.
October 4 hardening
A critique of the spec sheet led to a live check, which found three silent failures the health check had missed: a dead microphone, an expired calendar token and a hosted vision fallback still live.
- The mic code never picks a device the audio layer cannot open, rescans from the failure loop and says "I can't hear you" once per outage. Health now reports real open, read and error state instead of whether a thread exists.
- Hosted vision is limited to arcade frames while a known emulator is in front. Desktop vision is local or fails loudly.
- Importing the main module no longer starts a second voice loop.
- Every self-patch becomes a git commit authored by Chloe, built from only the patched hunks, so one command lists everything she changed about herself. Watchdog reverts carry the failing reason.
- Facts the brain learns automatically record which turn they came from.
Voice reliability fixes
- Windows exclusive-mode and privacy settings had blocked every audio device; both were corrected.
- Recognition output is gated against hallucinated phrases on silence.
- Currency amounts are normalized before speech.
- Format-constrained tool calling stopped output drifting into another language.
