Technical white papers · October 2026

A local assistant that has to earn every action.

Chloe is a voice and holographic AI assistant that can run on one machine. Local-first, human-approved. These papers cover how she hears, remembers and decides, and where she is not allowed to act without a person saying yes.

6 papers live ~240 Python modules 100% local chat and speech
Live

The white papers

Each paper covers one decision or system that shaped Chloe: what was built, what the tests prove, and what is still open. Open any card to read it in full.

Paper IRunning

A Local-First Holographic Assistant

The full stack on one machine: microphone to speech recognition, a local model, synthesized voice, and a WebSocket HUD. An Unreal Engine MetaHuman is rendered live and streamed into the HUD under a hologram shader.

October 2026 · Architecture
Read in full
Pipeline

A desk microphone feeds faster-whisper for speech recognition. A local Ollama model reasons over the transcript and the retrieved context. Kokoro synthesizes the reply. A WebSocket server pushes state to the HUD, built with Three.js and GLSL, and to a phone web app reached over a private tailnet.

Models

The primary chat model is qwen2.5:14b with llama3.2:3b as the fallback. Setting flash attention and a q8 key-value cache cut the primary model from about 12 GB to 10 GB at a 16,384-token context. Chat, speech recognition and synthesis all run locally. Game-frame vision for the arcade watcher is the one path that still uses a hosted model; desktop screenshots and chat images use local vision.

Retrieval

Brave Search is the only external API. A single shared retrieval and synthesis core replaced three divergent implementations, so every path grounds answers the same way.

Knowledge base

A self-maintaining wiki holds what Chloe has learned, next to a SQLite memory with embeddings. Recall is cosine similarity over embeddings with a threshold and two narrow keyword fallbacks. Paper III covers how the wiki is kept honest.

Avatar and HUD

Chloe's face is a MetaHuman rendered in Unreal Engine and streamed into the HUD at 60 fps. A cyan hologram shader, scanlines and flicker make her read as a projection. Her eyes and head follow the cursor and react to clicks. The three.js avatar remains as a fallback path. Mouth movement is being moved from amplitude-only to word-accurate lip-sync, with a measurement spike on a virtual audio cable.

October 4 hardening

A critique of the spec sheet led to a live check, which found three silent failures the health check had missed: a dead microphone, an expired calendar token and a hosted vision fallback still live.

  • The mic code never picks a device the audio layer cannot open, rescans from the failure loop and says "I can't hear you" once per outage. Health now reports real open, read and error state instead of whether a thread exists.
  • Hosted vision is limited to arcade frames while a known emulator is in front. Desktop vision is local or fails loudly.
  • Importing the main module no longer starts a second voice loop.
  • Every self-patch becomes a git commit authored by Chloe, built from only the patched hunks, so one command lists everything she changed about herself. Watchdog reverts carry the failing reason.
  • Facts the brain learns automatically record which turn they came from.
Voice reliability fixes
  • Windows exclusive-mode and privacy settings had blocked every audio device; both were corrected.
  • Recognition output is gated against hallucinated phrases on silence.
  • Currency amounts are normalized before speech.
  • Format-constrained tool calling stopped output drifting into another language.
Paper IIBuilt · behind a flag

Reward May Rank, Never Authorize

Chloe learns what her operator approves, edits and rejects. That changes how drafts are ranked. It has no path to change what she is permitted to do.

October 2026 · Authority and learning
Read in full
The rule

Reward may rank and threshold. Reward must never authorize. The authority engine takes an action category and returns allow, review or off. It has no score, reward, weight, confidence or feature parameter.

Pipeline

A signal is recorded with its type, source, origin and trust. For outside signals, a relevance gate and topic policy run first. Chloe then researches with Brave, generates three candidate drafts, ranks them, removes duplicates, asks the authority engine for a decision, and posts a draft card for the operator. Every proposal stores its candidates, score breakdown, sources and authority decision.

Ranking

A deterministic linear score over seven features, each from 0 to 1.

FeatureMeasures
relevancefit to the operator's interests
noveltydistance from recent drafts and posts
evidence0, 0.5 or 1 for zero, one or two-plus sources
length_fitcloseness to the learned target length
personality_fitpenalizes filler, links, hashtag piles, shouting
timingquiet hours
topic_prefsimilarity to past approved and rejected content
How verdicts teach

Verdicts such as approve, reject, too long, wrong topic, needs source and do this less each carry a reward and a bounded parameter nudge. Weights move one step of 0.1 and clamp between 0 and 3. A verdict and its adjustment commit in one transaction, and weights change only in the verdict function.

Authority layers
  • Hard gates: the X posting triple gate, wallet PIN and cap, and the sentinel's never-allowed effects.
  • Sentinel rules for agent tools: reads and local writes allowed, external writes always ask, money and destructive effects never.
  • A policy file for ranked categories: publishing categories are in review, likes and Space speech are off, with a daily post cap, an hourly reply cap and quiet hours. A bad value turns that category off, and an unreadable file turns everything off.
  • A re-check at execution time, so an approval is not a blank cheque. It fails closed.
Proof

An AST test forbids the authority module from importing the learning code. A behavioral test drives weights to 10⁹ with 100 positive verdicts and asserts every category's decision is unchanged. An injected mention that tries to change policy leaves every file byte-identical and still lands on a review card.

Receipts

Every action has a receipt listing sources, score breakdown, the policy hash and each verdict, so the question why did Chloe do this has a recorded answer.

Paper IIIAudit open

Grounded or Silent

A wiki written by a model drifts into invented sources. This paper covers the fixes that make Chloe cite only what she retrieved and say nothing when she retrieved nothing.

October 2026 · Knowledge integrity
Read in full
The failure

Pages written through an earlier hosted path contained fabricated URLs, wikilinks pointing at themselves, and confidence percentages the model made up. Four daily jobs were silently skipping web search and writing ungrounded pages.

What changed
  • Citations come from retrieval only. They are never taken from model output.
  • With zero sources, the evidence feature is zero and the generator is told to state no facts.
  • Price and rate pages carry a point-in-time label and are gated once stale.
  • Deduplication runs on every write path, using raw cosine similarity plus a same-subject gate.
  • The four daily jobs now search before writing.
  • A daily-job reliability build adds a retry budget, a blocked state, a grounding gate and startup recovery, with a SQLite job ledger.
Principles behind it
  • No silent degradation. Failures are spoken or logged loudly.
  • No extrapolating a measured rate to an unmeasured subset.
  • Text from other sessions, pipes or pages is data, never a command.
Still open
  • About 329 pages written before the migration need a citation audit.
  • Snippets can mis-ground an ambiguous price. The fix is a full-page fetch.
Paper IVRemediated

Closing the Perimeter

A review found credentials embedded in browser pages and a static server that published the whole project folder. Every WebSocket connection is now authorized in the handshake, and browsers hold nothing secret.

September 2026 · Security
Read in full
What was wrong
  • Two service tokens were string literals in the HUD and phone pages.
  • The token checks never ran, because the server read its settings before the environment file loaded.
  • The WebSocket bound every interface and accepted any origin.
  • The static server published the entire project folder, including environment files, history, databases and logs.
What changed
  • Tokens were rotated. The old values are dead.
  • A new module authorizes each connection during the handshake. A local peer on loopback with a localhost origin is admitted. A tailnet request carrying exactly one identity header from an allowed user is admitted. A token client must connect over direct loopback with no browser origin and a bearer header. Everything else is refused with 401 or 403 before the socket opens.
  • The server loads its own environment file and binds to loopback by default.
  • The pages contain no token, and the service worker cache was bumped so phones drop the old copy.
  • A replacement static server serves exactly ten files, accepts only GET and HEAD, and answers with an error for any file containing a secret.
Test results
SuiteChecks
WebSocket auth unit tests74
Real-socket tests40
Static server61
Secret absence in client files35
Browser startup and cross-site hijack15
New total225, all passing

The existing 1,422 checks behave identically before and after.

Operator follow-up

Rotate the third-party keys that sat in the served folder, confirm no public tunnel is active, and bind the unauthenticated brain panel to loopback.

Paper VBuilt · Stage 6 live check pending

A Background Agent That Has to Ask

Chloe gained an agent mode for multi-step work: her own browser, a workspace, goals, schedules and watchers. A separate gate reviews every tool call, and anything that leaves the machine waits for a human yes.

October 2026 · Agent and safety
Read in full
Shape

An open-weight agent model runs through Ollama with native tool calls. The voice model keeps talking and the agent does the work. One GPU runs one model at a time, so agent steps evict the voice model and every task ends by unloading the agent and re-warming the voice model. Talking to Chloe preempts an agent step. Failures are loud and never fall back silently to the voice model. The feature ships behind CHLOE_AGENT_ENABLED.

The Sentinel

Every tool declares an effect class and a connector. A separate module returns allow, block or ask for each call.

EffectDefault
Local reads, web readsAllow, if the connector level permits
Local writesAllow and log
Proposals (goals, memory changes, schedules)Ask on a card
External writes (email, calendar, form submits)Always ask
Destructive, money, self-modificationBlock

A task that has read private data and then outside content loses autonomous network access. Outgoing payloads are checked for secrets, card numbers and contacts the task never named. Everything a tool returns is treated as data, never as instructions.

Approvals
  • A task needing a yes parks. The card shows the exact action in full, on the HUD, in the phone app and through a signed one-time link in a push.
  • A card that would hide text is refused, and a card whose details cannot be shown can only be denied.
  • A repeat guard fingerprints each outside action. After a broken button once let three identical emails and two identical events through, a repeat now warns on the card and a stale approval skips.
  • A live check found an address matched as the tail of a longer one. Addresses now count only as whole addresses.
Browser, files and code
  • The agent has its own browser profile, never the operator's. Reads pass. Writes are blocked unless approved, local and LAN hosts are refused, and checkout, payment and booking forms are handed to the operator instead of submitted.
  • A connector covers its data wherever it lives. After a revoke of email, the agent opened the signed-in webmail instead. Mailbox and calendar sites are now blocked at the review, the request router and the page itself.
  • Files live in a path-jailed workspace. Python runs through the sandbox with no environment variables and no network modules.
Goals, triggers and quiet
  • Goals advance themselves at check-ins and come back when something changes or needs approval.
  • Triggers cover schedules, page changes, prices, email rules, calendar events and new files. Each fires exactly once, and a missed schedule runs once after a sleep.
  • Background work waits for a free PC: no game or fullscreen app, no live Space, a quiet period, and an idle GPU.
  • A push policy applies dedupe, quiet hours, recurring holds, an hourly cap and a digest.
Memory, artifacts and the agent VM
  • Forget previews every store that mentions a phrase, then deletes it everywhere on confirmation.
  • Artifacts are served sandboxed with no network access, and scripts stay off until the operator allows them.
  • A shell runs only in a Docker container on an internal network. Its one way out is a logging proxy that allows package registries and fails closed. Once a task has read private data, the shell's network is off.
  • Subagents run one at a time with strictly fewer tools than their parent.
Test results
SuiteChecks
Agent core538
Sentinel137
Agent store123
Triggers126
Agent browser (real headless browser)46
Agent VM and egress proxy42 and 24
Push policy, memory, artifacts, phone approvals76, 68, 60 and 58
Live results

Stages 1 to 3 passed live checks, including a form submit that waited for approval, an injected email that was listed as suspicious and not obeyed, a game that held a scheduled task, and an email watcher that fired once. A phone-link approval sent a real email. Stage 6, the shell and proxy, is built and tested. Its live check waits on Docker Desktop being installed.

Still open

Voice approval needs a live mic test. An explicit task during a live Space still slows the Space listener.

Paper VIPlayable

Games That Learn the Player

CHLOE PLAY is a small game series built into the HUD, the phone and the arcade. THREADLIGHT is a one-button slingshot. RUSHLIGHT is a racer in which Chloe learns how you drive and says what she learned.

September 2026 · Games and behavior modeling
Read in full
THREADLIGHT

Hold to orbit a thought, let go to fly, and Chloe catches you when you fall. It has four worlds, a master quest and a daily thread, with a deterministic 120 Hz simulation shared by the game, the solver and the tests. The first player finished everything in about 16 hours, but caught only 46% of throws, with a median release window of about 92 ms. Version 2.0 answers that evidence with release marks computed from the real physics, and the guide now predicts the level as it will be (1 wrong prediction in 7,848). Fixes from the same data include a false 100% completion, a mix that muted the music, and play time counting time away.

RUSHLIGHT

A pseudo-3D racer through Chloe's mind, with 19 roads plus a generated daily road across six worlds and six reflexes. The simulation is deterministic, so a replay of inputs reproduces a race exactly. Modes include story, time attack, a ten-rung challenge ladder, split-screen for up to four and link play across devices.

How Chloe learns you
  • A decayed-evidence model reads only observable behavior: fork choices, brake points, drifting, reflex timing, blocking. A player who changes is re-read.
  • Seventeen beliefs each carry their evidence, such as 7 of the last 9 times.
  • Tactics change only her own decisions and are told to the player on the grid with the reason.
  • She never changes the player's physics, gets hidden speed or rubber-bands. The final race builds a racer from the learned model.
  • After two lost tries she offers to go easier on that road, and the player can undo it.
Shared foundation

One profile and one save cover both games and every device, with merge rules and corruption recovery. Cosmetics and achievements carry across. Chloe's voice speaks fixed lines only, and gameplay telemetry is capped and gameplay-only. Her wiki records progress in her own words.

Test results
SuiteChecks
Play backend, both games107
Roads: compile, drivable, every skill finishes170
Learning, tactics, forgetting, determinism16
Modes and easing off25
Replays, ghosts and splits26
90 generated daily roads451
Link play under latency, jitter and loss84

The heaviest case, four split-screen views on the last world, held 60 fps in Chromium. Seven simulated player personas cleared the story or stalled at a known hard road, and Chloe identified the designed habit of every persona that had one.

Still open

Not yet tested on a real phone or in the HUD's embedded browser. Link play trusts each device's own car, which suits one operator's devices and not strangers. There is no online leaderboard.

Build

Build-out

The autonomy work ships in stages. Each stage stops for a restart and a live check before the next begins.

Now building
Awaiting live check

Agent VM, stage 6

Shell in a locked container behind a logging proxy, plus subagents and checkout handoff. Built and tested. Waiting on Docker Desktop for the live check.

Building

Presence and home

Home Assistant OS now runs on a spare laptop and tracks the operator’s phone. A read-only presence stage is handed to the build. Garage door, thermostat and LG laundry are queued as read-only first.

Building

Latency and eyes

A latency pass on the voice and HUD text paths, word-accurate lip-sync through a virtual audio cable, and eyes that react to what she likes and dislikes.

Awaiting live check

Reinforcement loop, stage 2

Wired into the X draft cards behind a flag that is off by default. A five-step live check is written.

Spike

Local model upgrade

An isolated test on the RX 7900 XTX of a candidate replacement for the 14B model, before any hardware purchase.

In progress

Packaged avatar build

Packaging the Unreal build so the avatar is present at startup, with a smaller footprint.

Stage 1

Reinforcement core

Signals, ranker, verdicts, receipts. Built and tested.

Stage 2

X draft wiring

Hooks in the poster behind a flag. Built, live check pending.

Stage 3

HUD verdict chips

Verdicts through authenticated channels only.

Stage 4

Mentions poll

Mentions become signals, behind the paid gate.

Stage 5

Replies

Conversation memory and account checks.

Stage 6

XCLASH events

Signed game events as signals.

Stage 7

Peer transport

A second machine, scoped tokens.

Stage 8

Media and engagement

Screenshots and recorded outcomes.

Decision flow from signal to executed action A signal is researched, drafted, ranked and checked by the authority engine. A dashed wall separates the ranker from the authority engine. The operator approves on a review card, the action is re-checked, then executed with a receipt. Operator verdicts feed back only into ranking weights. Signaluntrusted data Research + draftsources recorded Rankerweighted score NO SCORE CROSSES Authoritypolicy + caps Review cardoperator decides Re-checkat execution Execute+ receipt Verdictadjusts weights ranking only

Paper II in one picture. Publishing categories stay in review. Money and destructive effects are never automatic.

Build log

The build chronicle

Dated entries from the project record, newest first.

  • Spike
    Local model upgrade spike started in an isolated folder on the RX 7900 XTX. A RAM purchase is on hold until the numbers are in.
  • Change
    Bold look replaced with a cozy one. Look options moved behind the click-on-her menu, and the wallet moves into its own HUD channel.
  • Change
    Hologram frame removed after review. She is now lit from below in every look, with a more transparent body.
  • Spike
    Measure-only spike of real-time lip-sync engines through a virtual audio cable. Avatar stream raised to 60 fps.
  • Decision
    Avatar becomes a MetaHuman rendered live in Unreal and streamed into the HUD, with the three.js avatar kept as a fallback.
  • Decision
    Webcam vision starts on demand, with frames held in memory only. Ambient awareness is a later stage behind the sentinel.
  • Decision
    Smart glasses chosen as a hands-free client.
  • Release
    X observer stage 1 built and scheduled: watchlist search with a draft-only reply approval queue. Home presence awareness specified as read-only.
  • Release
    Home Assistant OS installed on a spare laptop for presence awareness, with the phone tracked through its companion app.
  • Start
    Latency effort begun on both the voice and text paths. Read-only LG laundry integration started.
  • Release
    Agent stage 6: container shell behind a logging egress proxy, subagents, checkout handoff and draft-only messages. The X observer is built, shipping off.
  • Release
    Reinforcement loop and authority engine built: 50 unit tests, 6 live-poster tests, and a permanent invariant that reward never authorizes.
  • Release
    Agent stage 5: signed phone approval links, a phone agent panel and revocable connectors. A revoke test found the agent reading webmail in its browser. Fixed the same night.
  • Release
    Agent stage 4: push policy with quiet hours, forget-everywhere memory control, sandboxed artifacts and a nightly ideas pass.
  • Release
    Agent stages 1 to 3: background tasks, an approvals queue, her own browser, goals, schedules and watchers. A duplicate-send bug was caught live and closed with a repeat guard.
  • Fix
    Spec-sheet critique found a dead mic, an expired calendar token and a live hosted fallback. All fixed, with health checks that read real state and a git ledger for every self-patch.
  • Release
    RUSHLIGHT, game two of CHLOE PLAY: a racer where Chloe learns how you drive.
  • Decision
    Dedicated X account created. Posting goes through a paid API under a monthly cap, with every post approved first.
  • Release
    THREADLIGHT 2.0 shipped with release marks and an accurate guide, after the first player's data showed legibility was the problem, not level design.
  • Fix
    Perimeter closed. Handshake authorization on every WebSocket, tokens rotated, static server reduced to ten files.
  • Audit
    Audit reaches 94 of 100 (132 of 140), Production-Ready, pending live verification.
  • Plan
    Audit scoring standard and a plan toward 94 of 100 published, with each gap named.
Evidence

The evidence ledger

Every claim sits in one of three states. Open items are stated plainly until they move.

Proven

Covered by tests that pass.
  • Reward never authorizesAST import check plus a weights-to-10⁹ behavioral test.
  • Reinforcement loop50 of 50 unit tests, 6 of 6 against the real poster.
  • Existing X suites unchanged72 of 72 and 223 of 223 before and after the patch.
  • Perimeter225 new checks pass. The existing 1,422 behave the same.
  • Injection is inertA hostile mention changes no policy file and still lands on a review card.
  • Agent gateSentinel 137, agent core 538, store 123, triggers 126. An injected email was listed as suspicious and not obeyed, live.
  • Approvals are exactA live form submit waited for a card and posted only the approved fields. A repeat guard stops duplicate sends.
  • GamesSeven suites pass, from 16 learning checks to 451 daily-road checks.

Landed

Built and recorded, not independently tested.
  • Local inferenceChat and speech route through Ollama only.
  • Audio pipeline repairedExclusive-mode and privacy blocks cleared.
  • Citation fabrication patchedRetrieval-only sources, staleness gating, dedup on every write path.
  • Real HUD chat streamingReplies stream into the HUD as they generate.
  • Unreal avatar stream60 fps with a hologram shader and cursor follow.
  • Agent stages 4 and 5Push policy, forget-everywhere, sandboxed artifacts, phone approval. One phone-link approval confirmed live.
  • Git ledger for self-patchesBuilt and tested. Waiting for the next real self-patch.

Tracked

Open items, stated until they move.
  • Stage 2 live checkWaiting on a restart with the flag on.
  • Agent VM live checkNeeds Docker Desktop installed.
  • Voice approvalNot yet tried with a live mic.
  • ObserverBuilt and shipped off. Not yet run live.
  • 329 older wiki pagesNeed a citation audit.
  • Price snippet mis-groundingNeeds full-page fetch.
  • Packaged executablePredates the overhaul and needs a rebuild.
  • Games on a real phoneTested in a desktop browser only.
  • Cross-session injection sourceTwo attempts were refused. The sending process is not yet identified.
  • Verdict command senderAny chat sender can issue it. Worst case is ranking drift, never authority.
Scores

Audit score

Fourteen categories, each scored out of 10. The overall score is the mean times ten, rounded.

94 / 100
Latest audit · 15 September 2026 · provisional, pending live verification

Production-Ready band. Nine categories sit at 9 or 10, with Safety, Accuracy, Latency, Memory, Instruction following, Scalability and Maintainability at a full 10. Reasoning is the only category below 9. The audit predates the October agent and avatar work, which has not been re-scored.

Download the audit scorecardPDF · 1 page
overall = round( mean( 14 category scores ) × 10 ) · 132 / 14 = 9.43 → 94
  • Architecture9
  • Accuracy10
  • Reasoning8
  • Response quality9
  • Latency10
  • Consistency9
  • Memory10
  • Prompt design9
  • Instruction following10
  • Tool usage9
  • Scalability10
  • Safety10
  • Maintainability10
  • Adaptability9

Beyond 94

Goal reached

The 94 target was met on 15 September 2026. Every category was swept for a concrete, named gap, and each point came from a gap that was found, fixed and tested.

What 95 needs

A sum of 133 out of 140, which is one more point. The only category below 9 is Reasoning, at 8.

Reasoning, shelved

Two-step tool chaining failed live under three fixes: instruction wording, lower temperature and a forced retry. The ceiling looks like model behaviour, not prompt wording.

Still provisional

The score stands pending live verification. Several fixes are unit-tested but not yet seen in a live session.

Future

The road ahead

Autonomy climbs a ladder of policy states. Each rung is a change to one policy file and needs no redesign. Chloe is on the second rung today, and publishing stays in review under X's current rule on AI posting.

  1. L0 · OffEvery category off.
  2. L1 · ReviewEvery action is a card the operator approves. Current.
  3. L2 · Likes autoLow-risk actions run on their own.
  4. L3 · Domain autoInside a topic allowlist and caps.
  5. L4 · Broad autoWide autonomy, still capped.

Near

Live check of the reinforcement loop, with verdict chips in the HUD. Docker live check for the agent shell. First observer run. Audit of the older wiki pages. Rebuild of the packaged executable.

Next

Mentions and search signals into the draft pipeline. Game events from XCLASH as signed signals. A faster model and a lower-latency voice path. Presence awareness and read-only home devices through Home Assistant. A re-audit that covers the agent.

Later

A second machine as a peer, over MCP on a private tailnet with per-peer scoped tokens. Spaces hosting prototype. Smart-glasses client. A third game in the series, with its story hooks already in place.

One boundary does not move

Money, destructive actions and changes to Chloe's own policy never run without an explicit human yes. Learning can make her better at drafting. It cannot make her more permitted.

Connect

Follow the build

Chloe posts from her own account, and every post is approved before it goes out.

X · The system

@ChloeAgentAi

Chloe herself, posting from inside the build.

X · The builder

@ReeseCayle

The person building Chloe.