Architecture overview
How DeskMate is put together — processes, the event protocol, the two loops, memory, and where each piece lives in the repository.
This is the short version for someone reading the code for the first time. The full design, in Chinese, is docs/architecture.md; the decisions behind it are the ADRs, and the vocabulary is CONTEXT.md. Those three stay in Chinese. If you open a pull request, please use the terms they define.
One core, several front ends
DeskMate is one core with several front ends. The core owns all state and all intelligence. Everything else is thin.
| Process | What it is | What it does |
|---|---|---|
| Main | Electron main process | Windows, tray, global shortcuts, system permissions, starting and supervising the core. Holds no business logic. |
| Core | Electron utilityProcess (TypeScript) | The event bus, the conversation loop, the deliberation loop, actions and their gating, perception, memory, emotion, model routing, MCP client, local models. |
| Audio | Hidden BrowserWindow | Microphone capture with echo cancellation, TTS playback, and the playback clock that mouth movement follows. |
| Stage | Transparent BrowserWindow | Renders the avatar and the subtitle bubble; hit-tests pixels so clicks pass through transparent areas. |
| Console | Normal BrowserWindow (React, shadcn) | Characters, memory, tools, tasks, models, settings. It never touches data directly; everything goes through the core. |
Audio is its own process because echo cancellation needs the microphone and the speaker output in the same audio context. It is separate from the stage so closing the avatar doesn’t end the conversation.
If the core crashes, the main process restarts it with exponential backoff (1 s up to 30 s) and the core recovers from persisted state. Five crashes within a minute stops the restarts and the tray offers a manual one.
The event protocol
The core and the front ends talk only through typed events, defined in src/shared/protocol and shared by every process.
type Envelope<T extends EventType> = {
v: 1;
id: string; // ULID
ts: number; // core monotonic clock, ms
characterId?: string;
type: T;
payload: PayloadOf<T>;
}; - Events say what happened or what a front end asks for. The core’s internal state is never exposed directly.
- A front end that wants data sends a query (
queries.ts): a question and an answer on the same port. Queries aren’t broadcast and aren’t logged. - Every event is appended to
logs/events.jsonlfor debugging and replay. - High-rate data — audio chunks, mouth frames, playback position — skips the bus and goes over two direct links with the audio process.
Two loops
- The conversation loop (
src/core/conversation/) is what you talk to. It is fast: it takes a finished user turn (or a decision to speak up), calls the character’sconversemodel, streams the reply into sentences, and hands each sentence to TTS. It can be interrupted at any point; the part that was actually heard is what gets remembered. Its model output can carry markers such as[affect joy]and[delegate]. - The deliberation loop (
src/core/deliberation/) is what does work. A[delegate]from the conversation loop becomes a task: a tool-calling loop on thedeliberatemodel, one at a time. Its result goes back to the conversation loop to be said in the character’s own voice. The deliberation loop never speaks.
The voice path is cascaded — VAD, turn detection, ASR, the model, TTS — rather than speech-to-speech. ADR-0001 has the reasons.
Actions
Everything the character can do is an action from some action provider: MCP servers, the built-in computer provider, and built-ins like recall and look. They register into one registry in a common shape, and one gate decides whether a call may run. See Tools and action providers.
Memory
Memory is per character, in one SQLite file (memory.sqlite), in three layers:
- Episodes — what happened, in a stretch of time the character lived through, summarized when it ends.
- Facts — durable statements about the user, each able to be replaced or expire.
- Relationship — a running sense of how the two of you stand.
Retrieval uses an embedding model, which you pick in settings (default bge-m3); changing it re-embeds everything in the background while the old vectors keep serving. While the character is idle it “dreams”: consolidating episodes into facts.
Emotion and autonomy
Each character has an affect with two axes, valence and arousal, that drifts back to a baseline with a half-life. The avatar shows it, and the voice carries it. Whether and when the character speaks up on its own is driven by a separate drive system (boredom and the need for company, tuned by urge in character.json) and gated by the autonomy level (quiet, companion, propose, autonomous), which is a user setting that applies to every character.
Where things are
src/main/ Electron main process
src/core/ the core
conversation/ conversation loop
deliberation/ tasks
actions/ MCP, computer use, gating
characters/ characters, avatar import, character packs
memory/ memory store, retrieval, dreaming
models/ model services and routing
voice/ TTS vendors
local-models/ model manifest and downloader
src/renderer/ stage, audio and console renderers
src/shared/ protocol types, i18n dictionaries
native/ the two Swift helpers
assets/ built-in avatar, default character, known avatars
scripts/ model download, icons, built-in avatar
website/ this site Further reading
Last updated on