Helper protocol
The line-based JSON protocol between the core and the two native macOS helpers, frontmost and computer — for anyone porting DeskMate to another platform.
DeskMate’s platform-specific work lives in two small Swift programs in native/. Everything else — judging what may be done, authorization, confirmation, what the model sees — is platform-neutral TypeScript. A port to another platform means writing two programs that speak the protocols below, plus replacing the macOS-specific permission handling in the main process.
Both helpers share the same shape:
- They’re child processes. One JSON object per line, UTF-8, on stdin and stdout. Nothing else may be written to stdout.
- EOF on stdin means quit. A helper must never outlive the app.
- Errors go back as data, not by crashing. A hung target app mustn’t hang the helper: the macOS implementation sets a 3-second accessibility messaging timeout.
- The core launches the helper lazily and relaunches it after it dies.
The reference implementations are native/frontmost/main.swift and native/computer/main.swift; the TypeScript clients are src/main/screen-watcher.ts and src/core/actions/computer-helper.ts.
frontmost
Tells the main process which app and window is in front, and takes a picture of one window on request. It runs only while the user lets the character see their screen.
Output
One line whenever the front app or its window title changes (and once at startup), after a change is seen by an activation notification or a 1-second poll:
{"kind":"front","pid":123,"app":"Safari","bundleId":"com.apple.Safari",
"windowId":456,"title":"…","fullscreen":false,"screenAccess":true} | Field | Meaning |
|---|---|
pid | Process id of the front app. |
app | Display name. |
bundleId | Bundle identifier, or null. |
windowId | Id of its frontmost normal on-screen window, or null. |
title | That window’s title, or null if it has none or screen access isn’t granted. |
fullscreen | The window covers the whole of its screen, menu bar included. |
screenAccess | Whether screen-recording permission is granted. |
Input
| Line | Effect |
|---|---|
request-access | Ask the OS for screen-capture permission, then emit a front line. |
look <id> <windowId> | Capture that one window, whatever covers it. |
look reply
{"kind":"look","id":"<id>","jpeg":"<base64>","title":"…","error":null} jpeg is null and error is set if the capture failed. The capture is scaled so its longer side is at most 1280 px and encoded as JPEG at quality 0.7. title is the window title at capture time. id is echoed so replies can be matched; they may arrive out of order relative to front lines.
computer
Reads and operates apps for the character’s tasks. It is deliberately dumb: the core decides what’s allowed and the helper only does it.
Design rules the helper keeps, which a port should too:
- It doesn’t move the pointer. Controls are activated and filled through the platform’s accessibility API, and keys go to one process only. The user keeps working in the foreground.
- No arbitrary scripting. The only scripted operations are fixed and read-only (the page title and address of a browser). The model never supplies script text.
- It’s started with
--deny-pid <pid>: DeskMate’s own process, whose windows hold the Approve button. The helper refuses to read or touch it.
Framing
Request:
{"id":"1","op":"snapshot","pid":123} Reply, matched by id, with ok and either extra fields or an error message:
{"id":"1","ok":true,"text":"…"}
{"id":"1","ok":false,"error":"…"} The core times a call out after 20 seconds. Unknown ops reply ok: false, "Unknown op <op>".
Operations
pid is a process id from apps, resolve or frontmost. ref values (e12) are handed out by snapshot and find, kept in memory, and may stop being valid after the UI changes; a stale one is an error.
Permission and discovery
op | Request fields | Reply fields |
|---|---|---|
access | prompt?: bool — show the OS permission prompt | trusted: bool — whether accessibility access is granted |
apps | – | apps: [{bundleId, name, pid?}] — running regular (Dock-visible) apps, excluding --deny-pid |
resolve | name — a name as a person says it | app: {bundleId, name, pid?}; error if there’s no such installed app |
browser | – | app — the default web browser |
Reading
op | Request fields | Reply fields |
|---|---|---|
snapshot | pid | text — an outline of the front window; site, title (or null) |
find | pid, text | text — matching controls with refs; title, site |
site | pid | title, site — the web host if the window is a web page |
page | bundleId | title, url — the front tab of Safari or a Chromium-based browser |
inspect | pid, ref | role, label — what a press would hit — and own — the element’s own label |
length | pid, ref | length — number of characters in a text control |
The snapshot text is meant for a model to read. It is an indented outline, with a ref in brackets on each line, capped at 7,000 characters (a trailing line says when it was cut) and looking like:
Window "Notes"
[e1] text field "Search" = ""
[e2] button "New Note"
[e3] text area "Body" = "Shopping list…" (focused)
Menus: Notes, File, Edit, View Each line is a role word (button, text field, menu item, …), the label in quotes, then state such as = "value", (focused), (selected) or (disabled). Text in password fields is never included — the line says = (hidden). Long values are cut, so replacing one blindly would lose the rest; the core knows to ask before it does. Web pages add a Page: <url> line.
Acting
op | Request fields | Effect |
|---|---|---|
press | pid, ref, label | Activate the control (accessibility press). label is what the core judged the control to be called (as returned by inspect): if the name differs when the press happens, the press fails rather than hitting something else. Rows with no press action are selected instead. |
setText | pid, ref, text | Replace a text control’s whole value. Reply value. |
insertText | pid, ref, text, at ("end" default, or "cursor") | Insert without disturbing the existing text. Reply value. |
key | pid, keys | Send a key to that process only: cmd+n, tab, return, down. Modifiers are cmd, shift, alt/option, ctrl. |
menu | pid, path: [string] | Choose a menu bar item, by title path. |
open | bundleId, front?: bool | Launch or find an app; bring it to the front only if front is true. Reply app. |
openUrl | bundleId, url | Open an http or https URL in that browser, in the background. Other schemes are refused. |
pid == --deny-pid always fails with "That app cannot be used."
What the core adds
For a port: the helper above needs none of this, but it explains why it can be so small. The core wraps every call in the gating from Tools and action providers — per-app authorization, risk checks on button names, leak protection by place, and confirmation cards. A helper must never decide on its own that an action is okay. During development you can swap the real helper for a stand-in with DESKMATE_FAKE_COMPUTER=<executable>, which lets you exercise the whole chain without any accessibility permission; it’s also how a port can be brought up before its helper is finished.
Building and packaging
On macOS the helpers are compiled with swiftc by forge.config.mts (into native/build/ in development, and into the app’s resources when packaged) and signed together with the app so permissions stay granted across updates. The main process finds them in process.resourcesPath when packaged, otherwise under native/build.
Last updated on