A chat assistant returns text that a person must then act on; the work of producing the document, updating the spreadsheet, or sending the reply remains theirs. OpenRomeo is a cross-platform desktop agent built on the opposite premise: the unit of output is a finished artefact in the user's own workspace — a formatted document, an updated workbook, a triaged inbox — rather than a message describing one. This paper describes the system that premise requires. Four components carry it. A turn engine runs the model-tool loop asynchronously with blocking provider and tool calls off the event loop, executes read-only calls concurrently while keeping writes and shell commands strictly ordered, and can be interrupted mid-stream, mid-tool, or between iterations; on restart it detects tool calls left unanswered by a previous process and re-processes them rather than resuming into an inconsistent transcript. A permission layer classifies every tool by intrinsic risk — read, local write, execution, or off-machine side effect — and decides allow, deny, or ask against the session mode and the workspace root, with the decision function deliberately separated from the surface that renders the prompt. An Inbox generalises approval to the unattended case: rather than acting alone or stalling invisibly, a running agent parks its request as an item that any surface can resolve exactly once, under a first-responder-wins contract. A skill layer adopts the Anthropic SKILL.md format with progressive disclosure, so only skill names and descriptions occupy context while full instructions load on demand; the built-in document suite is overridable by name from a user or workspace folder. Credentials and conversations remain on the machine, with an optional cloud broker used only for OAuth consent — a boundary the paper states precisely, including what the broker does see. OpenRomeo is a fork of OpenWorker, and this paper distinguishes inherited architecture from what the fork adds.
The dominant form of AI assistance is conversational: a person describes a task, a model returns text, and the person performs the task. For questions this is the right shape. For work it is not, because the deliverable — the report in the correct template, the spreadsheet with the numbers actually updated, the reply actually sent — is what the task consists of, and text describing it leaves the labour where it started.
Closing that gap requires an agent to touch things: the file system, the terminal, and third-party services holding real data. Each capability that makes the agent useful is the same capability that makes it dangerous, and the interesting design problem is not obtaining permissions but structuring them — deciding what may proceed silently, what must be asked, what happens when no one is watching, and what remains true if the machine is turned off mid-task.
OpenRomeo is a desktop application for macOS and Windows addressing that problem. It is a fork of OpenWorker by Andrew Ng and contributors, itself built on aisuite; the agent engine, permission model, and connector platform originate there, and this paper marks the boundary where it matters. Section 2 describes the architecture, Section 3 the turn engine, Section 4 permissions, Section 5 the Inbox, Section 6 skills, Section 7 the provider layer, Section 8 the credential boundary, and Section 9 the limitations.
The system is three processes with one direction of dependency.
The desktop shell supervises a local agent server rather than embedding the agent, so the interface can be reloaded, closed, or replaced without disturbing a running task. Because the server is an ordinary local service, the same backend drives the packaged desktop app and a browser during development, and long-running work survives interface restarts — a property the recovery behaviour in Section 3.3 depends on.
| Component | Responsibility |
|---|---|
| Agent backend (Python) | Turn engine, providers, tool registry, permissions and risk, connectors and MCP, skills, memory, automations, audit log, Inbox |
| Desktop surface | React interface inside a Tauri shell; supervises the server process |
| Speech sidecar (Rust) | Local speech-to-text for voice input |
| Packaging | macOS disk images including an Intel cross-build, Windows installer and MSI, update manifest |
One user request becomes many model-tool iterations, continuing until the model stops requesting tools, a rail trips, or the user interrupts. Three properties of this loop are worth stating.
When a model requests several tools at once, executing them in sequence wastes time and executing them all in parallel is unsafe. The engine splits the difference along the risk classification of Section 4: read-only calls run concurrently, while writes and shell commands remain strictly ordered. Reading six files at once is safe because the operations do not interact; writing two files concurrently is not, because their ordering may be the difference between a correct and a corrupted workspace. Blocking provider and tool calls are dispatched off the event loop, so the interface stays responsive during a long model call.
An agent that cannot be stopped is not trustworthy regardless of how good its judgement is. Interruption is handled at each place a turn can be waiting: between streamed chunks of a model response, inside a running tool via registered interrupt hooks, and at the loop checkpoint between iterations. A stop request therefore takes effect promptly rather than at the next convenient boundary, which for a long shell command or a slow API call would be indistinguishable from ignoring it.
Conversations with tool calls have a structural invariant: every assistant tool call must be followed by its result. A process that dies mid-tool — a crash, a quit, an evicted engine — leaves a transcript violating that invariant, and naively resuming produces either a provider error or a model reasoning from a request whose outcome it never saw. The engine instead inspects the trailing assistant message on resume, identifies tool calls with no answer, and re-processes them. Turn-ending conditions such as an error or an interruption are persisted as display-only markers, so a stray retry cannot re-answer a turn that already completed.
A bounded iteration count acts as the outer rail against loops that make no progress.
Earlier designs of this kind maintain hardcoded sets of dangerous tool names inside the permission logic. OpenRomeo's backend instead treats risk as a declared property of a tool, read through a single classification function — which means adding a tool does not require editing the permission engine, and an unknown tool is classified conservatively rather than escaping the gate.
| Class | Meaning | Treatment |
|---|---|---|
read | No side effects | Always allowed; eligible for concurrent execution |
write_local | Mutates the workspace | Path-scoped to the working root and gated by mode |
exec | Runs commands | Mode-gated; command prefixes refine the decision |
external | Side effects off the machine | Gated, and the hook for unattended Inbox routing |
Effective risk resolves in a fixed order: a user-local override, then the by-name table for vetted built-in tools, then tool metadata declaring that approval is required, and otherwise read. The override exists chiefly so that a deliberately conservative default — notably for tools arriving over MCP — can be relaxed by the user rather than by the vendor.
Risk is then combined with a session mode. Discuss and Plan are read-only, differing only in that Plan drives an explore-then-propose-then-execute contract; Interactive is the default, permitting reads and asking before writes and commands; Auto allows without prompting but remains path-scoped; Custom is Interactive with a configured allow-list. Approvals may be granted once or as a task-scoped standing rule, and when such a rule permits a call the decision carries the rule with it, so the audit log and the interface can state which grant applied.
One structural choice deserves emphasis: the permission engine only decides. It does not prompt, block, or render. The turn engine routes a needs-user decision to whichever surface is attached and records the outcome. That separation is what makes Section 5 possible — the same decision can be answered by a desktop dialog, a queue item, or a message from another device, without the policy layer knowing which.
Interactive approval assumes someone is present. Scheduled and unattended runs break that assumption, and the usual resolutions are both bad: act without asking, or stall silently until someone returns.
The Inbox is a durable, cross-session queue of what agents need from the user — an approval, a question, a notification, or a request to be granted a directory. When a permission decision requires a person and no one is attached, the request becomes an Inbox item and the agent suspends until it is resolved.
The contract is explicit about the race it must survive. Each item moves from pending to resolved exactly once; resolution is idempotent and first-responder-wins. This matters because the same item may be answerable from several places at once — the application, a messaging connector, or the composer after resuming the session — and two simultaneous answers must not produce two actions. The Inbox is the store of record and other surfaces are transports of the same items, rather than each surface maintaining its own notion of what is pending.
The result is that an unattended agent has a third option beyond acting alone and giving up: it can wait, visibly, with the reason for waiting attached.
Skills teach the agent repeatable procedures. OpenRomeo adopts the Anthropic Agent Skills SKILL.md format — a folder with YAML frontmatter naming and describing the skill, optionally constraining the tools it may use, followed by a body of instructions and any resources or scripts — so skills written for other tools in that ecosystem work without translation.
The loading strategy is progressive disclosure, and it addresses a real constraint. Injecting every skill's full instructions at session start consumes context proportional to the number of installed skills, most of which are irrelevant to the current request. Instead only the catalog — each skill's name and description — is present at the start, and a skill's body is loaded on demand when the agent decides it applies. Context cost therefore scales with the number of skills used, not the number installed.
Four skills ship built in: docx, pptx, xlsx, and pdf — the document suite that makes the deliverable premise of Section 1 concrete. Resolution is by name, with a user-level directory and a workspace-level directory each able to override a built-in: a team with house formatting rules for spreadsheets replaces xlsx in the workspace folder and every session in that project uses it. New sessions pick up changes without a restart, and removing a skill means deleting its folder.
Model access is the user's own. Keys are supplied per provider and verified with a single read-only call, or a local runtime is used instead and no key is required at all; usage bills to the user's provider account, since the application intermediates nothing.
The registry curates models across the major hosted providers, open-weight inference services, and local runtimes, with vision enabled on the multimodal flagships. Two details make the abstraction usable rather than merely broad. Arbitrary model strings are accepted with conservative capability fallbacks, so a model released after the application was built can be selected without an update — the fallback assumes less capability rather than more, which fails toward a degraded experience instead of a broken one. And provider credential resolution is deliberately kept separate per provider rather than falling back across them, so a key configured for one service is never silently used to reach another.
Beyond model choice, the tool surface spans twenty-five-plus first-party connectors — source control, messaging, issue tracking, documents, mail and calendar, CRM — alongside anything reachable over the Model Context Protocol, plus the local file system and terminal. Connector and tool actions are written to a durable local audit log with secret-bearing fields stripped.
There is no account, no sign-up, and no subscription for the application itself. Conversations, tokens, and keys live in a local secret store under the platform's per-user configuration directory. There is no OpenRomeo server holding user data, because there is no OpenRomeo server.
Integrations may be connected in two ways, and the distinction is the honest part of the story. A manually pasted token is fully local: the credential is entered in the application and never transits a third party. One-click OAuth instead uses an optional upstream cloud broker to conduct the consent handshake; long-lived tokens are written only to the user's machine, but they do pass through the broker during consent and refresh, one connector uses short-lived cloud-minted tokens by design, and a broker sign-in leaves connection metadata and opt-out session telemetry with the broker. Skipping the broker entirely is supported, and manual token entry is available for almost every integration.
Stating this precisely is the point. “Local-first” is frequently used to mean “mostly local, with an unexamined exception”; the useful version names the exception, says what it can observe, and offers a path that avoids it.
Approval fatigue is unsolved. Risk classification reduces prompting for reads, and standing rules reduce repetition within a task, but a user in Interactive mode on a write-heavy task still answers many prompts — and a user who tires of answering will switch to Auto, which is precisely the outcome the gate exists to avoid.
Distribution builds are unsigned. macOS reports downloaded copies as damaged under Gatekeeper quarantine and requires a manual attribute removal; Windows shows a SmartScreen warning. Both are documented workarounds, and both teach users to bypass a security prompt — an unfortunate lesson that only notarisation and code signing properly resolve.
The broker is a real boundary. For OAuth-connected integrations, tokens transit a third party during consent and refresh. This is disclosed rather than eliminated.
Output quality is the model's. The engine governs what an agent may do and when it must ask; it does not make a weak model produce a strong document. The deliverable premise raises the cost of a poor result, because a wrong file in the workspace is more consequential than a wrong paragraph in a chat window.
Much of the foundation is inherited. The agent engine, permission model, and connector platform come from OpenWorker and aisuite. The fork's contributions are the frontier-model lineup, the built-in document skills, image generation, and packaging — not the architecture beneath them.
Making an assistant produce finished work rather than descriptions of work is a permissions problem before it is a modelling problem. OpenRomeo's answer is to make risk a declared property that a single classifier reads, to separate deciding from prompting so the same decision can be answered by a person present or absent, to give an unattended agent the option of waiting visibly rather than acting alone or hanging, and to keep the loop interruptible at every point it can wait and recoverable when the process dies mid-tool. The document skills make the premise concrete, and the credential boundary is stated with its exceptions named. What remains genuinely open is approval fatigue: the gate is only as good as the attention of the person answering it, and no amount of classification manufactures that attention.