spike-probe

Orchestrator

Runs Claude Code agents unattended, with one server to command them, watch them live, and keep what they produce. The design is in docs/design/2026-10-06-brainstorm.md; the decisions are in docs/adr/.

Building

The Nix flake dev shell provides the Go toolchain, so Go need not be installed on the host:

nix develop -c go build ./... nix develop -c go vet ./...

Module path: github.com/sebnow/orchestrator. Binaries live under cmd/ (cmd/server, cmd/daemon). Create a package only when it has code.

nix develop -c go test ./... runs offline. The live tests, behind the live build tag, run the real claude on PATH under the owner's login and spend subscription quota:

nix develop -c go test -tags live ./...

The browser tests, behind the browser build tag, load the GUI in a headless browser and check what htmx does with it. They run offline against an in-process server and a fake daemon:

nix develop -c go test -tags browser ./internal/server/

They use $ORCHESTRATOR_BROWSER when it is set, or else the first of chromium, google-chrome-stable and google-chrome on PATH. The dev shell provides chromium on Linux and Google Chrome on macOS, as google-chrome-stable and google-chrome, because nixpkgs builds Chromium only for Linux. A failing step saves a screenshot, which go test keeps when given -artifacts.

Running

The server stores daemons, tasks and their events, and serves the GUI. The daemon runs the tasks with Claude Code, so claude must be installed and logged in for the user who runs it: the user who starts the daemon, or the harness user with -harness-user (see Running the harness as another user). The daemon clones a task's repository, and pushes the task's branch to it, with the git on the daemon's PATH, so git must be installed on every daemon's machine. A missing git shows only when a task with a repository starts, and fails it. Build both from the repository root:

nix develop -c go build ./cmd/server nix develop -c go build ./cmd/daemon

The paths below are examples; use any writable location.

On a server reachable from the internet

The server serves TLS. Daemons authenticate with client certificates from the server's own certificate authority, and the owner with a token (daemon authentication, owner authentication). On the server's machine, create the CA, the server's certificate for every name and address that daemons and browsers reach it by, and the owner token:

./server init-ca -pki-dir ~/.local/state/orchestrator/pki ./server issue-server-cert -pki-dir ~/.local/state/orchestrator/pki \ -host orchestrator.example -host 203.0.113.7 ./server issue-owner-token -db ~/.local/state/orchestrator/server.db

issue-owner-token prints the token once; keep it somewhere retrievable, such as a password manager. Running it again replaces the token and logs out every browser. Start the server:

./server -listen :8443 -db ~/.local/state/orchestrator/server.db \ -tls-cert ~/.local/state/orchestrator/pki/server.crt \ -tls-key ~/.local/state/orchestrator/pki/server.key \ -client-ca ~/.local/state/orchestrator/pki/ca.crt

For each daemon, issue a certificate named after the daemon's id:

./server issue-daemon-cert -pki-dir ~/.local/state/orchestrator/pki -id vps-1

It writes daemon.crt, daemon.key and ca.crt to daemons/vps-1/ under the -pki-dir. Copy the three files and a daemon binary built for that machine to one directory on it, with daemon.key readable only by the user who runs the daemon, and start the daemon from that directory:

./daemon -server https://orchestrator.example:8443 \ -cert daemon.crt -key daemon.key -ca ca.crt \ -state-dir ~/.local/state/orchestrator/daemon

Open https://orchestrator.example:8443/ and log in with the owner token. The browser warns about the server's certificate until it trusts ca.crt. Scripts call the owner API under /v1/tasks with the header Authorization: Bearer <token>.

Whoever holds ca.key can issue any daemon's certificate, and a server certificate that the daemons trust. The running server does not read it, so it can be kept elsewhere between issuing certificates.

Server and daemon certificates last a year, the CA ten. To renew a certificate, delete its .crt and .key, issue it again, and restart the program that uses it, after copying a daemon's new files to its machine. The server does not check revocation. To shut out a daemon before its certificate expires, run init-ca with a new -pki-dir, issue the server's and every remaining daemon's certificate from it, restart the server and those daemons with the new files, and have browsers trust the new ca.crt.

On one machine, for development

Run the server and the daemon in separate terminals:

./server init-ca -pki-dir ~/.local/state/orchestrator/pki ./server issue-daemon-cert -pki-dir ~/.local/state/orchestrator/pki -id laptop ./server -insecure-loopback -db ~/.local/state/orchestrator/server.db ./daemon -server http://127.0.0.1:8080 \ -cert ~/.local/state/orchestrator/pki/daemons/laptop/daemon.crt \ -key ~/.local/state/orchestrator/pki/daemons/laptop/daemon.key \ -state-dir ~/.local/state/orchestrator/daemon

Then open http://127.0.0.1:8080/. With -insecure-loopback the server serves plain HTTP and authenticates nobody, so anyone who can reach it can start tasks that run commands on the daemon's machine. It refuses to start unless -listen is a loopback IP address. The daemon still needs its certificate, which names it, but not -ca.

Running the harness as another user

By default the daemon runs claude, and with it the agent's tools, as its own OS user, so the agent can read the daemon's key, its state and its journals. With -harness-user NAME the daemon runs claude, and every command that touches a workspace, as the unprivileged user NAME through sudo, and file permissions keep the agent from reading the daemon's key, state and journals (harness user). The agent can still read its own Claude Code login, and every task's workspace on that daemon, since all tasks share the one harness user. It can also see in the process list the URL of the daemon's MCP gateway, which serves every task's spawn_task, send_message and permission tools on loopback without authentication.

The daemon starts each command as sudo -n -u NAME -D DIR VAR=value... -- COMMAND ARG..., in a session of its own so that sudo has no terminal. As the harness user it runs:

• claude, in the task's workspace;
• git, to clone, to set up the task's branch, its pre-push hook and the clone's local configuration, to read the clone's status and history and the remote's branches, and to push;
• rm -rf, to delete a workspace.

The daemon does not run git in a workspace as itself, because git runs commands that a repository's configuration and hooks name (git help git, section SECURITY), and the agent can write every workspace. The harness user creates and owns the workspaces.

-workspace-dir is required with -harness-user, because the harness user cannot enter the state directory, where workspaces go by default. The directory must exist and be owned by the harness user. When it starts, the daemon lists this directory and deletes the workspaces of tasks it does not know, so the daemon's user needs permission to read it. -claude must be an absolute path that the harness user can run; the daemon runs claude --version as that user through sudo when it starts, so a missing sudoers rule stops it there. When it starts, the daemon also looks up git and rm on its PATH, logs their absolute paths, and runs them by those paths.

The sudoers rule lets the daemon's user run those three paths as the harness user only, without a password, and nothing as root. With the daemon running as orchestrator, the harness user orch-agent, and the paths below, write it with sudo visudo -f /etc/sudoers.d/orchestrator:

Cmnd_Alias ORCH_CLAUDE = /usr/local/bin/claude Cmnd_Alias ORCH_GIT = /usr/bin/git Cmnd_Alias ORCH_RM = /bin/rm Cmnd_Alias ORCH_HARNESS = ORCH_CLAUDE, ORCH_GIT, ORCH_RM Defaults!ORCH_HARNESS !requiretty, umask=0077 Defaults!ORCH_CLAUDE env_keep += "PATH SSH_AUTH_SOCK" Defaults!ORCH_GIT env_keep += "PATH SSH_AUTH_SOCK GIT_TERMINAL_PROMPT GIT_CONFIG_GLOBAL GIT_CONFIG_NOSYSTEM GIT_SSH_COMMAND" orchestrator ALL = (orch-agent) CWD=* NOPASSWD: ORCH_HARNESS

Replace the three paths with the value of -claude and the output of command -v git rm run as the daemon's user. CWD=* lets the daemon choose the directory with -D, since the daemon's user may not be able to enter a workspace the harness user owns; it needs sudo 1.9.3 or later (sudoers(5), "Chdir_Spec"). umask=0077 makes the workspaces readable by the harness user only: sudo runs the command with the union of the daemon's umask and this one (sudoers(5), "umask"). sudo 1.9.17p2's visudo -c accepted these lines; they have not been tried on a machine with a harness user.

sudo resets the environment and sets HOME, MAIL, SHELL, LOGNAME and USER for the harness user (sudoers(5), "Command environment"). The daemon passes the rest as follows:

• PATH: the daemon's, in sudo's own environment. sudo keeps it because env_keep lists it, unless sudoers sets secure_path, whose value then replaces it (sudoers(5), "env_reset").
• SSH_AUTH_SOCK: when the daemon has an ssh agent, the path of the socket the daemon relays to it (below), in sudo's own environment. sudo keeps it because env_keep lists it.
• GIT_TERMINAL_PROMPT=0, GIT_CONFIG_GLOBAL=/dev/null, GIT_CONFIG_NOSYSTEM=1 and GIT_SSH_COMMAND=ssh -o BatchMode=yes, for git only: on sudo's command line, as VAR=value before --. sudo refuses to run a command given a variable that sudoers does not allow, with "sorry, you are not allowed to set the following environment variables" (sudoers(5), "Denied command log entries"), so env_keep lists them for git, and a rule without them stops git rather than letting it read the harness user's git configuration.

The daemon's ssh agent socket is open only to the daemon's user. When the daemon has SSH_AUTH_SOCK, it listens on a socket of its own, /tmp/orchestrator-agent-*/agent-*.sock, for as long as it runs, and relays each connection to its agent byte for byte. The socket has mode 0666 and a random name, in a directory of mode 0711, so the harness user can connect without a group in common with the daemon's user. Any local user who learns the socket's path can use the daemon's ssh keys; the directory's mode keeps others from listing it to learn the name. Without SSH_AUTH_SOCK the harness gets no ssh agent. ssh runs as the harness user, so known_hosts must list the repository's host in that user's ~/.ssh or in the system's.

The harness user logs in to Claude Code once on each machine, and tasks spend that account's quota; the daemon does not manage the login. On Linux Claude Code keeps it under the user's home, in ~/.claude. On macOS, see step 5 of the checklist below.

sudo relays SIGTERM to the command it runs but not SIGKILL (sudo(8), "Signal handling"), and the daemon's user cannot signal the harness user's processes. When the daemon kills a harness in this mode, as it does when a stopping harness has not exited within 30 seconds, it sends sudo SIGTERM and closes the harness's input instead; a harness that ignores both keeps running. A daemon that restarts and finds a harness its previous run left sends that harness's sudo SIGTERM.

Checklist for a machine with a daemon user orchestrator and a harness user orch-agent. It has not been run.

1. Create the harness user, such as with sudo useradd --create-home orch-agent on Linux or sudo sysadminctl -addUser orch-agent on macOS.
2. Install claude where orch-agent can run it, write the sudoers rule above with that path, and check it as orchestrator with sudo -n -u orch-agent -D / -- /usr/local/bin/claude --version.
3. Create the workspace directory, such as with sudo install -d -o orch-agent -m 0755 /srv/orchestrator/workspaces on Linux. macOS's system volume is read-only; use a directory such as /Users/Shared/orchestrator-workspaces there.
4. Log orch-agent in to Claude Code: sudo -u orch-agent -i, then claude and /login. In the same shell, add the repository host's key to ~/.ssh/known_hosts, such as with ssh-keyscan HOST >> ~/.ssh/known_hosts, after comparing its fingerprint with the one the host publishes.
5. On macOS, confirm where Claude Code keeps its login and that a process started through sudo can read it. As orchestrator, sudo -n -u orch-agent -D / -- /usr/local/bin/claude -p 'Reply OK' replies rather than asking to log in.
6. As orchestrator, with its ssh agent running, start the daemon with -harness-user orch-agent -workspace-dir /srv/orchestrator/workspaces -claude /usr/local/bin/claude and the usual flags. The log has a line "running tasks as the harness user" with the user, the git and rm paths, and ssh_agent=true.
7. Start a task with a repository and a prompt that commits a file. The task page shows the branch pushed.
8. ps -o user,pid,ppid,command -ax | grep claude shows claude run by orch-agent, its parent a sudo process.
9. In sudo -u orch-agent -i, cat the daemon's key and ls its state directory fail with "Permission denied". Ask the agent to run ssh-add -l: it lists the daemon's keys.
10. With a task running, sudo -u orch-agent kill -TERM <claude pid> ends claude, and the task page shows the harness's exit.
11. With another task running, kill -TERM <sudo pid> as orchestrator ends both sudo and claude.

Using the GUI

The dashboard lists the tasks that need attention, every task with the agent it was started as, the account's quota reading, and the daemons with their labels. Its form starts a task with:

• the agent to start it as, if any (see Agents);
• its prompt, and an optional repository and ref;
• the model;
• the daemon to run it on, or any connected daemon, and the labels its daemon must have (see Placement);
• its priority, and whether it is filler (see Scheduling);
• its pause limits, to acknowledge and to clean up.

A field left blank takes the agent's value, or else the default: the server's model, no labels, normal priority, and pause limits of 1 minute to acknowledge and 5 to clean up. The filler box can only make a task filler; an agent whose tasks are filler makes the task filler whether it is ticked or not, and only the owner API's "filler": false overrides that.

Each task page:

• shows the transcript;
• asks for permission when the agent wants to run a tool and the server runs with -permissions ask;
• shows the branch the daemon pushed the task's work to (see Tasks);
• lists the tasks it spawned, with each one's agent, state and latest report, and links its parent;
• has buttons to pause, resume, interrupt or stop the task.

Claude Code can run subagents of its own within a task's turn, with its Agent tool, which Claude Code 2.1.289 listed as Task. The daemon runs claude with --forward-subagent-text, so a subagent's text and thinking reach the transcript as well as its tool calls, and the task page nests them under the tool call that started the subagent, in a list titled by the call's description that can be folded away. Such subagents run inside the task's own process: the scheduler does not see them, and they share the task's slot and daemon (see the agent model note).

The Agents page, linked from the top of every page, lists the agents and creates, edits and deletes them. Each daemon's row on the dashboard links to the daemon's page, which shows the facts it reported and sets the labels the owner gives it (see Placement).

A stopped or failed task's page, and a failed task among those that need attention, have a Dismiss button. A dismissed task no longer needs attention and is left out of the dashboard's list of every task, which says how many dismissed tasks it leaves out; /?dismissed=show includes them, marked as dismissed. Scripts dismiss a task with POST /v1/tasks/{task}/dismiss, which answers 409 unless the task is stopped or failed.

Flags

Server flags:

• -db (required): the SQLite database file, created with its directory when missing. It holds every daemon, task, event and command, the owner token's hash and the login sessions.
• -listen: the address to serve on, 127.0.0.1:8080 by default.
• -tls-cert, -tls-key (required unless -insecure-loopback): the server's certificate and key, from issue-server-cert.
• -client-ca (required unless -insecure-loopback): the CA certificate, ca.crt from init-ca, that daemons' certificates are verified against.
• -insecure-loopback: serve plain HTTP without authentication; only with a loopback IP address as -listen, and without -tls-cert, -tls-key and -client-ca.
• -default-model: the model of a task started without one, haiku by default.
• -slots-per-daemon: how many tasks each daemon runs at once, 2 by default.
• -filler-threshold: the utilization of the account's five-hour quota window, from 0 to 1, below which filler tasks run; 0.5 by default (see Scheduling).
• -low-threshold: the same for low-priority tasks; 0.85 by default.
• -daemon-timeout: how long a daemon may go unseen, with no command stream open, before it is lost and its tasks move to other daemons (see Lost daemons); 10 minutes by default, and at least a minute, as a daemon retries every 30 seconds at most.
• -permissions: who answers the agents' requests to run a tool (permission policy). allow-all, the default, has the server allow every request as soon as it arrives, and the task's transcript says it was allowed by policy. With ask, each request waits, without limit, for the owner to answer it on the task page or through the owner API. Changing it does not answer requests already waiting.

Without -insecure-loopback, the server also refuses to start until an owner token has been issued into its database.

Server subcommands, each printing its usage with -h:

• init-ca -pki-dir DIR: create the CA as ca.crt and ca.key.
• issue-server-cert -pki-dir DIR -host NAME_OR_IP...: issue server.crt and server.key, valid for every -host.
• issue-daemon-cert -pki-dir DIR -id DAEMON: issue a daemon's certificate into daemons/DAEMON/, with a copy of ca.crt. The id is up to 128 letters, digits, ., _ and -, other than . or ..; once the daemon has connected, the new-task form offers it by this id.
• issue-owner-token -db FILE: print a new owner token and keep its hash in place of the previous token's.

The first three refuse to replace an existing certificate or key.

Daemon flags:

• -server (required): the server's base URL. Plain http:// is accepted only for a loopback IP address.
• -cert, -key (required): the daemon's certificate and key from issue-daemon-cert. The certificate's name is the daemon's id.
• -ca (required for https://): the CA certificate to verify the server with.
• -state-dir (required): created when missing. It holds state.json, which records what each task needs to be resumed, one journal per task under journal/, and, unless -workspace-dir is given, each task's working directory under workspaces/<task>/. Claude Code keeps its sessions outside it: Claude Code 2.1.289 on macOS kept them under ~/.claude/projects/ of the user running it (resume spike).
• -workspace-dir: the directory holding each task's workspace, <task>/; workspaces/ under -state-dir by default. Required with -harness-user; it must then exist, be owned by the harness user, and be readable by the daemon's user.
• -harness-user: the OS user that runs claude and every workspace command, through sudo. Empty, the default, runs them as the daemon's own user (see Running the harness as another user).
• -claude: the claude executable, claude on PATH by default; with -harness-user, the absolute path the sudoers rule names. The daemon runs claude --version when it starts, as the user who runs tasks, and exits if that fails. Tasks use the Claude Code login of that user, the daemon's own or the harness user, and spend that account's quota.
• -git-identity: name and email for the commits an agent makes, as Name <email>; orchestrator <orchestrator@localhost> by default.

Tasks

Without a repository, a task starts in an empty directory. With one, it starts in a clone on the branch orchestrator/<task id>, created at the ref, or checked out from the repository when it has that branch already. The daemon accepts a repository given as an https:// or ssh:// URL with a host, or as an ssh address such as git@github.com:owner/repo.git. A local path, file:// or git:// fails the task, as does a ref that starts with -.

The daemon delivers a task's work by pushing its branch to the repository; nothing of the work goes to the server (work delivery). At the end of every turn, when the branch holds commits that the repository's copy of it lacks, the daemon pushes it with a plain git push, never forced. After a push, and after any turn that leaves files uncommitted, pushed or not, the task page shows the branch, its commit, how many commits the branch holds beyond the ref, how many files the agent left uncommitted, and any push error. The dashboard's task list shows each task's branch and marks a failed push. The daemon adds to the task's system prompt that the agent must commit its work on the branch and never push. A pre-push hook in the clone refuses to push any other ref, and the daemon refuses to push the ref the task started from or the repository's default branch. Work the agent did not commit is not delivered. A child task gets a branch of its own, starting at its parent's ref.

The daemon's git runs with prompts off and ignores the user's and the system's git configuration, credential helpers and URL rewrites included (GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1 GIT_TERMINAL_PROMPT=0), for cloning and for pushing, and runs ssh in batch mode. ssh authenticates with the daemon user's ssh setup; the expected form is an ssh agent, with SSH_AUTH_SOCK in the daemon's environment, so the daemon never needs a key file it can read. known_hosts must already list the host, since batch mode fails the clone or the push on an unknown host key rather than waiting. Pushing needs those credentials on every daemon's machine, with write access to the repository; a repository over https that needs credentials fails to clone, as no credential helper applies. The harness and the agent it runs inherit the daemon's environment, the ssh agent included, so the agent's own git can authenticate over ssh the same way, or, with -harness-user, get the daemon's ssh agent through a socket the daemon relays (see Running the harness as another user); the agent is told not to push, and a pre-push hook in the clone refuses any ref but the task branch, so the daemon pushes for it. The agent's own