sutando-observer

Sutando

-FFD700?logo=github&logoColor=white)

My AI Stand — Realtime by Day, Rewriting Itself by Night. Summon my AI superpower.

Voice, vision, screen, meetings, calls when I'm engaged. Learns my patterns, ships its own code when I'm not. Runs across my Macs, interacts with people & their Stands.

It belongs entirely to you.

🛠 Open source: this repo — clone, build, run locally on your own Mac. 🍎 Native app preview: sutando.ai — packaged Mac app, request access.

No pay-per-token core API key required. Sutando runs through the Claude Code or Codex CLI session you select, with optional service credentials only for the capabilities you enable.

Named after Stands from JoJo's Bizarre Adventure — a personal spirit that fights on your behalf. Like a Stand, Sutando starts unnamed. As it learns your style and earns real capabilities, it names itself and generates its own avatar — your Stand, unique to you.

https://github.com/user-attachments/assets/a86ec34e-3b26-4011-824c-d2d124753c25

24 tool calls. 6 tasks. 7 minutes. All by voice from a phone. Demo by @liususan091219. Watch on YouTube (full quality) →

⭐ If Sutando is useful to you, star this repo — it's how other people find it.

🤖 The AI agent that presented live	📺 ep.007 — 50 days. 600+ PRs. #1 trending	📺 Vision talk at UC Berkeley

Sutando in action

<!-- wire-list:start (auto-generated by scripts/regen-wire-list.py; do not edit between markers) -->

• The Pentagon Declassified a UAP Case Its Own Analysts Rate 40% "Unrecognized", Oct 2023 Orb Incident
• We Measured Our Agent's First 48 Hours on Claude Fable 5 — Then It Got Pulled
• An AI agent caught a bug humans missed — phantom voice sessions, found via anomaly detection
• Anthropic's $150M Bet: 1,000 Paid Fellows to Build AI Inside Nonprofits
• OpenAI's AI Just Disproved Paul Erdős's 80-Year-Old Math Conjecture

<!-- wire-list:end -->

@sutando-ai channel · Sutando WIRE playlist

What can you do with it?

Talk while you work. You're looking at a doc. You say "make this paragraph shorter." Sutando sees your screen, rewrites the paragraph, and replaces the original text directly.

Join meetings for you. "Join my 2pm call." It reads your calendar and joins — Zoom via the desktop app, Google Meet via the browser — with computer audio. It can also dial in by phone when you ask. It takes screenshots to identify participants, does live research when someone asks a question, and writes you a summary when the call ends. Meeting access is gated — it messages you on Telegram asking for approval before enabling task delegation.

Make calls for you. "Call her and leave a message." Sutando looks up the contact, dials the number, has the conversation, and reports back — while you keep working. It can even make concurrent calls while in a meeting.

Work from your phone. Call Sutando and say "summon." It opens Zoom with screen sharing — join from your phone to see its screen in real time. "What's on my screen?" — it takes a screenshot and tells you. "Fix the typo in that file" — done. You scroll, switch apps, navigate — all by voice while walking around.

Get better on its own. When you're not giving it tasks, Sutando runs an autonomous build loop — it monitors its own health, detects patterns in how you work, discovers new skills, and builds missing capabilities. Most of Sutando's code was written this way. It learns from your corrections and adapts over time.

Remember everything — and act on it. You have an idea while walking. Say it out loud. Sutando captures it, tags it, and saves it as a searchable note. If there's something actionable, it starts working on it right away or queues it for the next free cycle.

Reach you anywhere. Voice, Zoom, Google Meet, Telegram, Discord (text + voice channel), web, phone, or email — same agent, same memory, any channel.

Scale across machines. Plug in a second Mac and Sutando sets it up — the original agent opens a Discord channel, sends setup commands, and migrates services. The new machine handles phone calls 24/7 while your laptop stays portable. No migration scripts needed — the two agents coordinate the handoff themselves.

Status: Alpha

This is an early-stage project. Honest status:

We're looking for contributors to help test and harden these capabilities. If you try something and it breaks, open an issue.

How it works

See Sutando architecture boundaries for the normative definitions of core, adapters, apps, skills, tooling, and workspace state.

The core agent's loop is a cron job that fires /proactive-loop every 15 minutes (*/15 * * * * in the per-host crons.json). On Claude, Sutando backs that off to every 30 minutes when 7-day quota utilization reaches 80%, restoring the configured cadence once an authoritative routed reading drops below it; missing, stale, rejected, or unrouted telemetry holds the slower cadence and reports why. Each pass keeps a persistent watcher on tasks/ via Claude Code's Monitor tool, so tasks are processed on arrival rather than on the tick, and also runs health checks and picks the next build-log item.

Four processes work together:

• Voice agent (Gemini Live, WebSocket on :9900) — listens and talks in real time for browser voice.
• Web client (com.sutando.web-client.plist, HTTP on :8080) — separate launchd service that serves the browser UI. The browser then connects directly to the voice agent's WebSocket on :9900 — the web client is not in the WebSocket data path.
• Conversation server (Gemini Live, Twilio WebSocket on :3100) — same role as the voice agent for inbound and outbound phone calls.
• Core agent (Claude Code or Codex CLI) — executes tasks with full system access. The persistent CLI session provides an interactive terminal and runs Sutando's task watcher and scheduled work.

Voice agent and conversation server handle conversation-scope actions with inline tools — in-process calls that round-trip instantly (describe the screen, hang up, send DTMF, read the clipboard/current time, capture a screenshot). For anything outside that scope they write to tasks/; core reads them, executes, and writes to results/, which each channel speaks or messages back. Telegram and Discord bridges only use the tasks/ path.

Quick start

Prerequisites — bash src/startup.sh checks that these are installed and refuses to boot otherwise:

• Claude Code or Codex CLI — whichever you select
• Node.js (brew install node)
• Python 3 (brew install python3)
• fswatch (brew install fswatch) — auto-installs via Homebrew on first start

Also required — sign in to the selected agent CLI. Startup checks the configured Claude or Codex home before launching background services and fails with the matching login remedy. SUTANDO_SKIP_AUTH_PREFLIGHT=1 bypasses this once for recovery; the runtime launcher still checks again before replacing the core session.

Recommended, but not checked at boot — macOS 15+ and Node.js 22+.

bash src/verify-setup.sh covers this second list — it checks the Node version and whether your CLI is actually authenticated. Run it if startup succeeds but the core doesn't.

Optional — each unlocks one feature and degrades alone:

• Gemini API key — voice (text/core paths work without it)
• pip3 install discord.py / slack_bolt — Discord / Slack bridges (Telegram needs no package)
• ffmpeg (brew install ffmpeg) — subtitle-burn, video-concat, recording handoff
• tmux (brew install tmux) — Sutando.app watcher auto-restart; the core starts without it
• git — vault sync, self-upgrade, commit provenance
• Twilio account + ngrok — phone calls and SMS

Full list with the line enforcing each, plus what to vendor when embedding Sutando in another application: External runtime dependencies.

# Clone
git clone https://github.com/sonichi/sutando.git
cd sutando

# Configure optional integrations (skip for text/core-only use)
cp .env.example .env
# Add GEMINI_API_KEY only if you want voice

# Start everything on macOS / Linux — core, app, and dashboard
./start.sh

# Start everything on Windows
pwsh -File src/startup.ps1

That is the whole first run. start.sh is a thin front door: it delegates to src/startup.sh --with-app and opens the dashboard once it answers. Extra arguments pass straight through (./start.sh --runtime codex). Set SUTANDO_OPEN_DASHBOARD=0 to skip the browser, or SUTANDO_DASHBOARD_URL to point it elsewhere. If the dashboard never comes up the core still starts — the browser open is backgrounded and can never gate it.

src/startup.sh remains the supported lower-level entry, and is what you want when there is no desktop to open things on:

# Headless core only — no app, no browser
bash src/startup.sh

# Core plus the macOS menu-bar app, still no browser
bash src/startup.sh --with-app

Either path starts the core services (voice agent, phone conversation server, web client, dashboard, and API) and the autonomous loop. The browser UI is at http://localhost:8080 and the dashboard at http://localhost:7844; src/startup.sh never opens a browser for you.

The macOS menu-bar app is opt-in and separate. Plain bash src/startup.sh never touches it, so the core stays headless. --with-app builds, signs, and launches the bundle; a failure there is reported and never stops the core.

Auto-start at login is a further, explicit opt-in. Neither ./start.sh nor --with-app installs a launchd job — running the app and having macOS resurrect it forever are different decisions. When you do want it, run the installer from the checkout you actually use: it records that path in the LaunchAgent, so installing from a temporary worktree leaves you with a login job pointing at a directory that will be deleted.

To manage the app on its own — build only, launch once, or supervise — use its installer directly:

bash scripts/install-menu-bar-app.sh              # build + sign, print next steps
bash scripts/install-menu-bar-app.sh --launch     # …and open it now
bash scripts/install-menu-bar-app.sh --supervise  # …and auto-start it at login

First run needs Accessibility granted in System Settings → Privacy & Security. Run the installer from the checkout you actually use: it records that path in the launchd job, so running it from a temporary worktree pins the app to a directory that will be deleted.

Why Sutando runs with elevated permissions. Autonomous voice-driven work means startup.sh launches the selected core CLI with unattended approvals and full local access — permission prompts would otherwise break the voice-in / answer-out flow. In exchange:

• It's local. Sutando runs entirely on your Mac. No remote control plane, no third party with write access.
• You control the audience. 3-tier access gating means owner / verified / unverified callers get different capability bands on phone, Discord, and Telegram. Set VERIFIED_CALLERS in .env before going live.
• Actions are auditable. Every task lands in tasks/ + results/, and service activity is written to logs/*.log. Use the Core CLI terminal while it works to watch in real time.
• Hooks are your brake pedal. git-rules-guard.sh (see $CLAUDE_CONFIG_DIR/hooks) pops a Discord approval DM for any public write (push / PR / issue comment) regardless of transport. Reject with 👎 to block.

Keep the Core CLI terminal reachable — quota exhaustion or an unrecognized CLI prompt can leave the core agent waiting for you to respond. See Codex core setup to select or roll back the Codex runtime.

Why macOS 15+? The setup scripts assume the Sequoia System Settings layout for granting TCC permissions (Screen Recording, Accessibility, Input Monitoring). Earlier macOS versions may work for the headless parts (proactive loop, Discord/Telegram bridges) but aren't tested.

macOS permissions — on first run, macOS will ask you to grant Screen Recording, Accessibility, and Microphone access. See Security for what each permission is used for.

Windows support

Sutando started life on macOS and most of its app-automation surface — AppleScript-driven Chrome/QuickTime control, Cmd+Ctrl+F fullscreen, the Sutando menu-bar Swift app — has no portable Windows equivalent. The core Sutando loop nevertheless runs on Windows; what's there is the headless agent: voice, screen capture, clipboard, notifications, the task bridge, the dashboard, and the messaging bridges.

Works on Windows:

• Voice agent (Gemini Live WebSocket on :9900) — talk to Sutando in the browser
• Web client (:8080) and Dashboard (:7844) and Agent API (:7843)
• Screen capture (:7845) — uses PowerShell System.Drawing.Bitmap instead of screencapture
• Task bridge — file-based; uses a PowerShell FileSystemWatcher shim in place of fswatch
• Clipboard (Get-Clipboard / Set-Clipboard) and desktop notifications (balloon-tip)
• Telegram, Discord, Slack bridges (any feature that runs in the core agent)
• Capture screen + describe screen tools

Dashboard and /tasks/active use platform process probes, not a fixed pgrep path. Unavailable process probes do not abort either response: the dashboard reports unavailable status and /tasks/active returns null for watcher state. The macOS-only Sutando app reports as not running on Windows. Proactive orphan recovery uses the shared, non-signalling process-identity probe: only confirmed dead owners release claims; live or uninspectable owners keep them.

switch_app and pwsh -File scripts/open-app.ps1 "Calculator" identify Windows apps by registered app ID or exact executable path and verify foreground focus. They reuse existing windows, restore minimized ones, and fail explicitly if no interactive desktop exists or Windows refuses focus. Bundled services include the same native backend; no window-title guessing or simulated keystrokes are used.

Returns a macOSOnly error on Windows (the voice agent stays up; Gemini tells the user):

• press_key, type_text, volume, brightness, fullscreen, slide_control, toggle_tasks
• scroll, switch_tab, close_tab, open_url, click, point_at (browser AppleEvents)
• join_gmeet, call_contact (Chrome AppleScript)
• screen_record, play_video, pause_video, resume_video, replay_video, close_video, scroll_and_describe (QuickTime + Chrome)
• Sutando.app menu-bar shortcuts (Swift/Cocoa), Twilio + ngrok auto-launch from startup

Windows quickstart:

# Clone (PowerShell)
git clone https://github.com/sonichi/sutando.git
cd sutando

# Configure
Copy-Item .env.example .env
# Edit .env in your editor; set GEMINI_API_KEY

# Install dependencies + start everything
pwsh -File src/startup.ps1

# Stop everything
pwsh -File src/stop.ps1

# Restart
pwsh -File src/restart.ps1

The Windows scripts mirror their .sh twins:

• src/startup.ps1 — launches voice agent + web client + dashboard + agent API + screen capture (+ optional bridges)
• src/restart.ps1 — stops everything, then starts (matches restart.sh); it runs detached so a restart requested from chat survives stopping its caller, logging to <workspace>/logs/restart.log
• src/stop.ps1 — stops everything without restarting (matches stop.sh; shortcut for restart.ps1 -StopOnly)
• src/notify.ps1 "msg" — desktop notification + Discord DM (matches notify.sh)
• src/watch-tasks-stream.ps1 — task-folder watcher; emits TASK_FILE: … per new file

Prerequisites:

• Windows 10/11
• PowerShell 7+ (winget install Microsoft.PowerShell — pwsh shim)
• Node.js 22+ from nodejs.org
• Python 3.11+ from python.org (used by the dashboard, agent API, and bridges)
• Claude Code installed and logged in (claude once)

Fresh Windows installs use npm ci --ignore-scripts and require the shipped runtime build. Repository paths and workspace names may contain spaces and Unicode.

Workspace: identical contract as macOS — defaults to <repo>/workspace/; override via sutando.config.local.json (see docs/workspace-config.md).

What's not ported (and why):

• Sutando.app menu bar — Swift / AppKit, no Windows equivalent. The global ⌃C / ⌃V / ⌃M shortcuts aren't available; use the web client UI instead.
• AppleScript-driven app automation — Windows has no equivalent of System Events that's portable from the CLI. UIAutomation via PowerShell could replace some of this if there's demand.
• Phone-call flow — startup.ps1 skips Twilio + ngrok auto-launch. If you want phone calls on Windows, start ngrok manually and set WEBHOOK_BASE_URL in .env.
• macOS permissions block — Windows has no TCC; screen capture and microphone "just work" once you grant Chrome microphone access.

Task-loop architecture (Windows-specific). macOS Claude Code exposes a Monitor tool that streams stdout from a long-running command (e.g. bash src/watch-tasks-stream.sh) and wakes the agent on every TASK_FILE: event. Claude Code 2.1.168 on Windows does not include the Monitor tool (verified: not in the agent's tool list, and the literal string "Monitor" is absent from claude.exe). Without Monitor, there's no push-based file-watch primitive available to the agent, so the long-running sutando-core TUI would only pick up new tasks on its */5 proactive-loop cron tick — fine for autonomous work, far too slow for chat.

The Windows port works around this with src/task-dispatcher.ps1, a standalone process auto-launched by src/startup.ps1. It uses FileSystemWatcher to watch tasks/, claims new files via atomic rename, and runs each one through claude --print as a one-shot subprocess. Each queued chat task starts as soon as the dispatcher is available; inference and delivery time depend on the model and connection. The long-running core still handles autonomous proactive-loop work + cron jobs; the dispatcher only intercepts user-driven chat tasks.

Task context and authorization: owner turns and authenticated Discord collaborator turns resume a session per channel. Collaborators require both a verified task envelope and a current access entry. Other non-owner turns use codex exec --sandbox read-only; when Codex is unavailable, they are refused. Owner and collaborator turns in the same channel share conversational context.

The dispatcher holds an exclusive lifetime lock and publishes results atomically. After a crash or forced restart, abandoned claims are archived with an interruption result instead of being retried: actions may already have run. Check their outcome before resubmitting. Restart stops the dispatcher process tree, including its in-flight CLI child.

Try saying:

• "What's on my screen?" — takes a screenshot and describes it
• "Summon my computer to zoom" — opens Zoom with screen sharing, join from your phone
• "Join my next meeting" — checks your calendar and joins
• "Take a note: my first idea" — saves a searchable note
• "Tutorial" — walks you through all capabilities step by step

Verify your setup (optional):

bash src/verify-setup.sh

Troubleshooting:

• Browser shows blank page? Services may still be starting — wait 5 seconds and refresh
• Microphone not working? Chrome will ask for permission on first connect — click Allow
• Voice agent not responding? Check logs/voice-agent.log for errors. Common causes:
• GEMINI_API_KEY not set or invalid in .env — get one at ai.google.dev
• Port 9900 already in use — run lsof -i :9900 to check
• npm install failed? Make sure Node.js 22+ is installed: node --version
• Gemini 429 errors? Your shell may have a stale GEMINI_API_KEY overriding .env — run unset GEMINI_API_KEY then restart
• Screen recording produces 0-second files? screencapture -v needs a TTY. Sutando uses ffmpeg instead — make sure it's installed: brew install ffmpeg
• Something broke? Run bash src/restart.sh — this kills all services and restarts fresh
• Sutando acting confused, contradicting itself, or giving stale answers after a long session? Restart the selected core CLI session to reset its context.
• Still stuck? Join the official Discord — real humans and community-run agents answer support questions there.
• Phone call answers with "We are sorry, an error has occurred"? The conversation server (skills/phone-conversation/scripts/conversation-server.ts, port 3100) isn't running. Run bash src/startup.sh or bash src/restart.sh to relaunch all services.

Shutting down:

bash src/restart.sh    # stops all services (voice agent, web client, API, bridges, etc.)
pkill -x Sutando # stop the menu bar app

Exiting startup.sh alone does NOT stop background services. Always use restart.sh (or kill-all.sh if available) to cleanly shut everything down.

Uninstalling:

1. Stop all services: bash src/restart.sh && pkill -x Sutando
2. Remove the repo: rm -rf ~/Desktop/sutando (or wherever you cloned it)
3. Remove config: rm -rf $CLAUDE_CONFIG_DIR/projects/*sutando*
4. Remove npm packages (optional): the repo uses local node_modules/ — deleted with the repo
5. Remove any tools you installed during setup (e.g. imsg, wacli) via the package manager you used to install them.
6. If you installed the OS-supervised health checks: bash src/install-health-check-launchd.sh --uninstall (