restart-ultron-backend

ULTRON_cBrain

A personal AI assistant with Claude as its brain. It runs on your own Mac (or Windows PC), for one person, in a browser page: a chat on the left, a stage in the middle (the reactor orb, or a canvas with images, 3D models, maps, videos, tables…), and status panels on the right. You can type to it or talk to it, and it can talk back.

It runs on your Claude Pro login through Claude Code, so there's no API bill. The backend only listens on 127.0.0.1; your phone and other computers reach it privately through Tailscale.

Contents

1. What it looks like
2. Setup
3. What it can do
4. How it works

1. What it looks like

Texting

The chat (COMMS) is on the left, the reactor orb in the middle, and on the right your Pro usage (5-hour and weekly limits, Ultron's own tokens, the model picker), a live log and the terminal panel. The top bar shows status, time and which brain is in use.

Texting with a canvas open

Anything bigger than a short answer (maps, tables, drafts, emails, events, images, 3D models, videos) opens as a CANVAS tab on the stage, so the chat reply stays short. Each reply has a badge with the model that answered and why (here: Sonnet, the default).

Speaking

Click the mic (or say "Hey Ultron") and the page switches to voice mode: the chat, log and terminal panels step aside, the orb takes the stage, and what you said and Ultron's answer appear as captions under it. The orb and its button follow the real state (listening, processing, speaking). TYPE in the top right goes back to typing.

Speaking with a canvas open

In voice mode a canvas takes the whole left side and the orb drifts under the usage panel, so you see the results while you keep talking.

2. Setup

⚠️ THIS SETUP WAS MADE FOR AN ANDROID PHONE

Some things may be different for iPhone users. Anything that only works on Android is marked (android phone).

2.1 What you need

Required

On a Mac, Homebrew makes the rest easy. On Windows, use winget.

Optional apps (only for the features that use them)

2.2 Keys and accounts

Every key is optional: without one, only the feature that needs it is off. They all go in .env.

Never add ANTHROPIC_API_KEY. If it's set, Claude Code bills that key instead of using your Pro subscription. Ultron removes it from its own environment anyway.

The claude.ai connectors (Gmail, Google Calendar, Google Drive, TickTick, Canva, Spotify, …) need no keys: connect them once at claude.ai/settings/connectors and Ultron gets them through your Claude login.

Everything else in .env (thinking effort, which connectors to load, quiet hours, voice, Amazon site, 3D library, OmniRoute models, …) is explained line by line in .env.example.

2.3 Install

1. Claude Code, signed in with your Pro account:

claude auth status

It should show "subscriptionType": "pro" (or max).

2. Get the code. Keep it in your home folder, not Desktop or Documents (macOS restricts those for background programs):

git clone https://github.com/erte-del/ULTRON_cBrain.git ~/ULTRON_cBrain

3. Backend (Python 3.13), from the project folder:

uv venv --python 3.13 backend/.venv

VIRTUAL_ENV=backend/.venv uv pip install -r backend/requirements.txt

On Windows (PowerShell):

uv venv --python 3.13 backend\.venv
uv pip install --python backend\.venv\Scripts\python.exe -r backend\requirements.txt

There's no pip inside this environment: always install with uv pip install.

4. Frontend:

cd frontend && npm install && npm run build

5. Settings: copy .env.example to .env and fill in the keys you want (section 2.2).

2.4 Optional one-time setups (Mac)

Each script is safe to run again; finished steps are skipped.

The voice mode itself (speech-to-text and Ultron's voice) needs no script: its models (~250 MB + ~350 MB) download the first time you turn voice on.

2.5 Sign-ins and permissions only you can do

• Chrome: sign in to Teams and Amazon, then View → Developer → Allow JavaScript from Apple Events (homework, Amazon and flights read Chrome this way).
• Spotify and WhatsApp: sign in to the apps. WhatsApp also needs Ultron allowed in System Settings → Privacy & Security → Accessibility.
• Contacts: add your Google account in System Settings → Internet Accounts with Contacts on.
• Screen overlay: allow Screen Recording the first time macOS asks.
• The first time Ultron uses Chrome, Spotify, Contacts or notifications, macOS asks: allow each once.

2.6 Run

The easy way: Ultron.app. Build it once with scripts/make_app.sh, then double-click Ultron.app. It puts an orb in the menu bar, starts Ultron and opens http://127.0.0.1:8000. Quit from the orb's menu stops Ultron. Logs go to backend/storage/ultron.log.

Always on: scripts/autostart.sh on starts Ultron at login and restarts it if it crashes (off undoes it; the log is then ~/Library/Logs/Ultron.log). On Windows: scripts\autostart.ps1 on, which also listens for "Hey Ultron" with no window open.

For development, in two terminals (the page reloads as you edit):

cd backend && .venv/bin/python main.py

cd frontend && npm run dev

Then open http://127.0.0.1:5173. To test the brain without the page: cd backend && .venv/bin/python chat_cli.py (/haiku, /sonnet, /opus switch models).

Tests:

cd backend && .venv/bin/python -m unittest discover tests

2.7 Your phone and Windows PC (optional)

Phone. Ultron has no password, so it never listens on your Wi-Fi. Install Tailscale on the Mac and the phone (same account), then on the Mac:

tailscale serve --bg 8000

Put the https://….ts.net address it prints in .env as JARVIS_REMOTE_ORIGIN, restart Ultron, and open that address on your phone. Never use tailscale funnel: that would put Ultron on the public internet.

Terminal from another device (e.g. your MacBook at school). Set JARVIS_TERMINAL_USER in .env to your Tailscale login (the email you sign in to Tailscale with), restart Ultron, and open the tailscale serve address on that device. Ask Ultron "open a terminal" and it appears on that device's canvas, running on the Mac. Optionally brew install tmux on the Mac: then a closed tab or dropped Wi-Fi only detaches, and reopening the tab reattaches to the same session (type exit to end it). Check that tailscale serve passes the login through: the terminal test in backend/tests/test_terminal.py covers the rule, but a real remote connection is the only proof of the header.

brew install tmux

Windows PC (controlled from the Mac). Paste windows/SETUP_PROMPT.md into Claude Code on the PC (or run windows/install.ps1). It installs Python, Git and Tailscale with winget, starts the small PC agent at logon, and prints the two lines (JARVIS_PC_URL, JARVIS_PC_TOKEN) for the Mac's .env.

Windows PC (running Ultron itself). One script does all of 2.3 and 2.6 (installs uv and Node if missing, the Python packages, the page build, .env, autostart):

powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1

Or step by step: follow 2.3 with the PowerShell commands. Winget installs:

winget install -e --id astral-sh.uv
winget install -e --id OpenJS.NodeJS
winget install -e --id Git.Git
winget install -e --id Gyan.FFmpeg
winget install -e --id Tailscale.Tailscale

Mac-only features (Apple Maps, Contacts, WhatsApp/Chrome automation, the sandboxed Python, FLUX/Wan on Apple Silicon, Ultron.app) are off on Windows.

Windows PC ("Hey Ultron" opens the Mac's Ultron). Ultron has one brain, on the Mac. Instead of running Ultron on the PC, the PC can just listen for "Hey Ultron" and open the Mac's page (over Tailscale, like the phone) full screen in its own Edge window, already in voice mode, and Ultron says "Awake and ready, sir." Speech is checked on the PC (Silero VAD + faster-whisper small.en), never sent anywhere; while the window is open the page listens and the PC doesn't. It uses backend\.venv (nothing new to install; the model, ~250 MB, downloads once into windows\whisper\).

powershell -ExecutionPolicy Bypass -File scripts\autostart.ps1 off   # the full Ultron off on the PC
copy windows\wake_pc.env.example windows\wake_pc.env                 # then set ULTRON_URL
backend\.venv\Scripts\python.exe windows\wake_pc.py --check          # does the Mac's page load?
powershell -ExecutionPolicy Bypass -File windows\wake_pc.ps1 on      # start at logon ("off" to stop)

The Mac needs tailscale serve --bg 8000 and JARVIS_REMOTE_ORIGIN set to the same address. Log: windows\wake_pc.log. Words in the same breath ("Hey Ultron, play music") aren't passed on yet: say the request again once the window is open.

2.8 Back up and move to a new Mac

scripts/backup.sh writes everything that isn't on GitHub (.env, memory, saved chats, jobs, school notes, images, 3D models, videos, uploads) to ~/Ultron-backup-<date>.tgz. It holds your keys: move it by AirDrop or USB, never email.

On the new Mac: do 2.1–2.3 (skip copying .env.example), then scripts/backup.sh restore ~/Ultron-backup-<date>.tgz, the optional scripts from 2.4, and the sign-ins from 2.5. Run scripts/autostart.sh off on the old Mac first, or two Ultrons will both run your scheduled jobs. For a new Tailscale address, update JARVIS_REMOTE_ORIGIN.

Setting up a dedicated Mac mini? The full checklist (apps, accounts, permissions and every feature to test) is in mac mini setup.md.

3. What it can do

How it asks. Looking things up and everyday actions (reminders, events just for you, notes, memory, music, your cart) run straight away. Anything that reaches other people (emails, WhatsApp, invites, posting), touches something you marked important, or uses a developer connector (Supabase, Vercel, Shopify, …) shows a confirmation card first. Scheduled jobs can only look things up.

Talking and thinking

• Chat with streaming replies and a model badge on every answer.
• Voice mode: Ultron hears when you stop talking, writes it down (Groq Whisper, or faster-whisper on this computer), and answers out loud sentence by sentence (Kokoro voice, ~2 s to the first word). Spoken answers are 1–3 sentences; anything longer goes on the canvas. Whisper's habit of inventing "Thank you" on silence or hiss is filtered (voice/stt.py: silent audio isn't transcribed, and a lone "thank you" from quiet audio is dropped; if your mic is very quiet and real thank-yous vanish, lower QUIET). Say "yes" / "no" to confirmation cards. Talk over it to interrupt; "stop" just stops it. A card still waiting when you interrupt ends as "Stopped — not done", and Ultron is told how its cards ended, so it never guesses whether something (an Instagram post) went out. Optional: ElevenLabs as the voice (JARVIS_TTS=elevenlabs, a key and a voice id in .env). Kokoro takes over when ElevenLabs runs out or fails. "How much voice is left?" (voice_usage) shows the characters used and about how many replies remain.
• "Hey Ultron" wake word, checked on this computer so nothing leaves it until you say it.
• Smart model choice: Haiku for quick things, Sonnet by default, Opus when you say "think hard". Sonnet can hand a hard task to Opus (ask_expert).
• Web search with clickable source chips.
• Look at your screen: share your screen and ask about what's on it, in a small floating window (⌃⌥U in Ultron.app).
• Read your files: drop images, PDFs, text, code, CSV or JSON into the chat.
• Three brains: Claude (Pro), OmniRoute (Gemini, Groq, …) or Ollama (a model on this computer, free and offline), switched with the gear icon. Ollama chat has no web search and only your everyday connectors (JARVIS_OLLAMA_CONNECTORS). What it may change is JARVIS_OLLAMA_CHANGES: chat (default) lets it add reminders, notes and memories in your chats, none makes it look things up only.

• Ultron can change how the page looks, for one chat. Every message tells it your window size and which layout you're seeing, and its change_ui tool restyles the page (panel sizes, hidden panels, colors, fonts, a bigger canvas for a wide table, map or video) when you ask or when something shows better another way. The reset button in the top bar (next to New chat; lit while the look is changed) puts the standard look back, and every new chat, loaded chat or brain switch starts there too. The top bar and approval cards can't be hidden, whatever the style sheet says. Windows setup: nothing to set up.

Knowing you

• Memory: preferences, people, projects, decisions. See, edit or delete them all in the memory panel. Passwords, card numbers and keys are always refused.

Memory v2 (steps 1-4: linked memory, smart saves, better recall)

Ultron's memory now links what it knows. Everything runs on your local Ollama; memories never go to a hosted embedding service.

• Smart save. When Ultron remembers something it first looks at the 5 most similar memories. A near-copy is merged (the old wording stays in the memory's history); a plan whose date moved ("test moved to 19 Nov") replaces the old one, which is archived with the reason, not deleted; a disagreement is saved but flagged (a contradicts link) and Ultron asks you which is right. The check is quick (a rule first, then the local judge with a short timeout). If the local model is down, a cheap Claude model may judge (only the few candidate memories, never family/health/money, #private or "important" ones, and only under JARVIS_JOBS_MAX_USAGE); otherwise the memory is saved as it is and checked in the nightly pass.
• Links (storage/memory_links.json): same person / project / topic from the names and themes each memory mentions (found by the local model, cached), plus supersedes / contradicts from the judge. A link you reject isn't made again.
• Recall finds by words and meaning, favours recent, often-useful and sure memories, and shows each hit's linked neighbours with their dates ("EasyTV pricing" also brings the Supabase upgrade note). It records last_used_at / use_count. Without the embedding model it uses word matching only.
• Fields. Each memory in memory.json also has last_used_at, use_count, confidence, expires_at, entities, topics, pinned, archived, history, sensitive. Old sources are renamed: chat/you -> user_said, conversation -> conversation_summary (inferred is for Ultron's own guesses, never above what you said). Embeddings, links, insights and settings sit in separate files beside memory.json (memory_embeddings.json, memory_links.json, memory_insights.json, memory_state.json), all git-ignored.
• Obsidian mirror only lists memories in use (archived ones stay in memory.json); Ultron's guesses appear under their own "guesses" heading with their confidence.
• Kill switch & dry run. JARVIS_MEMORY_V2=0 goes back to the old flat memory (no judge, no links, word-match recall). JARVIS_MEMORY_DRY_RUN=1 makes the nightly pass compute but write nothing.
• Try it on a copy first: cd backend && .venv/bin/python memory_cli.py dry-run "EasyTV pricing" Supabase runs the whole migration on a temp copy and prints what it would do (labels, links, expiry dates, duplicates, recall before/after). memory_cli.py migrate does it for real (copies memory.json to storage/backups/ first; safe to repeat). The nightly pass (3 am, memory_jobs.py) only starts after migrate has been run once.
• Wake-up context (step 5). The system prompt lists memories pinned first, then those expiring within 7 days, then the newest (same ~1,500-character budget). Because the system prompt is built before you say anything, the memories (plus one linked neighbour each) that relate to your first message of a chat are added to that message's [Now: …] prefix instead (max ~1,000 characters, skipping ones already listed; none with the kill switch off).
• Pattern finder (step 6, memory_insights.py). Each night, deadlines (expiring within 7 days) and flagged contradictions come from rules; recurring topics, unfinished things, commitments and decisions to revisit come from the local model reading the last 30 days of conversation notes and memories. Each insight has evidence, a confidence and a surface_when; below 0.6 it is stored but never shown. Memories that are sensitive, #private or name something marked important, and #private notes, are never read. Notes are passed as data (emails can be quoted in them), and the output is checked (no secrets, real evidence only, at most 8 pieces of evidence, confidence capped at 0.8). Insights never become memories. memory_cli.py insights lists them.
• Surfacing (step 7, memory_surface.py). In chat, a matching insight may become ONE Related: line at the end of a reply: never in voice replies, one per reply, each once per conversation. Proactive nudges are off until you run memory_cli.py nudges on (the app switch comes in step 9). They only go out while you are at the computer, outside quiet hours, at most 2 a day (60 min apart), 1 per topic per week, never after you dismissed the insight twice, and on a day you have a test only about that test (found from memories with an expiry date today that mention a test/exam). They use the same notify.route as watchers (a toast in the open page, else Telegram); family/health/money topics get neutral wording. Dry-run (JARVIS_MEMORY_DRY_RUN=1) and the kill switch send nothing. The caps live in memory_state.json under caps.
• Decay (step 8, memory_decay.py). Expired memories are archived (not deleted; unpinned only). Guesses nobody used for 30 days lose 0.1 confidence a week. Merges and archives are only proposed (memory_cli.py proposals, approve ID, reject ID); a rejected proposal isn't made again.
• In the app (step 9). The memory button now has three tabs: memories (as before, plus a ★ to pin a memory so it is always in view and never expires, and "guess" / "until <date>" labels), connections (every link, with × to remove one for good) and insights (each pattern with its confidence, why it was found and its evidence, buttons Useful / Not useful / Forget, patterns under 0.6 shown greyed as "not shown", this week's merge / archive suggestions with Do it / No, and the Quiet nudges switch, off by default). The memory.list message gained links, insights, proposals, proactive and the new memory fields; old fields are unchanged.
• Tools (step 10). `recall