token-usage

Agentic Research Pipeline

Agent pipelines for investment research at scale: build thematic company universes, extract schema-defined data points from disclosures, and analyse portfolios — designed for up to ~4,000 companies per run.

Two design rules hold everywhere: the LLM plans, deterministic code computes, and every citation is re-verified programmatically against the source document rather than trusted from the model's self-report.

Python (FastAPI + Typer CLI) backend, React/TypeScript (Vite) frontend, Postgres + pgvector as the system of record (required).

Functions

Architecture

backend/   Python (FastAPI + Typer CLI) — every agent pipeline
frontend/  React + TypeScript (Vite) — authoring, monitoring, review UI
Postgres   system of record (+ pgvector): companies, runs, review, facts, documents, embeddings, holdings
runs/      run side files with no table: snapshots, events, restatement candidates
taxonomies/ portfolios/ reports/ report_templates/ data/documents/
frameworks/ versioned decision frameworks + the tables they are applied to
indices/   index construction calibrations (versioned, effective-dated) + reviews

• The Anthropic SDK (anthropic) is the LLM client, behind one narrow interface (arp/llm/base.py) — schema-forced structured output, a bounded validation-retry loop, and a disk-backed response cache.
• LangGraph models each per-company multi-step flow as an explicit state graph (the classification debate, the extractor/verifier pair, …). Batch fan-out, checkpointing and resumability stay outside the graphs, in the Postgres run store (arp/db/runs.py).
• LlamaIndex backs chunking and BM25 evidence selection; hybrid (BM25 + local multilingual embedding) retrieval is on by default.
• arp/grounding.py is independent of all three and re-verifies every citation against the original document text.

Every run type is checkpointed and resumable — each company's result is committed to Postgres in one transaction as it finishes, and a resume skips every result key already stored, so an interrupted batch picks back up without redoing completed work. Runs can be cancelled cooperatively (arp runs cancel) and resumed later.

Setup

Windows / corporate proxy / VS Code + conda: see docs/INSTALLATION.md, which ships scripts under scripts/windows/ that automate everything below.

# Backend
cd backend
python3 -m venv .venv && source .venv/bin/activate   # Python 3.11
pip install -e ".[dev]"
cp .env.example .env            # fill in ARP_ANTHROPIC_API_KEY; ARP_POSTGRES_DSN is preset
docker compose up -d postgres   # required: Postgres + pgvector
docker compose up -d neo4j      # optional: company graph for the theme exposure layers
arp db init-postgres            # schema; re-run after every upgrade
pytest -q                       # needs ARP_TEST_POSTGRES_DSN (a separate, empty database)
uvicorn arp.api.main:app --reload            # API on :8000

# Frontend
cd frontend && npm install
cp .env.example .env            # VITE_API_BASE, VITE_SUPERSET_URL
npm run dev                                   # UI on :5173

Sign-in

Every API route except /api/health (and the proxy-voting routes) needs a signed-in user, and every review, ratification and approval is recorded against that user.

Current default: dev mode (ARP_AUTH_MODE=dev). Every request from 127.0.0.1/::1 is signed in as an approver named ARP_DEV_USER (default dev) without a token; the sidebar shows "Signed in as dev (approver)". Use it only on your own machine, or with the provided docker-compose.yml, which publishes the API on 127.0.0.1 only and adds the Docker bridge range (172.16.0.0/12) to ARP_DEV_TRUSTED_NETWORKS because the browser's requests reach the container from the bridge, not loopback. That range also covers every other container on the compose network (e.g. Superset): while dev mode is on, they get dev approver access too. On Docker hosts whose compose subnets come from 192.168.0.0/16 (check docker network inspect), adjust ARP_DEV_TRUSTED_NETWORKS to match. The bypass trusts the socket address: never run dev mode behind a proxy on the same host (every proxied request would arrive from loopback) and never publish the port on a LAN-reachable address. Requests from anywhere else still need a token. Dev mode also refuses browser requests from foreign origins (an Origin header not in ARP_ALLOWED_ORIGINS gets no bypass, so another website cannot act as you), and the API only answers the host names in ARP_TRUSTED_HOSTS (default localhost, 127.0.0.1, [::1], backend), which blocks DNS rebinding; this host check applies to every route, voting included. Dev mode has a single user, so four-eyes steps (calibration approval, co-sign) need local mode.

Switching to local mode (real per-person sign-in):

1. cp config/users.example.json config/users.json and give each person a long random token (openssl rand -hex 24), a unique user_id, a name and a role: viewer (read), analyst (change), approver (also approve calibrations and co-sign). Tokens and user_ids must be unique and non-empty, or the API refuses to start. config/users.json is gitignored; ARP_USERS_FILE points elsewhere.
2. Set ARP_AUTH_MODE=local in backend/.env (Docker: in the backend service's environment: in docker-compose.yml, and remove ARP_DEV_TRUSTED_NETWORKS there).
3. Restart the API (also after every later edit of the users file: it is read once).
4. Each person pastes their token into the sidebar; the UI keeps it in the browser's localStorage until they sign out (or the API rejects it).
5. CLI commands that write (arp index calibration-save, arp index approve, …) need ARP_CLI_TOKEN set to the person's token, in either mode.

ARP_ALLOWED_ORIGINS is the CORS allow-list for the browser UI, as JSON, e.g. ARP_ALLOWED_ORIGINS='["https://arp.example.com"]'.

[dev] is enough to run the app, but not enough for a green test suite — the reporting tests render through headless Chromium and a few storage tests need the optional backends. For the full suite, as CI installs it:

pip install -e ".[dev,postgres,opensearch,object_storage,emerging_themes]"
python -m playwright install chromium

Budget ~1.8 GB and around ten minutes: docling pulls in torch, torchvision, transformers, onnxruntime and opencv (217 packages for [dev] alone). Nothing is downloaded at import time — docling and fastembed are both imported lazily — but they are hard dependencies, so the install pays for them regardless.

No conda needed. environment.yml and scripts/windows/setup.ps1 use it to supply Python 3.11 and Node 22, but any 3.11 interpreter works; uv venv --python 3.11 fetches one in seconds if the system Python is a different version. Python 3.11 is what CI and environment.yml pin and the only version verified here.

On a first extraction or theme-matching run, hybrid retrieval downloads a ~120 MB embedding model (it is on by default); ARP_HYBRID_RETRIEVAL_ENABLED=false keeps a first run fully offline.

Docker is an alternative to both:

cp backend/.env.example backend/.env
docker compose up backend frontend --build    # same ports, state in named volumes

There is no authentication in front of either path today — see docs/CORPORATE_READINESS_PLAN.md.

Contributors: pre-commit install enables the secret-scanning hook that also runs in CI.

Data handling. Document text and company data go to Anthropic's API; outbound requests reach SEC EDGAR, GDELT, regulatory RSS and crawled IR sites. Everything else stays on local disk. Review this against your organization's data-handling policy before pointing it at real holdings.

Superset BI

Optional. Reads the bi views arp db init-postgres creates over the Postgres tables.

Set ARP_SUPERSET_PASSWORD, ARP_BI_READER_PASSWORD, SUPERSET_SECRET_KEY and SUPERSET_GUEST_TOKEN_JWT_SECRET in backend/.env. None has a default and the change-me placeholders are refused; the two Superset secrets need 32+ characters. Generate each with openssl rand -base64 42. ARP_SUPERSET_URL (default http://127.0.0.1:8088) and ARP_SUPERSET_USER (default arp_designer) have defaults; ARP_BI_SUPERSET_DB_HOST (default postgres:5432) is the host:port Superset uses to reach Postgres.

docker compose --env-file backend/.env up -d postgres superset   # first start builds superset/Dockerfile
arp db init-postgres                                             # schema + `bi` views
arp portfolio seed-demo                                          # demo holdings into Postgres
arp bi bootstrap                                                 # views, bi_reader role, Superset database, datasets, metrics, descriptions, templates (idempotent)
arp golden-set bi                                                # planner eval; needs an API key, not run in CI

arp bi bootstrap re-applies the bi views (including holdings_history), registers the six datasets and their metrics, and copies descriptions from backend/arp/bi/catalog.py. Each run overwrites the dataset description and the descriptions of catalogue-named columns, so edit them there, not in Superset. It then provisions the standard dashboards in backend/arp/bi/templates/ (today arp-risk-exposure: 8 charts, native filters Fund, Sector and Country) and prints {"templates": {"arp-risk-exposure": "created" | "rebuilt" | "unchanged"}} with the rest of its output. The dashboard is created unpublished; publish it in Superset. One that has at least the template's charts is left alone. One with fewer is deleted and rebuilt (its old charts stay), and the rebuild is unpublished again. Existing deployments must re-run arp bi bootstrap after upgrading: the AI designer needs every catalogue dataset, including holdings_history.

Upgrading to the monthly run's published datasets: run arp db init-postgres (creates the bi_published table and its bi views), then arp bi bootstrap (bi_reader is granted per existing object, and the climate, alerts, triggers and company-profile datasets and dashboards are registered there). Until then Company Profile embeds fail closed with a 502 ("Superset has no dataset bi.alerts"). The published datasets fill only when a monthly run (arp portfolio monthly-run --month YYYY-MM) succeeds, and a month runs only with an ok holdings load for every portfolio and an ok ESG load for that month (upload it, or arp portfolio esg-pull --month YYYY-MM). Holdings Intake shows what a month is still missing.

News comes from Refinitiv News over the Refinitiv Data Platform: set ARP_NEWS_API_CLIENT_ID and ARP_NEWS_API_CLIENT_SECRET (an RDP service account; ARP_NEWS_API_URL defaults to https://api.refinitiv.com), then pull from Data Hub · Feeds ("Pull now") or on a schedule with arp portfolio news-pull. Only issuers with a permid in the security master are asked for, and each story is tied to them by PermID exact match only (see backend/arp/portfolio/news/api_source.py). ESG can be pulled the same way from Feeds once ARP_ESG_API_URL and ARP_ESG_API_TOKEN are set.

Set VITE_SUPERSET_URL in frontend/.env, then open Risk Monitoring, Dashboards (Superset). This one tab replaces Pivot Explorer, Generative BI and Superset BI (old links land on it), and it is the default Risk Monitoring tab. The in-app Standard Analytics tab is gone; Monitoring & Alerts, Company Profiles and Ask the Portfolio are unchanged. The picker lists every Superset dashboard whose slug starts with arp- and opens on arp-risk-exposure. To add a dashboard you built by hand, set its slug to arp-<name> in Superset's dashboard Properties and reload the tab. The weighted-average climate pivots Pivot Explorer had are available through the climate API endpoints.

The UI embeds a draft through POST /api/bi/embed-token {dashboard_id}, which answers {token, embedded_id} (service.embed_token returns (embedded_id, token)). Only arp- dashboards (slug arp-..., the scratch one and hand-built ones included) are embeddable: another dashboard id gets 403, an unknown one 404.

Caveats:

• Drafts only. Publishing a dashboard is a human step in Superset.
• The superset/Dockerfile build (adds psycopg2 to apache/superset:5.0.0) is unverified end to end: the sandbox proxy's TLS blocked it, and it was tested with an equivalent local image. Run docker compose build superset on a normal machine before relying on it.
• Embedding is verified in Chromium only.
• The first-embed lock is in-process, so run a single uvicorn worker.
• Seed demo holdings (arp portfolio seed-demo) before the views have anything to show.
• Postgres 15 or later is assumed (compose runs pg16). Before 15, every role, bi_reader included, gets CREATE on schema public through PUBLIC, which bootstrap's REVOKE does not remove.
• frame-ancestors lists only http://localhost:5173 and http://127.0.0.1:5173 (the dev UI). Add production origins in superset/superset_config.py.

Projects

A project is a folder under ARP_PROJECTS_DIR (default projects/, git-ignored) holding replayable data sources (uploaded DWS constituent .xlsx files plus the assumed notional) and the dashboards stored for it. The data files are user data and are never committed. Opening a project re-imports its files into Postgres (tagged project:<id>, portfolio ids <project>-dws-<fund isin>) and provisions its dashboards, both idempotently.

UI (Risk Monitoring, Dashboards): pick a project in the selector or choose New project, upload the files and notional, and the project opens automatically. A generated dashboard is saved to the project with Save to project. Dashboards you built by hand in Superset (slug arp-...) appear under Other arp- dashboards and can be exported into a project from there; opening the project re-imports one that is missing (always unpublished, never over an existing slug). Saving the same title again is an explicit Save and rebuilds that dashboard in Superset (only the dashboard is deleted, charts stay; it comes back unpublished); opening a project never changes an existing dashboard.

arp project create alpha --name "Alpha review"
arp project add-data alpha Constituent_IE00B4L5Y983.xlsx --notional-eur 50000000
arp project list
arp project open alpha          # import data, provision dashboards; prints created / rebuilt / unchanged

API (/api/projects): GET "", POST "", GET /{id}, POST /{id}/data (multipart: file, notional_eur), POST /{id}/open, POST /{id}/dashboards (save a designed dashboard plan), POST /{id}/dashboards/export (store a hand-built dashboard).

Scoping: bi.holdings and bi.holdings_history have a trailing project_id column taken from the portfolio tag. Project dashboards carry the chart filter project_id == <id> and their native filters (Fund, Sector, Country) are pre-filtered the same way, so options list only that project's data. The STANDARD dashboards (such as arp-risk-exposure) stay unfiltered and show ALL data in the shared Postgres, every opened project included. Securities and companies are global and keyed by ISIN: the last import wins for sector and country labels, so a conflicting file in project B can change project A's sector/country breakdowns (holdings themselves never mix). Hand-built dashboards are stored as Superset export bundles and are not auto-scoped; filter them on project_id yourself if they must be.

Upgrade: re-run arp bi bootstrap so the views gain project_id. Project dashboards created before the native-filter scoping fix keep global filter options until rebuilt: delete the dashboard in Superset and open the project again.

Limits: the only data kind is the DWS constituent .xlsx; uploads are .xlsx up to 50 MB; there is no project delete, versioning or authentication; opening needs the Postgres portfolio backend. ARP_BI_READER_PASSWORD (not a placeholder) is needed only to re-import stored hand-built bundles.

CLI

The CLI drives the same pipelines as the API and is the intended path for large unattended batches. arp --help lists every command; a representative slice:

arp theme decompose "Electrification" --out theme.json
arp theme run --theme theme.json --universe companies.csv
arp taxonomy create "Electrification" --method industry_anchored
arp extract run --schema schema.json --universe companies.csv
arp extract financials-run --universe companies.csv
arp transition-barrier scores --region China --pillar Regulation
arp emerging-themes run --universe companies.csv
arp replicate backtest --spec spec.json --prices prices.csv --tickers universe.csv
arp climate waci --group-by portfolio_id
arp decision derive --source transition_plan_run --run-id <run_id> --save   # no API key: zero LLM calls
arp decision score --source transition_barrier --region "European Union"  # entity = sector, not company
arp decision score --dataset <dataset_id>