Native desktop automation with xa11y actions and Jev decisions; record and replay natural-language YAML tests

<img src="docs/banner.webp" alt="plain: a coral square with a lowercase p and a dot, next to the word plain." width="100%" />
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT license" /></a> <img src="https://img.shields.io/badge/node-%3E%3D22-brightgreen.svg" alt="Node 22 or newer" /> <img src="https://img.shields.io/badge/browser-Playwright-45ba4b.svg" alt="Driven by Playwright" />
Write browser, desktop and mobile tests in plain English. Run the YAML specs from the command line. Or let a coding agent explore a flow through MCP and save it as a spec.
<img src="docs/demo/demo.gif" alt="A browser is at the left and a terminal is at the right. The terminal shows the buy a backpack spec. plain runs it with Jev. The browser logs in to saucedemo.com, adds a backpack to the cart, fills in the checkout form and completes the order. All 15 steps pass. Then a no-judge run replays the steps from the lock file with no Jev call. The steps pass again." width="100%" />
Jev, a small decision model, selects the element that a target describes and judges each claim. Jev reads the accessibility tree. Playwright, xa11y or Appium do the actions. When a browser check fails, plain sends the viewport screenshot to Jev and prints what it shows. That report does not change the result.
https://github.com/user-attachments/assets/e5665a10-3690-403a-ab8c-6bb04e9f6d89
agent mode, run modes and lock files, what Jev costs, and how it compares with a classic test framework.</sub></p>
| Browser | Desktop | Mobile | |
|---|---|---|---|
| Plugin and CLI | plain | plain-computer | plain-mobile |
| Actions by | Playwright | xa11y | Appium |
| Spec target | url: a website | app: a running application | platform, device, app: an installed app |
| Platforms | Chromium or Chrome | macOS (tested), Windows, Linux | iOS, Android |
The desktop and mobile engines are beta. Use the browser engine for a paid pilot.
You need Node 22 or newer, npm, and a TypeSafe API key (a Vercel AI Gateway key also works).
To let your coding agent do the setup, give it this prompt:
Set up plain for me. Read https://raw.githubusercontent.com/gabe4coding/plain/main/docs/getting-started.mdx and ask me which engines I need: browser, desktop or mobile. Then do the guide's steps for this agent host and the setup pages it links for my engines, and install the tools that are missing.
Ask me before a step that needs sudo, an Apple ID, a license or a system permission. If the API key is missing, tell me which file to put it in and wait. Do not ask me to paste the key in chat.
At the end, run the first browser spec of the guide to check the setup. Tell me what is still missing, and if I must start a new session.
To do the setup yourself:
~/.config/plain/.env. All plugins and CLIs read this file. TYPESAFE_API_KEY=<your key>
/plugin marketplace add gabe4coding/plain
/plugin install plain@plain-marketplace
In Codex:
codex plugin marketplace add gabe4coding/plain
codex plugin add plain@plain-marketplace
todo.yaml. Then run it with npx. On the first run, npx installs the @gabe4coding/plain npm package and plain installs Chromium. npx -p @gabe4coding/plain plain todo.yaml
Getting started shows the desktop and mobile plugins, the key options and all environment variables.
A spec names its target and lists its steps. A browser spec starts with goto: url is only the base for goto.
name: add a todo
url: https://demo.playwright.dev/todomvc
steps:
- goto: /
- fill: { target: "the new todo input", value: "buy milk" }
- press: Enter
- expect: "a todo item named 'buy milk' is listed"
Describe each element by its role and visible text. Write each claim as a specific statement about the current state. The phrasing guide shows how.
In a comparison on six browser workflows, an agent with plain used 25–52% less time than with Playwright MCP. The API cost changed from 24% lower to 7% higher, by model.
| Guide | Contents |
|---|---|
| Getting started | Requirements, API key, plugin install, first specs, environment variables. |
| Spec reference | Spec format, browser steps, reusable flows, browser context, login state. |
| Phrasing | Targets, claims, thresholds, inconclusive results. |
| Hooks | Setup and teardown, test data in ${hooks.*}. |
| Running suites | CLI flags, config file, selection, retries, run modes, exit codes. |
| Reporting | Text, JSONL, JUnit and JSON output. |
| Artifacts | Screenshots, traces, debug dumps. |
| Security and data | What a run sends, temp files, secrets. |
| Web pilot runbook | First specs in CI, nightly runs, what to measure. |
| CI and containers | GitHub Action, GitLab, Docker. |
| Agent mode | Browser MCP tools, save, your real browser. |
| Snapshots | Raw, compact and smart snapshot modes. |
| Desktop | Desktop setup, permissions, specs and tools. |
| Mobile | Appium setup, mobile specs, gestures and tools. |
| Performance | Measured cost and speed, links to the benchmark reports. |
| Development | Build, tests, native smoke tests, plugin packaging. |
The same author maintains these plugins. Each plugin installs and works alone.
hooks/session-pane/register.ts 109 lines1// The mods API is typed by the declarations Claude Code writes into .claude-plugin/types/ for the installed version;
2// nothing type-checks this module at build time, so the hook arguments stay `any` here.
3import { ENGINE, PLUGIN } from './config.ts';
4import { apply, emptyState, jsonOf, payload, saveName, type State } from './model.ts';
5import { render, type Elements } from './view.ts';
6
7/** The pane and its command are named after the plugin, so each plugin's copy has its own. */
8const PANE = PLUGIN;
9const COMMAND = `${PLUGIN}-pane`;
10/** Every tool of this plugin's MCP server, which has the plugin's name: `mcp__plugin_plain_plain__step`. */
11const TOOL = new RegExp(`^mcp__plugin_${PLUGIN}_${PLUGIN}__(\\w+)$`);
12/** The plugin's own MCP server, named as `$.mcp.call` takes it: the tool-name spelling without `mcp__` (the server has the plugin's name, as TOOL assumes). */
13const SERVER = `plugin_${PLUGIN}_${PLUGIN}`;
14
15let state: State = emptyState();
16/** What the user typed in the Spec field; undefined follows the goal. */
17let typedPath: string | undefined;
18/** Whether this session's first `open` already showed the pane: later ones never reopen it. */
19let shown = false;
20
21const specPath = (): string => typedPath?.trim() || saveName(state.goal);
22const message = (error: unknown): string => (error instanceof Error ? error.message : String(error));
23const errorText = (result: any): string => result?.content?.find((block: any) => block?.type === 'text')?.text ?? 'unknown error';
24
25/** Opens the pane; true when it is drawn. One opened unasked waits undrawn below 144 terminal columns. */
26async function show($: any): Promise<boolean> {
27 const opened = await $.ui.open({ id: PANE, title: PLUGIN });
28 return opened?.isPlaced !== false;
29}
30
31/** Asks the plugin's own server to write the passing steps: the same session Claude is driving. */
32async function save($: any, path: string): Promise<void> {
33 try {
34 const result = await $.mcp.call(SERVER, 'save', { path });
35 const data = jsonOf(result);
36 $.ui.toast(result?.isError ? `Save failed: ${errorText(result)}` : `Saved ${data?.steps ?? '?'} steps to ${data?.path ?? path}`);
37 } catch (error) {
38 $.ui.toast(`Save failed: ${message(error)}`);
39 }
40}
41
42async function copy($: any): Promise<void> {
43 if (!state.lastFailure) return;
44 try {
45 const copied = await $.ui.copy({ text: JSON.stringify(state.lastFailure, null, 2) });
46 $.ui.toast(copied?.isCopied ? 'Copied the last failure' : `Copy failed: ${copied?.reason ?? 'nothing was copied'}`);
47 } catch (error) {
48 $.ui.toast(`Copy failed: ${message(error)}`);
49 }
50}
51
52export function register(on: any): void {
53 on('session.start', async ($: any, e: any, next: any) => {
54 await $.command.register({ name: COMMAND, description: `Show the ${PLUGIN} session pane`, immediate: true });
55 return next(e);
56 });
57
58 on('command.run', { command: COMMAND }, async ($: any) => {
59 await show($);
60 return {};
61 });
62
63 // Observe only: the result goes back to Claude exactly as the server sent it, whatever happens in here.
64 on('tool.call', { tool: TOOL }, async ($: any, e: any, next: any) => {
65 const outcome = await next(e);
66 try {
67 const data = payload(outcome);
68 if (data) {
69 const tool = TOOL.exec(e.tool)![1];
70 const goal = state.goal;
71 // The event is the tool input: its arguments are fields of `e` beside `tool` (url, step, steps, claims, ...).
72 state = apply(state, ENGINE, tool, e, data);
73 // A new flow gets its own spec name: a path typed for the last one would overwrite that spec.
74 if (tool === 'open' && (ENGINE === 'native' || state.goal !== goal)) typedPath = undefined;
75 $.ui.invalidate('ui.render');
76 if (tool === 'open' && !shown) {
77 shown = true;
78 void show($)
79 .then((drawn) => drawn || $.ui.toast(`Run /${COMMAND} to show the ${PLUGIN} session pane`))
80 .catch((error: unknown) => $.ui.toast(`${PLUGIN} pane: ${message(error)}`));
81 }
82 }
83 } catch (error) {
84 $.ui.toast(`${PLUGIN} pane: ${message(error)}`);
85 }
86 return outcome;
87 });
88
89 on('ui.render', { component: 'Pane' }, async ($: any, e: any, next: any) => {
90 if (e.requestId !== PANE) return next(e);
91 return render(state, $.ui.resolve(e) as Elements, {
92 columns: e.props?.bodyColumns ?? 60,
93 savePath: specPath(),
94 onSavePath: (path) => {
95 typedPath = path;
96 },
97 onSave: (path) => {
98 // Enter in the field submits its text: keep it as the typed path, as typing does.
99 if (path?.trim()) {
100 typedPath = path;
101 $.ui.invalidate('ui.render');
102 }
103 return save($, specPath());
104 },
105 onCopy: () => copy($),
106 });
107 });
108}
109hooks/session-pane/config.ts 4 lines1// Generated by scripts/build-plugins.mjs from mods/session-pane/: do not edit.
2export const PLUGIN: string = "plain-computer";
3export const ENGINE: 'browser' | 'native' = "native";
4hooks/session-pane/model.ts 140 lines1export type Status = 'pass' | 'fail' | 'inconclusive' | 'error' | 'skipped';
2export type Engine = 'browser' | 'native';
3export type Row = { label: string; status?: Status; tokens: number; detail?: string; dim?: true };
4export type Failure = { step: unknown; status: Status; detail?: string; notes?: unknown; url?: string; changed?: unknown };
5export type State = { target?: string; goal?: string; rows: Row[]; tokens: number; lastFailure?: Failure };
6type Data = Record<string, unknown>;
7
8/** Older rows are dropped, so a long session does not grow the pane without bound. */
9export const MAX_ROWS = 200;
10const STATUSES: readonly Status[] = ['pass', 'fail', 'inconclusive', 'error', 'skipped'];
11
12export const emptyState = (): State => ({ rows: [], tokens: 0 });
13
14const str = (value: unknown): string | undefined => (typeof value === 'string' && value ? value : undefined);
15const num = (value: unknown): number => (typeof value === 'number' && Number.isFinite(value) ? value : 0);
16const asStatus = (value: unknown): Status => (STATUSES.includes(value as Status) ? (value as Status) : 'error');
17const capped = (rows: Row[]): Row[] => rows.slice(-MAX_ROWS);
18
19/** The JSON object in an MCP result: its structured content, its first text block, or a JSON string. */
20export function jsonOf(value: unknown): Data | null {
21 if (typeof value === 'string') {
22 try {
23 return jsonOf(JSON.parse(value));
24 } catch {
25 return null;
26 }
27 }
28 if (Array.isArray(value)) return jsonOf(value.find((block) => block?.type === 'text')?.text);
29 if (!value || typeof value !== 'object') return null;
30 const record = value as Data;
31 if (record.structuredContent !== undefined) return jsonOf(record.structuredContent);
32 if (record.content !== undefined) return jsonOf(record.content);
33 return record;
34}
35
36/** What a plain tool returned, as `next(e)` hands it to a `tool.call` hook; null when refused or failed. */
37export function payload(outcome: unknown): Data | null {
38 if (!outcome || typeof outcome !== 'object') return null;
39 const { deny, isError, result, text } = outcome as Data;
40 if (deny !== undefined || isError) return null;
41 if (result && typeof result === 'object' && (result as Data).isError) return null;
42 return jsonOf(text) ?? jsonOf(result);
43}
44
45/** The words of a step's value: the string itself, or its target/claim field. */
46function words(value: unknown): string {
47 if (typeof value === 'string' || typeof value === 'number') return String(value);
48 if (Array.isArray(value)) return value.map(words).filter(Boolean).join('; ');
49 if (value && typeof value === 'object') {
50 const fields = value as Data;
51 for (const key of ['target', 'that', 'url', 'to', 'key']) {
52 const field = str(fields[key]);
53 if (field) return field;
54 }
55 return words(Object.values(fields).find((field) => typeof field === 'string'));
56 }
57 return '';
58}
59
60/** `click "Save"` from `{ click: "Save" }`: the step kind, then the words naming its target or claim. */
61export function stepLabel(step: unknown): string {
62 if (!step || typeof step !== 'object') return 'step';
63 const entry = Object.entries(step).find(([key]) => key !== 'optional');
64 if (!entry) return 'step';
65 const [kind, value] = entry;
66 const text = words(value);
67 return text ? `${kind} "${text}"` : kind;
68}
69
70type Outcome = Data & { step: unknown };
71
72/** Adds one row per step outcome; a failing outcome (neither pass nor skipped) becomes the last failure. */
73function withSteps(state: State, outcomes: Outcome[], batchChange?: unknown): State {
74 let next = state;
75 for (const outcome of outcomes) {
76 const status = asStatus(outcome.status);
77 const tokens = num(outcome.jevTokens);
78 const detail = status === 'pass' ? undefined : str(outcome.detail);
79 const row: Row = { label: stepLabel(outcome.step), status, tokens, ...(detail ? { detail } : {}) };
80 next = {
81 ...next,
82 target: str(outcome.url) ?? next.target,
83 tokens: next.tokens + tokens,
84 rows: capped([...next.rows, row]),
85 lastFailure: status === 'pass' || status === 'skipped' ? next.lastFailure : {
86 step: outcome.step,
87 status,
88 detail: str(outcome.detail),
89 notes: outcome.notes,
90 url: str(outcome.url),
91 changed: outcome.changed ?? batchChange,
92 },
93 };
94 }
95 return next;
96}
97
98/** Tools that read without acting: a dim row naming what they read. */
99const READS: Record<string, (args: Data) => string | undefined> = {
100 ask: (args) => (Array.isArray(args.claims) ? args.claims.join('; ') : undefined),
101 read: (args) => str(args.question),
102 find: (args) => str(args.target),
103 snapshot: (args) => str(args.within),
104};
105
106/** The pane state after one plain call: the tool's short name, its arguments and its result JSON. */
107export function apply(state: State, engine: Engine, tool: string, args: Data, data: Data): State {
108 if (tool === 'open') {
109 const target = str(data.url) ?? str(data.name) ?? str(data.app) ?? str(args.url) ?? str(args.app) ?? str(args.device) ?? state.target;
110 if (engine === 'native') return { ...emptyState(), target, goal: str(args.goal) };
111 const goto: Row = { label: `goto "${str(args.url) ?? target ?? ''}"`, status: 'pass', tokens: 0 };
112 return { ...state, target, goal: str(args.goal) ?? state.goal, rows: capped([...state.rows, goto]) };
113 }
114 if (tool === 'step') return withSteps(state, [{ ...data, step: args.step }]);
115 if (tool === 'batch') {
116 const steps = Array.isArray(args.steps) ? args.steps : [];
117 const results = Array.isArray(data.results) ? (data.results as Data[]) : [];
118 const stepAt = (index: unknown) => (typeof index === 'number' ? steps[index] : undefined);
119 return withSteps(state, results.map((result) => ({ ...result, step: stepAt(result.index) })), data.changed);
120 }
121 const read = READS[tool];
122 if (!read) return state;
123 const tokens = num(data.jevTokens);
124 const what = read(args);
125 const row: Row = { label: what ? `${tool} "${what}"` : tool, tokens, dim: true };
126 return { ...state, tokens: state.tokens + tokens, rows: capped([...state.rows, row]) };
127}
128
129/** Rows that are steps (reads are not), and how many of them passed. */
130export function counts(state: State): { steps: number; passed: number } {
131 const steps = state.rows.filter((row) => row.status !== undefined);
132 return { steps: steps.length, passed: steps.filter((row) => row.status === 'pass').length };
133}
134
135/** A spec file name from the flow's goal: `read-the-discussion.yaml`. */
136export function saveName(goal: string | undefined): string {
137 const slug = (goal ?? '').toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/^-+|-+$/g, '').slice(0, 48).replace(/-+$/, '');
138 return `${slug || 'plain-session'}.yaml`;
139}
140hooks/session-pane/view.ts 66 lines1import { counts, type State, type Status } from './model.ts';
2
3/** The element factories `$.ui.resolve(e)` returns; only the four both the terminal and Desktop draw. */
4export type Elements = Record<'Box' | 'Text' | 'Button' | 'Input', (props: Record<string, unknown>) => unknown>;
5export type Controls = {
6 columns: number;
7 savePath: string;
8 onSavePath: (path: string) => void;
9 /** With a path from the field's Enter; without one from the button, which uses the latest typed path. */
10 onSave: (path?: string) => unknown;
11 onCopy: () => unknown;
12};
13
14const ICON: Record<Status, string> = { pass: '✓', fail: '✗', inconclusive: '?', error: '!', skipped: '–' };
15const COLOR: Partial<Record<Status, string>> = { pass: 'green', fail: 'red', inconclusive: 'yellow', error: 'red' };
16
17/** Columns left free at the right: a docked terminal pane (2.1.287) clips about two columns inside `bodyColumns`. */
18export const EDGE = 2;
19
20/** `text` cut to `width` characters, with an ellipsis when cut. */
21export function fit(text: string, width: number): string {
22 return text.length <= width ? text : text.slice(0, Math.max(0, width - 1)) + '…';
23}
24
25/** `left`, then spaces, then `right` ending at `width`; `left` is cut first. */
26export function spread(left: string, right: string, width: number): string {
27 if (!right) return fit(left, width);
28 const start = fit(left, Math.max(1, width - right.length - 1));
29 return start + ' '.repeat(Math.max(1, width - start.length - right.length)) + right;
30}
31
32/** The pane: target and goal, one line per row (detail under a non-pass step), totals, then the controls. */
33export function render(state: State, el: Elements, controls: Controls): unknown {
34 const { Box, Text, Button, Input } = el;
35 const width = Math.max(20, controls.columns - EDGE);
36 const line = (text: string, style: Record<string, unknown> = {}) => Text({ ...style, children: [text] });
37
38 const children: unknown[] = [line(fit(state.target ?? 'No page open yet', width), { bold: true })];
39 if (state.goal) children.push(line(fit(`goal: ${state.goal}`, width), { dimColor: true }));
40 children.push(line(' '));
41 if (state.rows.length === 0) children.push(line('Waiting for the first step…', { dimColor: true }));
42 for (const row of state.rows) {
43 const icon = row.status ? ICON[row.status] : '·';
44 const color = row.status && COLOR[row.status];
45 const style = row.dim ? { dimColor: true } : color ? { color } : {};
46 children.push(line(spread(`${icon} ${row.label}`, row.tokens ? `${row.tokens} tk` : '', width), style));
47 if (row.detail) children.push(line(fit(` ${row.status}: ${row.detail}`, width), { dimColor: true }));
48 }
49
50 const { steps, passed } = counts(state);
51 children.push(line(' '));
52 children.push(line(fit(`${steps} steps · ${passed} pass · ${state.tokens} Jev tokens`, width), { dimColor: true }));
53 children.push(Input({
54 key: 'save-path',
55 label: 'Spec',
56 value: controls.savePath,
57 submitLabel: 'save',
58 onInput: controls.onSavePath,
59 onSubmit: (path: string) => controls.onSave(path),
60 }));
61 const buttons = [Button({ key: 'save', label: 'Save as spec', onPress: () => controls.onSave() })];
62 if (state.lastFailure) buttons.push(Button({ key: 'copy', label: 'Copy last failure', onPress: () => controls.onCopy() }));
63 children.push(Box({ flexDirection: 'row', columnGap: 2, children: buttons }));
64 return Box({ flexDirection: 'column', children });
65}
66