skyward
Skyward for Claude Code: the `/sky` panel over the daemon's computes, plus a skill for the Python SDK, the `sky` CLI and scripts run with `sky run`.
newpanecommandnetworktimer
Skyward for Claude Code: the `/sky` panel over the daemon's computes, plus a skill for the Python SDK, the `sky` CLI and scripts run with `sky run`.
<img src="https://github.com/gabfssilva/skyward/blob/main/docs/logo.svg?raw=true" alt="Skyward" width="140">
<strong>Cloud accelerators with a single decorator</strong>
<a href="https://github.com/gabfssilva/skyward/actions/workflows/tests.yml"><img src="https://github.com/gabfssilva/skyward/actions/workflows/tests.yml/badge.svg" alt="CI"></a> <a href="https://pypi.org/project/skyward/"><img src="https://img.shields.io/pypi/v/skyward" alt="PyPI"></a> <a href="https://pypi.org/project/skyward/"><img src="https://img.shields.io/pypi/pyversions/skyward" alt="Python"></a> <a href="https://github.com/gabfssilva/skyward/blob/main/LICENSE"><img src="https://img.shields.io/github/license/gabfssilva/skyward" alt="License"></a>
<img src="https://github.com/gabfssilva/skyward/blob/main/docs/demo.gif?raw=true" alt="Skyward Demo" width="800">
Skyward is a Python library for ephemeral accelerator compute. Spin up cloud accelerators, run your code, and tear them down automatically. No infrastructure to manage, no idle costs.
# pi.py
import skyward as sky
@sky.app(
provider=sky.Salad(priority="high"),
accelerator=sky.accelerators.RTX_3090(),
image=sky.Image(pip=["torch", "numpy"]),
)
def estimate_pi(n: int = 100_000_000) -> dict[str, str | float]:
"""Estimate pi from N random points, on a GPU."""
import torch
points = torch.rand(2, n, device="cuda")
inside = (points[0] ** 2 + points[1] ** 2 <= 1).sum().item()
return {"gpu": torch.cuda.get_device_name(), "pi": 4 * inside / n}sky server start
sky run pi.py --n 500000000sky.app declares the machines a function runs on. sky run provisions them, parses the command line against the function's signature, runs the function there, prints what it returned as JSON, and tears the machines down.
sky.Compute is the same compute as a context manager, for a program that dispatches functions itself:
import skyward as sky
@sky.function
def train(epochs: int) -> dict:
import torch
model = torch.nn.Linear(100, 10).cuda()
optimizer = torch.optim.Adam(model.parameters())
for epoch in range(epochs):
loss = model(torch.randn(32, 100, device="cuda")).sum()
loss.backward()
optimizer.step()
return { "final_loss": loss.item() }
with sky.Compute(
provider=sky.AWS(),
accelerator=sky.accelerators.T4(),
image=sky.Image(pip=["torch"])
) as compute:
result = train(epochs=100) >> compute
print(result)>> executes on one node, @ broadcasts to all, & runs in parallel. No job configs, no YAML.uv add "skyward[all]"The base package is what a node installs. The SDK, the daemon, the sky command and each provider SDK are extras; getting started lists them.
Full documentation at gabfssilva.github.io/skyward.
MIT
hooks/register.tsx 1248 lines1import type { EngineInterface, HttpInit, HttpResponse, Register, RenderElement, Timer } from 'claude-code'
2
3const DAEMON = 'http://127.0.0.1:17590'
4const PANE_ID = 'skyward'
5const REFRESH_MS = 3000
6const LOG_LINES = 20
7const SPARK_STEP_MS = 30_000
8const SPARK_BUCKETS = 20
9const SPARK_GLYPHS = '▁▂▃▄▅▆▇█'
10const NO_SAMPLES: readonly null[] = Array.from({ length: SPARK_BUCKETS }, () => null)
11const NODES_PER_PAGE = 4
12const IDLE_GPU_PCT = 5
13// Four buckets of 30s: a gap between two tasks is not idleness.
14const IDLE_BUCKETS = 4
15const NODE_STAGES = ['requested', 'provisioning', 'connecting', 'bootstrapping', 'ready', 'draining', 'lost', 'deleting', 'deleted', 'failed']
16// The states the daemon counts a node alive in; a node in any other is already on its way out.
17const DRAINABLE = new Set(['requested', 'provisioning', 'connecting', 'bootstrapping', 'ready'])
18const TASKS_PER_PAGE = 5
19const WRITE_ATTEMPTS = 5
20const NAME_WIDTH = 16
21const MARQUEE_MS = 250
22const MARQUEE_PAUSE_FRAMES = 8
23
24type Compute = {
25 id: string
26 name: string | null
27 state: string
28 nodes: Node[]
29 nodesReady: number
30 nodesTotal: number
31 provider: string
32 accelerator: string | null
33 acceleratorCount: number
34 rate: number
35 createdAt: number
36 ended: boolean
37 cost: number | null
38 lastError: string | null
39 bounds: Bounds
40 /** Each node's newest reading of each metric, keyed node/name — present when the read asked for it. */
41 latest: Map<string, number>
42}
43
44// `spec.nodes`, with an unset `min` or `max` read as `initial`, as the reconciler reads it.
45type Bounds = { initial: number; min: number; max: number }
46
47type Node = {
48 id: string
49 rank: number
50 state: string
51 address: string | null
52 machine: string | null
53 market: string | null
54 price: number | null
55 launchedAt: number | null
56 terminatedAt: number | null
57 lastError: string | null
58}
59
60type LogLine = { at: number; node: string | null; type: string; text: string; error: boolean }
61
62// `cursors` is the way back from the newest page: the log only pages toward older entries.
63type LogFilter = { node: string | null; task: string | null; term: string; cursors: readonly string[] }
64type LogPage = { path: string; lines: LogLine[]; next: string | null }
65
66// `node`, `startedAt` and `error` come from the latest attempt.
67type Task = {
68 id: string
69 sha: string
70 name: string | null
71 state: string
72 node: string | null
73 attempts: number
74 submittedAt: number
75 startedAt: number | null
76 finishedAt: number | null
77 error: string | null
78}
79
80type TaskOrder = 'state' | 'submitted' | 'finished'
81// Tasks also page forward only, so `cursors` is the way back to the first page.
82type TaskQuery = { order: TaskOrder; fn: string | null; cursors: readonly string[] }
83type TaskPage = { items: Task[]; total: number | null; next: string | null; counts: Map<string, number> }
84
85// What the compute is as a whole: its nodes counted by stage, the mean of each metric over the nodes still
86// alive, and whether it is doing anything. `idle` is null when no node reported gpu_util in the window, and
87// `capped` means the whole window was idle, so it began before the window did.
88type Fleet = {
89 stages: readonly (readonly [string, number])[]
90 averages: ReadonlyMap<string, number>
91 running: number
92 idle: { since: number; capped: boolean } | null
93}
94
95// `listed` is `nodes` cut to `group`, and what `page` and `pageCount` count.
96type Detail = {
97 compute: Compute
98 fleet: Fleet
99 nodes: Node[]
100 group: NodeGroup | null
101 listed: Node[]
102 page: number
103 pageCount: number
104 latest: Map<string, number>
105 sparks: Map<string, (number | null)[]>
106 tasks: TaskPage
107 log: LogPage
108}
109
110type DetailView = { kind: 'detail'; id: string; page: number; group: NodeGroup | null; expanded: ReadonlySet<string>; tasks: TaskQuery; log: LogFilter }
111type View = { kind: 'list' } | DetailView
112
113type Step = { step: 'asking' } | { step: 'sending' } | { step: 'failed'; reason: string }
114// One confirmation or form is open at a time, so asking for one closes the other. A kill's `id` is the node's; a
115// scale's `min` and `max` are what was typed, empty for the bound it was opened on.
116type Action = ({ kind: 'delete'; id: string } | { kind: 'kill'; id: string } | { kind: 'scale'; id: string; min: string; max: string; invalid: boolean }) & Step
117
118// A write outlives the drawing that pressed it, so it goes through the command's engine rather than the render's.
119type Writes = {
120 remove: (id: string) => Promise<void>
121 scale: (c: Compute, min: number, max: number) => Promise<void>
122 drain: (compute: string, node: string) => Promise<void>
123}
124
125type Snapshot =
126 | { kind: 'loading' }
127 | { kind: 'down'; reason: string }
128 | { kind: 'list'; live: Compute[]; recent: Compute[]; at: number }
129 | { kind: 'detail'; detail: Detail; at: number }
130
131const STATE_COLORS = new Map([
132 ['requested', 'yellow'],
133 ['provisioning', 'yellow'],
134 ['connecting', 'yellow'],
135 ['bootstrapping', 'yellow'],
136 ['ready', 'green'],
137 ['degraded', 'red'],
138 ['draining', 'magenta'],
139 ['deleting', 'magenta'],
140])
141
142// Dark enough for white text on light and dark themes; no pure green, red or yellow, which the states use.
143const RANK_COLORS = ['#3b6fd8', '#8e5bd6', '#1f8f8f', '#c7702a', '#c2427a', '#6b8e23', '#546e7a', '#8d6e63']
144const RANK_WIDTH = 5
145
146const NEXT_TASK_ORDER: Record<TaskOrder, TaskOrder> = { state: 'submitted', submitted: 'finished', finished: 'state' }
147// The rest (cancelled, timed_out, indeterminate) is shown as what the total leaves over.
148const COUNTED_TASK_STATES = ['running', 'queued', 'succeeded', 'failed']
149const TASK_GLYPHS = new Map([
150 ['running', '●'],
151 ['queued', '◌'],
152 ['succeeded', '✓'],
153 ['failed', '✗'],
154 ['timed_out', '✗'],
155 ['cancelled', '○'],
156 ['indeterminate', '?'],
157])
158const TASK_COLORS = new Map([
159 ['running', 'green'],
160 ['failed', 'red'],
161 ['timed_out', 'red'],
162 ['indeterminate', 'magenta'],
163])
164
165const STATE_GLYPHS = new Map([
166 ['requested', '◌'],
167 ['provisioning', '◐'],
168 ['ready', '●'],
169 ['degraded', '●'],
170 ['deleting', '◑'],
171 ['deleted', '○'],
172])
173
174const at = (value: unknown, ...path: string[]): unknown =>
175 path.reduce<unknown>((current, key) => {
176 if (typeof current !== 'object' || current === null) return undefined
177 const field: unknown = Object.getOwnPropertyDescriptor(current, key)?.value
178 return field
179 }, value)
180const str = (value: unknown): string | undefined => (typeof value === 'string' ? value : undefined)
181const num = (value: unknown): number | undefined => (typeof value === 'number' ? value : undefined)
182const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : [])
183const time = (value: unknown): number | null => {
184 const ms = Date.parse(str(value) ?? '')
185 return Number.isNaN(ms) ? null : ms
186}
187const items = <T,>(body: unknown, parse: (item: unknown) => T | undefined): T[] =>
188 list(at(body, 'items')).flatMap((item) => {
189 const parsed = parse(item)
190 return parsed === undefined ? [] : [parsed]
191 })
192const metricKey = (node: string, name: string): string => `${node}/${name}`
193const metricNode = (key: string): string => key.slice(0, key.lastIndexOf('/'))
194
195function parseCompute(item: unknown): Compute | undefined {
196 const id = str(at(item, 'id'))
197 const state = str(at(item, 'status', 'state'))
198 const createdAt = time(at(item, 'created_at'))
199 if (id === undefined || state === undefined || createdAt === null) return undefined
200
201 const spec = at(item, 'spec', 'specs', '0')
202 // A compute carries the node holding each rank, and the ones that held one before when the read asked for them.
203 const nodes = list(at(item, 'nodes'))
204 .flatMap((node) => {
205 const parsed = parseNode(node)
206 return parsed === undefined ? [] : [parsed]
207 })
208 .sort((a, b) => Number(a.terminatedAt !== null) - Number(b.terminatedAt !== null) || a.rank - b.rank)
209 const alive = nodes.filter((node) => node.terminatedAt === null)
210 const initial = num(at(item, 'spec', 'nodes', 'initial')) ?? 0
211 return {
212 id,
213 state,
214 nodes,
215 nodesReady: alive.filter((node) => node.state === 'ready').length,
216 nodesTotal: alive.length,
217 createdAt,
218 name: str(at(item, 'name')) ?? null,
219 provider: str(at(item, 'provider', 'name')) ?? str(at(spec, 'provider', 'name')) ?? '?',
220 accelerator: str(at(item, 'offer', 'accelerator')) ?? str(at(spec, 'accelerator')) ?? null,
221 acceleratorCount: num(at(item, 'offer', 'accelerator_count')) ?? num(at(spec, 'accelerator_count')) ?? 1,
222 rate: num(at(item, 'rate')) ?? 0,
223 ended: time(at(item, 'ended', 'at')) !== null,
224 cost: num(at(item, 'cost')) ?? null,
225 lastError: str(at(item, 'status', 'last_error', 'message')) ?? null,
226 bounds: { initial, min: num(at(item, 'spec', 'nodes', 'min')) ?? initial, max: num(at(item, 'spec', 'nodes', 'max')) ?? initial },
227 latest: new Map(
228 list(at(item, 'nodes')).flatMap((node) => {
229 const id = str(at(node, 'id'))
230 const metrics = at(node, 'metrics')
231 if (id === undefined || typeof metrics !== 'object' || metrics === null) return []
232 return Object.entries(metrics).flatMap(([name, gauge]): [string, number][] => {
233 const value = num(at(gauge, 'value'))
234 return value === undefined ? [] : [[metricKey(id, name), value]]
235 })
236 }),
237 ),
238 }
239}
240
241function parseNode(item: unknown): Node | undefined {
242 const id = str(at(item, 'id'))
243 const rank = num(at(item, 'rank'))
244 const state = str(at(item, 'state'))
245 if (id === undefined || rank === undefined || state === undefined) return undefined
246 return {
247 id,
248 rank,
249 state,
250 address: str(at(item, 'address')) ?? null,
251 machine: str(at(item, 'machine')) ?? null,
252 market: str(at(item, 'market')) ?? null,
253 price: num(at(item, 'price_per_hour')) ?? null,
254 launchedAt: time(at(item, 'launched_at')),
255 terminatedAt: time(at(item, 'terminated_at')),
256 lastError: str(at(item, 'last_error', 'message')) ?? null,
257 }
258}
259
260function parseLogLine(item: unknown): LogLine | undefined {
261 const type = str(at(item, 'type'))
262 const when = time(at(item, 'at'))
263 if (type === undefined || when === null) return undefined
264 const data = at(item, 'data')
265 const error = str(at(data, 'error'))
266 const text =
267 type === 'node.console'
268 ? str(at(data, 'content')) ?? ''
269 : type === 'node.phase'
270 ? `${str(at(data, 'phase')) ?? '?'} ${str(at(data, 'event')) ?? '?'}`
271 : type
272 return {
273 at: when,
274 type,
275 node: str(at(data, 'node')) ?? null,
276 text: error === undefined ? text : `${text}: ${error}`,
277 error: error !== undefined,
278 }
279}
280
281function parseTask(item: unknown): Task | undefined {
282 const id = str(at(item, 'id'))
283 const sha = str(at(item, 'function', 'sha256'))
284 const state = str(at(item, 'state'))
285 const submittedAt = time(at(item, 'submitted_at'))
286 if (id === undefined || sha === undefined || state === undefined || submittedAt === null) return undefined
287 const executions = list(at(item, 'executions'))
288 const latest = executions.at(-1)
289 return {
290 id,
291 sha,
292 state,
293 submittedAt,
294 name: str(at(item, 'function', 'name')) ?? null,
295 node: str(at(latest, 'node_id')) ?? null,
296 attempts: executions.length,
297 startedAt: time(at(latest, 'started_at')),
298 finishedAt: time(at(item, 'finished_at')),
299 error: str(at(latest, 'error', 'message')) ?? null,
300 }
301}
302
303const age = (ms: number): string => {
304 if (ms < 60_000) return `${Math.max(0, Math.floor(ms / 1000))}s`
305 const minutes = Math.floor(ms / 60_000)
306 if (minutes < 60) return `${minutes}m`
307 const hours = Math.floor(minutes / 60)
308 return hours < 48 ? `${hours}h` : `${Math.floor(hours / 24)}d`
309}
310const price = (value: number | null): string => (value === null ? '-' : `$${value.toFixed(2)}/h`)
311const gib = (mb: number): string => (mb / 1024).toFixed(1)
312const rate = (kbps: number): string => (kbps >= 1000 ? `${(kbps / 1000).toFixed(1)}Mb/s` : `${Math.round(kbps)}kb/s`)
313const spark = (buckets: (number | null)[]): string =>
314 buckets
315 .map((value) => (value === null ? ' ' : SPARK_GLYPHS.charAt(Math.max(0, Math.min(7, Math.floor((value / 100) * 8))))))
316 .join('')
317const clock = (ms: number): string => new Date(ms).toTimeString().slice(0, 8)
318const money = (value: number | null): string => (value === null ? '—' : value.toFixed(2))
319const gpu = (c: Compute): string =>
320 c.accelerator === null ? '—' : c.acceleratorCount > 1 ? `${c.acceleratorCount}×${c.accelerator}` : c.accelerator
321const deletable = (c: Compute): boolean => !c.ended && c.state !== 'deleting' && c.state !== 'deleted'
322const NODE_GROUPS = ['active', 'bootstrapping', 'stopping', 'stopped', 'failed'] as const
323type NodeGroup = (typeof NODE_GROUPS)[number]
324// A node given up on keeps the error that condemned it through `deleting` and `deleted`, so on the way out the error,
325// not the state, tells a failure from a node that was drained or scaled away.
326const nodeGroup = (n: Node): NodeGroup => {
327 switch (n.state) {
328 case 'ready':
329 return 'active'
330 case 'requested':
331 case 'provisioning':
332 case 'connecting':
333 case 'bootstrapping':
334 return 'bootstrapping'
335 case 'lost':
336 case 'failed':
337 return 'failed'
338 default:
339 return n.lastError !== null ? 'failed' : n.state === 'deleted' ? 'stopped' : 'stopping'
340 }
341}
342const inGroup = (nodes: Node[], group: NodeGroup | null): Node[] => (group === null ? nodes : nodes.filter((n) => nodeGroup(n) === group))
343// The group filtered on is kept at zero, so the one way back to every node stays drawn.
344const nodeCounts = (nodes: Node[], filtered: NodeGroup | null): (readonly [NodeGroup, number])[] =>
345 NODE_GROUPS.map((group) => [group, inGroup(nodes, group).length] as const).filter(([group, count]) => count > 0 || group === filtered)
346const hourly = (c: Compute): number | null => (c.ended ? null : c.rate)
347const averaged = (f: Fleet): string => {
348 const mean = (name: string): number | undefined => f.averages.get(name)
349 const pct = (name: string, label: string): string | null => {
350 const value = mean(name)
351 return value === undefined ? null : `${label} ${Math.round(value)}%`
352 }
353 const used = mean('mem_used_mb')
354 const total = mean('mem_total_mb')
355 return [
356 pct('cpu', 'cpu'),
357 pct('gpu_util', 'gpu'),
358 used === undefined || total === undefined ? null : `mem ${gib(used)}/${gib(total)} GB`,
359 pct('disk_used_pct', 'disk'),
360 ]
361 .filter((part) => part !== null)
362 .join(' · ')
363}
364// An empty field keeps the bound the form was opened on.
365const count = (text: string, current: number): number | null => {
366 const typed = text.trim()
367 if (typed === '') return current
368 return /^\d+$/.test(typed) ? Number(typed) : null
369}
370const sum = (values: (number | null)[]): number | null =>
371 values.reduce<number | null>((total, value) => (total === null || value === null ? null : total + value), 0)
372
373// Pauses at both ends, so a name reads from its start before it scrolls and rests on its end before jumping back.
374const marquee = (text: string, width: number, now: number): string => {
375 const overflow = text.length - width
376 if (overflow <= 0) return text
377 const frame = Math.floor(now / MARQUEE_MS) % (overflow + 2 * MARQUEE_PAUSE_FRAMES)
378 const offset = Math.min(overflow, Math.max(0, frame - MARQUEE_PAUSE_FRAMES))
379 return text.slice(offset, offset + width)
380}
381
382type Column = {
383 title: string
384 width: number
385 right: boolean
386 // Lower ranks stay longest when the pane is too narrow for every column.
387 rank: number
388 cell: (c: Compute) => { text: string; color?: string }
389 total?: (live: Compute[]) => string
390}
391
392const COLUMNS: Column[] = [
393 {
394 title: 'STATE',
395 // A glyph and up to 100/100 nodes.
396 width: 9,
397 right: false,
398 rank: 1,
399 cell: (c) => ({
400 text: c.ended ? STATE_GLYPHS.get('deleted') ?? '○' : `${STATE_GLYPHS.get(c.state) ?? '●'} ${c.nodesReady}/${c.nodesTotal}`,
401 color: STATE_COLORS.get(c.state),
402 }),
403 },
404 { title: 'GPU', width: 10, right: false, rank: 2, cell: (c) => ({ text: gpu(c) }) },
405 { title: 'PROVIDER', width: 9, right: false, rank: 6, cell: (c) => ({ text: c.provider }) },
406 { title: '$/H', width: 7, right: true, rank: 4, cell: (c) => ({ text: money(hourly(c)) }), total: (live) => money(sum(live.map(hourly))) },
407 { title: 'COST', width: 7, right: true, rank: 3, cell: (c) => ({ text: money(c.cost) }), total: (live) => money(sum(live.map((c) => c.cost))) },
408 { title: 'AGE', width: 4, right: true, rank: 5, cell: (c) => ({ text: age(Date.now() - c.createdAt) }) },
409]
410
411const TABLE_GAP = 2
412
413const fitColumns = (columns: number): Column[] => {
414 const kept = new Set<Column>()
415 let used = NAME_WIDTH
416 for (const column of [...COLUMNS].sort((a, b) => a.rank - b.rank)) {
417 if (used + TABLE_GAP + column.width > columns) break
418 used += TABLE_GAP + column.width
419 kept.add(column)
420 }
421 return COLUMNS.filter((column) => kept.has(column))
422}
423
424// The daemon refuses with `{code, message}`; a body without a message is shown as it came.
425function refusal(response: HttpResponse): Error {
426 let body: unknown
427 try {
428 body = JSON.parse(response.text)
429 } catch {
430 body = undefined
431 }
432 return new Error(`HTTP ${response.status} ${str(at(body, 'message')) ?? response.text.slice(0, 200)}`)
433}
434
435async function getJson($: EngineInterface, path: string): Promise<unknown> {
436 const response = await $.http.fetch(`${DAEMON}${path}`)
437 if (!response.ok) throw refusal(response)
438 const body: unknown = JSON.parse(response.text)
439 return body
440}
441
442async function loadList($: EngineInterface): Promise<Snapshot> {
443 const [live, recent] = await Promise.all([getJson($, '/v1/computes?live=true'), getJson($, '/v1/computes?live=false&limit=5')])
444 return { kind: 'list', live: items(live, parseCompute), recent: items(recent, parseCompute), at: Date.now() }
445}
446
447const logPath = (id: string, filter: LogFilter): string => {
448 const cursor = filter.cursors.at(-1)
449 return [
450 `/v1/events/log?compute=${id}&limit=${LOG_LINES}`,
451 filter.node === null ? '' : `&node=${filter.node}`,
452 filter.task === null ? '' : `&task=${filter.task}`,
453 filter.term === '' ? '' : `&contains=${encodeURIComponent(filter.term)}`,
454 cursor === undefined ? '' : `&cursor=${cursor}`,
455 ].join('')
456}
457
458async function loadLog($: EngineInterface, path: string): Promise<LogPage> {
459 const body = await getJson($, path)
460 return { path, lines: items(body, parseLogLine), next: str(at(body, 'next_cursor')) ?? null }
461}
462
463async function loadTasks($: EngineInterface, id: string, query: TaskQuery): Promise<TaskPage> {
464 const filter = `/v1/tasks?compute=${id}${query.fn === null ? '' : `&function=${encodeURIComponent(query.fn)}`}`
465 const cursor = query.cursors.at(-1)
466 const [pageBody, ...countBodies] = await Promise.all([
467 getJson($, `${filter}&limit=${TASKS_PER_PAGE}&order=${query.order}${cursor === undefined ? '' : `&cursor=${cursor}`}`),
468 ...COUNTED_TASK_STATES.map((state) => getJson($, `${filter}&state=${state}&limit=1`)),
469 ])
470 const parsed = items(pageBody, parseTask)
471 const total = num(at(pageBody, 'total')) ?? null
472 // A last page exactly TASKS_PER_PAGE long still carries a cursor, and it leads to an empty page.
473 const last = total !== null && query.cursors.length * TASKS_PER_PAGE + parsed.length >= total
474 return {
475 items: parsed,
476 total,
477 next: last ? null : str(at(pageBody, 'next_cursor')) ?? null,
478 counts: new Map(COUNTED_TASK_STATES.map((state, i) => [state, num(at(countBodies[i], 'total')) ?? 0])),
479 }
480}
481
482async function loadDetail($: EngineInterface, target: DetailView, shown: LogPage | undefined): Promise<Snapshot> {
483 const { id, page: requestedPage } = target
484 const start = (Math.floor(Date.now() / SPARK_STEP_MS) - (SPARK_BUCKETS - 1)) * SPARK_STEP_MS
485 const base = `/v1/computes/${id}`
486 const path = logPath(id, target.log)
487 // The log is append-only, so a page behind a cursor never changes and only the newest page is fetched again.
488 const unchanged = target.log.cursors.length > 0 && shown?.path === path ? shown : undefined
489 // One read is the compute, the node holding each rank, the ones replaced before them, and each node's newest gauges.
490 const [computeBody, tasks, log] = await Promise.all([
491 getJson($, `${base}?include=nodes.replaced,nodes.metrics`),
492 loadTasks($, id, target.tasks),
493 unchanged ?? loadLog($, path),
494 ])
495 const compute = parseCompute(computeBody)
496 if (compute === undefined) throw new Error(`unexpected compute payload for ${id}`)
497
498 // The nodes are filtered and cut into pages here. The metrics are the whole compute's rather than the page's: the
499 // header averages them and reads idleness off the window, and a page's sparklines read the same two maps.
500 const nodes = compute.nodes
501 const listed = inGroup(nodes, target.group)
502 const latest = compute.latest
503 const pageCount = Math.max(1, Math.ceil(listed.length / NODES_PER_PAGE))
504 const page = Math.min(requestedPage, pageCount - 1)
505 const seriesBody = nodes.length === 0 ? undefined : await getJson($, `${base}/metrics?since=${start}&step=${SPARK_STEP_MS}&agg=avg&name=cpu&name=gpu_util`)
506
507 const sparks = new Map<string, (number | null)[]>()
508 for (const series of list(at(seriesBody, 'series'))) {
509 const node = str(at(series, 'node'))
510 const name = str(at(series, 'name'))
511 if (node === undefined || name === undefined) continue
512 const values = list(at(series, 'values'))
513 const buckets = Array.from({ length: SPARK_BUCKETS }, (): number | null => null)
514 list(at(series, 'at')).forEach((bucketAt, i) => {
515 const ms = num(bucketAt)
516 const value = num(values[i])
517 if (ms === undefined || value === undefined) return
518 const index = Math.floor((ms - start) / SPARK_STEP_MS)
519 if (index >= 0 && index < SPARK_BUCKETS) buckets[index] = value
520 })
521 sparks.set(metricKey(node, name), buckets)
522 }
523
524 const alive = nodes.filter((n) => n.terminatedAt === null)
525 const across = (pick: (node: Node) => number | null | undefined): number | undefined => {
526 const values = alive.flatMap((n) => {
527 const value = pick(n)
528 return value === undefined || value === null ? [] : [value]
529 })
530 return values.length === 0 ? undefined : values.reduce((total, value) => total + value, 0) / values.length
531 }
532
533 const averages = new Map<string, number>()
534 for (const name of new Set([...latest.keys()].map((key) => key.slice(key.lastIndexOf('/') + 1)))) {
535 const value = across((n) => latest.get(metricKey(n.id, name)))
536 if (value !== undefined) averages.set(name, value)
537 }
538
539 // The window is the sparkline's own: the last bucket to reach the threshold is when the fleet last did work, and
540 // no bucket reaching it means the idleness began before the window did.
541 const window = Array.from({ length: SPARK_BUCKETS }, (_, i) => across((n) => sparks.get(metricKey(n.id, 'gpu_util'))?.[i]))
542 const busy = window.reduce<number>((found, value, i) => (value !== undefined && value >= IDLE_GPU_PCT ? i : found), -1)
543 const idle =
544 window.every((value) => value === undefined) || SPARK_BUCKETS - 1 - busy < IDLE_BUCKETS
545 ? null
546 : { since: start + (busy + 1) * SPARK_STEP_MS, capped: busy < 0 }
547
548 const fleet: Fleet = {
549 stages: NODE_STAGES.map((state) => [state, alive.filter((n) => n.state === state).length] as const).filter(([, count]) => count > 0),
550 averages,
551 running: tasks.counts.get('running') ?? 0,
552 idle,
553 }
554
555 return { kind: 'detail', detail: { compute, fleet, nodes, group: target.group, listed, page, pageCount, latest, sparks, tasks, log }, at: Date.now() }
556}
557
558// The reconciler's own writes also move the revision, so a refused If-Match is read again and sent again, under the same
559// key when the write carries one.
560async function writeCompute($: EngineInterface, id: string, init: HttpInit): Promise<void> {
561 for (let attempt = 1; ; attempt++) {
562 const revision = num(at(await getJson($, `/v1/computes/${id}`), 'revision'))
563 if (revision === undefined) throw new Error(`unexpected compute payload for ${id}`)
564 const response = await $.http.fetch(`${DAEMON}/v1/computes/${id}`, { ...init, headers: { ...init.headers, 'If-Match': `"${revision}"` } })
565 if (response.ok) return
566 if (response.status !== 412 || attempt === WRITE_ATTEMPTS) throw refusal(response)
567 }
568}
569
570const deleteCompute = ($: EngineInterface, id: string): Promise<void> =>
571 writeCompute($, id, { method: 'DELETE', headers: { 'Idempotency-Key': crypto.randomUUID() } })
572
573// `initial` goes back as it was: it is the size the pool opened at, which only its creation decides.
574const scaleCompute = ($: EngineInterface, c: Compute, min: number, max: number): Promise<void> =>
575 writeCompute($, c.id, {
576 method: 'PATCH',
577 headers: { 'Content-Type': 'application/json' },
578 body: JSON.stringify({ nodes: { initial: c.bounds.initial, min, max } }),
579 })
580
581async function drainNode($: EngineInterface, compute: string, node: string): Promise<void> {
582 const response = await $.http.fetch(`${DAEMON}/v1/computes/${compute}/nodes/${node}`, {
583 method: 'DELETE',
584 headers: { 'Idempotency-Key': crypto.randomUUID() },
585 })
586 if (!response.ok) throw refusal(response)
587}
588
589export const register: Register = (on) => {
590 let view: View = { kind: 'list' }
591 let snapshot: Snapshot = { kind: 'loading' }
592 let loading: View | undefined
593 let timers: Timer[] = []
594 let refresh: (() => void) | undefined
595 let writes: Writes | undefined
596 let focusKey: string | undefined
597 let notice: string | undefined
598 let scrolling = false
599 let action: Action | undefined
600
601 on('session.start', async ($, e, next) => {
602 await $.command.register({ name: 'sky', description: 'Skyward computes panel' })
603 return next(e)
604 })
605
606 on('command.run', { command: 'sky' }, async ($) => {
607 if (timers.length > 0) {
608 await $.ui.close({ id: PANE_ID })
609 return { text: 'Skyward panel hidden' }
610 }
611 await $.ui.open({ id: PANE_ID, title: 'Skyward' })
612 const tick = async () => {
613 const target = view
614 if (loading === target) return
615 loading = target
616 const result = await (target.kind === 'list' ? loadList($) : loadDetail($, target, snapshot.kind === 'detail' ? snapshot.detail.log : undefined))
617 .catch((error: unknown): Snapshot => ({ kind: 'down', reason: error instanceof Error ? error.message : String(error) }))
618 .finally(() => {
619 if (loading === target) loading = undefined
620 })
621 if (view !== target) return
622 snapshot = result
623 $.ui.invalidate('ui.render')
624 }
625 refresh = () => void tick()
626 writes = {
627 remove: (id) => deleteCompute($, id),
628 scale: (c, min, max) => scaleCompute($, c, min, max),
629 drain: (compute, node) => drainNode($, compute, node),
630 }
631 timers = [
632 $.clock.every(REFRESH_MS, refresh),
633 $.clock.every(MARQUEE_MS, () => {
634 if (scrolling) $.ui.invalidate('ui.render')
635 }),
636 ]
637 refresh()
638 return { text: 'Skyward panel shown' }
639 })
640
641 on('ui.close', { id: PANE_ID }, async ($, e, next) => {
642 const result = await next(e)
643 timers.forEach((timer) => timer.cancel())
644 timers = []
645 refresh = undefined
646 writes = undefined
647 view = { kind: 'list' }
648 snapshot = { kind: 'loading' }
649 action = undefined
650 return result
651 })
652
653 on('ui.render', { component: 'Pane', requestId: PANE_ID }, ($, e) => {
654 const { Box, Button, Text } = $.ui.resolve(e)
655 const now = Date.now()
656
657 const show = (next: View, focus: string) => {
658 view = next
659 snapshot = { kind: 'loading' }
660 focusKey = focus
661 notice = undefined
662 action = undefined
663 refresh?.()
664 $.ui.invalidate('ui.render')
665 }
666
667 // Keeps the current snapshot on screen until the page arrives, and both buttons drawn so the focus ring stays put.
668 const turnPage = (page: number, pageCount: number) => {
669 if (view.kind !== 'detail' || page < 0 || page >= pageCount) return
670 view = { ...view, page }
671 refresh?.()
672 }
673
674 const filterLog = (log: LogFilter, at?: { page: number; group: NodeGroup | null }) => {
675 if (view.kind !== 'detail') return
676 if (at !== undefined && (at.page !== view.page || at.group !== view.group)) focusKey = 'log-node'
677 view = { ...view, log, ...at }
678 notice = undefined
679 refresh?.()
680 $.ui.invalidate('ui.render')
681 }
682
683 // Like a page turn, the list changes when the next read arrives, and the buttons pressed stay drawn meanwhile.
684 const filterNodes = (group: NodeGroup) => {
685 if (view.kind !== 'detail') return
686 view = { ...view, group: view.group === group ? null : group, page: 0 }
687 refresh?.()
688 $.ui.invalidate('ui.render')
689 }
690
691 const updateTasks = (tasks: TaskQuery) => {
692 if (view.kind !== 'detail') return
693 view = { ...view, tasks }
694 refresh?.()
695 $.ui.invalidate('ui.render')
696 }
697
698 const toggleLogTask = (taskId: string) => {
699 if (view.kind !== 'detail') return
700 filterLog({ ...view.log, task: view.log.task === taskId ? null : taskId, cursors: [] })
701 }
702
703 const toggleLogNode = (nodeId: string) => {
704 if (view.kind !== 'detail') return
705 filterLog({ ...view.log, node: view.log.node === nodeId ? null : nodeId, cursors: [] })
706 }
707
708 const toggleNode = (nodeId: string) => {
709 if (view.kind !== 'detail') return
710 const expanded = new Set(view.expanded)
711 if (!expanded.delete(nodeId)) expanded.add(nodeId)
712 view = { ...view, expanded }
713 refresh?.()
714 $.ui.invalidate('ui.render')
715 }
716
717 const setAction = (next: Action | undefined, focus?: string) => {
718 action = next
719 if (focus !== undefined) focusKey = focus
720 $.ui.invalidate('ui.render')
721 }
722
723 // A deleted compute and a drained node lose their controls once the next read shows it, so those stay in
724 // `sending` until then; a scaled compute keeps its controls and is let go at once.
725 const perform = (sending: Action, focus: string, write: (daemon: Writes) => Promise<void>) => {
726 const daemon = writes
727 if (daemon === undefined) return
728 setAction(sending, focus)
729 write(daemon).then(
730 () => {
731 if (action === sending && sending.kind === 'scale') action = undefined
732 refresh?.()
733 },
734 (error: unknown) => {
735 if (action !== sending) return
736 action = { ...sending, step: 'failed', reason: error instanceof Error ? error.message : String(error) }
737 $.ui.invalidate('ui.render')
738 },
739 )
740 }
741
742 // The first press only asks, with the ring on cancel, so a second Enter does not delete.
743 const removeControls = (c: Compute): RenderElement | null => {
744 if (!deletable(c)) return null
745 const pending = action?.kind === 'delete' && action.id === c.id ? action : undefined
746 if (pending?.step === 'asking') {
747 return (
748 <Box flexDirection="row" flexWrap="wrap" columnGap={1}>
749 <Text color="red">delete this compute?</Text>
750 <Button key="delete-confirm" label="yes, delete" onPress={() => perform({ kind: 'delete', id: c.id, step: 'sending' }, 'back', (daemon) => daemon.remove(c.id))} />
751 <Button key="delete-cancel" label="cancel" onPress={() => setAction(undefined, 'delete')} />
752 </Box>
753 )
754 }
755 if (pending?.step === 'sending') return <Text dimColor>deleting…</Text>
756 return <Button key="delete" label="delete" onPress={() => setAction({ kind: 'delete', id: c.id, step: 'asking' }, 'delete-cancel')} />
757 }
758
759 // Reads the drafts from `action` as it is now, not as it was drawn: two fields typed into between draws would
760 // otherwise each write back the other's stale text.
761 const draftScale = (typed: { min?: string; max?: string }) => {
762 if (action?.kind !== 'scale' || action.step !== 'asking') return
763 action = { ...action, ...typed }
764 }
765
766 const applyScale = (c: Compute, typed: { min?: string; max?: string }) => {
767 if (action?.kind !== 'scale' || action.id !== c.id || action.step !== 'asking') return
768 const asked = { ...action, ...typed }
769 const min = count(asked.min, c.bounds.min)
770 const max = count(asked.max, c.bounds.max)
771 if (min === null || max === null || min > max) return setAction({ ...asked, invalid: true })
772 if (min === c.bounds.min && max === c.bounds.max) return setAction(undefined, 'scale')
773 perform({ ...asked, step: 'sending' }, 'back', (daemon) => daemon.scale(c, min, max))
774 }
775
776 // Mobile has no Input, so a compute is scaled there from the CLI or the web console. A field empties on Enter, so
777 // one holding text that was refused shows it back as its placeholder.
778 const scaleControls = (c: Compute): RenderElement | null => {
779 if (!deletable(c) || e.surface === 'mobile') return null
780 const pending = action?.kind === 'scale' && action.id === c.id ? action : undefined
781 if (pending?.step === 'sending') return <Text dimColor>scaling…</Text>
782 if (pending?.step !== 'asking') {
783 return <Button key="scale" label="scale" onPress={() => setAction({ kind: 'scale', id: c.id, min: '', max: '', invalid: false, step: 'asking' }, 'scale-min')} />
784 }
785 const { Input } = $.ui.resolve(e)
786 return (
787 <Box flexDirection="row" flexWrap="wrap" columnGap={1}>
788 <Input key="scale-min" label="min" placeholder={pending.min === '' ? String(c.bounds.min) : pending.min} submitLabel="scale" onInput={(text) => draftScale({ min: text })} onSubmit={(text) => applyScale(c, { min: text })} />
789 <Input key="scale-max" label="max" placeholder={pending.max === '' ? String(c.bounds.max) : pending.max} submitLabel="scale" onInput={(text) => draftScale({ max: text })} onSubmit={(text) => applyScale(c, { max: text })} />
790 <Button key="scale-apply" label="apply" onPress={() => applyScale(c, {})} />
791 <Button key="scale-cancel" label="cancel" onPress={() => setAction(undefined, 'scale')} />
792 {pending.invalid ? <Text color="red">whole numbers, min ≤ max</Text> : null}
793 </Box>
794 )
795 }
796
797 scrolling = false
798 const columns = fitColumns(e.props.bodyColumns)
799 const tableWidth = columns.reduce((width, column) => width + TABLE_GAP + column.width, NAME_WIDTH)
800
801 const cell = (column: Column, text: string, props: { color?: string; dim?: boolean; bold?: boolean }): RenderElement => (
802 <Box width={column.width} flexShrink={0}>
803 <Text wrap="truncate" color={props.color} dimColor={props.dim} bold={props.bold}>
804 {column.right ? text.padStart(column.width) : text}
805 </Text>
806 </Box>
807 )
808
809 const tableRow = (name: RenderElement, cells: RenderElement[]): RenderElement => (
810 <Box flexDirection="row" gap={TABLE_GAP}>
811 <Box width={NAME_WIDTH} flexShrink={0}>{name}</Box>
812 {cells}
813 </Box>
814 )
815
816 const computeRow = (c: Compute): RenderElement => {
817 const name = c.name ?? c.id
818 if (name.length > NAME_WIDTH) scrolling = true
819 return (
820 <Box flexDirection="column">
821 {tableRow(
822 <Button key={c.id} plain label={marquee(name, NAME_WIDTH, now)} dimColor={c.ended} onPress={() => show({ kind: 'detail', id: c.id, page: 0, group: null, expanded: new Set(), tasks: { order: 'state', fn: null, cursors: [] }, log: { node: null, task: null, term: '', cursors: [] } }, 'back')} />,
823 columns.map((column) => {
824 const { text, color } = column.cell(c)
825 return cell(column, text, { color: c.ended ? undefined : color, dim: c.ended })
826 }),
827 )}
828 {c.lastError !== null && !c.ended ? <Text color="red" wrap="truncate">{` ${c.lastError}`}</Text> : null}
829 </Box>
830 )
831 }
832
833 const rule = (): RenderElement => <Text dimColor>{'─'.repeat(tableWidth)}</Text>
834
835 const computeTable = (live: Compute[], recent: Compute[]): RenderElement => (
836 <Box flexDirection="column">
837 {tableRow(<Text dimColor>NAME</Text>, columns.map((column) => cell(column, column.title, { dim: true })))}
838 {rule()}
839 {live.length === 0 ? <Text dimColor>no live computes</Text> : live.map(computeRow)}
840 {live.length === 0
841 ? null
842 : [
843 rule(),
844 tableRow(
845 <Text dimColor>total</Text>,
846 columns.map((column) => cell(column, column.total?.(live) ?? '', { bold: true })),
847 ),
848 ]}
849 {recent.length === 0 ? null : [<Text> </Text>, tableRow(<Text dimColor>recent</Text>, []), ...recent.map(computeRow)]}
850 </Box>
851 )
852
853 const computeHeader = (c: Compute, f: Fleet): RenderElement => {
854 const means = averaged(f)
855 const quiet = f.idle === null ? null : `${f.idle.capped ? '≥' : ''}${age(now - f.idle.since)}`
856 const tasks = f.running === 0 ? 'no task' : `${f.running} running`
857 // A compute whose gpu is working, or that has no gpu to read, is only called idle by its tasks.
858 const idleness =
859 c.ended || (quiet === null && (f.running > 0 || f.averages.has('gpu_util')))
860 ? null
861 : quiet === null
862 ? { text: `idle · ${tasks}`, color: 'yellow' }
863 : f.running > 0
864 ? { text: `gpu idle ${quiet} · ${tasks}`, color: 'red' }
865 : { text: `idle ${quiet} · ${tasks}`, color: 'yellow' }
866 return (
867 <Box flexDirection="column">
868 <Box flexDirection="row" gap={2}>
869 <Text bold wrap="truncate">{c.name ?? c.id}</Text>
870 <Text color={STATE_COLORS.get(c.state)}>{`${STATE_GLYPHS.get(c.state) ?? '●'} ${c.state}`}</Text>
871 </Box>
872 <Text dimColor wrap="truncate">
873 {[
874 f.stages.length === 0 ? `${c.nodesReady}/${c.nodesTotal} nodes` : f.stages.map(([state, count]) => `${count} ${state}`).join(', '),
875 c.accelerator === null ? c.provider : `${c.provider} ${gpu(c)}`,
876 c.ended ? `cost $${money(c.cost)}` : `$${money(c.cost)} spent`,
877 c.ended ? null : `$${money(hourly(c))}/h`,
878 age(now - c.createdAt),
879 ]
880 .filter((part) => part !== null)
881 .join(' · ')}
882 </Text>
883 {means === '' ? null : <Text dimColor wrap="truncate">{`avg ${means}`}</Text>}
884 {idleness === null ? null : <Text color={idleness.color} wrap="truncate">{idleness.text}</Text>}
885 {c.lastError !== null ? <Text color="red" wrap="truncate">{c.lastError}</Text> : null}
886 </Box>
887 )
888 }
889
890 const rankBadge = (rank: number | undefined, muted = false): RenderElement =>
891 rank === undefined || muted ? (
892 <Text dimColor>{rank === undefined ? ' #? ' : ` #${rank} `}</Text>
893 ) : (
894 <Text bold color="#ffffff" backgroundColor={RANK_COLORS[rank % RANK_COLORS.length]}>{` #${rank} `}</Text>
895 )
896
897 const nodeBlock = (n: Node, d: Detail, sparkStart: number, open: boolean, logging: boolean): RenderElement => {
898 const terminated = n.terminatedAt !== null
899 const metric = (name: string) => d.latest.get(metricKey(n.id, name))
900 const both = (a: number | undefined, b: number | undefined, f: (a: number, b: number) => string) =>
901 a === undefined || b === undefined ? null : f(a, b)
902 const one = (a: number | undefined, f: (a: number) => string) => (a === undefined ? null : f(a))
903
904 const stats = (parts: [string, string | null][], indent = 4): RenderElement => (
905 <Box flexDirection="row" flexWrap="wrap" columnGap={2} paddingLeft={indent}>
906 {parts.map(([label, value]) =>
907 value === null ? null : (
908 <Box flexShrink={0}>
909 <Text><Text dimColor>{`${label} `}</Text>{value}</Text>
910 </Box>
911 ),
912 )}
913 </Box>
914 )
915
916 const trend = (label: string, name: string, parts: [string, string | null][]): RenderElement | null =>
917 metric(name) === undefined ? null : (
918 <Box flexDirection="row" gap={2}>
919 <Box flexShrink={0}>
920 <Text>{` ${label} ${spark((d.sparks.get(metricKey(n.id, name)) ?? NO_SAMPLES).slice(sparkStart))}`}</Text>
921 </Box>
922 <Box flexShrink={1}>{stats(parts, 0)}</Box>
923 </Box>
924 )
925
926 const life =
927 n.launchedAt === null ? null : n.terminatedAt === null ? `up ${age(now - n.launchedAt)}` : `ran ${age(n.terminatedAt - n.launchedAt)}`
928 const tail = [life, n.price === null ? null : `@ ${price(n.price)}`, n.market === null ? null : `(${n.market.replace('_', ' ')})`]
929 .filter((part) => part !== null)
930 .join(' ')
931
932 const killing = action?.kind === 'kill' && action.id === n.id ? action : undefined
933 const kill = terminated || !DRAINABLE.has(n.state) ? null : killing?.step === 'sending' ? (
934 <Text dimColor>killing…</Text>
935 ) : (
936 <Button key={`kill:${n.id}`} plain label="kill" dimColor={killing?.step !== 'asking'} onPress={() => setAction({ kind: 'kill', id: n.id, step: 'asking' }, `kill-cancel:${n.id}`)} />
937 )
938
939 return (
940 <Box flexDirection="column">
941 <Box flexDirection="row" gap={1}>
942 <Button key={`node:${n.id}`} plain label={open ? '▾' : '▸'} onPress={() => toggleNode(n.id)} />
943 <Box flexShrink={0}>{rankBadge(n.rank, terminated)}</Box>
944 <Text>
945 <Text dimColor>is </Text>
946 <Text color={terminated ? undefined : STATE_COLORS.get(n.state)} dimColor={terminated}>{n.state}</Text>
947 <Text dimColor>{tail === '' ? '' : ` ${tail}`}</Text>
948 </Text>
949 <Box flexShrink={0} paddingLeft={1}>
950 <Button key={`logs:${n.id}`} plain label="logs" dimColor={!logging} onPress={() => toggleLogNode(n.id)} />
951 </Box>
952 {kill === null ? null : <Box flexShrink={0} paddingLeft={1}>{kill}</Box>}
953 </Box>
954 {killing?.step === 'asking' ? (
955 <Box flexDirection="row" flexWrap="wrap" columnGap={1} paddingLeft={4}>
956 <Text color="red">{`kill #${n.rank}?`}</Text>
957 <Button key={`kill-confirm:${n.id}`} label="yes, kill" onPress={() => perform({ kind: 'kill', id: n.id, step: 'sending' }, `logs:${n.id}`, (daemon) => daemon.drain(d.compute.id, n.id))} />
958 <Button key={`kill-cancel:${n.id}`} label="cancel" onPress={() => setAction(undefined, `kill:${n.id}`)} />
959 <Text dimColor>it drains first, and a new node takes its rank if the compute still wants it</Text>
960 </Box>
961 ) : null}
962 {killing?.step === 'failed' ? <Text color="red">{` kill failed: ${killing.reason}`}</Text> : null}
963 {n.lastError !== null ? <Text color="red">{` ${n.lastError}`}</Text> : null}
964 {terminated
965 ? null
966 : stats([
967 ['cpu', one(metric('cpu'), (v) => `${Math.round(v)}%`)],
968 ['gpu', one(metric('gpu_util'), (v) => `${Math.round(v)}%`)],
969 ['memory', both(metric('mem_used_mb'), metric('mem_total_mb'), (used, total) => `${gib(used)}/${gib(total)} GB`)],
970 ['disk', one(metric('disk_used_pct'), (v) => `${Math.round(v)}%`)],
971 ])}
972 {open && !terminated
973 ? [
974 trend('cpu', 'cpu', [['net', both(metric('net_rx_kbps'), metric('net_tx_kbps'), (rx, tx) => `↓${rate(rx)} ↑${rate(tx)}`)]]),
975 trend('gpu', 'gpu_util', [
976 ['vram', both(metric('gpu_mem_mb'), metric('gpu_mem_total_mb'), (used, total) => `${gib(used)}/${gib(total)} GB`)],
977 ['temp', one(metric('gpu_temp_c'), (v) => `${Math.round(v)}°C`)],
978 ['power', one(metric('gpu_power_w'), (v) => `${Math.round(v)} W`)],
979 ]),
980 ]
981 : null}
982 {open ? (
983 <Box paddingLeft={4}>
984 <Text dimColor>{[n.address, n.id, n.machine === null ? null : `machine ${n.machine}`].filter((part) => part !== null).join(' · ')}</Text>
985 </Box>
986 ) : null}
987 </Box>
988 )
989 }
990
991 const body = (): RenderElement => {
992 switch (snapshot.kind) {
993 case 'loading':
994 return <Text dimColor>loading…</Text>
995 case 'down':
996 return (
997 <Box flexDirection="column">
998 <Text color="red" wrap="truncate">{`daemon request failed at ${DAEMON}: ${snapshot.reason}`}</Text>
999 <Text dimColor>sky server start</Text>
1000 </Box>
1001 )
1002 case 'list':
1003 return (
1004 <Box flexDirection="column" gap={1}>
1005 <Text dimColor>{`${DAEMON} · updated ${clock(snapshot.at)}`}</Text>
1006 {computeTable(snapshot.live, snapshot.recent)}
1007 </Box>
1008 )
1009 case 'detail': {
1010 const d = snapshot.detail
1011 const ranks = new Map(d.nodes.map((n) => [n.id, n.rank]))
1012 const first = d.page * NODES_PER_PAGE
1013 const pageNodes = d.listed.slice(first, first + NODES_PER_PAGE)
1014 const counts = nodeCounts(d.nodes, d.group)
1015 // Columns before any node on the page had a sample are blank for all of them, so they are dropped.
1016 const onPage = new Set(pageNodes.map((n) => n.id))
1017 const sparkStart = Math.min(
1018 SPARK_BUCKETS,
1019 ...[...d.sparks]
1020 .filter(([key]) => onPage.has(metricNode(key)))
1021 .map(([, buckets]) => buckets.findIndex((value) => value !== null))
1022 .filter((i) => i >= 0),
1023 )
1024 return (
1025 <Box flexDirection="column" gap={1}>
1026 <Box flexDirection="column">
1027 {computeHeader(d.compute, d.fleet)}
1028 <Text dimColor>{`${d.compute.id} · updated ${clock(snapshot.at)}`}</Text>
1029 </Box>
1030 <Box flexDirection="column">
1031 <Box flexDirection="row" gap={2}>
1032 {counts.length === 0 ? (
1033 <Text bold>Nodes (0)</Text>
1034 ) : (
1035 <Box flexDirection="row" flexWrap="wrap">
1036 <Text bold>Nodes (</Text>
1037 {counts.map(([group, count], i) => [
1038 i === 0 ? null : <Text bold> · </Text>,
1039 <Button key={`nodes-group:${group}`} plain label={`${count} ${group}${d.group === group ? ' ✕' : ''}`} dimColor={d.group !== null && d.group !== group} onPress={() => filterNodes(group)} />,
1040 ])}
1041 <Text bold>)</Text>
1042 </Box>
1043 )}
1044 {d.pageCount > 1 ? (
1045 <Box flexDirection="row" gap={1}>
1046 <Button key="nodes-prev" plain label="‹ prev" dimColor={d.page === 0} onPress={() => turnPage(d.page - 1, d.pageCount)} />
1047 <Text dimColor>{`${first + 1}–${first + pageNodes.length} of ${d.listed.length}`}</Text>
1048 <Button key="nodes-next" plain label="next ›" dimColor={d.page === d.pageCount - 1} onPress={() => turnPage(d.page + 1, d.pageCount)} />
1049 </Box>
1050 ) : null}
1051 </Box>
1052 {d.listed.length === 0 ? <Text dimColor>{d.group === null ? 'no nodes' : `no ${d.group} nodes`}</Text> : pageNodes.map((n) => nodeBlock(n, d, sparkStart, view.kind === 'detail' && view.expanded.has(n.id), view.kind === 'detail' && view.log.node === n.id))}
1053 </Box>
1054 {view.kind === 'detail' ? tasksSection(d, view, ranks) : null}
1055 {view.kind === 'detail' ? logSection(d, view, ranks) : null}
1056 </Box>
1057 )
1058 }
1059 }
1060 }
1061
1062 const taskRow = (t: Task, query: TaskQuery, ranks: Map<string, number>, logging: boolean): RenderElement => {
1063 const state = t.state.replace('_', ' ')
1064 const took = t.startedAt === null ? null : age((t.finishedAt ?? now) - t.startedAt)
1065 const status =
1066 t.state === 'queued'
1067 ? [`queued ${age(now - t.submittedAt)}`]
1068 : t.finishedAt === null
1069 ? [took === null ? state : `${state} ${took}`]
1070 : [took === null ? state : `${state} ${t.state === 'succeeded' ? 'in' : 'after'} ${took}`, `${age(now - t.finishedAt)} ago`]
1071 const name = t.name
1072 return (
1073 <Box flexDirection="column">
1074 <Box flexDirection="row" gap={1}>
1075 <Box width={1} flexShrink={0}>
1076 <Text color={TASK_COLORS.get(t.state)} dimColor={!TASK_COLORS.has(t.state)}>{TASK_GLYPHS.get(t.state) ?? '·'}</Text>
1077 </Box>
1078 <Box width={RANK_WIDTH} flexShrink={0}>{t.node === null ? null : rankBadge(ranks.get(t.node))}</Box>
1079 <Box flexShrink={0}>
1080 {name === null ? (
1081 <Text dimColor>{t.sha.slice(0, 12)}</Text>
1082 ) : (
1083 // The filtered page may not hold this row, so the ring is handed to the sort button, which every page draws.
1084 <Button key={`task-fn:${t.id}`} plain label={name} onPress={() => {
1085 focusKey = 'tasks-sort'
1086 updateTasks({ ...query, fn: query.fn === name ? null : name, cursors: [] })
1087 }} />
1088 )}
1089 </Box>
1090 <Text dimColor>{[...status, t.attempts > 1 ? `${t.attempts} attempts` : null].filter((part) => part !== null).join(' · ')}</Text>
1091 {t.state === 'queued' ? null : (
1092 <Box flexShrink={0} paddingLeft={1}>
1093 <Button key={`task-logs:${t.id}`} plain label="logs" dimColor={!logging} onPress={() => toggleLogTask(t.id)} />
1094 </Box>
1095 )}
1096 </Box>
1097 {t.error === null ? null : (
1098 <Box paddingLeft={RANK_WIDTH + 3}>
1099 <Text color="red" wrap="wrap">{t.error}</Text>
1100 </Box>
1101 )}
1102 </Box>
1103 )
1104 }
1105
1106 const tasksSection = (d: Detail, target: DetailView, ranks: Map<string, number>): RenderElement => {
1107 const query = target.tasks
1108 const page = d.tasks
1109 const first = query.cursors.length * TASKS_PER_PAGE
1110 const count = (state: string) => page.counts.get(state) ?? 0
1111 const other = (page.total ?? 0) - COUNTED_TASK_STATES.reduce((sum, state) => sum + count(state), 0)
1112 const summary = [...COUNTED_TASK_STATES.map((state) => `${count(state)} ${state}`), other > 0 ? `${other} other` : null]
1113 .filter((part) => part !== null)
1114 .join(' · ')
1115 const next = page.next
1116 const order = NEXT_TASK_ORDER[query.order]
1117 const functionInput = (): RenderElement | null => {
1118 if (e.surface === 'mobile') return null
1119 const { Input } = $.ui.resolve(e)
1120 return (
1121 <Input key="tasks-function" label="function" placeholder={query.fn ?? 'all'} submitLabel="filter" onSubmit={(value) => updateTasks({ ...query, fn: value.trim() === '' ? null : value.trim(), cursors: [] })} />
1122 )
1123 }
1124 return (
1125 <Box flexDirection="column">
1126 <Box flexDirection="row" flexWrap="wrap" columnGap={2}>
1127 <Text bold>{`Tasks (${page.total ?? '?'})`}</Text>
1128 <Text dimColor>{summary}</Text>
1129 </Box>
1130 <Box flexDirection="row" flexWrap="wrap" columnGap={2}>
1131 <Button key="tasks-sort" plain label={`sort: ${query.order} ›`} onPress={() => updateTasks({ ...query, order, cursors: [] })} />
1132 {functionInput()}
1133 <Box flexDirection="row" gap={1}>
1134 <Button key="tasks-prev" plain label="‹ prev" dimColor={query.cursors.length === 0} onPress={() => updateTasks({ ...query, cursors: query.cursors.slice(0, -1) })} />
1135 <Text dimColor>{`${page.items.length === 0 ? first : `${first + 1}–${first + page.items.length}`} of ${page.total ?? '?'}`}</Text>
1136 <Button key="tasks-next" plain label="next ›" dimColor={next === null} onPress={() => {
1137 if (next !== null) updateTasks({ ...query, cursors: [...query.cursors, next] })
1138 }} />
1139 </Box>
1140 </Box>
1141 {page.items.length === 0 ? <Text dimColor>{query.fn === null ? 'no tasks' : `no ${query.fn} tasks`}</Text> : null}
1142 {page.items.map((t) => taskRow(t, query, ranks, target.log.task === t.id))}
1143 </Box>
1144 )
1145 }
1146
1147 const logSection = (d: Detail, target: DetailView, ranks: Map<string, number>): RenderElement => {
1148 const filter = target.log
1149 const newest = filter.cursors.length === 0
1150 const next = d.log.next
1151 const pickNode = (value: string) => {
1152 const text = value.trim().replace(/^#/, '')
1153 if (text === '') return filterLog({ ...filter, node: null, cursors: [] })
1154 // A replaced node leaves its rank to the next one, so a rank can name several machines: the live one wins, then the latest.
1155 const matches = d.nodes.filter((n) => n.rank === Number(text))
1156 const node = matches.find((n) => n.terminatedAt === null) ?? matches.sort((a, b) => (b.launchedAt ?? 0) - (a.launchedAt ?? 0))[0]
1157 if (node === undefined) {
1158 notice = `no node #${text}`
1159 $.ui.invalidate('ui.render')
1160 return
1161 }
1162 // A node the state filter hides is shown by dropping the filter.
1163 const group = target.group === null || nodeGroup(node) === target.group ? target.group : null
1164 filterLog({ ...filter, node: node.id, cursors: [] }, { page: Math.floor(inGroup(d.nodes, group).indexOf(node) / NODES_PER_PAGE), group })
1165 }
1166 // Mobile has no Input; there each node's logs button is the node filter. A field empties on Enter and is only
1167 // refilled by a value that differs from the last one drawn, so the filter in force is shown as the placeholder.
1168 const inputs = (): RenderElement[] => {
1169 if (e.surface === 'mobile') return []
1170 const { Input } = $.ui.resolve(e)
1171 return [
1172 <Input key="log-node" label="node" placeholder={filter.node === null ? 'all' : `#${ranks.get(filter.node) ?? '?'}`} submitLabel="go" onSubmit={pickNode} />,
1173 <Input key="log-term" label="filter" placeholder={filter.term === '' ? 'term' : filter.term} submitLabel="search" onSubmit={(value) => filterLog({ ...filter, term: value.trim(), cursors: [] })} />,
1174 ]
1175 }
1176 return (
1177 <Box flexDirection="column">
1178 <Box flexDirection="row" flexWrap="wrap" columnGap={2}>
1179 <Text bold>Log</Text>
1180 {inputs()}
1181 {filter.task === null ? null : (
1182 <Button key="log-task" plain label={`task ${filter.task} ✕`} onPress={() => {
1183 focusKey = 'log-older'
1184 filterLog({ ...filter, task: null, cursors: [] })
1185 }} />
1186 )}
1187 {notice === undefined ? null : <Text color="red">{notice}</Text>}
1188 </Box>
1189 <Box flexDirection="row" gap={1}>
1190 <Button key="log-older" plain label="‹ older" dimColor={next === null} onPress={() => {
1191 if (next !== null) filterLog({ ...filter, cursors: [...filter.cursors, next] })
1192 }} />
1193 <Text dimColor>{newest ? 'live' : `page ${filter.cursors.length + 1}`}</Text>
1194 <Button key="log-newer" plain label="newer ›" dimColor={newest} onPress={() => filterLog({ ...filter, cursors: filter.cursors.slice(0, -1) })} />
1195 <Button key="log-latest" plain label="latest »" dimColor={newest} onPress={() => filterLog({ ...filter, cursors: [] })} />
1196 </Box>
1197 {d.log.lines.length === 0 ? <Text dimColor>{filter.node === null && filter.task === null && filter.term === '' ? 'no events' : 'no matching events'}</Text> : null}
1198 {d.log.lines.map((l) => (
1199 <Box flexDirection="row" gap={1}>
1200 <Box width={8} flexShrink={0}><Text dimColor>{clock(l.at)}</Text></Box>