Agent requests and recovery

Markdown · Agent index

Yapwing projects are directories containing project.json and optional collected assets. The JSON is data. Use the installed yapwing CLI or the TypeScript SDK to inspect, validate and edit it. Both routes use the same Rust commands and native media pipeline as the app. To set up the command, see Use a coding agent.

Inspect, check, apply

yapwing inspect /absolute/project
yapwing check /absolute/project --request /absolute/request.json
yapwing apply /absolute/project --request /absolute/request.json

Check/apply also accept –request - to read stdin. Requests are bounded to 32 MiB. A request contains projectId, expectedRevision, operationId and commands. An optional undoGroup string joins consecutive authored requests with the same value into one undo step; each request still saves and advances the revision. Any other committed request, including undo, redo and metadata, ends the group; a rejected request leaves it unchanged. An authored request that changes nothing is accepted at the same revision, adds no undo step and leaves the group as it was. An optional actionName (1–40 characters, no control characters or edge spaces) names the request’s undo step in place of the default from its first command, so Undo and Redo read, for example, “Replace All”. Copy the ID and revision from inspect; choose a new stable operation ID for each edit. The wire contract documents every command. The usual limit is 128 commands. A larger batch may contain only relink_asset commands for distinct assets, up to the project’s 10,000-asset limit, so Collect All can commit its verified locators atomically. All other batches retain the 128-command limit; the 32 MiB bound applies to every request.

Check validates a fork of the current core, including its undo/redo state. It does not commit the proposed edit or adopt a changed external project file. Apply validates and commits the whole command batch as one undo step. Metadata commands preserve authored undo/redo and cannot share a batch with edits.

When a request’s result is uncertain, retry the exact request with the same operation ID. The store binds durable receipts to the project and payload. A changed payload using that ID fails. A new request with an old revision fails. Do not respond to a revision conflict by blindly changing expectedRevision: inspect the current state and reconsider the intended edit.

The app serves a private local Unix socket with owner-only permissions and a same-user peer check. It retains an advisory project owner lock through shutdown. A CLI finding that lock routes to the app; a failed connection does not fall back to an independent writer. Closed-project commands hold the same lock. Multiple live callers still compete through revision-checked core transactions.

Successful responses have ok:true and value. Failures have ok:false and error with code/message. Inspect’s value contains snapshot and statuses; check returns a proposed Snapshot; apply returns snapshot, receipt and replayed.

View and edit

yapwing view prints the project as a .yapwing.ts file: one clip() block per clip, one say() line per spoken sentence, and text() lines for on-screen text. It is TypeScript syntax so editors and coding agents read it well, but Yapwing only parses it as data and never runs it. Template literals with ${}, spreads, functions and any other code are refused.

yapwing view /absolute/project > /absolute/edit.yapwing.ts
yapwing view /absolute/project --outline
yapwing edit /absolute/project --file /absolute/edit.yapwing.ts [--dry-run]
yapwing edit /absolute/project --ops --file /absolute/ops.ts [--dry-run]

edit diffs the file against the revision it names, turns the difference into ordinary wire commands, and sends them as one apply request, so it reaches the app when the project is open and the edit is one undo step, named “Agent Edit” in the Undo and Redo menu items. While the project is open in the app, what an applied request added or changed is marked there for the person (words and cut chips in the transcript, clips and overlays on the timeline) until they edit that item, undo the request or choose View → Clear Agent Marks. Marks are view state in the window only: they never enter the project, the reply or inspect, and an undo or redo request clears the marks of what it steps back instead of adding any. Each step is validated against a private fork of the core first; if any step fails, nothing is applied and the reply lists the failing step. A file whose revision is no longer current is refused as stale. The default operation ID is derived from the revision and the file contents, so retrying the same edit is idempotent. --ops takes operation calls such as cut_fillers() or fix("s6", "flower", "flour") instead of a whole file; the operations are listed in the skill. The Yapwing skill documents the file rules for coding agents. In the app, Help → Use with Coding Agents… copies it to ~/.claude/skills/yapwing, a project’s .claude/skills/yapwing or ~/.agents/skills/yapwing (Codex), records its version in .yapwing-skill.json, offers an update when the app ships a newer one, and backs up a changed copy to yapwing-skill-backups before replacing it. It can also link ~/.local/bin/yapwing to the app’s CLI, without sudo.

The view names the output format in its first line, such as portrait 9:16 (1080x1920); the outline names it only when it is not portrait. Change it with the wire command set_output (preset: portrait, vertical45, square or landscape), one undo step named “Format”; the app’s View → Format menu and toolbar pop-up send the same command. The agent op format(preset) sends that one command as an agent edit (one undo step), so the chat and yapwing edit --ops can change it too. It takes the four names and the words people use for them ("9:16"/"vertical" portrait, "4:5", "1:1", "16:9"/"youtube" landscape, or a menu title such as "Square 1:1"); an unknown name is refused naming the four, and the current format is a no-op. A clip’s transform.reframe: "subject" follows its main subject; see the wire contract. The agent op reframe(clip, on) turns it on or off for a whole video clip in one undo step, keeping the rest of the clip’s framing; the outline marks such clips “follows subject”. Photos, 360° clips and text are refused. The app finds the subject on this Mac, and export analyzes followed clips first.

In the outline, [12.5] is when a sentence starts, in seconds of the edited video, and [cut] marks a spoken sentence already cut. A sentence that holds only a pause, such as the picture of a clip without speech, shows the time its pause starts. It is in the video, so the view and the outline do not mark it as cut or count it as cut.

Hosts that have measured a source on this Mac may pass signals with an agent call ({"method": "outline", "signals": {"assets": {ASSET_ID: {"silences": [Range, ...], "shots": [Range, ...], "quality": [Quality, ...]}}}}). Every member is optional. Ranges are in source time, ordered and nonoverlapping. quality[i] describes shots[i], so when present it has one entry per shot: {"sharpness", "faceQuality"?, "eyesOpen"?, "stability"}, each from 0 to 1 with higher better, and the face scores are absent for a shot without a face. The outline then lists the longest silent stretches that play and, on a “Shots” line, the shots that play in timeline order (the first 20, then how many more) with their scores, by clip handle; clips whose picture is off have no shots. The planner can read signals in that call. Signals are never saved in the project or in the compiled request. Callers that omit them see exactly the same outline.

The library measures every source’s sound and every video’s shots on this Mac, and the agent session’s signals method ({"method": "signals"}, no params) returns them in exactly that shape for sources whose bytes are still the ones measured; it is empty until analysis has finished. yapwing edit and yapwing view --outline fetch and pass them for you, and the app’s chat passes them with every call. remove_dead_air(pad?) uses them: it removes measured silence from the pauses between words, keeping pad seconds (default 0.1, at most 1) of quiet next to speech, as one undo step that moves pause edges like tighten_pauses. A pause keeps one stretch, so it can lose silence that reaches a word without losing any sound. When a silence has sound on both sides, the pause drops the sound between one word and that silence (the shorter side) only if it is at most 0.15 s, and otherwise is left alone. It never cuts inside a word, skips clips whose sound is off, only ever shortens a pause, and repeating it changes nothing. Without measurements for the reel’s clips it fails and says so.

fit_length(seconds, keep?) shortens the whole video to at most seconds, landing as close under it as practical, as one undo step. Candidates come in priority order, and a tier is used only once the earlier ones are used up: measured dead air (as remove_dead_air, skipped when nothing is measured), filler words, earlier takes of a retaken sentence (a sentence whose words nearly all recur in, or that starts, one of the next two sentences, unless each of the two says a content word the other doesn’t), pauses over 0.4 s shortened toward 0.4 s, then whole sentences by least content (the sum over a sentence’s distinct words of each word’s weight over how often the video says it, where a content word weighs 1 and a function word or greeting such as “the”, “yeah” or “hi” weighs 0.1, so short filler-ish sentences and repeated words go first and a long sentence of distinct content words goes last). Within the dead air, filler and pause tiers it takes the smallest candidate that alone reaches the target, else the largest that keeps the video above it, so one big trim never lands far under the target: the pause that reaches is shortened only as far as the target needs, and a dead air trim or filler that would land more than the close band (below) under the target is left for the later tiers. Pauses and dead air in a clip without speech, such as a silent screen recording, go only after all the others, so that clip keeps its picture unless the target needs it. The two whole-sentence tiers choose a set, so one long sentence or a whole clip is not cut when shorter ones land nearer the target. Close means within half a second under the target, or 5% of targets over 10 s. Earlier takes are cut as a set that lands close, else as many as keep the video above the target, leaving the rest to the later tiers. Whole sentences are cut as the set that loses the least content (then the fewest) of those that land close, so several short lines such as “Yeah.” and “It’s really funny.” go before one long sentence that says what the video is about, else as the set that removes the least beyond the target. The video’s last sentence counts as more than all the others together, so it is cut only when no set of the other sentences lands close: the cut comes from before the closing line and the video does not end mid-thought. The opening sentence has no such hold; a greeting is among the first lines to go (on a long reel cut by minutes, the least-content sentences go first and only the last few minutes of the cut are fitted this way). It measures the real length after each step. Sentences in keep (handles, ranges or lists) are never cut whole, though fillers and pauses inside them can go, and it never cuts every sentence: when nothing in keep is left, at least one sentence stays (the video’s final one unless only other sentences land close; name it in keep to hold it whatever the target). A video already short enough is left alone. An unreachable target changes nothing and the error names the shortest reachable length; on success the message gives the new length and each tier’s cuts.

make_rough_reel(sources?) builds a first cut from library videos as one undo step. It adds the sources (asset ids, clip ids or video names) to the end of the video in the order given, so pass them in recording order (yapwing library lists each video’s recording date); without sources it uses every library video (not pictures) not yet in the reel, in library order. Each video is trimmed to its speech with 0.15 s to spare. Across all of them, a line said again (token Jaccard of at least 0.6 with the line before it, or a false start the next line completes) keeps only its last complete take. Parallel lines are not retakes: lines that differ only in a counted word (“step one” and “step two”), and lines where each says a content word the other doesn’t (“Add the flour to the bowl.” and “Add the sugar to the bowl.”). A corrected word (“make bread” then “bake bread”: one letter apart, or the same stem), an added or dropped word, or changed small words (“I’m going to” then “I’ll”) still make a take. Filler words in the new clips are cut, and so is their measured dead air (as remove_dead_air; skipped when nothing is measured). A video whose every line is said again later is not added at all, and neither is a video not yet in the reel whose every line is a take of a line the reel already says, so running it again does not add, on its own, a video an earlier rough reel left out as earlier takes (add such a video by hand to use it whole). Only whole videos are left out this way: a line in a new video that repeats one already in the reel stays when the video says something new too. Clips already in the reel are left alone. The message names what it kept and quotes each dropped take with its video. Pictures, 360° videos, camera originals without their prepared video and videos without transcribed speech are left out and named; when none of the sources has speech nothing changes.

Preview renders

yapwing preview /absolute/project --range s3-s5 [--frames 4] [--audio] [--expected-revision N]
yapwing preview /absolute/project --range 12-18.5

preview lets an agent see and hear what it just edited. It renders the current render plan through the same composition and audio mix as export: --frames N (1–8, default 4) JPEG stills at the output aspect, 540 px wide, taken from the middle of N equal parts of the range, and with --audio an AAC snippet (48 kHz stereo, at most 30 s from the range start, levelled as export levels the whole reel: the same gain and, when export limits, the same look-ahead limiter). --range takes any sentence target the operations accept (s3, s3-s5, last2, from60, stable sentence-… ids) or timeline seconds START-END (12-18.5 or 0:12-0:18.5); a sentence range spans its first live word to its last. A sentence divided by a split spans every piece, while a piece handle (s4b, s4b-s6) covers only that stretch of the timeline. Without --range the whole video is previewed.

Files go into a new folder under the user’s temporary directory, never the project; delete it when done. The reply’s value has directory, revision, projectId, format, range, sentences, start/end (seconds), output, stills (path, frame, time, width, height) and audio (path, start, end, samples, truncated) or null.

Preview is read-only: it changes no project bytes and does not adopt an external edit. It works on a closed project (the CLI renders it itself) and through the open app. --expected-revision and --project-id are optional checks; a different current revision fails with stale_revision and names the current one, so view the project again before judging the stills. The SDK exposes preview({range, frames, audio, projectId, expectedRevision}), and the agent socket method is preview with params range, frames, audio, projectId and expectedRevision.

Connect over MCP

Claude Code or Codex can edit the project open in the app over the Model Context Protocol without the CLI. Help → Use with Coding Agents… copies the add command, which runs the yapwing helper inside the app as a stdio MCP server:

claude mcp add yapwing -- '/Applications/Yapwing.app/Contents/Helpers/yapwing' mcp
codex mcp add yapwing -- '/Applications/Yapwing.app/Contents/Helpers/yapwing' mcp

The copied path is the running app’s Contents/Helpers/yapwing in single quotes, so the shell expands nothing in it. Add it once: there is no port or token, so the entry stays valid across launches until the app moves.

yapwing mcp reads newline-delimited JSON-RPC 2.0 from stdin and writes one response line per request to stdout; it writes nothing else there, and its logs go to stderr. Each tool call goes to the running app over its fixed user-only socket (/private/tmp/yapwing-UID/app.sock), which the app opens at launch and removes at quit and which passes the call to the open project’s endpoint. Only the current macOS user can connect: the folder is 0700, the socket 0600, and both ends check the peer’s user ID. The app holds an exclusive lock on app.lock beside the socket while it listens, so a second running Yapwing leaves the first one’s socket alone; isolated (--defaults-suite) and --smoke-test launches don’t open it. To reach an isolated launch, run yapwing mcp --project PROJECT (the endpoint of the Yapwing that has PROJECT open) or yapwing mcp --socket PATH; while nothing listens there, tool calls fail with project_not_open. While Yapwing isn’t open, initialize, ping and tools/list still answer, and a tool call fails with app_not_open. Requests run concurrently, notifications/cancelled cancels one (its response is not sent), and the process exits when stdin ends, after requests in flight finish. A line that isn’t JSON gets -32700 with a null id; a line over 32 MiB gets -32600.

Over HTTP is the alternative for a client that can’t start a command. That server starts the first time Help → Use with Coding Agents… is opened and runs until Yapwing quits, on an address and token made for that launch; the sheet’s Over HTTP row copies its commands:

claude mcp add --transport http yapwing http://127.0.0.1:PORT/mcp --header "Authorization: Bearer TOKEN"
export YAPWING_MCP_TOKEN=TOKEN && codex mcp add yapwing --url http://127.0.0.1:PORT/mcp --bearer-token-env-var YAPWING_MCP_TOKEN

Codex reads the token from YAPWING_MCP_TOKEN when it starts, so run its command in the shell that starts Codex. Copy again after every relaunch.

The tools wrap the agent methods above, and each result’s text is exactly the JSON the matching CLI command prints (view returns the view text):

Tool	Arguments	Same as
inspect	none	yapwing inspect
view	outline?	yapwing view [--outline]
edit	source, ops?, dryRun?, operationId?	yapwing edit --file (source is the file’s text)
check, apply	request (a command transaction)	yapwing check/apply --request
preview	range?, frames?, audio?, projectId?, expectedRevision?	yapwing preview; the stills also come back as JPEG images
library	query?, offset?, limit?	yapwing library
search	query, limit?, shows?	yapwing search [--shows]
export	output, projectId, expectedRevision	yapwing export

A failed command is a tool result with isError: true whose text is the same {ok: false, error: {code, message}} the CLI prints, so stale_revision, project_mismatch and the rest mean what they always do. With no project open, calls fail with no_project; over stdio with the app closed, with app_not_open. Tools act on the project open in the window; projects that aren’t open use the CLI.

Both transports serve the same protocol versions, tools and results. HTTP security:

• It listens on 127.0.0.1 only, on a port the system picks when it starts, and refuses any peer that isn’t loopback.
• Every request needs Authorization: Bearer TOKEN. The token is 32 random bytes made when the server starts, kept only in memory and in the copied commands, compared in constant time, and never logged or written by Yapwing. A new launch has a new port and token. Missing or wrong tokens get HTTP 401.
• DNS rebinding is refused: the Host header must be 127.0.0.1:PORT or localhost:PORT, and an Origin header, when sent, must be one of those as http:// (HTTP 403). There is no CORS.
• Headers are limited to 16 KiB and bodies to 16 MiB (HTTP 431, 413). A request must arrive whole within 30 seconds, with no read waiting more than 10 (HTTP 408), and at most 16 connections are open at once. Header lines must end in CRLF; bare CR or LF, folded lines and repeated Host, Origin, Authorization, Content-Length or MCP-Protocol-Version headers get HTTP 400.

Protocol: JSON-RPC 2.0 over POST /mcp, one JSON response per request (Content-Type: application/json). initialize negotiates 2025-11-25, 2025-06-18 or 2025-03-26 (an unknown version gets the newest), and a request whose MCP-Protocol-Version header names another version gets HTTP 400. Notifications get 202. There are no sessions and no server-sent stream, so GET and DELETE answer 405. ping, tools/list and tools/call are served; batches, parse errors and unknown methods or tools get JSON-RPC errors (-32600, -32700, -32601, -32602).

Import, collect, export

yapwing init /absolute/project
yapwing import /absolute/project /absolute/clip.mov --project-id PROJECT_ID --expected-revision 0 --operation-id import-unique-id
yapwing collect /absolute/project --project-id PROJECT_ID --expected-revision CURRENT_REVISION --operation-id collect-unique-id
yapwing export /absolute/project --project-id PROJECT_ID --expected-revision CURRENT_REVISION --output /absolute/new-export.mp4

Replace the IDs and revisions with inspect results. Import references the original file, probes it, then hashes/transcribes it off the main thread. Import waits for speech by default; –no-wait requires an open app. The CLI exits 2 if media was imported but a transcription failed, preserving the complete inspection response. Retry speech with transcribe PROJECT –asset-id ID –expected-revision N. The CLI can obtain projectId itself if –project-id is omitted; agents should supply their inspected identity explicitly.

Import –index N inserts main clips. For one overlay, use –overlay-parent ID –at NUM/DEN, with an exact source time inside that parent. Image overlays start at that time and last five seconds, clamped to the free part of the parent’s trimmed source interval (another media overlay at that time is refused with overlay_collision; text uses its own lane and may overlap it), and start centered at canvas width; video overlays start at the supplied source time and end within the parent.

Collect copies all registered assets, verifies source identity and destination hashes, then commits relative locators in one metadata transaction. Originals remain intact. Collection waits for imported assets to have verified content hashes; asset_verification_pending means to let import verification finish and inspect again before retrying. Existing destinations are never overwritten. Prepared import and collection operations live under .yapwing/operations, so a retry after interruption reuses the exact original request. Keep the whole project directory, including its .yapwing metadata, when moving it. An interrupted or conflicted collection may leave verified unused copies without changing project references.

Export uses the inspected render-plan revision and preserves an existing destination. It returns exact logical/output timing, written frame/sample counts, color policy and font identity. Export levels the mix to −14 LUFS with one constant gain and keeps true peak at or under −1 dBTP, holding the loudest peaks down with a look-ahead limiter (at most 6 dB) when the gain would carry them past it; loudness reports the target, the measured integrated loudness and true peak, the gain in dB and, when the limiter ran, limiter with its ceiling, look-ahead, release and largest reduction (loudness is absent when the reel has no audio, and the gain is 0 for silence). A later edit does not alter an export already prepared from its snapshot. Export has no mutation receipt: if interrupted, inspect the requested output path and choose a fresh name before retrying.

While it runs, yapwing export writes progress to stderr as one JSON object per line; stdout still carries only the single JSON reply, so parse stdout alone (do not merge the streams with 2>&1). A closed project reports {"fraction":0.42,"stage":"render","type":"progress"} lines: stage loudness (the measuring pass, absent for a silent reel) then render, each a line when it starts and then at most one every half second that advances at least 1 %, ending on render 1.0 after a successful export. When the app has the project open it renders without reporting, so the CLI prints {"stage":"waiting","type":"progress"} at once and every 2 s until the reply, then render 1.0. Other commands write nothing to stderr, and yapwing mcp never reports export progress on stdout.

Source library and analysis

yapwing import /absolute/project /absolute/source.mov --library --project-id PROJECT_ID --expected-revision N --operation-id library-unique-id
yapwing library /absolute/project --query "water" --offset 0 --limit 20
yapwing search /absolute/project --query "bake bread" --limit 20
yapwing search /absolute/project --query "red car" --shows
yapwing analyze /absolute/project --asset-id ASSET_ID --project-id PROJECT_ID --expected-revision N
yapwing link-prepared /absolute/project --asset-id RAW_ID --prepared-asset-id PREPARED_ID --project-id PROJECT_ID --expected-revision N

Library import creates source membership without timeline placement. Explicitly prepared full-sphere video additionally uses --projection equirectangular. Import replies retain snapshot and statuses and add importedAssetIDs in accepted command order, including reused sources. Library aliases coalesce; repeated main placements repeat the selected ID. These IDs remain stable on replay, whose snapshot can include later edits or completed transcription. Use the exact returned ID when linking a prepared video: a file can have separate flat and spherical interpretations. Older tools that omit this field cannot identify the selection reliably from its path or project asset order. Raw INSV remains unplaced until a prepared video is imported and linked. Linking is an idempotent library sidecar operation bound to project identity/revision; it does not change authored Undo history. It has no mutation receipt, so inspect the current association after an interrupted response.

library is read-only and restores verified cached evidence without starting analysis. Its result has items, total, nextOffset and the observed revision; follow nextOffset until null. A changed source cannot expose its old analysis. Queries are bounded UTF-8 strings (4 KiB), offsets are nonnegative safe integers, and limits are 1–128. Analysis retries derived work for one source or the library and waits by default; --no-wait requires the open app to own the job. See the library guide for the app’s automatic analysis policy.

search is read-only and finds spoken sentences across placeable library videos, best first, with the app’s ranking: each query word must begin a word (case and accents ignored), then phrase proximity, then recency. The query needs at least one letter or number (4 KiB at most) and the limit is 1–128 (default 20). The reply is {hits, revision}; each hit has assetId, sentence (its index in the source), text (corrected words), sourceName, start/end seconds for reading, the exact source range, and placement, the range with 0.15 s handles clamped to the video. To place a hit, append the asset and trim the new occurrence to placement in one transaction, as the app does.

--shows (wire shows: true) also searches what the videos show with the optional picture model, after the sentences; when both overflow, each keeps at least half the limit. Every hit in a shows reply has kind: "says" for a sentence, "shows" for a picture match; a reply without shows is unchanged and has no kind. A "shows" hit has assetId, frame (the keyframe part index), sourceName, score (cosine similarity, higher is closer), start/end seconds, source and placement (exactly the stretch, no handles). The reply adds shows: {"state": "ready", "indexedSources": N} once the model is in place, otherwise {"state": "unavailable" | "notDownloaded", "message"} with only sentences (unavailable only from a build that does not offer the model). Agents never download the model; only the person can, from the Library’s search results. Videos are embedded after their shots step, so a recently added video can be missing until analysis finishes. Picture embeddings are cached per source and model. shows must be a boolean.

The SDK exposes search({query, limit, shows}), library({query, offset, limit}), analyze({projectId, expectedRevision, assetId, wait}), and linkPrepared(rawId, preparedId, {projectId, expectedRevision}). All return the paged library shape. A library item carries title when the video has a Yapwing name; set or clear it with set_asset_title (title: null shows the file name again), which never renames the source file. Source metadata and exact ranges retain their independent provenance and validation.

Text editing and media identity

Select exact IDs from renderPlan.words/elements. cut_units removes the selected visible word/pause coverage. correct_words changes the caption/transcript label over its existing source atoms; it does not insert audio. reset_words removes every correction touching the named atoms, showing the recognized words again (the app’s Use Original Caption). Corrections and timing groups can make one unit span several tokens. Do not infer new cut times from display text. Pause handles and deleted units have dedicated commands; ordinary manual clip trims remain separate from transcript omissions.

Imported source fingerprints bind immutable transcription metadata. Changing a linked recording is an error, not a request to silently retime existing edits. Reimport a replacement recording separately. Asset collection changes locators while retaining the same measured identity.

The live app may adopt a valid authored external JSON edit as one undo step. Malformed, stale, or metadata-changing edits are preserved and reported as conflicts. Prefer commands because they preserve explicit intent and receipts.