Aura/docs/authoring-guide.md

# Authoring guide: op-scripts, process documents, campaign documents

`docs/project-layout.md` describes the shape of a project and the arc of a
research session; this document is the practical companion for the three
JSON artifact kinds you author headlessly along that arc:

1. an **op-script** (role 6a) — builds a node graph (a strategy blueprint)
   through `aura graph build`;
2. a **process document** (role 5) — a named validation/eval methodology
   through `aura process validate|introspect|register`;
3. a **campaign document** (role 6b) — experiment intent (instruments ×
   windows × strategy × axes × process) through `aura campaign
   validate|introspect|register|run`.

Each section below is a worked, verified example — every command shown was
run against this repo and the output is transcribed, not invented. The *why*
of this three-artifact split (closed-vocabulary data, never a logic DSL)
lives in the design ledger (`docs/design/INDEX.md`, C20/C25) and the
glossary; this document only teaches the *shape*.

## 1. Op-scripts — building a strategy blueprint by hand

An op-script is a JSON **array of ops**, replayed in order to construct a
node graph. `aura graph build` reads the op-script from **stdin** (there is
no file argument) and prints the canonical blueprint envelope to stdout:

```
$ aura graph build < smacross.json > blueprint.json
```

Nodes are referenced by an **identifier** (given by `add`, see below); ports
are dotted `<identifier>.<port>` on both sides of a wire.

### The seven ops

| op | JSON shape | does |
|---|---|---|
| `source` | `{"op":"source","role":<str>,"kind":<ScalarKind>}` | reserve a bound root **source** role of `kind` — a real input the harness feeds (e.g. `"price"`). |
| `input` | `{"op":"input","role":<str>}` | reserve an open root **input** role (kind inferred from the slots it feeds) — for a fragment meant to be wired by an *enclosing* graph. A standalone document built with `aura graph build` finalizes as a closed root, so an `input` role that is never bound refuses at the end: `finalize: root input role <name> is unbound`. |
| `add` | `{"op":"add","type":<TypeId>,"name":<str>?,"bind":{<param>:<Scalar>}?}` | instantiate a node of a type in the closed vocabulary (`aura graph introspect --vocabulary`). `name` becomes the node's identifier for later ops (default: the type's own lowercase label — two unnamed nodes of the same type then collide). `bind` sets zero or more of its params. |
| `feed` | `{"op":"feed","role":<str>,"into":[<port>, …]}` | fan a previously-declared role into one or more interior input slots, all-or-nothing (a failing target leaves none of the batch wired). |
| `connect` | `{"op":"connect","from":<port>,"to":<port>}` | wire one interior output field to one interior input slot. A `connect` that would close a dataflow cycle is rejected immediately — the only legal feedback path is an explicit delay/state node (domain invariant 5). |
| `expose` | `{"op":"expose","from":<port>,"as":<str>}` | promote an interior output field to a boundary output under the alias `as` — the only op whose name key is a real *alias* (contrast `add`'s `name`, which is an identifier, not a rename). |
| `gang` | `{"op":"gang","as":"channel_length","into":["channel_hi.length","channel_lo.length"]}` | Fuse two or more sibling params into ONE public knob: the member addresses leave the sweepable param space and `as` replaces them; the bound or swept value fans out to every member at bootstrap. Members must share one scalar kind and stay open (un-bound). |

Value forms are the typed-tag representations used everywhere in this
family of artifacts:

- a bind value is `{"I64":2}` / `{"F64":0.5}` / `{"Bool":true}` /
  `{"Timestamp":<i64 ms>}`;
- a `kind` field is the capitalized `ScalarKind` name: `"I64"` | `"F64"` |
  `"Bool"` | `"Timestamp"`.

### Worked example: an SMA-crossover bias strategy

```json
[
  {"op": "source", "role": "price", "kind": "F64"},
  {"op": "add", "type": "SMA", "name": "fast"},
  {"op": "add", "type": "SMA", "name": "slow"},
  {"op": "feed", "role": "price", "into": ["fast.series", "slow.series"]},
  {"op": "add", "type": "Sub", "name": "sub"},
  {"op": "connect", "from": "fast.value", "to": "sub.lhs"},
  {"op": "connect", "from": "slow.value", "to": "sub.rhs"},
  {"op": "add", "type": "Bias", "name": "bias"},
  {"op": "connect", "from": "sub.value", "to": "bias.signal"},
  {"op": "expose", "from": "bias.bias", "as": "bias"}
]
```

(`fast.length`, `slow.length`, and `bias.scale` are all left unbound here on
purpose — three open campaign axes, see §3. This is the milestone fieldtest
corpus's own example, verified below; byte-identical to the on-disk
`fieldtests/milestone-research-artifacts/mra_1_strategy_smacross.json`.)

A `bind` in an `add` op pins that param to a fixed value and **removes it
from the sweepable param space**: a bound param is closed — it never shows
up in `--params`/`--list-axes`, and no campaign axis and no `aura sweep
--axis` can reopen it. `bind` is for a constant the strategy never varies;
leave a param unbound, as all three are here, to keep it open for sweeping.

A param therefore has three states: **open** (a sweepable axis), **bound**
(closed by a `bind` — gone from the axis namespace), and **ganged** (open, but
fused with its siblings under ONE public knob declared by a `gang` op; the
member addresses are unbindable and only the gang's own single-segment name —
wrapped like any knob, e.g. `graph.channel_length` — appears in `--list-axes`).

### Commands

```
$ aura graph build < smacross.json
{"format_version":1,"blueprint":{"name":"graph","nodes":[...],"edges":[...],
 "input_roles":[{"name":"price",...,"source":"F64"}],"output":[...]}}
```

The built blueprint renders visually, too: `aura graph blueprint.json` emits an
interactive HTML DAG so a mis-wire is visible before any run (`aura graph` with
no file renders the built-in sample; a named-but-unreadable file is a usage
error). Introspection is build-free wherever possible:

```
$ aura graph introspect --vocabulary        # one node type per line
$ aura graph introspect --node SMA          # ports + params of one type
SMA
  in   series:F64
  out  value:F64
  param length:I64  (bind {"I64": <v>})
$ aura graph introspect --unwired < partial.json   # open slots of a partial op-script
sub.rhs:F64
$ aura graph introspect --content-id smacross.json           # SHA-256 of the canonical form
597d719b7ac607158cda3e68cd497387620397a5e93087e23da512876dafba9a
$ aura graph introspect --content-id smacross.json --identity-id  # + debug-name-blind identity id, combinable
597d719b7ac607158cda3e68cd497387620397a5e93087e23da512876dafba9a
41bab46ce78356eeab2d2a4e03daaf2117eb970a1c3ef880264553bf662453a4
$ aura graph introspect --params smacross.json   # the raw param-space namespace (what a campaign's axes bind against)
fast.length:I64
slow.length:I64
bias.scale:F64
```

These printed names are the **raw param-space namespace**: op-script params
and a campaign document's `strategies[].axes` keys (§3) share this one raw
form. There is a third, *wrapped* surface: the dissolved `aura sweep
<blueprint> --axis` CLI (glossary `sweep`) accepts only the names `aura
sweep <blueprint> --list-axes` prints — the raw name prefixed with the
root-composite instance name, `graph.<param>` — never the raw form; the
campaign document the sweep sugar generates still stores the raw form
(#210):

```
raw   (--params, campaign axes):  fast.length
wrapped (--list-axes, --axis):    graph.fast.length
```

A ganged knob's raw address has one path segment less than a member address
would — it sits at the composite's own level, like a role name (e.g.
`channel_length`, not `channel_hi.length`) — and wraps identically (`graph.channel_length`).

`--content-id`, `--identity-id`, `--params`, and `graph register` all accept
**either** shape: the raw op-script array or an already-built `#155`
blueprint envelope (object) — shape-discriminated automatically, so you
never have to build first just to hash or register:

```
$ aura graph register smacross.json      # inside a project (Aura.toml present)
registered blueprint 597d719b7ac607158cda3e68cd497387620397a5e93087e23da512876dafba9a (…/runs/blueprints/597d…json)
```

The printed content id is the address a campaign document's `strategies[].ref`
points at (§3).

## 2. Process documents — a validation methodology (role 5)

A process document is a named, versionable pipeline of **std stage blocks**
over a shared metric vocabulary. Discover the block vocabulary and each
block's typed slots headlessly:

```
$ aura process introspect --vocabulary
std::sweep         evaluate the campaign's axes-space; reduce members to R metrics; optionally select a winner (the selection group metric+select is all-or-nothing; omitted = selection-free, terminal-stage-only)
std::gate          filter survivors by a conjunction of typed metric predicates
std::walk_forward  rolling in-sample optimize + out-of-sample test
std::monte_carlo   R-bootstrap over realised R (terminal annotator): ...
std::generalize    cross-instrument worst-case floor (terminal annotator): ...
$ aura process introspect --block std::sweep
std::sweep — evaluate the campaign's axes-space; reduce members to R metrics; optionally select a winner (the selection group metric+select is all-or-nothing; omitted = selection-free, terminal-stage-only)
  metric               optional, metric name (see metric_vocabulary)
  select               optional, select rule: argmax | plateau:mean | plateau:worst
  deflate              optional, bool
```

`metric` and `select` form one **selection group**: all-or-nothing (a
document naming one without the other is refused, "the selection group is
all-or-nothing"), `deflate` composes only when the group is present. Omit
the group entirely for a **selection-free sweep** — the family itself is
the result, no winner is chosen. A selection-free sweep is only legal as
the pipeline's *last* stage (the executor's preflight refuses one followed
by any other block, since a downstream stage would have no nominee to
consume):

```json
{
  "format_version": 1,
  "kind": "process",
  "name": "explore-only-sweep",
  "pipeline": [ { "block": "std::sweep" } ]
}
```

The executor records this stage's family (every member run) but no
`StageSelection` and no nominee — recording a winner here would fabricate a
selection intent the document never expressed.

### Worked example: full v2 pipeline (sweep → gate → walk-forward → Monte-Carlo → generalize)

```json
{
  "format_version": 1,
  "kind": "process",
  "name": "mra-full-v2-sweep-gate-wf-mc-generalize",
  "description": "Full v2 anti-false-discovery pipeline.",
  "pipeline": [
    { "block": "std::sweep", "metric": "sqn", "select": "argmax", "deflate": true },
    { "block": "std::gate", "all": [ { "metric": "expectancy_r", "cmp": "gt", "value": 0.0 } ] },
    { "block": "std::walk_forward", "in_sample_ms": 1209600000, "out_of_sample_ms": 604800000, "step_ms": 604800000, "mode": "rolling", "metric": "sqn", "select": "argmax" },
    { "block": "std::monte_carlo", "resamples": 1000, "block_len": 5 },
    { "block": "std::generalize", "metric": "expectancy_r" }
  ]
}
```

(`description` is optional; everything else in the envelope — `format_version`,
`kind`, `name`, `pipeline` — is required, as `aura process introspect --unwired`
over a bare `{}` will tell you.)

```
$ aura process validate mra_2_process_full_v2.json
process document valid (intrinsic): 5 pipeline blocks, 1 gate predicates
$ aura process register mra_2_process_full_v2.json    # inside a project
registered process cd91270ca61ad42f56939231a7803a1d6d7aaa2b70bf79cde9da9284683b86b9 (…/runs/processes/cd91…json)
```

### The metric vocabulary — discoverable, not hand-copied

`aura process introspect --metrics` (equivalently `aura campaign introspect
--metrics` — same roster) enumerates every metric name annotated with its
role, so you never have to guess which metrics a `select`/`gate`/`generalize`
field will accept:

```
$ aura process introspect --metrics
expectancy_r             rankable | gate | generalize
win_rate                 gate
avg_win_r                gate
avg_loss_r               gate
profit_factor            gate
max_r_drawdown           gate
sqn                      rankable | gate | generalize
sqn_normalized           rankable | gate | generalize
net_expectancy_r         rankable | gate | generalize
n_trades                 gate
n_open_at_end            gate
total_pips               rankable | gate
max_drawdown             rankable | gate
bias_sign_flips          rankable | gate
deflated_score           annotation
overfit_probability      annotation
neighbourhood_score      annotation
```

Read the three tags as three separate questions about one metric name:

- **`rankable`** — can this name be used as a `std::sweep` / `std::walk_forward`
  `select` metric (the field a winner is chosen by)? This is the small
  subset every strategy run always produces cheaply.
- **`gate`** — can this name be used in a `std::gate` predicate (a per-member
  filter)? This roster is a superset of `rankable` — most emitted metrics
  are filterable even when they make a poor ranking criterion.
- **`| generalize`** suffix — is this name usable as `std::generalize`'s
  `metric` (needs an R-expectancy-shaped metric to floor across instruments)?

That is why, for example, `profit_factor` shows up tagged only `gate`: every
member's metrics table carries it (so you can gate on it, e.g. "keep only
`profit_factor > 1.0`"), but it is not in the small ranking-eligible roster,
so a `std::sweep` with `"select": "argmax"` cannot select on it, and
`std::generalize` cannot floor across instruments on it either. Attempting
either produces the same intrinsic refusal that names the metric and points
back at this verb (`aura process introspect --metrics`) rather than leaving
you to search the glossary.

## 3. Campaign documents — experiment intent over instruments and windows

A campaign document names the data (instruments × windows), one or more
strategies (by blueprint content/identity id) with their param axes, a
process reference, and what to persist/emit.

```
$ aura campaign introspect --unwired bare.json   # bare.json contains just {}
open slot: format_version (required, must be 1)
open slot: kind (required, must be "campaign")
open slot: name (required, string)
open slot: data (required section: instruments + windows)
open slot: risk (optional, list of stop regimes { vol: { length, k } }; absent = one default regime)
open slot: strategies (required, non-empty list of { ref, axes })
open slot: process.ref (required, content id of a process document)
open slot: seed (required, non-negative integer)
open slot: presentation (required section: persist_taps (equity | exposure | r_equity | net_r_equity) + emit)
```

### Worked example: two instruments, one strategy, four axis points, two stop regimes

```json
{
  "format_version": 1,
  "kind": "campaign",
  "name": "mra-ger40-fra40-smacross-full-v2",
  "seed": 42,
  "data": {
    "instruments": ["GER40", "FRA40"],
    "windows": [ { "from_ms": 1725148800000, "to_ms": 1727740800000 } ]
  },
  "risk": [
    { "vol": { "length": 3, "k": 1.5 } },
    { "vol": { "length": 3, "k": 3.0 } }
  ],
  "strategies": [
    {
      "ref": { "content_id": "597d719b7ac607158cda3e68cd497387620397a5e93087e23da512876dafba9a" },
      "axes": {
        "fast.length": { "kind": "I64", "values": [2, 4] },
        "slow.length": { "kind": "I64", "values": [8, 16] },
        "bias.scale": { "kind": "F64", "values": [0.5] }
      }
    }
  ],
  "process": { "ref": { "content_id": "cd91270ca61ad42f56939231a7803a1d6d7aaa2b70bf79cde9da9284683b86b9" } },
  "presentation": { "persist_taps": ["equity", "r_equity"], "emit": ["family_table", "selection_report"] }
}
```

The `strategies[].ref.content_id` and `process.ref.content_id` are exactly
the ids printed by `aura graph register` / `aura process register` above —
a campaign never inlines a strategy or process body, only its content id.
Each axis name (`fast.length`, …) must name an open param of the referenced
blueprint (`aura graph introspect --params`, §1) and declare that param's
`ScalarKind`.

The optional `risk` list is the campaign's structural risk axis: every cell
runs under every listed stop regime, so cells differ by execution discipline,
never by signal — the regime's stop defines the risk unit R. Absent or empty,
the matrix runs one implicit default regime (the same vol regime the
orchestration verbs bind when their stop flags are omitted).

### Validate — three tiers, honest degradation

```
$ aura campaign validate mra_3_campaign_full_v2.json          # outside any project
campaign document valid (intrinsic): 1 strategy(ies), 3 axes (4 points), 2 instrument(s), 1 window(s), 2 regime(s) — 4 cell(s)
referential checks skipped (no Aura.toml found up from /home/…)
```

Inside a project, with the strategy blueprint and process both already
registered, the same command runs two further tiers:

```
$ aura campaign validate mra_3_campaign_full_v2.json          # inside a project, refs registered
campaign document valid (intrinsic): 1 strategy(ies), 3 axes (4 points), 2 instrument(s), 1 window(s), 2 regime(s) — 4 cell(s)
campaign document valid (referential): all references resolve, axes are in the param space
campaign document valid (executable): pipeline shape and static guards pass
```

- **intrinsic** — the document's own shape is well-formed (always checked).
- **referential** — every `ref` resolves in the project's store and every
  axis names a real, correctly-typed open param (needs a project).
- **executable** — the process pipeline's static guards pass against this
  campaign's shape (e.g. `std::generalize` needs ≥ 2 instruments) — a
  data-free preflight, so "valid" here means "runnable" (needs a project;
  it does not fetch or touch market data).

### Register and run

```
$ aura campaign register mra_3_campaign_full_v2.json
registered campaign 42edebd2159de708009ba21e1ed4aea2cffabc373bf761c5765f79c190b677bd (…/runs/campaigns/42ed…json)
$ aura campaign run 42edebd2159de708009ba21e1ed4aea2cffabc373bf761c5765f79c190b677bd
{"family_id":"42edebd2-0-GER40-w0-s0-0","report":{...}}
{"family_id":"42edebd2-0-GER40-w0-s0-0-r1","report":{...}}
…
{"campaign_run":{"campaign":"42edebd2…","process":"cd9127…","run":0,"seed":42,
  "cells":[{"strategy":"597d719b…","instrument":"GER40","window_ms":[...],
    "regime":{"vol":{"length":3,"k":1.5}},"stages":[
    {"block":"std::sweep",...},{"block":"std::gate","survivor_ordinals":[0,1,2]},
    {"block":"std::walk_forward",...},{"block":"std::monte_carlo","bootstrap":{"pooled_oos":{...}}}]},
   {"strategy":"597d719b…","instrument":"GER40","regime":{"vol":{"length":3,"k":3.0}},"regime_ordinal":1,...},
   {"strategy":"597d719b…","instrument":"FRA40",...},
   {"strategy":"597d719b…","instrument":"FRA40","regime_ordinal":1,...}],
  "generalizations":[{"strategy_ordinal":0,"window_ordinal":0,
    "generalization":{"selection_metric":"expectancy_r","n_instruments":2,
      "worst_case":0.0436…,"sign_agreement":2,"per_instrument":[["GER40",0.0436…],["FRA40",0.0484…]]},
    "winners":[...]},
   {"strategy_ordinal":0,"window_ordinal":0,"regime_ordinal":1,...}],
  "trace_name":"42edebd2-0"}}
```

With two stop regimes every (instrument, window) cell runs twice — the four
cells above are the "4 cell(s)" the validate summary counted. The second
regime's family ids and trace dirs carry the `-r1` ordinal suffix (the
default/first regime stays unsuffixed), each cell record names its regime,
and generalization is keyed per regime — regimes are compared, never pooled.

`aura campaign run` is register-then-run sugar for a `.json` file, but the
canonical address is always the content id — running a bare file the first
time registers it implicitly. `aura campaign runs` lists stored
realizations; `aura campaign runs <id>` dumps the bare stored record(s) (not
the `{"campaign_run": …}` emit wrapper above). If `presentation.persist_taps`
is non-empty, the run also persists the named taps under
`runs/traces/<trace_name>/…`, chartable with `aura chart`.