Brummel/Aura
1
0
Fork 0
Code Issues 31 Pull Requests Actions Packages Projects Releases Wiki Activity

Guarantee additive forward-compatibility of the blueprint format (optional-ignore vs load-bearing-must-understand) #156

New Issue
Closed
opened 2026-06-29 12:58:41 +02:00 by Brummel · 3 comments
Brummel commented 2026-06-29 12:58:41 +02:00
Owner
Copy Link
Copy Source

The blueprint format must extend additively: later additions land without breaking existing serializations. aura already runs the additive serde discipline elsewhere — verified: 15 #[serde(default)] and 14 skip_serializing_if across crates/, serde(alias) for field renames (crates/aura-analysis/src/lib.rs:72), no deny_unknown_fields anywhere (rg deny_unknown_fields crates -> 0 matches), and a dedicated shim home at crates/aura-registry/src/compat.rs. The gap: no format-version envelope exists today (rg 'format_version|schema_version' crates -> 0 matches); run identity is carried only by commit.

Two tiers, and the danger is the middle — silently ignoring something load-bearing:

Tier 1 — additive-optional (ignore / default-safe): new optional fields and annotations via serde default + skip_serializing_if, no deny_unknown_fields. Determinism rule (C1, the determinism/reproducibility contract in docs/design/INDEX.md): a new optional field's default equals the prior behaviour, so an old blueprint loaded by a new engine reproduces the same graph.

Tier 2 — load-bearing (must-understand): a new node type, edge semantics, or structural-axis kind. An old reader refuses cleanly and names the cause (unknown node type, or format_version above its horizon) — never silently builds a different graph (C1 / C18 reproducibility, docs/design/INDEX.md). Optional finer mechanism: a required flag per extension section, so a non-required unknown section is skipped (Tier 1) and a required one refuses (Tier 2) — the PNG critical-vs-ancillary chunk model.

Acceptance (validated property):

  • A Tier-1 addition leaves an existing blueprint's load + run byte-identical (golden).
  • A Tier-2 unknown element (unknown node type, or too-new format_version) yields a clean, named, exit-coded refusal — never a panic and never a silent graph change (refuse-don't-guess).

Depends on the serialization + loader issue #155 (the envelope and canonical form live there).

The blueprint format must extend additively: later additions land without breaking existing serializations. aura already runs the additive serde discipline elsewhere — verified: 15 `#[serde(default)]` and 14 `skip_serializing_if` across crates/, `serde(alias)` for field renames (crates/aura-analysis/src/lib.rs:72), no `deny_unknown_fields` anywhere (`rg deny_unknown_fields crates` -> 0 matches), and a dedicated shim home at crates/aura-registry/src/compat.rs. The gap: no format-version envelope exists today (`rg 'format_version|schema_version' crates` -> 0 matches); run identity is carried only by commit. Two tiers, and the danger is the middle — silently ignoring something load-bearing: Tier 1 — additive-optional (ignore / default-safe): new optional fields and annotations via serde `default` + `skip_serializing_if`, no `deny_unknown_fields`. Determinism rule (C1, the determinism/reproducibility contract in docs/design/INDEX.md): a new optional field's default equals the prior behaviour, so an old blueprint loaded by a new engine reproduces the same graph. Tier 2 — load-bearing (must-understand): a new node type, edge semantics, or structural-axis kind. An old reader refuses cleanly and names the cause (unknown node type, or format_version above its horizon) — never silently builds a different graph (C1 / C18 reproducibility, docs/design/INDEX.md). Optional finer mechanism: a `required` flag per extension section, so a non-required unknown section is skipped (Tier 1) and a required one refuses (Tier 2) — the PNG critical-vs-ancillary chunk model. Acceptance (validated property): - [ ] A Tier-1 addition leaves an existing blueprint's load + run byte-identical (golden). - [ ] A Tier-2 unknown element (unknown node type, or too-new format_version) yields a clean, named, exit-coded refusal — never a panic and never a silent graph change (refuse-don't-guess). Depends on the serialization + loader issue #155 (the envelope and canonical form live there).
Brummel added this to the Topology-as-data: blueprint serialization & loader (C24) milestone 2026-06-29 12:58:41 +02:00
Brummel added the feature label 2026-06-29 12:58:41 +02:00
Brummel commented 2026-06-29 15:04:51 +02:00
Author
Owner
Copy Link
Copy Source

Scope reconciliation after #155 shipped (cycle 0087, d5602ec)

The #155 first cut already delivers a substantial part of this issue's
must-understand gate. Verified against the shipped code
(crates/aura-engine/src/blueprint_serde.rs):

  • format_version envelope exists (BLUEPRINT_FORMAT_VERSION = 1, the
    top-level BlueprintDoc).
  • Tier-2 clean refusal already works and is tested: a too-new
    format_version → LoadError::UnsupportedVersion (test
    unsupported_version_fails_named); an unknown node type →
    LoadError::UnknownNodeType (test unknown_node_type_fails_named). Neither
    panics; neither silently builds a different graph (C1/C18-safe).
  • Tier-1 additive serde discipline is the format's mechanism: every DTO
    optional uses serde(default, skip_serializing_if = ...), no
    deny_unknown_fields; the canonical omit-defaults form makes an absent
    optional byte-identical to the pre-extension bytes.

So this issue's remaining substantive scope is narrower than the body
implies:

  1. A Tier-1 forward-additive-compat demonstration + golden: a new optional
    field added later loads an old serialization unchanged (and leaves its
    content id stable — shared with #158). #155 proved omit-defaults, not yet the
    forward direction (new reader, old bytes).
  2. The optional PNG required-vs-ancillary section mechanism (a required
    flag per extension section): genuinely new — the format today is a flat node
    list with no "sections", so this is only worth building when a real ancillary
    extension appears.
  3. Exit-coded refusal: a CLI concern — there is no CLI path that loads a
    blueprint yet (that surface is #157/#159). The typed LoadError is the
    library-level half; the exit-code mapping waits on that surface.

Recommendation for whoever picks this up: rescope to item 1 (a small tdd-sized
forward-compat golden), and treat items 2-3 as deferred-until-needed (2: a real
ancillary extension; 3: the CLI load surface). Not a blocker on anything.

## Scope reconciliation after #155 shipped (cycle 0087, `d5602ec`) The #155 first cut already delivers a substantial part of this issue's must-understand gate. Verified against the shipped code (`crates/aura-engine/src/blueprint_serde.rs`): - **`format_version` envelope** exists (`BLUEPRINT_FORMAT_VERSION = 1`, the top-level `BlueprintDoc`). - **Tier-2 clean refusal already works and is tested**: a too-new `format_version` → `LoadError::UnsupportedVersion` (test `unsupported_version_fails_named`); an unknown node type → `LoadError::UnknownNodeType` (test `unknown_node_type_fails_named`). Neither panics; neither silently builds a different graph (C1/C18-safe). - **Tier-1 additive serde discipline** is the format's mechanism: every DTO optional uses `serde(default, skip_serializing_if = ...)`, no `deny_unknown_fields`; the canonical omit-defaults form makes an absent optional byte-identical to the pre-extension bytes. So this issue's **remaining substantive scope** is narrower than the body implies: 1. A **Tier-1 forward-additive-compat** demonstration + golden: a *new* optional field added later loads an *old* serialization unchanged (and leaves its content id stable — shared with #158). #155 proved omit-defaults, not yet the forward direction (new reader, old bytes). 2. The optional **PNG required-vs-ancillary section mechanism** (a `required` flag per extension section): genuinely new — the format today is a flat node list with no "sections", so this is only worth building when a real ancillary extension appears. 3. **Exit-coded** refusal: a CLI concern — there is no CLI path that loads a blueprint yet (that surface is #157/#159). The typed `LoadError` is the library-level half; the exit-code mapping waits on that surface. Recommendation for whoever picks this up: rescope to item 1 (a small `tdd`-sized forward-compat golden), and treat items 2-3 as deferred-until-needed (2: a real ancillary extension; 3: the CLI load surface). Not a blocker on anything.
Brummel commented 2026-06-29 21:47:35 +02:00
Author
Owner
Copy Link
Copy Source

Cycle 0088 (#157) scaling note: the engine Op ↔ wire DTO needs the same must-understand discipline

The construction service (cycle 0088) keeps the engine Op enum serde-free and
deserializes the JSON op-list through a CLI-side DTO (OpDoc,
crates/aura-cli/src/graph_construct.rs) that maps into Op. The From<OpDoc> for Op
is exhaustive over OpDoc, but the reverse direction is unguarded: a future engine
Op variant — e.g. the construction-arg ops the format extensions here will add
(LinComb / CostSum / SimBroker / Session) — is not compiler-forced to gain a
wire encoding in OpDoc. It would simply be unreachable from the document format,
silently, until someone notices.

When the additive-extension work here lands construction-arg builders into the
serialized set, the op-list DTO should extend in lockstep — and the same
optional-ignore vs. load-bearing-must-understand tiering this issue defines for the
blueprint format applies to the op-list wire format too. A low scaling note, recorded
by the cycle-0088 architect drift review; non-blocking today (the current Op
vocabulary is fully covered by OpDoc).

## Cycle 0088 (#157) scaling note: the engine Op ↔ wire DTO needs the same must-understand discipline The construction service (cycle 0088) keeps the engine `Op` enum serde-free and deserializes the JSON op-list through a CLI-side DTO (`OpDoc`, `crates/aura-cli/src/graph_construct.rs`) that maps into `Op`. The `From<OpDoc> for Op` is exhaustive over `OpDoc`, but the **reverse direction is unguarded**: a future engine `Op` variant — e.g. the construction-arg ops the format extensions here will add (`LinComb` / `CostSum` / `SimBroker` / `Session`) — is **not** compiler-forced to gain a wire encoding in `OpDoc`. It would simply be unreachable from the document format, silently, until someone notices. When the additive-extension work here lands construction-arg builders into the serialized set, the op-list DTO should extend in lockstep — and the same optional-ignore vs. load-bearing-must-understand tiering this issue defines for the blueprint format applies to the op-list wire format too. A low scaling note, recorded by the cycle-0088 architect drift review; non-blocking today (the current `Op` vocabulary is fully covered by `OpDoc`).
Brummel commented 2026-06-30 13:06:54 +02:00
Author
Owner
Copy Link
Copy Source

Design reconciliation (specify, #156)

Spec being produced: docs/specs/0090-blueprint-forward-compat.md. One load-bearing fork is decided here before the spec is written.

Fork — granularity of the must-understand (Tier-2) refusal. The issue floats an optional finer mechanism: a required flag per extension section (the PNG critical-vs-ancillary chunk model), which would let a load-bearing addition refuse on an old reader without bumping the global format_version.

Decision (derived): do not build the per-section required-flag mechanism now. The single invariant "any must-understand change bumps format_version" is adopted instead.

Basis (derived, not a user decision):

  • The version envelope already delivers the must-understand refusal at version granularity — blueprint_from_json refuses any format_version != BLUEPRINT_FORMAT_VERSION cleanly (LoadError::UnsupportedVersion, blueprint_serde.rs:162), and an out-of-vocabulary node refuses as LoadError::UnknownNodeType (:128). Both are already green-tested (unsupported_version_fails_named, unknown_node_type_fails_named).
  • There is no current Tier-2 section to validate a required-flag scheme against; building the finer mechanism now is speculative structure with no consumer. It can be added additively if a real sub-version Tier-2 addition ever needs finer granularity than a version bump.
  • C1 reproducibility holds either way, so the simpler invariant carries no correctness cost.

Net discipline this cycle codifies: Tier-1 (additive-optional) = serde-default silent-ignore, no version bump, with the C1 constraint that a new optional field defaults to prior behaviour (an old blueprint reproduces the same graph); Tier-2 (must-understand: a new node type, edge semantics, or structural-axis kind) = bump format_version. The remaining unproven property — forward-tolerance of an unknown optional field on the current loader — becomes a green pin, and the discipline is written into the ledger (C24) and the LoadError doc.

## Design reconciliation (specify, #156) Spec being produced: `docs/specs/0090-blueprint-forward-compat.md`. One load-bearing fork is decided here before the spec is written. **Fork — granularity of the must-understand (Tier-2) refusal.** The issue floats an optional finer mechanism: a required flag per extension section (the PNG critical-vs-ancillary chunk model), which would let a load-bearing addition refuse on an old reader *without* bumping the global `format_version`. **Decision (derived): do not build the per-section required-flag mechanism now.** The single invariant "any must-understand change bumps `format_version`" is adopted instead. Basis (derived, not a user decision): - The version envelope already delivers the must-understand refusal at version granularity — `blueprint_from_json` refuses any `format_version != BLUEPRINT_FORMAT_VERSION` cleanly (`LoadError::UnsupportedVersion`, `blueprint_serde.rs:162`), and an out-of-vocabulary node refuses as `LoadError::UnknownNodeType` (`:128`). Both are already green-tested (`unsupported_version_fails_named`, `unknown_node_type_fails_named`). - There is no current Tier-2 *section* to validate a required-flag scheme against; building the finer mechanism now is speculative structure with no consumer. It can be added additively if a real sub-version Tier-2 addition ever needs finer granularity than a version bump. - C1 reproducibility holds either way, so the simpler invariant carries no correctness cost. Net discipline this cycle codifies: Tier-1 (additive-optional) = serde-default silent-ignore, no version bump, with the C1 constraint that a new optional field defaults to prior behaviour (an old blueprint reproduces the same graph); Tier-2 (must-understand: a new node type, edge semantics, or structural-axis kind) = bump `format_version`. The remaining unproven property — forward-tolerance of an unknown optional field on the current loader — becomes a green pin, and the discipline is written into the ledger (C24) and the `LoadError` doc.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
BLOCKER
Core functionality blocked; preempts other selection
 bug
Observable misbehaviour
 feature
Substantial extension; no full spec required
 idea
Droppable; no commitment
No Label feature
Milestone
No items
No Milestone Topology-as-data: blueprint serialization & loader (C24)
Projects
Clear projects
No project
Assignees
Clear assignees
claude (Claude) Brummel
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Brummel/Aura#156
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?