RustAst/docs/bytecode-architecture.md

Bytecode Generation Architecture

This document describes the transition from a pure tree-walking AST interpreter to a hybrid execution model using specialized bytecode.

1. Overview: The Hybrid Model

The MYC VM operates in a hybrid mode. While the Abstract Syntax Tree (AST) is the first-class citizen and primary representation, "hot" paths—specifically monomorphized functions and financial pipeline lambdas—are lowered into a linear bytecode representation.

---

2. The Integrated Transformation Pipeline

To ensure maximum performance and correct Tail Call Optimization (TCO), bytecode generation is integrated into the structural optimization phase.

Step 1: Specializer (Structural Optimization & Emission)

The Specializer is the core engine for code improvement.

• Inlining & Monomorphization: It first performs all structural changes (e.g., inlining function bodies). This is critical because inlining changes what constitutes a "tail position."
• Symbolic Emission: Once the structure of a function/block is final, the Specializer converts it into Symbolic Bytecode.
• Integrated TCO Analysis: During emission, the Specializer tracks the tail-call state locally. It emits OP_RECUR or OP_TAIL_CALL based on the final, inlined structure.
• Variable Mapping: Instructions still use VirtualId (e.g., OP_GET_VIRTUAL(V102)).

Step 2: Lowering (Address Resolution & Linking)

The Lowering pass acts as the "Linker". It does not change the structure anymore.

• Address Resolution: It walks the symbolic bytecode and replaces VirtualId with concrete StackOffset values.
• Stack Allocation: It calculates the final stack_size for the bytecode block.
• Finalization: It "freezes" the instruction vector for the ExecNode.

---

3. Why This Order? (Inlining vs. TCO)

TCO analysis must occur after structural optimizations like inlining. If TCO were analyzed before inlining, a tail-call in a small function might lose its tail position once embedded into a larger caller. By performing TCO analysis during bytecode emission (after inlining), we guarantee that the is_tail flags are always semantically correct.

---

4. Instruction Set Architecture (ISA) & TCO

The ISA uses specialized instructions to bypass the heavy tree-walking logic:

1. OP_RECUR:

   • Used for the again keyword.
   • Triggers an immediate jump to PC 0 within the same bytecode executor.
   • Repacks the current stack frame with new arguments.

2. OP_TAIL_CALL(Callee):

   • Used for function calls in tail positions.
   • If the callee is bytecode, it performs a fast-swap of the instruction vector.
   • If the callee is native/tree, it returns a TailCallRequest to the VM dispatcher.

---

5. The Hybrid VM Dispatcher

The VM's eval_internal remains the master controller. It dispatches to the Bytecode Executor when encountering a BoundKind::Bytecode node. The executor can "escape" back to the tree-walker for any operation it cannot handle natively (e.g., calling an opaque extension).

fn run_bytecode(instructions: &[Instruction], pc: &mut usize) -> Value {
    loop {
        match instructions[*pc] {
            Instruction::PushConst(v) => stack.push(v),
            Instruction::AddInt => {
                let b = stack.pop();
                let a = stack.pop();
                stack.push(a + b);
            }
            Instruction::TailCall(callee) => {
                // Return to dispatcher if it's a cross-world call
                return self.handle_tail_call(callee);
            }
            Instruction::CallTree(node) => {
                // Re-enter tree-walking for specific node
                let res = self.eval_internal(node)?;
                stack.push(res);
            }
            // ...
        }
        pc += 1;
    }
}