RustAst/docs/BNF.md

# Myc Language Specification & LLM Prompt Guide

This document defines the syntax (BNF) and semantics of the Myc language, a Lisp-like language designed for financial analysis with first-class abstract syntax trees (ASTs), closures, and a pipeline evaluation model. It serves as a foundational context for Large Language Models (LLMs) to understand and write idiomatic Myc code.

## 1. Syntax (BNF)

```ebnf
<program> ::= <expression>*

<expression> ::= <list>
               | <vector>
               | <record>
               | <quote>
               | <template>
               | <splice>
               | <placeholder>
               | <atom>

<atom> ::= <integer>
         | <float>
         | <string>
         | <keyword>
         | <identifier>
         | <field-accessor>
         | "..."

<list> ::= "(" <special-form> ")"
         | "(" <function-call> ")"

<special-form> ::= <if-form>
                 | <fn-form>
                 | <pipe-form>
                 | <again-form>
                 | <def-form>
                 | <assign-form>
                 | <do-form>
                 | <macro-form>

<function-call> ::= <expression> <expression>*

<if-form> ::= "if" <expression> <expression> [ <expression> ]
<fn-form> ::= "fn" <param-vector> <expression>
<param-vector> ::= "[" <pattern>* "]"
<pattern> ::= <identifier> | "[" <pattern>* "]"
<pipe-form> ::= "pipe" <expression> <expression>
<again-form> ::= "again" <expression>*
<def-form> ::= "def" <pattern> <expression>
<assign-form> ::= "assign" <expression> <expression>
<do-form> ::= "do" <expression>*
<macro-form> ::= "macro" <identifier> <param-vector> <expression>

<vector> ::= "[" <expression>* "]"
<record> ::= "{" ( <keyword> <expression> )* "}"

<quote> ::= "'" <expression>
<template> ::= "`" <expression>
<splice> ::= "~@" <expression>
<placeholder> ::= "~" <expression>

<integer> ::= [-]? [0-9]+
<float> ::= [-]? [0-9]+ "." [0-9]*
<string> ::= '"' [^"]* '"'
<keyword> ::= ":" <identifier>
<field-accessor> ::= "." <identifier>
<identifier> ::= <ident-start> <ident-char>*
```
*(Lexical Note: `<ident-start>` consists of alphabetic characters or `+-*/<=>!$%&_?.`. `<ident-char>` includes digits as well. Identifiers evaluating to a field accessor begin with a `.` followed by one or more valid identifier characters.)*

### Comments

```ebnf
<comment>     ::= ";"  " "? <line-text> <newline>
<doc-comment> ::= ";;" " "? <line-text> <newline>
```

**Rules:**
- Comments must be **whole lines** — they start at the beginning of a line. Inline comments (after code on the same line) are a compile error.
- Comment lines directly preceding an expression form a **comment block** attached to that expression.
- Blank lines between comment lines are preserved as separators within the block. The first and last line of a block must not be blank.
- `;;` (doc comment) before a `def` or `macro` registers the text in the symbol documentation registry (available via `--dump-docs`). Before other expressions it is allowed but has no registry effect.

## 2. Core Semantics & Data Types

- **Integers and Floats:** Standard 64-bit numeric types.
- **Strings:** Double-quoted text (e.g., `"Hello"`).
- **Keywords:** Prefixed with a colon (e.g., `:price`, `:volume`). They are interned constants often used as record keys.
- **Identifiers:** Variable bindings or function names. The special token `...` acts as a No-Op (`Nop`) in AST manipulation.
- **Field Accessors:** Prefixed with a dot (e.g., `.close`, `.name`). Evaluates to a first-class function that extracts the named field from a record. They can be passed to higher-order functions like `map` or used directly: `(.name user)`.

## 3. Data Structures & Series

- **Vectors (Tuples):** Enclosed in square brackets `[1 2 3]`. Evaluates to a vector/tuple structure.
- **Records:** Enclosed in curly braces with keyword keys and any expression as values `{:id 101 :name "Alice"}`. They provide O(1) field access using internal memory layouts. Structural equality `(= {:a 1} {:a 1})` is `true`.
- **Series:** A core concept for financial analysis. Series are "infinite" queues with a maximum length (lookback).
  - They are created via `(series lookback_limit)`. The element type is **inferred at compile time** from subsequent `push` calls (Hindley-Milner type inference) — no explicit schema is needed.
  - Example: `(series 100)` — the compiler infers the element type from usage.
  - **Indexing:** You access items in a series by calling it or an extracted field like a function with an integer index: `(my_series 0)`. **Crucially, index `0` represents the most recently pushed item.** Index `1` is the second most recent, and so on (lookback indexing).

## 4. Special Forms and Evaluation logic

- `(do expr...)`: Evaluates multiple expressions sequentially and returns the value of the last one.
- `(def pattern value)`: Binds a value to an identifier in the current scope. Supports deep pattern matching (destructuring) in the binding, e.g., `(def [a [b c]] [1 [2 3]])`.
- `(assign target value)`: Mutates an existing binding or field.
- `(if cond then else)`: Conditional evaluation. If `cond` is truthy, evaluates `then`, otherwise evaluates `else`. The `else` branch is optional.
- `(fn [params] body)`: Creates an anonymous closure. `params` uses a vector literal and supports destructuring natively. Example: `(fn [[x y]] (+ x y))`. Functions support lexical scoping (upvalues).
- `(again arg...)`: Triggers explicit Tail-Call Optimization (TCO), jumping back to the beginning of the closest enclosing function `(fn ...)` with new arguments. Example: `(again (- n 1) (* acc n))`. Similar to Clojure's `recur`.
- `(pipe inputs lambda)`: A specialized form designed for streaming or reactive evaluation. Takes a single expression resolving to inputs (often an array or a tuple) and passes them to the `lambda`. Heavily utilized in processing financial time series.
- `(call arg...)`: Any list where the head is not a special form evaluates the head as a function and calls it with the evaluated arguments.

## 5. Macros and Meta-Programming

The language provides a robust hygiene-aware macro system with explicit AST quotation forms:
- `(macro name [params] body)`: Defines a macro. During the compilation pass, macros run before normal execution.
- `'expr` (Quote): Returns the literal AST of `expr` without evaluating it.
- `` `expr `` (Template): Returns the AST of `expr` but allows selective evaluation inside it.
- `~expr` (Placeholder): Inside a `` ` `` template, evaluates `expr` and inserts its AST.
- `~@expr` (Splice): Inside a `` ` `` template, evaluates `expr` (must be a vector/tuple) and splices its elements directly into the enclosing list.
- `...` (Nop): Represents an empty or omitted AST node, heavily used in macro transformations like `(macro unless [c b] \`(if ~c ... ~b))`.

## 6. Standard Library (RTL) Overview

The Myc runtime environment provides a collection of built-in functions and macros:

### Core & Arithmetic
- **Math Operators:** `+`, `-`, `*`, `/`, `//` (integer division), `%` (modulo)
- **Comparison:** `=`, `<>`, `<`, `>`, `<=`, `>=`
- **Logic & Bitwise:** `and`, `or`, `xor`, `not`, `<<` (shift left), `>>` (shift right)

### Math
- **Unary:** `sqrt`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `exp`, `ln`, `log10`, `fract`
- **Rounding (Returns Int):** `round`, `floor`, `ceil`, `trunc`
- **Binary:** `atan2`, `pow`, `log`
- **Other:** `abs`, `min`, `max`, `make-random`

### Time & Dates
- `date`: Creates or parses a date.
- `now`: Returns the current timestamp.

### Series & Streaming
- `series`: Creates a new series. Takes a lookback limit; the element type is inferred at compile time from `push` calls.
- `push`: Pushes a new item into a series.
- `len`: Returns the current number of elements in a series.
- `create-random-ohlc`: Generates a mock Open-High-Low-Close data stream.
- `create-ticker`: Generates a mock ticker data stream.

### Standard Macros (Prelude)
- `(while cond body)`: Executes `body` repeatedly as long as `cond` is truthy.

## 7. Real-World Code Examples

**Records & First-Class Accessors:**
```clojure
(do
  (def user {:id 101 :name "Alice" :role :admin})
  ; Evaluates to :admin
  (def role (.role user))
)
```

**Financial Pipelines and Series (with Indexing):**
```clojure
(do
  ;; No schema needed — the compiler infers the element type from push calls.
  (def my_ticks (series 100))

  (push my_ticks {:price 10.5 :volume 100 :msg "A"})
  (push my_ticks {:price 11.2 :volume 200 :msg "B"})
  (push my_ticks {:price 15.5 :volume 300 :msg "C"})

  ;; Extract the price series specifically
  (def prices (.price my_ticks))

  ;; Series indexing: 0 is the newest (15.5), 1 is the previous (11.2)
  ;; Output: 26.7
  (+ (prices 0) (prices 1))
)
```

## 8. Common Mistakes & Pitfalls

This section is especially relevant for LLMs generating Myc code.

### Wrong keywords from other Lisps

| Wrong (Clojure/Scheme) | Correct Myc |
|---|---|
| `(let [x 1] ...)` | `(do (def x 1) ...)` |
| `(begin ...)` | `(do ...)` |
| `(lambda [x] ...)` | `(fn [x] ...)` |
| `(define x 1)` | `(def x 1)` |
| `(set! x 2)` | `(assign x 2)` |
| `(cond ...)` | nested `(if ...)` |
| `(not= a b)` | `(<> a b)` |

### Series indexing is reversed

Index `0` is the **most recently pushed** item, not the oldest. This is the opposite of normal array indexing:

```clojure
(push s {:v 1})
(push s {:v 2})
(push s {:v 3})
;; ((.v s) 0) => 3  (newest)
;; ((.v s) 1) => 2
;; ((.v s) 2) => 1  (oldest)
```

### Record keys must be keywords, not strings

```clojure
;; WRONG:
{"price" 10.5}

;; CORRECT:
{:price 10.5}
```

### `again` is only valid inside `fn`

`(again ...)` jumps back to the enclosing `fn`. Using it outside a function is a compile error.

```clojure
;; WRONG — top-level again:
(again 1 2)

;; CORRECT — inside fn:
(fn [n] (if (= n 0) "done" (again (- n 1))))
```

### `check_syntax` accepts only a single expression

The `check_syntax` tool uses the single-expression compiler pass. Wrap multiple expressions in `(do ...)`:

```clojure
;; WRONG:
(def x 1) (+ x 2)

;; CORRECT:
(do (def x 1) (+ x 2))
```

### Inline comments are not allowed

Comments must be **whole lines** starting with `;` or `;;`. Placing a comment after code on the same line is a compile error.

```clojure
;; WRONG — inline comment causes a compile error:
(def x 42) ; this is x

;; CORRECT — comment on its own line before the expression:
; this is x
(def x 42)
```

### Field accessors are functions, not property syntax

```clojure
;; WRONG — not valid syntax:
user.name

;; CORRECT — field accessor called as a function:
(.name user)

;; Also correct — extract accessor first, then call:
(def get-name .name)
(get-name user)
```

**Recursive Function with Tail-Call Optimization:**
```clojure
(do
  ;; Factorial using explicit TCO via (again ...) — like Clojure's recur
  (def factorial
    (fn [n acc]
      (if (= n 0)
        acc
        (again (- n 1) (* acc n)))))

  ;; Output: 3628800
  (factorial 10 1)
)
```

**Macro Definition:**
```clojure
(do
  ;; Define a macro that inverts a condition
  (macro unless [c body]
    `(if ~c ... ~body))

  ;; Output: "executed"
  (unless false "executed")
)
```

**Series with while Loop:**
```clojure
(do
  (def ticks (series 100))
  (def i 0)

  ;; Push 5 prices into the series
  (while (< i 5)
    (do
      (push ticks {:price (* i 1.5)})
      (assign i (+ i 1))))

  ;; Read back the two most recent prices
  (def p (.price ticks))
  ;; newest + second-newest
  (+ (p 0) (p 1))
)
```