RustAst/docs/BNF.md

Myc Language Specification & LLM Prompt Guide

This document defines the syntax (BNF) and semantics of the Myc language, a Lisp-like language designed for financial analysis with first-class abstract syntax trees (ASTs), closures, and a pipeline evaluation model. It serves as a foundational context for Large Language Models (LLMs) to understand and write idiomatic Myc code.

1. Syntax (BNF)

<program> ::= <expression>*

<expression> ::= <list>
               | <vector>
               | <record>
               | <quote>
               | <template>
               | <splice>
               | <placeholder>
               | <atom>

<atom> ::= <integer>
         | <float>
         | <string>
         | <keyword>
         | <identifier>
         | <field-accessor>
         | "..."

<list> ::= "(" <special-form> ")"
         | "(" <function-call> ")"

<special-form> ::= <if-form>
                 | <fn-form>
                 | <pipe-form>
                 | <again-form>
                 | <def-form>
                 | <assign-form>
                 | <do-form>
                 | <macro-form>

<function-call> ::= <expression> <expression>*

<if-form> ::= "if" <expression> <expression> [ <expression> ]
<fn-form> ::= "fn" <param-vector> <expression>
<param-vector> ::= "[" <pattern>* "]"
<pattern> ::= <identifier> | "[" <pattern>* "]"
<pipe-form> ::= "pipe" <expression> <expression>
<again-form> ::= "again" <expression>*
<def-form> ::= "def" <pattern> <expression>
<assign-form> ::= "assign" <expression> <expression>
<do-form> ::= "do" <expression>*
<macro-form> ::= "macro" <identifier> <param-vector> <expression>

<vector> ::= "[" <expression>* "]"
<record> ::= "{" ( <keyword> <expression> )* "}"

<quote> ::= "'" <expression>
<template> ::= "`" <expression>
<splice> ::= "~@" <expression>
<placeholder> ::= "~" <expression>

<integer> ::= [-]? [0-9]+
<float> ::= [-]? [0-9]+ "." [0-9]*
<string> ::= '"' [^"]* '"'
<keyword> ::= ":" <identifier>
<field-accessor> ::= "." <identifier>
<identifier> ::= <ident-start> <ident-char>*

(Lexical Note: <ident-start> consists of alphabetic characters or +-*/<=>!$%&_?.. <ident-char> includes digits as well. Identifiers evaluating to a field accessor begin with a . followed by one or more valid identifier characters.)

Comments

<comment>     ::= ";"  " "? <line-text> <newline>
<doc-comment> ::= ";;" " "? <line-text> <newline>

Rules:

• Comments must be whole lines — they start at the beginning of a line. Inline comments (after code on the same line) are a compile error.
• Comment lines directly preceding an expression form a comment block attached to that expression.
• Blank lines between comment lines are preserved as separators within the block. The first and last line of a block must not be blank.
• ;; (doc comment) before a def or macro registers the text in the symbol documentation registry (available via --dump-docs). Before other expressions it is allowed but has no registry effect.

2. Core Semantics & Data Types

• Integers and Floats: Standard 64-bit numeric types.
• Strings: Double-quoted text (e.g., "Hello").
• Keywords: Prefixed with a colon (e.g., :price, :volume). They are interned constants often used as record keys.
• Identifiers: Variable bindings or function names. The special token ... acts as a No-Op (Nop) in AST manipulation.
• Field Accessors: Prefixed with a dot (e.g., .close, .name). Evaluates to a first-class function that extracts the named field from a record. They can be passed to higher-order functions like map or used directly: (.name user).

3. Data Structures & Series

• Vectors (Tuples): Enclosed in square brackets [1 2 3]. Evaluates to a vector/tuple structure.
• Records: Enclosed in curly braces with keyword keys and any expression as values {:id 101 :name "Alice"}. They provide O(1) field access using internal memory layouts. Structural equality (= {:a 1} {:a 1}) is true.
• Series: A core concept for financial analysis. Series are "infinite" queues with a maximum length (lookback).
  • They are created via (series lookback_limit). The element type is inferred at compile time from subsequent push calls (Hindley-Milner type inference) — no explicit schema is needed.
  • Example: (series 100) — the compiler infers the element type from usage.
  • Indexing: You access items in a series by calling it or an extracted field like a function with an integer index: (my_series 0). Crucially, index 0 represents the most recently pushed item. Index 1 is the second most recent, and so on (lookback indexing).

4. Special Forms and Evaluation logic

• (do expr...): Evaluates multiple expressions sequentially and returns the value of the last one.
• (def pattern value): Binds a value to an identifier in the current scope. Supports deep pattern matching (destructuring) in the binding, e.g., (def [a [b c]] [1 [2 3]]).
• (assign target value): Mutates an existing binding or field.
• (if cond then else): Conditional evaluation. If cond is truthy, evaluates then, otherwise evaluates else. The else branch is optional.
• (fn [params] body): Creates an anonymous closure. params uses a vector literal and supports destructuring natively. Example: (fn [[x y]] (+ x y)). Functions support lexical scoping (upvalues).
• (again arg...): Triggers explicit Tail-Call Optimization (TCO), jumping back to the beginning of the closest enclosing function (fn ...) with new arguments. Example: (again (- n 1) (* acc n)). Similar to Clojure's recur.
• (pipe inputs lambda): A specialized form designed for streaming or reactive evaluation. Takes a single expression resolving to inputs (often an array or a tuple) and passes them to the lambda. Heavily utilized in processing financial time series.
• (call arg...): Any list where the head is not a special form evaluates the head as a function and calls it with the evaluated arguments.

5. Macros and Meta-Programming

The language provides a robust hygiene-aware macro system with explicit AST quotation forms:

• (macro name [params] body): Defines a macro. During the compilation pass, macros run before normal execution.
• 'expr (Quote): Returns the literal AST of expr without evaluating it.
• `expr (Template): Returns the AST of expr but allows selective evaluation inside it.
• ~expr (Placeholder): Inside a ` template, evaluates expr and inserts its AST.
• ~@expr (Splice): Inside a ` template, evaluates expr (must be a vector/tuple) and splices its elements directly into the enclosing list.
• ... (Nop): Represents an empty or omitted AST node, heavily used in macro transformations like (macro unless [c b] \(if ~c ... ~b))`.

6. Standard Library (RTL) Overview

The Myc runtime environment provides a collection of built-in functions and macros:

Core & Arithmetic

• Math Operators: +, -, *, /, // (integer division), % (modulo)
• Comparison: =, <>, <, >, <=, >=
• Logic & Bitwise: and, or, xor, not, << (shift left), >> (shift right)

Math

• Unary: sqrt, sin, cos, tan, asin, acos, atan, exp, ln, log10, fract
• Rounding (Returns Int): round, floor, ceil, trunc
• Binary: atan2, pow, log
• Other: abs, min, max, make-random

Time & Dates

• date: Creates or parses a date.
• now: Returns the current timestamp.

Series & Streaming

• series: Creates a new series. Takes a lookback limit; the element type is inferred at compile time from push calls.
• push: Pushes a new item into a series.
• len: Returns the current number of elements in a series.
• create-random-ohlc: Generates a mock Open-High-Low-Close data stream.
• create-ticker: Generates a mock ticker data stream.

Standard Macros (Prelude)

• (while cond body): Executes body repeatedly as long as cond is truthy.

7. Real-World Code Examples

Records & First-Class Accessors:

(do
  (def user {:id 101 :name "Alice" :role :admin})
  ; Evaluates to :admin
  (def role (.role user))
)

Financial Pipelines and Series (with Indexing):

(do
  ;; No schema needed — the compiler infers the element type from push calls.
  (def my_ticks (series 100))

  (push my_ticks {:price 10.5 :volume 100 :msg "A"})
  (push my_ticks {:price 11.2 :volume 200 :msg "B"})
  (push my_ticks {:price 15.5 :volume 300 :msg "C"})

  ;; Extract the price series specifically
  (def prices (.price my_ticks))

  ;; Series indexing: 0 is the newest (15.5), 1 is the previous (11.2)
  ;; Output: 26.7
  (+ (prices 0) (prices 1))
)

8. Common Mistakes & Pitfalls

This section is especially relevant for LLMs generating Myc code.

Wrong keywords from other Lisps

Wrong (Clojure/Scheme)	Correct Myc
(let [x 1] ...)	(do (def x 1) ...)
(begin ...)	(do ...)
(lambda [x] ...)	(fn [x] ...)
(define x 1)	(def x 1)
(set! x 2)	(assign x 2)
(cond ...)	nested (if ...)
(not= a b)	(<> a b)

Series indexing is reversed

Index 0 is the most recently pushed item, not the oldest. This is the opposite of normal array indexing:

(push s {:v 1})
(push s {:v 2})
(push s {:v 3})
;; ((.v s) 0) => 3  (newest)
;; ((.v s) 1) => 2
;; ((.v s) 2) => 1  (oldest)

Record keys must be keywords, not strings

;; WRONG:
{"price" 10.5}

;; CORRECT:
{:price 10.5}

again is only valid inside fn

(again ...) jumps back to the enclosing fn. Using it outside a function is a compile error.

;; WRONG — top-level again:
(again 1 2)

;; CORRECT — inside fn:
(fn [n] (if (= n 0) "done" (again (- n 1))))

check_syntax accepts only a single expression

The check_syntax tool uses the single-expression compiler pass. Wrap multiple expressions in (do ...):

;; WRONG:
(def x 1) (+ x 2)

;; CORRECT:
(do (def x 1) (+ x 2))

Inline comments are not allowed

Comments must be whole lines starting with ; or ;;. Placing a comment after code on the same line is a compile error.

;; WRONG — inline comment causes a compile error:
(def x 42) ; this is x

;; CORRECT — comment on its own line before the expression:
; this is x
(def x 42)

Field accessors are functions, not property syntax

;; WRONG — not valid syntax:
user.name

;; CORRECT — field accessor called as a function:
(.name user)

;; Also correct — extract accessor first, then call:
(def get-name .name)
(get-name user)

Recursive Function with Tail-Call Optimization:

(do
  ;; Factorial using explicit TCO via (again ...) — like Clojure's recur
  (def factorial
    (fn [n acc]
      (if (= n 0)
        acc
        (again (- n 1) (* acc n)))))

  ;; Output: 3628800
  (factorial 10 1)
)

Macro Definition:

(do
  ;; Define a macro that inverts a condition
  (macro unless [c body]
    `(if ~c ... ~body))

  ;; Output: "executed"
  (unless false "executed")
)

Series with while Loop:

(do
  (def ticks (series 100))
  (def i 0)

  ;; Push 5 prices into the series
  (while (< i 5)
    (do
      (push ticks {:price (* i 1.5)})
      (assign i (+ i 1))))

  ;; Read back the two most recent prices
  (def p (.price ticks))
  ;; newest + second-newest
  (+ (p 0) (p 1))
)