RustAst/docs/BNF.md

Myc Language Specification & LLM Prompt Guide

This document defines the syntax (BNF) and semantics of the Myc language, a Lisp-like language designed for financial analysis with first-class abstract syntax trees (ASTs), closures, and a pipeline evaluation model. It serves as a foundational context for Large Language Models (LLMs) to understand and write idiomatic Myc code.

1. Syntax (BNF)

<program> ::= <expression>*

<expression> ::= <list>
               | <vector>
               | <record>
               | <quote>
               | <template>
               | <splice>
               | <placeholder>
               | <atom>

<atom> ::= <integer>
         | <float>
         | <string>
         | <keyword>
         | <identifier>
         | <field-accessor>
         | "..."

<list> ::= "(" <special-form> ")"
         | "(" <function-call> ")"

<special-form> ::= <if-form>
                 | <fn-form>
                 | <pipe-form>
                 | <again-form>
                 | <def-form>
                 | <assign-form>
                 | <do-form>
                 | <macro-form>

<function-call> ::= <expression> <expression>*

<if-form> ::= "if" <expression> <expression> [ <expression> ]
<fn-form> ::= "fn" <param-vector> <expression>
<param-vector> ::= "[" <pattern>* "]"
<pattern> ::= <identifier> | "[" <pattern>* "]"
<pipe-form> ::= "pipe" <expression> <expression>
<again-form> ::= "again" <expression>*
<def-form> ::= "def" <pattern> <expression>
<assign-form> ::= "assign" <expression> <expression>
<do-form> ::= "do" <expression>*
<macro-form> ::= "macro" <identifier> <param-vector> <expression>

<vector> ::= "[" <expression>* "]"
<record> ::= "{" ( <keyword> <expression> )* "}"

<quote> ::= "'" <expression>
<template> ::= "`" <expression>
<splice> ::= "~@" <expression>
<placeholder> ::= "~" <expression>

<integer> ::= [-]? [0-9]+
<float> ::= [-]? [0-9]+ "." [0-9]*
<string> ::= '"' [^"]* '"'
<keyword> ::= ":" <identifier>
<field-accessor> ::= "." <identifier>
<identifier> ::= <ident-start> <ident-char>*

(Lexical Note: <ident-start> consists of alphabetic characters or +-*/<=>!$%&_?.. <ident-char> includes digits as well. Identifiers evaluating to a field accessor begin with a . followed by one or more valid identifier characters.)

2. Core Semantics & Data Types

• Integers and Floats: Standard 64-bit numeric types.
• Strings: Double-quoted text (e.g., "Hello").
• Keywords: Prefixed with a colon (e.g., :price, :volume). They are interned constants often used as record keys.
• Identifiers: Variable bindings or function names. The special token ... acts as a No-Op (Nop) in AST manipulation.
• Field Accessors: Prefixed with a dot (e.g., .close, .name). Evaluates to a first-class function that extracts the named field from a record. They can be passed to higher-order functions like map or used directly: (.name user).

3. Data Structures & Series

• Vectors (Tuples): Enclosed in square brackets [1 2 3]. Evaluates to a vector/tuple structure.
• Records: Enclosed in curly braces with keyword keys and any expression as values {:id 101 :name "Alice"}. They provide O(1) field access using internal memory layouts. Structural equality (= {:a 1} {:a 1}) is true.
• Series: A core concept for financial analysis. Series are "infinite" queues with a maximum length (lookback).
  • They are created via the (series ...) function specifying a record layout (e.g., (series {:price :float :volume :int})).
  • Indexing: You access items in a series by calling it or an extracted field like a function with an integer index: (my_series 0). Crucially, index 0 represents the most recently pushed item. Index 1 is the second most recent, and so on (lookback indexing).

4. Special Forms and Evaluation logic

• (do expr...): Evaluates multiple expressions sequentially and returns the value of the last one.
• (def pattern value): Binds a value to an identifier in the current scope. Supports deep pattern matching (destructuring) in the binding, e.g., (def [a [b c]] [1 [2 3]]).
• (assign target value): Mutates an existing binding or field.
• (if cond then else): Conditional evaluation. If cond is truthy, evaluates then, otherwise evaluates else. The else branch is optional.
• (fn [params] body): Creates an anonymous closure. params uses a vector literal and supports destructuring natively. Example: (fn [[x y]] (+ x y)). Functions support lexical scoping (upvalues).
• (again arg...): Triggers explicit Tail-Call Optimization (TCO), jumping back to the beginning of the closest enclosing function (fn ...) with new arguments. Example: (again (- n 1) (* acc n)). Similar to Clojure's recur.
• (pipe inputs lambda): A specialized form designed for streaming or reactive evaluation. Takes a single expression resolving to inputs (often an array or a tuple) and passes them to the lambda. Heavily utilized in processing financial time series.
• (call arg...): Any list where the head is not a special form evaluates the head as a function and calls it with the evaluated arguments.

5. Macros and Meta-Programming

The language provides a robust hygiene-aware macro system with explicit AST quotation forms:

• (macro name [params] body): Defines a macro. During the compilation pass, macros run before normal execution.
• 'expr (Quote): Returns the literal AST of expr without evaluating it.
• `expr (Template): Returns the AST of expr but allows selective evaluation inside it.
• ~expr (Placeholder): Inside a ` template, evaluates expr and inserts its AST.
• ~@expr (Splice): Inside a ` template, evaluates expr (must be a vector/tuple) and splices its elements directly into the enclosing list.
• ... (Nop): Represents an empty or omitted AST node, heavily used in macro transformations like (macro unless [c b] \(if ~c ... ~b))`.

6. Standard Library (RTL) Overview

The Myc runtime environment provides a collection of built-in functions and macros:

Core & Arithmetic

• Math Operators: +, -, *, /, // (integer division), % (modulo)
• Comparison: =, <>, <, >, <=, >=
• Logic & Bitwise: and, or, xor, not, << (shift left), >> (shift right)

Math

• Unary: sqrt, sin, cos, tan, asin, acos, atan, exp, ln, log10, fract
• Rounding (Returns Int): round, floor, ceil, trunc
• Binary: atan2, pow, log
• Other: abs, min, max, make-random

Time & Dates

• date: Creates or parses a date.
• now: Returns the current timestamp.

Series & Streaming

• series: Creates a new series with a defined layout.
• push: Pushes a new item into a series.
• len: Returns the current number of elements in a series.
• create-random-ohlc: Generates a mock Open-High-Low-Close data stream.
• create-ticker: Generates a mock ticker data stream.

Standard Macros (Prelude)

• (while cond body): Executes body repeatedly as long as cond is truthy.

7. Real-World Code Examples

Records & First-Class Accessors:

(do
  (def user {:id 101 :name "Alice" :role :admin})
  (def role (.role user)) ; Evaluates to :admin
)

Financial Pipelines and Series (with Indexing):

(do
  ;; Create a series with a typed layout (Struct-of-Arrays under the hood)
  (def my_ticks (series {:price :float :volume :int :msg :text}))
  
  (push my_ticks {:price 10.5 :volume 100 :msg "A"})
  (push my_ticks {:price 11.2 :volume 200 :msg "B"})
  (push my_ticks {:price 15.5 :volume 300 :msg "C"})
  
  ;; Extract the price series specifically
  (def prices (.price my_ticks))

  ;; Series indexing: 0 is the newest (15.5), 1 is the previous (11.2)
  (+ (prices 0) (prices 1)) ;; Output: 26.7
)