Brummel
903c77a665
Refine the BNF for `series` to explicitly include the `lookback_limit` and `schema`. Introduce scope management within the `bind` function for `if/else` branches to ensure correct context handling during compilation. Add `Again` and `GetField` node kinds to the `Specializer`. Improve lexer to ignore invisible characters within identifiers, demonstrated by a new test case.
314 lines
12 KiB
Markdown
314 lines
12 KiB
Markdown
# Myc Language Specification & LLM Prompt Guide
|
|
|
|
This document defines the syntax (BNF) and semantics of the Myc language, a Lisp-like language designed for financial analysis with first-class abstract syntax trees (ASTs), closures, and a pipeline evaluation model. It serves as a foundational context for Large Language Models (LLMs) to understand and write idiomatic Myc code.
|
|
|
|
## 1. Syntax (BNF)
|
|
|
|
```ebnf
|
|
<program> ::= <expression>*
|
|
|
|
<expression> ::= <list>
|
|
| <vector>
|
|
| <record>
|
|
| <quote>
|
|
| <template>
|
|
| <splice>
|
|
| <placeholder>
|
|
| <atom>
|
|
|
|
<atom> ::= <integer>
|
|
| <float>
|
|
| <string>
|
|
| <keyword>
|
|
| <identifier>
|
|
| <field-accessor>
|
|
| "..."
|
|
|
|
<list> ::= "(" <special-form> ")"
|
|
| "(" <function-call> ")"
|
|
|
|
<special-form> ::= <if-form>
|
|
| <fn-form>
|
|
| <pipe-form>
|
|
| <again-form>
|
|
| <def-form>
|
|
| <assign-form>
|
|
| <do-form>
|
|
| <macro-form>
|
|
|
|
<function-call> ::= <expression> <expression>*
|
|
|
|
<if-form> ::= "if" <expression> <expression> [ <expression> ]
|
|
<fn-form> ::= "fn" <param-vector> <expression>
|
|
<param-vector> ::= "[" <pattern>* "]"
|
|
<pattern> ::= <identifier> | "[" <pattern>* "]"
|
|
<pipe-form> ::= "pipe" <expression> <expression>
|
|
<again-form> ::= "again" <expression>*
|
|
<def-form> ::= "def" <pattern> <expression>
|
|
<assign-form> ::= "assign" <expression> <expression>
|
|
<do-form> ::= "do" <expression>*
|
|
<macro-form> ::= "macro" <identifier> <param-vector> <expression>
|
|
|
|
<vector> ::= "[" <expression>* "]"
|
|
<record> ::= "{" ( <keyword> <expression> )* "}"
|
|
|
|
<quote> ::= "'" <expression>
|
|
<template> ::= "`" <expression>
|
|
<splice> ::= "~@" <expression>
|
|
<placeholder> ::= "~" <expression>
|
|
|
|
<integer> ::= [-]? [0-9]+
|
|
<float> ::= [-]? [0-9]+ "." [0-9]*
|
|
<string> ::= '"' [^"]* '"'
|
|
<keyword> ::= ":" <identifier>
|
|
<field-accessor> ::= "." <identifier>
|
|
<identifier> ::= <ident-start> <ident-char>*
|
|
```
|
|
*(Lexical Note: `<ident-start>` consists of alphabetic characters or `+-*/<=>!$%&_?.`. `<ident-char>` includes digits as well. Identifiers evaluating to a field accessor begin with a `.` followed by one or more valid identifier characters.)*
|
|
|
|
### Comments
|
|
|
|
```ebnf
|
|
<comment> ::= ";" " "? <line-text> <newline>
|
|
<doc-comment> ::= ";;" " "? <line-text> <newline>
|
|
```
|
|
|
|
**Rules:**
|
|
- Comments must be **whole lines** — they start at the beginning of a line. Inline comments (after code on the same line) are a compile error.
|
|
- Comment lines directly preceding an expression form a **comment block** attached to that expression.
|
|
- Blank lines between comment lines are preserved as separators within the block. The first and last line of a block must not be blank.
|
|
- `;;` (doc comment) before a `def` or `macro` registers the text in the symbol documentation registry (available via `--dump-docs`). Before other expressions it is allowed but has no registry effect.
|
|
|
|
## 2. Core Semantics & Data Types
|
|
|
|
- **Integers and Floats:** Standard 64-bit numeric types.
|
|
- **Strings:** Double-quoted text (e.g., `"Hello"`).
|
|
- **Keywords:** Prefixed with a colon (e.g., `:price`, `:volume`). They are interned constants often used as record keys.
|
|
- **Identifiers:** Variable bindings or function names. The special token `...` acts as a No-Op (`Nop`) in AST manipulation.
|
|
- **Field Accessors:** Prefixed with a dot (e.g., `.close`, `.name`). Evaluates to a first-class function that extracts the named field from a record. They can be passed to higher-order functions like `map` or used directly: `(.name user)`.
|
|
|
|
## 3. Data Structures & Series
|
|
|
|
- **Vectors (Tuples):** Enclosed in square brackets `[1 2 3]`. Evaluates to a vector/tuple structure.
|
|
- **Records:** Enclosed in curly braces with keyword keys and any expression as values `{:id 101 :name "Alice"}`. They provide O(1) field access using internal memory layouts. Structural equality `(= {:a 1} {:a 1})` is `true`.
|
|
- **Series:** A core concept for financial analysis. Series are "infinite" queues with a maximum length (lookback).
|
|
- They are created via the `(series lookback_limit schema)` function. The `schema` can be a record layout (e.g., `{:price :float}`) or a type keyword (e.g., `:float`).
|
|
- Example: `(series 100 {:price :float :volume :int})`.
|
|
- **Indexing:** You access items in a series by calling it or an extracted field like a function with an integer index: `(my_series 0)`. **Crucially, index `0` represents the most recently pushed item.** Index `1` is the second most recent, and so on (lookback indexing).
|
|
|
|
## 4. Special Forms and Evaluation logic
|
|
|
|
- `(do expr...)`: Evaluates multiple expressions sequentially and returns the value of the last one.
|
|
- `(def pattern value)`: Binds a value to an identifier in the current scope. Supports deep pattern matching (destructuring) in the binding, e.g., `(def [a [b c]] [1 [2 3]])`.
|
|
- `(assign target value)`: Mutates an existing binding or field.
|
|
- `(if cond then else)`: Conditional evaluation. If `cond` is truthy, evaluates `then`, otherwise evaluates `else`. The `else` branch is optional.
|
|
- `(fn [params] body)`: Creates an anonymous closure. `params` uses a vector literal and supports destructuring natively. Example: `(fn [[x y]] (+ x y))`. Functions support lexical scoping (upvalues).
|
|
- `(again arg...)`: Triggers explicit Tail-Call Optimization (TCO), jumping back to the beginning of the closest enclosing function `(fn ...)` with new arguments. Example: `(again (- n 1) (* acc n))`. Similar to Clojure's `recur`.
|
|
- `(pipe inputs lambda)`: A specialized form designed for streaming or reactive evaluation. Takes a single expression resolving to inputs (often an array or a tuple) and passes them to the `lambda`. Heavily utilized in processing financial time series.
|
|
- `(call arg...)`: Any list where the head is not a special form evaluates the head as a function and calls it with the evaluated arguments.
|
|
|
|
## 5. Macros and Meta-Programming
|
|
|
|
The language provides a robust hygiene-aware macro system with explicit AST quotation forms:
|
|
- `(macro name [params] body)`: Defines a macro. During the compilation pass, macros run before normal execution.
|
|
- `'expr` (Quote): Returns the literal AST of `expr` without evaluating it.
|
|
- `` `expr `` (Template): Returns the AST of `expr` but allows selective evaluation inside it.
|
|
- `~expr` (Placeholder): Inside a `` ` `` template, evaluates `expr` and inserts its AST.
|
|
- `~@expr` (Splice): Inside a `` ` `` template, evaluates `expr` (must be a vector/tuple) and splices its elements directly into the enclosing list.
|
|
- `...` (Nop): Represents an empty or omitted AST node, heavily used in macro transformations like `(macro unless [c b] \`(if ~c ... ~b))`.
|
|
|
|
## 6. Standard Library (RTL) Overview
|
|
|
|
The Myc runtime environment provides a collection of built-in functions and macros:
|
|
|
|
### Core & Arithmetic
|
|
- **Math Operators:** `+`, `-`, `*`, `/`, `//` (integer division), `%` (modulo)
|
|
- **Comparison:** `=`, `<>`, `<`, `>`, `<=`, `>=`
|
|
- **Logic & Bitwise:** `and`, `or`, `xor`, `not`, `<<` (shift left), `>>` (shift right)
|
|
|
|
### Math
|
|
- **Unary:** `sqrt`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `exp`, `ln`, `log10`, `fract`
|
|
- **Rounding (Returns Int):** `round`, `floor`, `ceil`, `trunc`
|
|
- **Binary:** `atan2`, `pow`, `log`
|
|
- **Other:** `abs`, `min`, `max`, `make-random`
|
|
|
|
### Time & Dates
|
|
- `date`: Creates or parses a date.
|
|
- `now`: Returns the current timestamp.
|
|
|
|
### Series & Streaming
|
|
- `series`: Creates a new series with a defined layout.
|
|
- `push`: Pushes a new item into a series.
|
|
- `len`: Returns the current number of elements in a series.
|
|
- `create-random-ohlc`: Generates a mock Open-High-Low-Close data stream.
|
|
- `create-ticker`: Generates a mock ticker data stream.
|
|
|
|
### Standard Macros (Prelude)
|
|
- `(while cond body)`: Executes `body` repeatedly as long as `cond` is truthy.
|
|
|
|
## 7. Real-World Code Examples
|
|
|
|
**Records & First-Class Accessors:**
|
|
```clojure
|
|
(do
|
|
(def user {:id 101 :name "Alice" :role :admin})
|
|
; Evaluates to :admin
|
|
(def role (.role user))
|
|
)
|
|
```
|
|
|
|
**Financial Pipelines and Series (with Indexing):**
|
|
```clojure
|
|
(do
|
|
;; Create a series with a typed layout (Struct-of-Arrays under the hood)
|
|
(def my_ticks (series {:price :float :volume :int :msg :text}))
|
|
|
|
(push my_ticks {:price 10.5 :volume 100 :msg "A"})
|
|
(push my_ticks {:price 11.2 :volume 200 :msg "B"})
|
|
(push my_ticks {:price 15.5 :volume 300 :msg "C"})
|
|
|
|
;; Extract the price series specifically
|
|
(def prices (.price my_ticks))
|
|
|
|
;; Series indexing: 0 is the newest (15.5), 1 is the previous (11.2)
|
|
;; Output: 26.7
|
|
(+ (prices 0) (prices 1))
|
|
)
|
|
```
|
|
|
|
## 8. Common Mistakes & Pitfalls
|
|
|
|
This section is especially relevant for LLMs generating Myc code.
|
|
|
|
### Wrong keywords from other Lisps
|
|
|
|
| Wrong (Clojure/Scheme) | Correct Myc |
|
|
|---|---|
|
|
| `(let [x 1] ...)` | `(do (def x 1) ...)` |
|
|
| `(begin ...)` | `(do ...)` |
|
|
| `(lambda [x] ...)` | `(fn [x] ...)` |
|
|
| `(define x 1)` | `(def x 1)` |
|
|
| `(set! x 2)` | `(assign x 2)` |
|
|
| `(cond ...)` | nested `(if ...)` |
|
|
| `(not= a b)` | `(<> a b)` |
|
|
|
|
### Series indexing is reversed
|
|
|
|
Index `0` is the **most recently pushed** item, not the oldest. This is the opposite of normal array indexing:
|
|
|
|
```clojure
|
|
(push s {:v 1})
|
|
(push s {:v 2})
|
|
(push s {:v 3})
|
|
;; ((.v s) 0) => 3 (newest)
|
|
;; ((.v s) 1) => 2
|
|
;; ((.v s) 2) => 1 (oldest)
|
|
```
|
|
|
|
### Record keys must be keywords, not strings
|
|
|
|
```clojure
|
|
;; WRONG:
|
|
{"price" 10.5}
|
|
|
|
;; CORRECT:
|
|
{:price 10.5}
|
|
```
|
|
|
|
### `again` is only valid inside `fn`
|
|
|
|
`(again ...)` jumps back to the enclosing `fn`. Using it outside a function is a compile error.
|
|
|
|
```clojure
|
|
;; WRONG — top-level again:
|
|
(again 1 2)
|
|
|
|
;; CORRECT — inside fn:
|
|
(fn [n] (if (= n 0) "done" (again (- n 1))))
|
|
```
|
|
|
|
### `check_syntax` accepts only a single expression
|
|
|
|
The `check_syntax` tool uses the single-expression compiler pass. Wrap multiple expressions in `(do ...)`:
|
|
|
|
```clojure
|
|
;; WRONG:
|
|
(def x 1) (+ x 2)
|
|
|
|
;; CORRECT:
|
|
(do (def x 1) (+ x 2))
|
|
```
|
|
|
|
### Inline comments are not allowed
|
|
|
|
Comments must be **whole lines** starting with `;` or `;;`. Placing a comment after code on the same line is a compile error.
|
|
|
|
```clojure
|
|
;; WRONG — inline comment causes a compile error:
|
|
(def x 42) ; this is x
|
|
|
|
;; CORRECT — comment on its own line before the expression:
|
|
; this is x
|
|
(def x 42)
|
|
```
|
|
|
|
### Field accessors are functions, not property syntax
|
|
|
|
```clojure
|
|
;; WRONG — not valid syntax:
|
|
user.name
|
|
|
|
;; CORRECT — field accessor called as a function:
|
|
(.name user)
|
|
|
|
;; Also correct — extract accessor first, then call:
|
|
(def get-name .name)
|
|
(get-name user)
|
|
```
|
|
|
|
**Recursive Function with Tail-Call Optimization:**
|
|
```clojure
|
|
(do
|
|
;; Factorial using explicit TCO via (again ...) — like Clojure's recur
|
|
(def factorial
|
|
(fn [n acc]
|
|
(if (= n 0)
|
|
acc
|
|
(again (- n 1) (* acc n)))))
|
|
|
|
;; Output: 3628800
|
|
(factorial 10 1)
|
|
)
|
|
```
|
|
|
|
**Macro Definition:**
|
|
```clojure
|
|
(do
|
|
;; Define a macro that inverts a condition
|
|
(macro unless [c body]
|
|
`(if ~c ... ~body))
|
|
|
|
;; Output: "executed"
|
|
(unless false "executed")
|
|
)
|
|
```
|
|
|
|
**Series with while Loop:**
|
|
```clojure
|
|
(do
|
|
(def ticks (series {:price :float}))
|
|
(def i 0)
|
|
|
|
;; Push 5 prices into the series
|
|
(while (< i 5)
|
|
(do
|
|
(push ticks {:price (* i 1.5)})
|
|
(assign i (+ i 1))))
|
|
|
|
;; Read back the two most recent prices
|
|
(def p (.price ticks))
|
|
;; newest + second-newest
|
|
(+ (p 0) (p 1))
|
|
)
|
|
```
|