This commit introduces fundamental changes to Myc's language structure
to enhance type safety and scoping clarity.
Key changes include:
- Introducing local scopes for `do` blocks.
- Separating statements (`def`, `assign`) from expressions, where
statements now have no return value and are only allowed within
blocks.
- A block now consists of multiple statements followed by a single final
expression.
- Stricter validation rules are enforced, such as disallowing
redefinition of local symbols in the same scope while allowing
shadowing of outer symbols.
- Compiler errors are generated for statements used in expression
contexts.
The implementation involved modifications to the AST, parser, binder
(scope-stack), and various compiler passes, along with VM runtime
optimizations.
The `Parameter` kind in `UntypedKind` was only used for identifiers that
were being declared or referenced. This commit renames it to
`Identifier` and updates all the necessary code to reflect this change.
This simplifies the AST and makes it more consistent.
Additionally, a new macro `repeat` has been added to
`src/ast/system.myc`.
The `#use` directive now supports referencing entire directories,
automatically including all `.myc` files within them. Additionally,
environments are now initialized with the current working directory as a
default search path, simplifying module resolution.
Introduces the `#use` preprocessor directive, allowing Myc scripts to
explicitly declare dependencies on other Myc libraries. This change
moves dependency management from a command-line argument to an in-script
declaration, ensuring scripts are self-contained and can correctly
resolve macros and other symbols.
Key features include:
- Declarations at the beginning of a file, evaluated before parsing.
- Support for relative paths and a new `->` separator.
- Automatic resolution of dependencies from specified search paths.
- Idempotent loading to prevent duplicate parsing and evaluation.
- No AST pollution; dependency management is a compile-time concern.
The compiler's lexer has been updated to recognize `#` as a comment
character, and the environment now manages search paths and loaded
modules. This lays the groundwork for more complex library structures
and improved code organization.
This commit introduces the BNF for the Myc language, along with its core
semantics, data types, and evaluation logic. It also details the macro
system and provides an overview of the standard library. This document
serves as a foundational context for LLMs to understand and generate Myc
code.
Introduces type specialization for reactive pipeline nodes and their
data buffers. This eliminates the overhead of generic `Value` enums and
`SharedValueSeries` when dealing with known scalar or record types.
The architecture shifts buffer instantiation from the VM to the runtime
(RTL), leveraging type information from the AST. A new `out_type` field
is added to `BoundKind::Pipe` to store the static type of the pipeline's
output, determined by the type checker.
This enables the creation of specialized `RingBuffer<T>` and
`SharedRecordSeries` (for Struct-of-Arrays layout) when the output type
is known, significantly improving memory usage and processing speed for
time-series data, especially in financial analysis.
Introduce a new field to `Environment` to store a list of pipeline
generator functions.
Add a `run_pipeline` method to `Environment` that iterates through and
executes all registered generators until they are exhausted.
Implement `ObservableStream` trait for `RootStream` and `PipeStream`.
Add a `StreamNode` wrapper for `ObservableStream` to be used as a script
object.
Register `create-random-ohlc` as a native function that creates a
`RootStream`, sets up a OHLC record layout, and registers a stateful
generator closure in the environment's pipeline generators.
This commit introduces a detailed analysis of the Delphi Reactive Stream
Pipeline architecture, contrasting it with the current Rust
implementation. It outlines the core concepts of Delphi's stateless
streams, `TStreamSignal`, `TRootStream`, `TPipeSource`, `TScalarSeries`,
and `TPipeStream`, emphasizing their roles in event propagation, state
management, and synchronization.
The document also details the integration of the `pipe` statement within
the Delphi AST and compiler pipeline, covering the parser, AST
structure, type checker, and evaluator.
Finally, it proposes a "Dual Series Architecture" for Rust,
differentiating between script-local `RecordSeries` (for accumulators)
and reactive `SharedSeries` (for the pipeline), and outlines a concrete
implementation plan.
Introduce the `series` module to handle time-series data storage and
manipulation. This module includes:
- `RingBuffer`: An efficient ring buffer for storing time-series data
with a configurable lookback.
- `ScalarSeries`: A type-safe series for primitive scalar values (f64,
i64, bool) optimized for performance.
- `ValueSeries`: A fallback series for non-scalar or mixed data types.
- `RecordSeries`: A "Struct of Arrays" implementation for records,
splitting fields into parallel series for efficient access.
Add a new Markdown document detailing the AST node structure as it
evolves through the compiler's different stages. This includes a diagram
illustrating the data flow and relationships between the generic
`Node<K, T>` structure and its specialized forms (`UntypedNode`,
`BoundNode`, `TypedNode`, `AnalyzedNode`). The document explains the
`kind` and `ty` payloads for each stage and their significance.
Also removes a deleted file related to a previous optimization plan and
adds new files for future optimization plans.
Introduces a new `RecordLayout` system for efficient and type-safe
record handling.
Key changes:
- `RecordLayout` struct with `O(1)` field lookup using FMap
optimization.
- Field accessors (`.name`) are now first-class callable values.
- Parser recognizes dot-prefixed identifiers as field accessors.
- Binder and TypeChecker handle field accessors.
- Optimizer transforms field accessor calls into specialized `GetField`
nodes.
- VM executes `GetField` efficiently by directly accessing record
values.
- Adds a new `records.myc` example showcasing record features.
- Improves comparison logic for records to use pointer equality for
layout.
This commit introduces a new AST analysis pass that identifies function
purity and recursion. This information is then used by the optimizer and
specializer to make more informed decisions, particularly regarding
inlining.
The `Analyzer` struct and its associated `Analysis` struct are
responsible for traversing the AST and collecting this data.
Key changes include:
- A new `analyzer` module is added to `ast::compiler`.
- `Analyzer::analyze` performs a two-pass traversal to collect
global-to-lambda mappings and then analyze purity and recursion.
- The `Optimizer` and `Specializer` are updated to accept and utilize
the `Analysis` data.
- Recursion checks in `Optimizer` and `Specializer` are replaced with
checks against the pre-computed `Analysis.is_recursive` set.
- The `Environment` now stores and passes the `Analysis` results to the
compiler stages.
Introduces a new optimizer pass that can "crack" closures, allowing for
more aggressive specialization. It also enables inlining of upvalues
that point to immutable global variables. This removes overhead for
higher-order functions and currying when arguments are statically
resolvable.
correctly initialize the `TypeChecker` with argument types during
macro expansion.
- Refactor `Specializer::compile` to perform type checking with provided
arguments before specialization and to correctly extract the return
type.
- Enhance the `Dumper` to introspect and display specialized closure
bodies.
- Update `LambdaCollector` to use `BoundNode` consistently.
- Modify `TypeChecker` to accept and inject specialized argument types
for lambdas.
This commit introduces support for AST macros and templates, enabling
users to define and use custom, reusable components within the visual
DSL.
Key changes include:
- A new `MacroRegistry` to manage macro definitions.
- A `MacroExpander` to process macro calls and expand templates.
- A `MacroEvaluator` trait for evaluating expressions during expansion.
- The `Expansion` node in `BoundKind` to preserve the original macro
call and its expanded form for debugging.
- Updates to the `Binder`, `Dumper`, and `UpvalueAnalyzer` to handle the
new macro constructs.
- New examples demonstrating various macro functionalities like
`unless`, splicing, and nested macros.
Michael Schimmel