Introduces `CallHooks` to enable extensions to the type inference and
AST finalization process. This allows for more dynamic and configurable
behavior without hardcoding symbol names in the compiler core.
Specifically, this commit adds:
- **`series_post_call`**: A hook for the `(series n)` function. It
ensures that the return type is a fresh `TypeVar` when the element
type is `Any`, allowing for proper type inference from subsequent
`push` calls.
- **`series_finalize`**: A hook for `(series n)` that rewrites the AST
node. It replaces the generic `series` call with a pre-configured
factory closure that allocates the correct series storage type (e.g.,
`ScalarSeries<f64>` for floats, `RecordSeries` for records). This
avoids runtime dispatch and relies on type information captured during
compilation.
- **`push_post_call`**: A hook for the `(push series val)` function. It
unifies the series element `TypeVar` with the pushed value's type. It
also handles numeric and record field promotion and updates the
`TypeVar` binding if a promotion occurs.
Introduce `ModuleLoader` to handle `#use` directives and dependency
resolution. This involves adding `tempfile` dependency, modifying
`Cargo.toml` and `Cargo.lock`, creating a new `module_loader.rs` file,
and updating `environment.rs` to use the new module.
Also, add comprehensive tests for `Diagnostics::format_errors`,
`CompilationResult`, and the new `ModuleLoader` functionality. These
tests cover various scenarios including empty diagnostics, multiple
errors, error formatting, compilation success/failure, and complex
dependency loading with topological ordering and search path resolution.
This commit introduces the `CompilationResult` struct to encapsulate the
outcome of a compilation pass. It holds either a successfully compiled
`TypedNode` or a `Diagnostics` object containing errors. Helper methods
for creating success or error results and for converting to a `Result`
are also included. This improves error handling and clarity in the
compilation process.
Introduces a new AST emitter module responsible for serializing AST
nodes back into Myc source code. This emitter preserves leading comments
and whitespace, enabling a round-trip guarantee for the parser and
emitter.
The documentation in `docs/BNF.md` has been updated to reflect the new
syntax rules for comments, including single-line comments (`;`) and
documentation comments (`;;`). The rules clarify that comments must be
whole lines and that inline comments are disallowed.
The lexer and AST node structures have been updated to accommodate
`CommentLine` variants, allowing for different types of comments
(regular, documentation, and blank lines) to be stored and processed.
A new test suite `tests/comment_roundtrip.rs` has been added to verify
the round-trip property of the parser and emitter across all example
`.myc` files. This test ensures that parsing source code, emitting it,
and then parsing the emitted code again results in the same output
string.
The `ast.rs` binary now includes a `--emit` flag that uses the new
emitter to output the parsed source code, facilitating debugging and
testing of the emitter itself.
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.
The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
This commit replaces the `UntypedNode` enum with the more accurately
named `SyntaxNode`. This change is primarily for clarity and better
reflects the role of these nodes as representing the structure of the
source code prior to semantic analysis.
The corresponding enum `UntypedKind` has also been renamed to
`SyntaxKind` to maintain consistency.
No functional changes are introduced by this refactoring; it is purely a
renaming and organizational update.
The TCO (Tail Call Optimization) module has been renamed to `lowering`.
This change better reflects the module's broader responsibility, which
includes not only TCO but also general AST transformations and
preparation for VM execution.
The `optimize` function has been renamed to `lower` to align with the
module's new name.
The `UpvalueAnalyzer` pass is no longer necessary as the binder now
directly tracks captures. This commit removes the `UpvalueAnalyzer`
struct and associated logic from `src/ast/compiler/upvalues.rs`.
The `Binder::bind_root` function now returns the `captures` map, which
is then processed by a new `CapturePass` in `src/ast/environment.rs`
before type checking. This consolidates capture logic within the binder
and its subsequent processing steps.
This commit introduces a new AST analysis pass that identifies function
purity and recursion. This information is then used by the optimizer and
specializer to make more informed decisions, particularly regarding
inlining.
The `Analyzer` struct and its associated `Analysis` struct are
responsible for traversing the AST and collecting this data.
Key changes include:
- A new `analyzer` module is added to `ast::compiler`.
- `Analyzer::analyze` performs a two-pass traversal to collect
global-to-lambda mappings and then analyze purity and recursion.
- The `Optimizer` and `Specializer` are updated to accept and utilize
the `Analysis` data.
- Recursion checks in `Optimizer` and `Specializer` are replaced with
checks against the pre-computed `Analysis.is_recursive` set.
- The `Environment` now stores and passes the `Analysis` results to the
compiler stages.
Introduces a new optimizer pass that can "crack" closures, allowing for
more aggressive specialization. It also enables inlining of upvalues
that point to immutable global variables. This removes overhead for
higher-order functions and currying when arguments are statically
resolvable.
Introduces `LambdaCollector` to gather lambda functions and populate the
function registry. This enables the `Specializer` to work with
user-defined functions.
The `Environment` struct is updated to manage the `function_registry`
and `monomorph_cache`, which are essential for the specialization
process.
The `link` method in `Environment` now incorporates lambda collection
and node specialization before applying TCO optimization. This ensures
that lambdas are properly processed and specialized for potential
performance gains.
The `Specializer`'s `new` constructor has been modified to accept and
initialize the `MonoCache` through an `Rc<RefCell<MonoCache>>`. This
allows the cache to be shared across different specialized functions.
Also includes minor refactoring and type adjustments in `specializer.rs`
for better clarity and consistency.
Update the testing section to clarify when warnings should be eliminated
and tests run.
Add the new `specializer` module to the compiler's public API.
Add the `type_registry` module to the `rtl` module's public API.
Introduces the `TypeChecker` struct and its associated `TypeContext` for
performing static type analysis on the abstract syntax tree.
This change includes:
- A new `type_checker.rs` module.
- Integration of `TypeChecker` into the `Environment::compile` method.
- Updates to `Environment` to manage global types.
- Renaming `bound_nodes.rs`'s `BoundNode` to `TypedNode` to reflect its
type-checked nature.
- Refinements in binder and macro expansion to accommodate type
information.
This commit introduces support for AST macros and templates, enabling
users to define and use custom, reusable components within the visual
DSL.
Key changes include:
- A new `MacroRegistry` to manage macro definitions.
- A `MacroExpander` to process macro calls and expand templates.
- A `MacroEvaluator` trait for evaluating expressions during expansion.
- The `Expansion` node in `BoundKind` to preserve the original macro
call and its expanded form for debugging.
- Updates to the `Binder`, `Dumper`, and `UpvalueAnalyzer` to handle the
new macro constructs.
- New examples demonstrating various macro functionalities like
`unless`, splicing, and nested macros.
The `UpvalueAnalyzer` now correctly identifies which lambdas capture
which variable declarations, rather than just marking declarations that
need boxing. This information is stored in a `capture_map`.
The `Binder` has been updated to use this new `capture_map` instead of a
`HashSet` of boxed declarations. The `DefLocal` node in `bound_nodes.rs`
now stores a `captured_by` field, which is a list of lambda identities
that capture the local variable.
A new `dumper` module has been added to provide a human-readable
representation of the bound AST, including information about captured
variables.
The `VM` has been updated to use the `captured_by` field to determine if
a local variable needs to be stored in a `Cell`, rather than relying on
a boolean `is_boxed` flag.
An example script `extreme_capture.myc` has been added to test deep
nesting and variable capture.
A new "Dump AST" button has been added to the UI, which uses the new
`Dumper` to display the bound AST.
The binder now uses an `UpvalueAnalyzer` to identify local variables
that are captured by nested functions. These identified variables are
marked with `is_boxed: true` during `DefLocal` node creation. The VM
then uses this flag to wrap such variables in a `Value::Cell` (using
`Rc<RefCell<_>>`) to ensure they can be mutated across function calls.
feat: Introduce boxing for captured variables
Add UpvalueAnalyzer to identify variables captured by nested lambdas.
Modify Binder to use the analyzer and mark captured local variables for
boxing.
Update BoundKind::DefLocal to include an `is_boxed` flag.
Update VM to box captured variables when they are defined.
Add tests for upvalue capture detection.
Adds a new `Binder` struct that traverses the AST and resolves variable
references to concrete `Address` types (Local, Upvalue, Global). This
information is crucial for the Virtual Machine's execution phase.
Introduces the `BoundKind` enum to represent the AST after binding.
The `VM` is updated to handle `BoundKind` nodes and execute the bound
AST.
It now manages a call stack, local variables, and closures for function
calls and upvalue capturing.
The `Environment` struct is enhanced to manage global variables and
provide
a unified interface for parsing, binding, and executing scripts. It also
includes basic standard library functions.
Brummel
Michael Schimmel