This commit introduces the `CompilationResult` struct to encapsulate the
outcome of a compilation pass. It holds either a successfully compiled
`TypedNode` or a `Diagnostics` object containing errors. Helper methods
for creating success or error results and for converting to a `Result`
are also included. This improves error handling and clarity in the
compilation process.
Introduce `allocate_slot` helper function to centralize the logic for
allocating new global slots. This improves code clarity and reduces
duplication by consolidating common operations like updating scopes,
types, purity, and values.
Introduces a new method `format_errors` to the `Diagnostics` struct.
This method iterates through all diagnostics, filters for errors, and
formats them into a newline-joined string. This simplifies error
reporting in compilation results.
This commit introduces support for unhygienic macro expansion using the
`~ident` syntax. When an identifier is prefixed with a tilde (`~`)
within a macro template, it is no longer subject to hygienic renaming.
Instead, it directly refers to a binding in the call site's environment.
This allows macros to interact with variables defined outside the
macro's scope, such as:
- Modifying call-site variables.
- Capturing call-site variables as upvalues in closures.
- Defining new variables that are visible in the call site.
A new test case `test_macro_tilde_nonparam_assign` has been added to
verify this functionality. Previously, unhygienic escapes were
implicitly handled by `SyntaxKind::Identifier` if they were not macro
parameters. This change makes this behavior explicit and correctly
handles non-parameter identifiers by returning a bare symbol.
Removes outdated tests for macro expansion, destructuring, and if branch
scope leaks. These tests were either superseded by newer, more accurate
tests or are no longer relevant to the current implementation of the
compiler's scoping rules. The remaining test correctly asserts that
defining the same variable in different 'if' branches is functional.
The `discover_globals` function has been refactored to better handle
pattern matching within definitions. It now uses a recursive helper
function `register_pattern` to correctly process identifiers and tuple
patterns. This ensures that all declared variables within a `Def` node
are registered with their corresponding scope and virtual ID.
Additionally, support for `Expansion` nodes has been added to ensure
that any nested global definitions within them are also discovered.
Refine the BNF for `series` to explicitly include the `lookback_limit`
and `schema`.
Introduce scope management within the `bind` function for `if/else`
branches to ensure correct context handling during compilation.
Add `Again` and `GetField` node kinds to the `Specializer`.
Improve lexer to ignore invisible characters within identifiers,
demonstrated by a new test case.
This commit updates all example files to use `assert_eq!` for verifying
output, replacing the previous `Output:` comments. This change makes the
examples more robust and self-testing.
The benchmark results in some examples have also been slightly adjusted,
reflecting minor performance variations after the refactoring.
Additionally, a runtime panic handling mechanism has been introduced in
the VM for function calls, which improves error reporting for unexpected
panics.
Introduce `free_slots` in `StackAllocator` to keep track of physical
slots that have been freed. When mapping a new virtual slot, the
allocator first attempts to reuse a slot from the `free_slots` vector
before allocating a new one.
This change also includes a new function `collect_scope_locals` that
identifies non-captured local variables defined within a block. After
these expressions are lowered, the allocator reclaims the physical slots
associated with these locals, making them available for reuse by
subsequent blocks.
A new test `test_slot_reuse_non_overlapping_scopes` is added to verify
that distinct scopes correctly reuse stack slots, ensuring optimal stack
usage.
The optimizer now correctly handles empty blocks by collapsing them into
Nop nodes. It also unwraps blocks containing a single expression, making
the AST more concise. This change improves AST simplification by
ensuring that redundant block structures are removed.
The explicit scope management within the `if`/`else` binding logic was
redundant.
The `bind` function recursively handles scope creation and destruction
for
expressions, making manual scope manipulation here unnecessary and
error-prone.
This commit introduces a new example script, `HMA.myc`, which
demonstrates the implementation and usage of the Hull Moving Average
(HMA) indicator.
The script includes factory functions for Simple Moving Average
(`make-sma`), Weighted Moving Average (`make-wma`), and the Hull Moving
Average itself (`make-hma`). It also includes a small example loop to
initialize and print HMA values.
Introduces a new AST emitter module responsible for serializing AST
nodes back into Myc source code. This emitter preserves leading comments
and whitespace, enabling a round-trip guarantee for the parser and
emitter.
The documentation in `docs/BNF.md` has been updated to reflect the new
syntax rules for comments, including single-line comments (`;`) and
documentation comments (`;;`). The rules clarify that comments must be
whole lines and that inline comments are disallowed.
The lexer and AST node structures have been updated to accommodate
`CommentLine` variants, allowing for different types of comments
(regular, documentation, and blank lines) to be stored and processed.
A new test suite `tests/comment_roundtrip.rs` has been added to verify
the round-trip property of the parser and emitter across all example
`.myc` files. This test ensures that parsing source code, emitting it,
and then parsing the emitted code again results in the same output
string.
The `ast.rs` binary now includes a `--emit` flag that uses the new
emitter to output the parsed source code, facilitating debugging and
testing of the emitter itself.
This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
Introduces `RtlDocEntry` and `RtlRegistration` structs to store and
manage documentation for native functions and constants. The
`Environment` struct is updated to include a registry for these
documentation entries.
The `register_native_fn` and `register_constant` methods now return an
`RtlRegistration` builder, allowing for optional chaining of `.doc()`,
`.description()`, and `.examples()` methods before the registration is
finalized when the builder is dropped.
Macros `rtl_fn!` and `rtl_const!` are introduced to simplify the
registration process with documentation.
The MCP server is extended with `get_rtl_docs` and `get_symbol_doc`
tools to expose this documentation externally.
Specific RTL functions and constants (e.g., `+`, `-`, `*`, `/`, `NaN`,
`true`, `false`, `date`, `now`, math functions, `series`, `push`, `len`,
`create-random-ohlc`, `create-ticker`, `pipe`) have been updated to
include their respective documentation using the new infrastructure.
The `Signature` and `StaticType` types have gained `to_doc_string`
methods for generating human-readable documentation strings.
Integrates the MCP server functionality into the AST tool binary. This
allows the tool to act as a server for LLM integration, enabling
communication over stdio. The server can list available bindings and
execute scripts.
This commit refactors the way series are represented and handled within
the AST.
Key changes include:
- Introducing `SeriesStorage` and `PushableStorage` traits to provide a
more
unified and type-safe interface for series data.
- Renaming `Object` trait and its methods to clarify that it's for RTL
extensions
other than series (like Streams).
- Updating the `Value` enum to have a distinct `Series` variant,
separating it
from `Object`.
- Adjusting various parts of the `VM` and `register` functions to work
with the
new series traits and `Value::Series` variant.
This change aims to improve the type system's clarity and safety when
dealing with
series data, aligning with Rust's best practices for trait design.
This commit introduces bidirectional type inference, enabling the
compiler to infer lambda parameter types based on the context of their
usage. This is particularly useful for functions like `pipe`, where a
lambda's signature can be deduced from the types of the streams it
operates on.
Key changes include:
- **`extract_lambda_param_hints` function:** This new helper function in
`type_checker.rs` is responsible for extracting expected parameter
types for a lambda based on the callee's signature and already known
argument types.
- **`check_lambda_with_param_hints` function:** This function allows
type-checking a lambda node using provided parameter type hints,
preserving upvalue types from the enclosing scope.
- **Call expression type checking:** The `check_call` function now
phases its argument type checking. Non-lambda arguments are typed
first, and then lambda arguments are typed using hints derived from
`extract_lambda_param_hints`.
- **`pipe_arg_hint_resolver`:** This new resolver for the `pipe`
function's `PolymorphicFn` type allows it to provide specific type
hints for its lambda argument based on the types of its stream inputs
(single stream, vector of streams, or tuple of streams).
- **`StaticType::PolymorphicFn` update:** The `PolymorphicFn` enum
variant has been extended to include `resolve_arg_hints`, enabling
functions to provide lambda parameter hints during bidirectional type
inference.
- **New tests:** Added tests to verify the correct inference of lambda
parameters for `pipe` with vector and tuple inputs, ensuring that
inner stream types are correctly propagated. Also, a test to confirm
`pipe` with an optional return still produces `Stream(Float)`.
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
This commit reorganizes the series-related code by moving the
`RingBuffer`, `ScalarSeries`, `ValueSeries`, `RecordSeries`,
`SeriesView`, and `SharedSeries` structs into a new
`src/ast/rtl/series/data.rs` file.
The `src/ast/rtl/series/mod.rs` file now contains the module
declaration, re-exports, and the `register` function for registering
series-related native functions with the environment. This improves code
organization and maintainability.
Replaces `Value::Object(Rc<dyn Object>)` for SyntaxNodes with a
dedicated `Value::Quote(Rc<SyntaxNode>)`. This provides better type
safety and clarity when dealing with AST nodes as values.
Refactor Value enum for Quote node
This commit introduces a new `Value::Quote` variant to represent quoted
AST nodes directly, replacing the generic `Value::Object` for this
purpose.
This change simplifies the handling of quoted nodes by:
- Directly storing an `Rc<SyntaxNode>` instead of relying on dynamic
downcasting from `Rc<dyn Object>`.
- Streamlining macro expansion logic by removing the need to check for
`SyntaxNode` within `Value::Object`.
- Improving type safety and explicitness in the `Value` enum.
Introduce a new trait `PushableSeries` to distinguish objects that can
have values pushed into them from those that are only readable series.
This clarifies the API and allows for more precise type checking.
The `Object` trait no longer has a default `push_value` implementation.
Instead, `PushableSeries` now provides this functionality.
`SeriesMember` has also been updated to require `PushableSeries` for
RecordSeries field buffers.
Previously, compiled closures were stored as `Value::Object` and then
downcasted. This commit introduces a dedicated `Value::Closure` enum
variant to represent compiled closures. This improves type safety and
simplifies handling of closures throughout the compiler and VM.
This change involves:
- Updating type definitions to use `Value::Closure`.
- Adjusting downcasting logic to directly access the `Closure` struct.
- Modifying `run_with_args` and `run_with_args_observed` to expect
`Value::Closure` for tail-call targets.
- Refining the handling of closures in the `Optimizer` and `Inliner` for
better type clarity.
The `Closure` struct has been moved from `ast/vm.rs` to a new module
`ast/closure.rs`. This improves the organization of the AST nodes. The
`Closure` struct represents a compiled function body along with its
captured upvalues, which is a key feature for Myc Script.
The `TailCallRequest` enum variant and its associated logic have been
removed. Instead, the VM now uses an `Option<(Rc<dyn Object>,
Vec<Value>)>` field named `tail_call` to manage tail-call requests. This
change simplifies the `Value` enum and centralizes tail-call handling
within the `VM` struct.
This commit introduces the `into_rc_any` and `push_value` methods to the
`Object` trait.
The `into_rc_any` method allows for efficient, zero-cost downcasting of
`Rc<Self>` to `Rc<dyn Any>`. This is a crucial step for enabling more
flexible dynamic dispatch and type manipulation within the VM.
The `push_value` method is implemented for series types (`ScalarSeries`,
`ValueSeries`, `RecordSeries`) to allow pushing values directly onto
them. This simplifies the `push` intrinsic function, removing the need
for manual downcasting within its implementation. The default
implementation for non-series objects panics, enforcing that only
mutable series support this operation.
Additionally, `ScalarValue` now includes a `from_value` method to
facilitate converting `Value` back into concrete scalar types. This
improves type safety and reduces boilerplate code when handling scalar
conversions.
The `Value::Object` variant in `ast/types.rs` has been updated to
provide a more specific display format for `SharedValueSeries`.
Previously, it would generically display the object's type name. This
change allows it to directly show the length of the `SharedValueSeries`
buffer, which is more informative.
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.
Specifically, the following changes were made:
- In `analyzer.rs`, `crate::ast::types::Identity` and
`crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
`crate::ast::types::SourceLocation`,
`crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
`SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
`RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
`AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
`StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
`PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
`build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
`Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
`build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.
The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
This commit refactors the test suite by removing the monolithic
`integration_test.rs` file.
The tests have been categorized and moved into dedicated modules within
the `tests/` directory:
- `tests/parser.rs`: Contains tests for parsing various literal values.
- `tests/destructuring.rs`: Houses tests related to destructuring
syntax.
- `tests/closures.rs`: Includes tests for closure behavior and related
optimizations.
- `tests/optimizer.rs`: Contains tests specifically targeting optimizer
logic and regressions.
- `tests/records.rs`: Holds tests for record syntax and associated
optimizations.
- `tests/rtl.rs`: Contains tests for runtime library operators and
functions.
- `tests/pipeline.rs`: Tests for pipeline functionality.
- `tests/macros.rs`: Tests for macro expansion and related issues.
- `tests/error_recovery.rs`: Includes tests for compiler error handling
and recovery mechanisms.
- `tests/examples.rs`: Verifies the execution of example scripts.
This reorganization improves test discoverability and maintainability.
The `lib.rs` file has also been updated to reflect these changes by
removing the reference to the old `integration_test` module.
The `type_checker.rs` file has had some tests removed, as they are now
covered by the more specific tests in `tests/error_recovery.rs`.
Introduce `collect_pattern_addrs` to gather all bound addresses from a
destructuring pattern. This enables the optimizer to remove unused
destructuring definitions, similar to how simple unused definitions are
handled. Added integration tests to verify this behavior and other
optimizer robustness scenarios.
The Binder and related types have been refactored to use the new
`NodeKind` enum instead of the previous `BoundKind`. This commit updates
all references to use the new structure, ensuring consistency across the
compiler's AST representation.
Key changes include:
- Replacing `BoundKind` with `NodeKind` in the Binder's `bind` and
`visit` methods.
- Updating pattern matching and field access to reflect the new enum
variants (e.g., `NodeKind::Def` instead of `BoundKind::Define`).
- Adjusting identifier bindings to use the new `IdentifierBinding` enum.
- Reflecting changes in `DefBinding`, `AssignBinding`, and
`LambdaBinding` structures.
- Ensuring all newly created nodes use `NodeKind` and the appropriate
metadata.
The `AssignBinding` struct now uses `Option<Address<L>>` to
differentiate between simple assignments and destructuring assignments.
For destructuring, the addresses are managed by the `Identifier` nodes
within the target pattern.
A new `GetField` node kind has been introduced to represent optimized
field access, which is generated during the lowering phase for the
`RuntimePhase`. This change aids in more efficient code generation for
accessing fields.
This commit introduces two new files:
- `.claudeignore`: Specifies files and directories that should be
ignored by Claude, such as snapshot files and the `target` directory.
- `docs/bytecode-architecture.md`: Documents the architecture for
bytecode generation, outlining the hybrid execution model, the
transformation pipeline (Specializer and Lowering), and the role of
Tail Call Optimization (TCO).
This commit introduces a new `NodeKind` enum that serves as a unified
representation for AST nodes across different compilation phases. It
replaces the separate `SyntaxKind` and `BoundKind` enums, allowing
phase-specific information to be attached through generic type
parameters.
This change simplifies the AST structure and makes it easier to manage
node information consistently throughout the compilation process. The
`NodeKind` enum now holds all possible AST node variants, with
phase-specific data encapsulated within `CompilerPhase` trait
implementations.
This commit reduces unnecessary cloning for fields like `addr`, `kind`,
and `field` within `BoundKind` and `VM::GetField`. This improves
performance by avoiding redundant memory allocations.
This commit refactors the AST substitution logic to correctly handle
nested expansions and improve the inlining of lambda expressions.
The changes include:
- Extracting the core value from potentially expanded AST nodes before
checking their kind.
- Ensuring that lambda expressions with empty upvalues are correctly
substituted.
- Adding support for inlining global get expressions as AST
substitutions.
- Improving the `UsageInfo` collection to correctly track assigned
values.
Introduce generic `CompilerPhase` trait to unify different stages of the
bound AST. Rename `LocalSlot` to `VirtualId` and introduce `StackOffset`
for clearer distinction between compile-time and run-time addressing.
Update type aliases and implementations to reflect these changes.
The VM's field access logic has been refactored to clone `Rc` pointers
for addresses and field identifiers. This change aims to improve
efficiency by avoiding unnecessary dereferencing and copying of values
when accessing fields within records and objects.
Additionally, the `Value::Object` handling in `BoundKind::GetField` has
been refined to more explicitly manage `RecordSeries` and `StreamNode`
types, ensuring that field access is handled polymorphically and
efficiently without copying underlying data structures.
This commit makes several changes related to how closures are handled:
- **Dumper:** Removes redundant introspection of closure ASTs, as this
is no longer relevant.
- **Optimizer:** Updates `try_inline` to accept `ExecNode` for
parameters and adds a check to ensure the `function_node` is a
`Lambda` before attempting inlining.
- **Environment:** Simplifies closure creation by passing an `ExecNode`
for parameters and ensuring the correct `stack_size` is used.
- **VM:** Adjusts `Closure` to store an `ExecNode` for parameters and
updates `VM::unpack` to accept an `ExecNode`.
Introduces a `StackAllocator` to manage local slot mapping during
lowering. This allows for efficient calculation of stack sizes for
lambdas and the root node. Tail position marking is now handled more
accurately.
Brummel
Michael Schimmel