This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
Introduces `RtlDocEntry` and `RtlRegistration` structs to store and
manage documentation for native functions and constants. The
`Environment` struct is updated to include a registry for these
documentation entries.
The `register_native_fn` and `register_constant` methods now return an
`RtlRegistration` builder, allowing for optional chaining of `.doc()`,
`.description()`, and `.examples()` methods before the registration is
finalized when the builder is dropped.
Macros `rtl_fn!` and `rtl_const!` are introduced to simplify the
registration process with documentation.
The MCP server is extended with `get_rtl_docs` and `get_symbol_doc`
tools to expose this documentation externally.
Specific RTL functions and constants (e.g., `+`, `-`, `*`, `/`, `NaN`,
`true`, `false`, `date`, `now`, math functions, `series`, `push`, `len`,
`create-random-ohlc`, `create-ticker`, `pipe`) have been updated to
include their respective documentation using the new infrastructure.
The `Signature` and `StaticType` types have gained `to_doc_string`
methods for generating human-readable documentation strings.
Integrates the MCP server functionality into the AST tool binary. This
allows the tool to act as a server for LLM integration, enabling
communication over stdio. The server can list available bindings and
execute scripts.
This commit refactors the way series are represented and handled within
the AST.
Key changes include:
- Introducing `SeriesStorage` and `PushableStorage` traits to provide a
more
unified and type-safe interface for series data.
- Renaming `Object` trait and its methods to clarify that it's for RTL
extensions
other than series (like Streams).
- Updating the `Value` enum to have a distinct `Series` variant,
separating it
from `Object`.
- Adjusting various parts of the `VM` and `register` functions to work
with the
new series traits and `Value::Series` variant.
This change aims to improve the type system's clarity and safety when
dealing with
series data, aligning with Rust's best practices for trait design.
This commit introduces bidirectional type inference, enabling the
compiler to infer lambda parameter types based on the context of their
usage. This is particularly useful for functions like `pipe`, where a
lambda's signature can be deduced from the types of the streams it
operates on.
Key changes include:
- **`extract_lambda_param_hints` function:** This new helper function in
`type_checker.rs` is responsible for extracting expected parameter
types for a lambda based on the callee's signature and already known
argument types.
- **`check_lambda_with_param_hints` function:** This function allows
type-checking a lambda node using provided parameter type hints,
preserving upvalue types from the enclosing scope.
- **Call expression type checking:** The `check_call` function now
phases its argument type checking. Non-lambda arguments are typed
first, and then lambda arguments are typed using hints derived from
`extract_lambda_param_hints`.
- **`pipe_arg_hint_resolver`:** This new resolver for the `pipe`
function's `PolymorphicFn` type allows it to provide specific type
hints for its lambda argument based on the types of its stream inputs
(single stream, vector of streams, or tuple of streams).
- **`StaticType::PolymorphicFn` update:** The `PolymorphicFn` enum
variant has been extended to include `resolve_arg_hints`, enabling
functions to provide lambda parameter hints during bidirectional type
inference.
- **New tests:** Added tests to verify the correct inference of lambda
parameters for `pipe` with vector and tuple inputs, ensuring that
inner stream types are correctly propagated. Also, a test to confirm
`pipe` with an optional return still produces `Stream(Float)`.
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
This commit reorganizes the series-related code by moving the
`RingBuffer`, `ScalarSeries`, `ValueSeries`, `RecordSeries`,
`SeriesView`, and `SharedSeries` structs into a new
`src/ast/rtl/series/data.rs` file.
The `src/ast/rtl/series/mod.rs` file now contains the module
declaration, re-exports, and the `register` function for registering
series-related native functions with the environment. This improves code
organization and maintainability.
Replaces `Value::Object(Rc<dyn Object>)` for SyntaxNodes with a
dedicated `Value::Quote(Rc<SyntaxNode>)`. This provides better type
safety and clarity when dealing with AST nodes as values.
Refactor Value enum for Quote node
This commit introduces a new `Value::Quote` variant to represent quoted
AST nodes directly, replacing the generic `Value::Object` for this
purpose.
This change simplifies the handling of quoted nodes by:
- Directly storing an `Rc<SyntaxNode>` instead of relying on dynamic
downcasting from `Rc<dyn Object>`.
- Streamlining macro expansion logic by removing the need to check for
`SyntaxNode` within `Value::Object`.
- Improving type safety and explicitness in the `Value` enum.
Introduce a new trait `PushableSeries` to distinguish objects that can
have values pushed into them from those that are only readable series.
This clarifies the API and allows for more precise type checking.
The `Object` trait no longer has a default `push_value` implementation.
Instead, `PushableSeries` now provides this functionality.
`SeriesMember` has also been updated to require `PushableSeries` for
RecordSeries field buffers.
Previously, compiled closures were stored as `Value::Object` and then
downcasted. This commit introduces a dedicated `Value::Closure` enum
variant to represent compiled closures. This improves type safety and
simplifies handling of closures throughout the compiler and VM.
This change involves:
- Updating type definitions to use `Value::Closure`.
- Adjusting downcasting logic to directly access the `Closure` struct.
- Modifying `run_with_args` and `run_with_args_observed` to expect
`Value::Closure` for tail-call targets.
- Refining the handling of closures in the `Optimizer` and `Inliner` for
better type clarity.
The `Closure` struct has been moved from `ast/vm.rs` to a new module
`ast/closure.rs`. This improves the organization of the AST nodes. The
`Closure` struct represents a compiled function body along with its
captured upvalues, which is a key feature for Myc Script.
The `TailCallRequest` enum variant and its associated logic have been
removed. Instead, the VM now uses an `Option<(Rc<dyn Object>,
Vec<Value>)>` field named `tail_call` to manage tail-call requests. This
change simplifies the `Value` enum and centralizes tail-call handling
within the `VM` struct.
This commit introduces the `into_rc_any` and `push_value` methods to the
`Object` trait.
The `into_rc_any` method allows for efficient, zero-cost downcasting of
`Rc<Self>` to `Rc<dyn Any>`. This is a crucial step for enabling more
flexible dynamic dispatch and type manipulation within the VM.
The `push_value` method is implemented for series types (`ScalarSeries`,
`ValueSeries`, `RecordSeries`) to allow pushing values directly onto
them. This simplifies the `push` intrinsic function, removing the need
for manual downcasting within its implementation. The default
implementation for non-series objects panics, enforcing that only
mutable series support this operation.
Additionally, `ScalarValue` now includes a `from_value` method to
facilitate converting `Value` back into concrete scalar types. This
improves type safety and reduces boilerplate code when handling scalar
conversions.
The `Value::Object` variant in `ast/types.rs` has been updated to
provide a more specific display format for `SharedValueSeries`.
Previously, it would generically display the object's type name. This
change allows it to directly show the length of the `SharedValueSeries`
buffer, which is more informative.
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.
Specifically, the following changes were made:
- In `analyzer.rs`, `crate::ast::types::Identity` and
`crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
`crate::ast::types::SourceLocation`,
`crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
`SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
`RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
`AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
`StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
`PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
`build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
`Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
`build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.
The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
This commit refactors the test suite by removing the monolithic
`integration_test.rs` file.
The tests have been categorized and moved into dedicated modules within
the `tests/` directory:
- `tests/parser.rs`: Contains tests for parsing various literal values.
- `tests/destructuring.rs`: Houses tests related to destructuring
syntax.
- `tests/closures.rs`: Includes tests for closure behavior and related
optimizations.
- `tests/optimizer.rs`: Contains tests specifically targeting optimizer
logic and regressions.
- `tests/records.rs`: Holds tests for record syntax and associated
optimizations.
- `tests/rtl.rs`: Contains tests for runtime library operators and
functions.
- `tests/pipeline.rs`: Tests for pipeline functionality.
- `tests/macros.rs`: Tests for macro expansion and related issues.
- `tests/error_recovery.rs`: Includes tests for compiler error handling
and recovery mechanisms.
- `tests/examples.rs`: Verifies the execution of example scripts.
This reorganization improves test discoverability and maintainability.
The `lib.rs` file has also been updated to reflect these changes by
removing the reference to the old `integration_test` module.
The `type_checker.rs` file has had some tests removed, as they are now
covered by the more specific tests in `tests/error_recovery.rs`.
Introduce `collect_pattern_addrs` to gather all bound addresses from a
destructuring pattern. This enables the optimizer to remove unused
destructuring definitions, similar to how simple unused definitions are
handled. Added integration tests to verify this behavior and other
optimizer robustness scenarios.
The Binder and related types have been refactored to use the new
`NodeKind` enum instead of the previous `BoundKind`. This commit updates
all references to use the new structure, ensuring consistency across the
compiler's AST representation.
Key changes include:
- Replacing `BoundKind` with `NodeKind` in the Binder's `bind` and
`visit` methods.
- Updating pattern matching and field access to reflect the new enum
variants (e.g., `NodeKind::Def` instead of `BoundKind::Define`).
- Adjusting identifier bindings to use the new `IdentifierBinding` enum.
- Reflecting changes in `DefBinding`, `AssignBinding`, and
`LambdaBinding` structures.
- Ensuring all newly created nodes use `NodeKind` and the appropriate
metadata.
The `AssignBinding` struct now uses `Option<Address<L>>` to
differentiate between simple assignments and destructuring assignments.
For destructuring, the addresses are managed by the `Identifier` nodes
within the target pattern.
A new `GetField` node kind has been introduced to represent optimized
field access, which is generated during the lowering phase for the
`RuntimePhase`. This change aids in more efficient code generation for
accessing fields.
This commit introduces two new files:
- `.claudeignore`: Specifies files and directories that should be
ignored by Claude, such as snapshot files and the `target` directory.
- `docs/bytecode-architecture.md`: Documents the architecture for
bytecode generation, outlining the hybrid execution model, the
transformation pipeline (Specializer and Lowering), and the role of
Tail Call Optimization (TCO).
This commit introduces a new `NodeKind` enum that serves as a unified
representation for AST nodes across different compilation phases. It
replaces the separate `SyntaxKind` and `BoundKind` enums, allowing
phase-specific information to be attached through generic type
parameters.
This change simplifies the AST structure and makes it easier to manage
node information consistently throughout the compilation process. The
`NodeKind` enum now holds all possible AST node variants, with
phase-specific data encapsulated within `CompilerPhase` trait
implementations.
This commit reduces unnecessary cloning for fields like `addr`, `kind`,
and `field` within `BoundKind` and `VM::GetField`. This improves
performance by avoiding redundant memory allocations.
This commit refactors the AST substitution logic to correctly handle
nested expansions and improve the inlining of lambda expressions.
The changes include:
- Extracting the core value from potentially expanded AST nodes before
checking their kind.
- Ensuring that lambda expressions with empty upvalues are correctly
substituted.
- Adding support for inlining global get expressions as AST
substitutions.
- Improving the `UsageInfo` collection to correctly track assigned
values.
Introduce generic `CompilerPhase` trait to unify different stages of the
bound AST. Rename `LocalSlot` to `VirtualId` and introduce `StackOffset`
for clearer distinction between compile-time and run-time addressing.
Update type aliases and implementations to reflect these changes.
The VM's field access logic has been refactored to clone `Rc` pointers
for addresses and field identifiers. This change aims to improve
efficiency by avoiding unnecessary dereferencing and copying of values
when accessing fields within records and objects.
Additionally, the `Value::Object` handling in `BoundKind::GetField` has
been refined to more explicitly manage `RecordSeries` and `StreamNode`
types, ensuring that field access is handled polymorphically and
efficiently without copying underlying data structures.
This commit makes several changes related to how closures are handled:
- **Dumper:** Removes redundant introspection of closure ASTs, as this
is no longer relevant.
- **Optimizer:** Updates `try_inline` to accept `ExecNode` for
parameters and adds a check to ensure the `function_node` is a
`Lambda` before attempting inlining.
- **Environment:** Simplifies closure creation by passing an `ExecNode`
for parameters and ensuring the correct `stack_size` is used.
- **VM:** Adjusts `Closure` to store an `ExecNode` for parameters and
updates `VM::unpack` to accept an `ExecNode`.
Introduces a `StackAllocator` to manage local slot mapping during
lowering. This allows for efficient calculation of stack sizes for
lambdas and the root node. Tail position marking is now handled more
accurately.
Replace manual resizing of global arrays with simple pushes. This
simplifies the logic and removes the need for index calculations within
the registration functions.
Change the `slots` field in `TypeContext` from a `Vec` to a `HashMap`.
This allows for sparse local variable storage and avoids the need to
pre-allocate for all slots, which is more efficient when not all locals
are used.
Add configuration for the repomix MCP server to the Gemini settings.json
file. This will allow Gemini to use repomix for code analysis and
understanding.
This commit replaces the `UntypedNode` enum with the more accurately
named `SyntaxNode`. This change is primarily for clarity and better
reflects the role of these nodes as representing the structure of the
source code prior to semantic analysis.
The corresponding enum `UntypedKind` has also been renamed to
`SyntaxKind` to maintain consistency.
No functional changes are introduced by this refactoring; it is purely a
renaming and organizational update.
Simplified the AST architecture by removing the overly complex Node<K,
T> structure and replacing it with specialized types for
each phase:
- Introduced UntypedNode in nodes.rs for the Parser and Macro system,
eliminating redundant ty: () initializations.
- Defined a new generic Node<T> in bound_nodes.rs for all compiler
stages, where T represents the phase-specific metadata.
- Updated the Binder to act as the bridge between UntypedNode and
Node<()>.
- Simplified type aliases: TypedNode, AnalyzedNode, and ExecNode now
leverage the streamlined Node<T> structure.
- Updated Parser, Macros, Type-Checker, Optimizer, and VM to reflect
the architectural changes.
- Fixed import chains and resolved needless borrows via clippy.
This change improves type safety, reduces boilerplate code in the
parser, and makes the compiler stages more idiomatic and
readable.
This commit refactors the purity analysis from using a
`HashMap<GlobalIdx, Purity>` to a `Vec<Purity>` indexed by
`GlobalIdx.0`.
This change simplifies the data structure and makes purity lookups more
efficient. The `Binder`'s `globals` field has also been removed as it
was not being used.
The `Binder` struct now accepts a `Rc<RefCell<HashMap<Symbol,
(GlobalIdx, Identity)>>>` for global names. This allows the binder to
directly access and manage global symbols during the binding process,
simplifying the logic and improving efficiency.
Additionally, `CompilerScope` now implements `Default`, and a type alias
`BindingResult` has been introduced for clarity. The `bind_root`
function signature has been updated to accept the `globals` argument.
The `Binder` and `FunctionCompiler` initialization has been refactored
to accept initial scopes and slot counts. This allows for more flexible
management of the compiler's state, particularly for the root scope.
The `bind_root` function now returns the final scopes and slot count of
the root function compiler, enabling the `Environment` to update its
state with these new bindings.
The global variable handling has been integrated into the scope
management, removing the direct reliance on a shared `HashMap` for
globals. This promotes a more consistent approach to symbol resolution.
The `LocalInfo` struct now includes a `purity` field to track the purity
of local variables and function arguments. This is initialized to
`Purity::Impure` for local variables and arguments within functions and
when registering native functions. Additionally, the `Environment`
struct is updated to include `root_scopes` and `root_slot_count` to
support parallel mode compilation by populating the root scope with
initial function and native function information.
Removes global variables and related logic from the Binder, simplifying
its responsibilities to managing local scopes and function compilers.
This change also streamlines the `define_variable` and `resolve_local`
methods within `FunctionCompiler` and removes unused fields from
`FunctionCompiler` and `Binder`.
Introduces `fixed_scope_idx` to `Binder` and `Environment` to track
immutable scopes.
This allows enforcing that definitions (`def`) cannot occur in scopes
that are considered fixed or immutable, such as the global scope or the
initial bootstrap scope.
The `FunctionCompiler` is updated to use `fixed_scope_idx` and `is_root`
to determine scope immutability.
The TCO (Tail Call Optimization) module has been renamed to `lowering`.
This change better reflects the module's broader responsibility, which
includes not only TCO but also general AST transformations and
preparation for VM execution.
The `optimize` function has been renamed to `lower` to align with the
module's new name.
The `calculate_stack_size` method was duplicated in `bound_nodes.rs` and
`tco.rs`. This commit extracts it into a single standalone function in
`tco.rs` to avoid duplication. The stack size is now calculated and
stored in the `ExecNode`'s `ty` field during the TCO optimization phase.
Replaces manual iteration for pushing `Value::Void` with `Vec::resize`
for more efficient stack management. This change improves performance by
reducing redundant operations and simplifies the code.
Store the pre-calculated stack size of lambda bodies in the
RuntimeMetadata. This allows the VM to directly access this information
without recalculating it.
Brummel
Michael Schimmel