This commit introduces a new test suite for HM Let-Polymorphism.
These tests verify that user-defined generic functions work correctly
across multiple series element types without type conflicts. This is
crucial for ensuring that the type inference system can handle
polymorphic functions properly, especially when they are applied to
different types. The tests cover scenarios like generic accessors,
runtime correctness with different series types, record series, and
higher-order functions. They also include a test for the value
restriction to ensure that series bindings are not generalized, which
would lead to incorrect type errors.
Introduces `CallHooks` to enable extensions to the type inference and
AST finalization process. This allows for more dynamic and configurable
behavior without hardcoding symbol names in the compiler core.
Specifically, this commit adds:
- **`series_post_call`**: A hook for the `(series n)` function. It
ensures that the return type is a fresh `TypeVar` when the element
type is `Any`, allowing for proper type inference from subsequent
`push` calls.
- **`series_finalize`**: A hook for `(series n)` that rewrites the AST
node. It replaces the generic `series` call with a pre-configured
factory closure that allocates the correct series storage type (e.g.,
`ScalarSeries<f64>` for floats, `RecordSeries` for records). This
avoids runtime dispatch and relies on type information captured during
compilation.
- **`push_post_call`**: A hook for the `(push series val)` function. It
unifies the series element `TypeVar` with the pushed value's type. It
also handles numeric and record field promotion and updates the
`TypeVar` binding if a promotion occurs.
This commit introduces implicit numeric widening from Int to Float and a
corresponding record promotion mechanism.
When pushing values into a series, if the series' element type is a
TypeVar,
and the pushed value is of a different numeric type (e.g., Int and
Float),
the TypeVar will be unified with the wider type (Float). This ensures
that
operations on series elements handle type differences gracefully.
The record promotion handles cases where two records are involved, and
their
fields have compatible types, either identical or through numeric
widening.
This allows for more flexible type inference in complex structures.
When an identifier is used as an upvalue within a nested scope (e.g., a
`while` loop), the type checker needs to apply the global substitution
map to resolve `TypeVar`s that might have been determined in outer
scopes. This ensures that the correct type is used even if
`ctx.set_type` couldn't propagate the change directly through the
upvalue address.
This fix also includes a regression test to specifically cover this
scenario with series and closures.
The `series` function no longer requires an explicit schema. The element
type is now inferred at compile time from subsequent `push` calls using
Hindley-Milner type inference. This simplifies series creation and
aligns with the project's goal of making the untyped AST simple while
the system handles complexity.
Introduce `unify_matched_overload` to handle type variable propagation
for overloaded functions. This ensures that when an overload is chosen,
its concrete parameter types are unified with the actual argument types.
This is crucial for resolving type variables nested within closures
called through overloaded functions.
Additionally, this commit:
- Allows `series` to infer schema type from `push` calls, simplifying
its signature.
- Adds a fallback to `ValueSeries` in `rtl::series` if schema injection
fails.
- Updates `StaticType::TypeVar` to return `Any` when accessed,
simplifying field access logic.
- Adjusts tests to reflect the simplified `series` signature.
The `series` constructor now only accepts the `lookback` limit. The
element type is inferred by the type checker and implicitly added as a
schema argument during compilation. This simplifies the API and
centralizes type inference.
This change also includes:
- Updates to example scripts (`HMA.myc`, `sma.myc`, `soa_series.myc`) to
reflect the new `series` signature.
- Modifications to the `TypeChecker` to handle the inferred schema
injection.
- An addition of a regression test for type inference through nested
closures with `series`.
- Removal of `#[allow(dead_code)]` from several `TypeChecker` methods as
they are now used.
- Update to `rtl/prelude.myc` macro `cache`.
- Update to `rtl/series/mod.rs` to reflect new signature.
- Update to `ast/types.rs` to allow `series` to be indexed with an
integer to retrieve its element type.
Introduces a `TypeChecker` struct with logic for Hindley-Milner type
inference.
This includes:
- `var_counter` for generating unique type variable IDs.
- `subst` for the global substitution map.
- `fresh_var` to create new type variables.
- `apply_subst` to resolve type variables recursively.
- `occurs` to check for cycles during unification.
- `unify` to merge types and extend the substitution.
The `StaticType` enum is extended with a `TypeVar(u32)` variant to
represent unresolved type variables.
The `Display` and `is_assignable_from` implementations for `StaticType`
are updated to handle `TypeVar`.
The `GlobalStore` was refactored to clearly distinguish between
immutable RTL values and mutable user-defined global slots.
The `Environment` struct now holds:
- `rtl_values`: An immutable `Rc<[Value]>` for pre-defined RTL values.
- `user_values`: An `Rc<RefCell<Vec<Value>>>` for user-defined mutable
globals.
The `GlobalStore` struct now takes both `rtl` and `user` as parameters
and uses `rtl_len` to determine which store to access. This change
separates concerns and better reflects the immutability of RTL values
after the bootstrap phase, aligning with the project's concurrency
rules.
Documentation in `docs/Analysis_Environment.md` was updated to reflect
these structural changes.
Replaces `Rc<RefCell<Vec<Value>>>` with a new `GlobalStore` struct for
accessing global variables. This provides a more structured way to
manage global values and allows for future optimizations like sharing
immutable RTL portions across environments. The `GlobalStore` also
includes an `rtl_len` field to distinguish between RTL and user global
slots, enabling debug-time checks for writes to immutable RTL slots.
This commit refactors the `Environment` struct to better manage the Rust
Runtime Library (RTL) bootstrap state.
A new `Rtl` struct is introduced to hold a frozen snapshot of the
initial RTL state, including scopes, types, purity, values, and
documentation. This snapshot is created once after `rtl::register()`
completes.
The `Environment::new()` constructor is updated to perform a two-phase
bootstrap: first, it registers RTL functions, and then it freezes the
scope 0 and captures the RTL state into the new `Rtl` struct.
A new constructor, `Environment::from_rtl()`, is added. This allows
creating independent execution environments that all share the same
underlying RTL state. This is crucial for isolation between different
execution contexts, as user definitions and loaded modules will not leak
between environments created from the same `Rtl` snapshot.
The `Environment::list_bindings` and `Environment::list_rtl_docs`
methods are updated to use the `rtl` field for accessing the RTL scope
and documentation, reinforcing the concept of a shared, immutable RTL
state.
A new test, `test_environments_from_shared_rtl_are_independent`, is
added to verify that environments created from the same `Rtl` snapshot
are indeed independent and do not leak user-defined bindings.
The `instantiate` method in `Environment` has been refactored to return
`Result<Rc<Closure>, String>` instead of `Rc<NativeFunction>`. This
change separates the creation of a `Closure` from its execution,
allowing the caller to manage the `VM` lifecycle and choose the
appropriate execution strategy (e.g., single run vs. repeated benchmark
iterations).
The `NativeFunction` type is now exclusively used for true Rust
intrinsics. The benchmark runner has also been updated to reuse a single
`VM` across iterations, improving performance by avoiding repeated VM
allocations.
The `dump_ast` method previously bypassed the `compile` method, which
meant it would silently ignore parsing errors. This commit ensures that
`dump_ast` now delegates to `compile` and correctly propagates any
parsing errors. This aligns the behavior of `dump_ast` with other
compilation steps and provides better feedback to the user.
Fix: Dump AST ignores parse errors
The `dump_ast` method now delegates to `compile(source).into_result()?`
to properly handle and report parse errors, ensuring consistency with
the compilation process.
Re-type-checks function templates with concrete argument types to
produce specialized TypedNodes for monomorphization. This process
leverages the `check_node_as_bound` method, which is generic enough to
handle different binding phases, allowing for fresh type inference from
the call site.
The module loading logic has been refactored to enhance readability and
maintainability. Key changes include:
* **Path Handling**: Simplified path construction using `join` for
better clarity and consistency.
* **Error Reporting**: Improved error messages during parsing by
utilizing `parser.diagnostics.format_errors()`.
* **Module Loading Logic**:
* The `loaded_modules` set now uses `insert()` to both add new
modules and efficiently check for duplicates, returning `false`
if the module was already loaded.
* The recursive call to `collect_dependencies` is now placed
before processing the current file's AST, ensuring a topological
order of dependency loading.
* **Helper Function Scope**: `extract_use_directives` is now a static
method as it doesn't depend on the `ModuleLoader` instance.
Introduce `ModuleLoader` to handle `#use` directives and dependency
resolution. This involves adding `tempfile` dependency, modifying
`Cargo.toml` and `Cargo.lock`, creating a new `module_loader.rs` file,
and updating `environment.rs` to use the new module.
Also, add comprehensive tests for `Diagnostics::format_errors`,
`CompilationResult`, and the new `ModuleLoader` functionality. These
tests cover various scenarios including empty diagnostics, multiple
errors, error formatting, compilation success/failure, and complex
dependency loading with topological ordering and search path resolution.
Move `RtlDocEntry`, `RtlRegistration`, and `PipelineGenerator` to a new
`docs` module within `src/ast/rtl/` and re-export them. This improves
organization and modularity of the RTL component.
This commit introduces the `CompilationResult` struct to encapsulate the
outcome of a compilation pass. It holds either a successfully compiled
`TypedNode` or a `Diagnostics` object containing errors. Helper methods
for creating success or error results and for converting to a `Result`
are also included. This improves error handling and clarity in the
compilation process.
Introduce `allocate_slot` helper function to centralize the logic for
allocating new global slots. This improves code clarity and reduces
duplication by consolidating common operations like updating scopes,
types, purity, and values.
Introduces a new method `format_errors` to the `Diagnostics` struct.
This method iterates through all diagnostics, filters for errors, and
formats them into a newline-joined string. This simplifies error
reporting in compilation results.
This commit introduces support for unhygienic macro expansion using the
`~ident` syntax. When an identifier is prefixed with a tilde (`~`)
within a macro template, it is no longer subject to hygienic renaming.
Instead, it directly refers to a binding in the call site's environment.
This allows macros to interact with variables defined outside the
macro's scope, such as:
- Modifying call-site variables.
- Capturing call-site variables as upvalues in closures.
- Defining new variables that are visible in the call site.
A new test case `test_macro_tilde_nonparam_assign` has been added to
verify this functionality. Previously, unhygienic escapes were
implicitly handled by `SyntaxKind::Identifier` if they were not macro
parameters. This change makes this behavior explicit and correctly
handles non-parameter identifiers by returning a bare symbol.
Removes outdated tests for macro expansion, destructuring, and if branch
scope leaks. These tests were either superseded by newer, more accurate
tests or are no longer relevant to the current implementation of the
compiler's scoping rules. The remaining test correctly asserts that
defining the same variable in different 'if' branches is functional.
The `discover_globals` function has been refactored to better handle
pattern matching within definitions. It now uses a recursive helper
function `register_pattern` to correctly process identifiers and tuple
patterns. This ensures that all declared variables within a `Def` node
are registered with their corresponding scope and virtual ID.
Additionally, support for `Expansion` nodes has been added to ensure
that any nested global definitions within them are also discovered.
Refine the BNF for `series` to explicitly include the `lookback_limit`
and `schema`.
Introduce scope management within the `bind` function for `if/else`
branches to ensure correct context handling during compilation.
Add `Again` and `GetField` node kinds to the `Specializer`.
Improve lexer to ignore invisible characters within identifiers,
demonstrated by a new test case.
This commit updates all example files to use `assert_eq!` for verifying
output, replacing the previous `Output:` comments. This change makes the
examples more robust and self-testing.
The benchmark results in some examples have also been slightly adjusted,
reflecting minor performance variations after the refactoring.
Additionally, a runtime panic handling mechanism has been introduced in
the VM for function calls, which improves error reporting for unexpected
panics.
Introduce `free_slots` in `StackAllocator` to keep track of physical
slots that have been freed. When mapping a new virtual slot, the
allocator first attempts to reuse a slot from the `free_slots` vector
before allocating a new one.
This change also includes a new function `collect_scope_locals` that
identifies non-captured local variables defined within a block. After
these expressions are lowered, the allocator reclaims the physical slots
associated with these locals, making them available for reuse by
subsequent blocks.
A new test `test_slot_reuse_non_overlapping_scopes` is added to verify
that distinct scopes correctly reuse stack slots, ensuring optimal stack
usage.
The optimizer now correctly handles empty blocks by collapsing them into
Nop nodes. It also unwraps blocks containing a single expression, making
the AST more concise. This change improves AST simplification by
ensuring that redundant block structures are removed.
The explicit scope management within the `if`/`else` binding logic was
redundant.
The `bind` function recursively handles scope creation and destruction
for
expressions, making manual scope manipulation here unnecessary and
error-prone.
This commit introduces a new example script, `HMA.myc`, which
demonstrates the implementation and usage of the Hull Moving Average
(HMA) indicator.
The script includes factory functions for Simple Moving Average
(`make-sma`), Weighted Moving Average (`make-wma`), and the Hull Moving
Average itself (`make-hma`). It also includes a small example loop to
initialize and print HMA values.
Introduces a new AST emitter module responsible for serializing AST
nodes back into Myc source code. This emitter preserves leading comments
and whitespace, enabling a round-trip guarantee for the parser and
emitter.
The documentation in `docs/BNF.md` has been updated to reflect the new
syntax rules for comments, including single-line comments (`;`) and
documentation comments (`;;`). The rules clarify that comments must be
whole lines and that inline comments are disallowed.
The lexer and AST node structures have been updated to accommodate
`CommentLine` variants, allowing for different types of comments
(regular, documentation, and blank lines) to be stored and processed.
A new test suite `tests/comment_roundtrip.rs` has been added to verify
the round-trip property of the parser and emitter across all example
`.myc` files. This test ensures that parsing source code, emitting it,
and then parsing the emitted code again results in the same output
string.
The `ast.rs` binary now includes a `--emit` flag that uses the new
emitter to output the parsed source code, facilitating debugging and
testing of the emitter itself.
This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
Introduces `RtlDocEntry` and `RtlRegistration` structs to store and
manage documentation for native functions and constants. The
`Environment` struct is updated to include a registry for these
documentation entries.
The `register_native_fn` and `register_constant` methods now return an
`RtlRegistration` builder, allowing for optional chaining of `.doc()`,
`.description()`, and `.examples()` methods before the registration is
finalized when the builder is dropped.
Macros `rtl_fn!` and `rtl_const!` are introduced to simplify the
registration process with documentation.
The MCP server is extended with `get_rtl_docs` and `get_symbol_doc`
tools to expose this documentation externally.
Specific RTL functions and constants (e.g., `+`, `-`, `*`, `/`, `NaN`,
`true`, `false`, `date`, `now`, math functions, `series`, `push`, `len`,
`create-random-ohlc`, `create-ticker`, `pipe`) have been updated to
include their respective documentation using the new infrastructure.
The `Signature` and `StaticType` types have gained `to_doc_string`
methods for generating human-readable documentation strings.
Integrates the MCP server functionality into the AST tool binary. This
allows the tool to act as a server for LLM integration, enabling
communication over stdio. The server can list available bindings and
execute scripts.
This commit refactors the way series are represented and handled within
the AST.
Key changes include:
- Introducing `SeriesStorage` and `PushableStorage` traits to provide a
more
unified and type-safe interface for series data.
- Renaming `Object` trait and its methods to clarify that it's for RTL
extensions
other than series (like Streams).
- Updating the `Value` enum to have a distinct `Series` variant,
separating it
from `Object`.
- Adjusting various parts of the `VM` and `register` functions to work
with the
new series traits and `Value::Series` variant.
This change aims to improve the type system's clarity and safety when
dealing with
series data, aligning with Rust's best practices for trait design.
This commit introduces bidirectional type inference, enabling the
compiler to infer lambda parameter types based on the context of their
usage. This is particularly useful for functions like `pipe`, where a
lambda's signature can be deduced from the types of the streams it
operates on.
Key changes include:
- **`extract_lambda_param_hints` function:** This new helper function in
`type_checker.rs` is responsible for extracting expected parameter
types for a lambda based on the callee's signature and already known
argument types.
- **`check_lambda_with_param_hints` function:** This function allows
type-checking a lambda node using provided parameter type hints,
preserving upvalue types from the enclosing scope.
- **Call expression type checking:** The `check_call` function now
phases its argument type checking. Non-lambda arguments are typed
first, and then lambda arguments are typed using hints derived from
`extract_lambda_param_hints`.
- **`pipe_arg_hint_resolver`:** This new resolver for the `pipe`
function's `PolymorphicFn` type allows it to provide specific type
hints for its lambda argument based on the types of its stream inputs
(single stream, vector of streams, or tuple of streams).
- **`StaticType::PolymorphicFn` update:** The `PolymorphicFn` enum
variant has been extended to include `resolve_arg_hints`, enabling
functions to provide lambda parameter hints during bidirectional type
inference.
- **New tests:** Added tests to verify the correct inference of lambda
parameters for `pipe` with vector and tuple inputs, ensuring that
inner stream types are correctly propagated. Also, a test to confirm
`pipe` with an optional return still produces `Stream(Float)`.
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
This commit reorganizes the series-related code by moving the
`RingBuffer`, `ScalarSeries`, `ValueSeries`, `RecordSeries`,
`SeriesView`, and `SharedSeries` structs into a new
`src/ast/rtl/series/data.rs` file.
The `src/ast/rtl/series/mod.rs` file now contains the module
declaration, re-exports, and the `register` function for registering
series-related native functions with the environment. This improves code
organization and maintainability.
Replaces `Value::Object(Rc<dyn Object>)` for SyntaxNodes with a
dedicated `Value::Quote(Rc<SyntaxNode>)`. This provides better type
safety and clarity when dealing with AST nodes as values.
Refactor Value enum for Quote node
This commit introduces a new `Value::Quote` variant to represent quoted
AST nodes directly, replacing the generic `Value::Object` for this
purpose.
This change simplifies the handling of quoted nodes by:
- Directly storing an `Rc<SyntaxNode>` instead of relying on dynamic
downcasting from `Rc<dyn Object>`.
- Streamlining macro expansion logic by removing the need to check for
`SyntaxNode` within `Value::Object`.
- Improving type safety and explicitness in the `Value` enum.
Introduce a new trait `PushableSeries` to distinguish objects that can
have values pushed into them from those that are only readable series.
This clarifies the API and allows for more precise type checking.
The `Object` trait no longer has a default `push_value` implementation.
Instead, `PushableSeries` now provides this functionality.
`SeriesMember` has also been updated to require `PushableSeries` for
RecordSeries field buffers.
Previously, compiled closures were stored as `Value::Object` and then
downcasted. This commit introduces a dedicated `Value::Closure` enum
variant to represent compiled closures. This improves type safety and
simplifies handling of closures throughout the compiler and VM.
This change involves:
- Updating type definitions to use `Value::Closure`.
- Adjusting downcasting logic to directly access the `Closure` struct.
- Modifying `run_with_args` and `run_with_args_observed` to expect
`Value::Closure` for tail-call targets.
- Refining the handling of closures in the `Optimizer` and `Inliner` for
better type clarity.
The `Closure` struct has been moved from `ast/vm.rs` to a new module
`ast/closure.rs`. This improves the organization of the AST nodes. The
`Closure` struct represents a compiled function body along with its
captured upvalues, which is a key feature for Myc Script.
The `TailCallRequest` enum variant and its associated logic have been
removed. Instead, the VM now uses an `Option<(Rc<dyn Object>,
Vec<Value>)>` field named `tail_call` to manage tail-call requests. This
change simplifies the `Value` enum and centralizes tail-call handling
within the `VM` struct.
This commit introduces the `into_rc_any` and `push_value` methods to the
`Object` trait.
The `into_rc_any` method allows for efficient, zero-cost downcasting of
`Rc<Self>` to `Rc<dyn Any>`. This is a crucial step for enabling more
flexible dynamic dispatch and type manipulation within the VM.
The `push_value` method is implemented for series types (`ScalarSeries`,
`ValueSeries`, `RecordSeries`) to allow pushing values directly onto
them. This simplifies the `push` intrinsic function, removing the need
for manual downcasting within its implementation. The default
implementation for non-series objects panics, enforcing that only
mutable series support this operation.
Additionally, `ScalarValue` now includes a `from_value` method to
facilitate converting `Value` back into concrete scalar types. This
improves type safety and reduces boilerplate code when handling scalar
conversions.
The `Value::Object` variant in `ast/types.rs` has been updated to
provide a more specific display format for `SharedValueSeries`.
Previously, it would generically display the object's type name. This
change allows it to directly show the length of the `SharedValueSeries`
buffer, which is more informative.
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.
Specifically, the following changes were made:
- In `analyzer.rs`, `crate::ast::types::Identity` and
`crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
`crate::ast::types::SourceLocation`,
`crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
`SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
`RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
`AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
`StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
`PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
`build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
`Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
`build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
Brummel