The module loading logic has been refactored to enhance readability and
maintainability. Key changes include:
* **Path Handling**: Simplified path construction using `join` for
better clarity and consistency.
* **Error Reporting**: Improved error messages during parsing by
utilizing `parser.diagnostics.format_errors()`.
* **Module Loading Logic**:
* The `loaded_modules` set now uses `insert()` to both add new
modules and efficiently check for duplicates, returning `false`
if the module was already loaded.
* The recursive call to `collect_dependencies` is now placed
before processing the current file's AST, ensuring a topological
order of dependency loading.
* **Helper Function Scope**: `extract_use_directives` is now a static
method as it doesn't depend on the `ModuleLoader` instance.
Introduce `ModuleLoader` to handle `#use` directives and dependency
resolution. This involves adding `tempfile` dependency, modifying
`Cargo.toml` and `Cargo.lock`, creating a new `module_loader.rs` file,
and updating `environment.rs` to use the new module.
Also, add comprehensive tests for `Diagnostics::format_errors`,
`CompilationResult`, and the new `ModuleLoader` functionality. These
tests cover various scenarios including empty diagnostics, multiple
errors, error formatting, compilation success/failure, and complex
dependency loading with topological ordering and search path resolution.
This commit introduces the `CompilationResult` struct to encapsulate the
outcome of a compilation pass. It holds either a successfully compiled
`TypedNode` or a `Diagnostics` object containing errors. Helper methods
for creating success or error results and for converting to a `Result`
are also included. This improves error handling and clarity in the
compilation process.
This commit introduces support for unhygienic macro expansion using the
`~ident` syntax. When an identifier is prefixed with a tilde (`~`)
within a macro template, it is no longer subject to hygienic renaming.
Instead, it directly refers to a binding in the call site's environment.
This allows macros to interact with variables defined outside the
macro's scope, such as:
- Modifying call-site variables.
- Capturing call-site variables as upvalues in closures.
- Defining new variables that are visible in the call site.
A new test case `test_macro_tilde_nonparam_assign` has been added to
verify this functionality. Previously, unhygienic escapes were
implicitly handled by `SyntaxKind::Identifier` if they were not macro
parameters. This change makes this behavior explicit and correctly
handles non-parameter identifiers by returning a bare symbol.
Refine the BNF for `series` to explicitly include the `lookback_limit`
and `schema`.
Introduce scope management within the `bind` function for `if/else`
branches to ensure correct context handling during compilation.
Add `Again` and `GetField` node kinds to the `Specializer`.
Improve lexer to ignore invisible characters within identifiers,
demonstrated by a new test case.
Introduce `free_slots` in `StackAllocator` to keep track of physical
slots that have been freed. When mapping a new virtual slot, the
allocator first attempts to reuse a slot from the `free_slots` vector
before allocating a new one.
This change also includes a new function `collect_scope_locals` that
identifies non-captured local variables defined within a block. After
these expressions are lowered, the allocator reclaims the physical slots
associated with these locals, making them available for reuse by
subsequent blocks.
A new test `test_slot_reuse_non_overlapping_scopes` is added to verify
that distinct scopes correctly reuse stack slots, ensuring optimal stack
usage.
The optimizer now correctly handles empty blocks by collapsing them into
Nop nodes. It also unwraps blocks containing a single expression, making
the AST more concise. This change improves AST simplification by
ensuring that redundant block structures are removed.
The explicit scope management within the `if`/`else` binding logic was
redundant.
The `bind` function recursively handles scope creation and destruction
for
expressions, making manual scope manipulation here unnecessary and
error-prone.
Introduces a new AST emitter module responsible for serializing AST
nodes back into Myc source code. This emitter preserves leading comments
and whitespace, enabling a round-trip guarantee for the parser and
emitter.
The documentation in `docs/BNF.md` has been updated to reflect the new
syntax rules for comments, including single-line comments (`;`) and
documentation comments (`;;`). The rules clarify that comments must be
whole lines and that inline comments are disallowed.
The lexer and AST node structures have been updated to accommodate
`CommentLine` variants, allowing for different types of comments
(regular, documentation, and blank lines) to be stored and processed.
A new test suite `tests/comment_roundtrip.rs` has been added to verify
the round-trip property of the parser and emitter across all example
`.myc` files. This test ensures that parsing source code, emitting it,
and then parsing the emitted code again results in the same output
string.
The `ast.rs` binary now includes a `--emit` flag that uses the new
emitter to output the parsed source code, facilitating debugging and
testing of the emitter itself.
This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
This commit introduces bidirectional type inference, enabling the
compiler to infer lambda parameter types based on the context of their
usage. This is particularly useful for functions like `pipe`, where a
lambda's signature can be deduced from the types of the streams it
operates on.
Key changes include:
- **`extract_lambda_param_hints` function:** This new helper function in
`type_checker.rs` is responsible for extracting expected parameter
types for a lambda based on the callee's signature and already known
argument types.
- **`check_lambda_with_param_hints` function:** This function allows
type-checking a lambda node using provided parameter type hints,
preserving upvalue types from the enclosing scope.
- **Call expression type checking:** The `check_call` function now
phases its argument type checking. Non-lambda arguments are typed
first, and then lambda arguments are typed using hints derived from
`extract_lambda_param_hints`.
- **`pipe_arg_hint_resolver`:** This new resolver for the `pipe`
function's `PolymorphicFn` type allows it to provide specific type
hints for its lambda argument based on the types of its stream inputs
(single stream, vector of streams, or tuple of streams).
- **`StaticType::PolymorphicFn` update:** The `PolymorphicFn` enum
variant has been extended to include `resolve_arg_hints`, enabling
functions to provide lambda parameter hints during bidirectional type
inference.
- **New tests:** Added tests to verify the correct inference of lambda
parameters for `pipe` with vector and tuple inputs, ensuring that
inner stream types are correctly propagated. Also, a test to confirm
`pipe` with an optional return still produces `Stream(Float)`.
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
Replaces `Value::Object(Rc<dyn Object>)` for SyntaxNodes with a
dedicated `Value::Quote(Rc<SyntaxNode>)`. This provides better type
safety and clarity when dealing with AST nodes as values.
Refactor Value enum for Quote node
This commit introduces a new `Value::Quote` variant to represent quoted
AST nodes directly, replacing the generic `Value::Object` for this
purpose.
This change simplifies the handling of quoted nodes by:
- Directly storing an `Rc<SyntaxNode>` instead of relying on dynamic
downcasting from `Rc<dyn Object>`.
- Streamlining macro expansion logic by removing the need to check for
`SyntaxNode` within `Value::Object`.
- Improving type safety and explicitness in the `Value` enum.
Previously, compiled closures were stored as `Value::Object` and then
downcasted. This commit introduces a dedicated `Value::Closure` enum
variant to represent compiled closures. This improves type safety and
simplifies handling of closures throughout the compiler and VM.
This change involves:
- Updating type definitions to use `Value::Closure`.
- Adjusting downcasting logic to directly access the `Closure` struct.
- Modifying `run_with_args` and `run_with_args_observed` to expect
`Value::Closure` for tail-call targets.
- Refining the handling of closures in the `Optimizer` and `Inliner` for
better type clarity.
The `Closure` struct has been moved from `ast/vm.rs` to a new module
`ast/closure.rs`. This improves the organization of the AST nodes. The
`Closure` struct represents a compiled function body along with its
captured upvalues, which is a key feature for Myc Script.
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.
Specifically, the following changes were made:
- In `analyzer.rs`, `crate::ast::types::Identity` and
`crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
`crate::ast::types::SourceLocation`,
`crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
`SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
`RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
`AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
`StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
`PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
`build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
`Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
`build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.
The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
This commit refactors the test suite by removing the monolithic
`integration_test.rs` file.
The tests have been categorized and moved into dedicated modules within
the `tests/` directory:
- `tests/parser.rs`: Contains tests for parsing various literal values.
- `tests/destructuring.rs`: Houses tests related to destructuring
syntax.
- `tests/closures.rs`: Includes tests for closure behavior and related
optimizations.
- `tests/optimizer.rs`: Contains tests specifically targeting optimizer
logic and regressions.
- `tests/records.rs`: Holds tests for record syntax and associated
optimizations.
- `tests/rtl.rs`: Contains tests for runtime library operators and
functions.
- `tests/pipeline.rs`: Tests for pipeline functionality.
- `tests/macros.rs`: Tests for macro expansion and related issues.
- `tests/error_recovery.rs`: Includes tests for compiler error handling
and recovery mechanisms.
- `tests/examples.rs`: Verifies the execution of example scripts.
This reorganization improves test discoverability and maintainability.
The `lib.rs` file has also been updated to reflect these changes by
removing the reference to the old `integration_test` module.
The `type_checker.rs` file has had some tests removed, as they are now
covered by the more specific tests in `tests/error_recovery.rs`.
Introduce `collect_pattern_addrs` to gather all bound addresses from a
destructuring pattern. This enables the optimizer to remove unused
destructuring definitions, similar to how simple unused definitions are
handled. Added integration tests to verify this behavior and other
optimizer robustness scenarios.
The Binder and related types have been refactored to use the new
`NodeKind` enum instead of the previous `BoundKind`. This commit updates
all references to use the new structure, ensuring consistency across the
compiler's AST representation.
Key changes include:
- Replacing `BoundKind` with `NodeKind` in the Binder's `bind` and
`visit` methods.
- Updating pattern matching and field access to reflect the new enum
variants (e.g., `NodeKind::Def` instead of `BoundKind::Define`).
- Adjusting identifier bindings to use the new `IdentifierBinding` enum.
- Reflecting changes in `DefBinding`, `AssignBinding`, and
`LambdaBinding` structures.
- Ensuring all newly created nodes use `NodeKind` and the appropriate
metadata.
The `AssignBinding` struct now uses `Option<Address<L>>` to
differentiate between simple assignments and destructuring assignments.
For destructuring, the addresses are managed by the `Identifier` nodes
within the target pattern.
A new `GetField` node kind has been introduced to represent optimized
field access, which is generated during the lowering phase for the
`RuntimePhase`. This change aids in more efficient code generation for
accessing fields.
This commit introduces a new `NodeKind` enum that serves as a unified
representation for AST nodes across different compilation phases. It
replaces the separate `SyntaxKind` and `BoundKind` enums, allowing
phase-specific information to be attached through generic type
parameters.
This change simplifies the AST structure and makes it easier to manage
node information consistently throughout the compilation process. The
`NodeKind` enum now holds all possible AST node variants, with
phase-specific data encapsulated within `CompilerPhase` trait
implementations.
This commit reduces unnecessary cloning for fields like `addr`, `kind`,
and `field` within `BoundKind` and `VM::GetField`. This improves
performance by avoiding redundant memory allocations.
This commit refactors the AST substitution logic to correctly handle
nested expansions and improve the inlining of lambda expressions.
The changes include:
- Extracting the core value from potentially expanded AST nodes before
checking their kind.
- Ensuring that lambda expressions with empty upvalues are correctly
substituted.
- Adding support for inlining global get expressions as AST
substitutions.
- Improving the `UsageInfo` collection to correctly track assigned
values.
Introduce generic `CompilerPhase` trait to unify different stages of the
bound AST. Rename `LocalSlot` to `VirtualId` and introduce `StackOffset`
for clearer distinction between compile-time and run-time addressing.
Update type aliases and implementations to reflect these changes.
This commit makes several changes related to how closures are handled:
- **Dumper:** Removes redundant introspection of closure ASTs, as this
is no longer relevant.
- **Optimizer:** Updates `try_inline` to accept `ExecNode` for
parameters and adds a check to ensure the `function_node` is a
`Lambda` before attempting inlining.
- **Environment:** Simplifies closure creation by passing an `ExecNode`
for parameters and ensuring the correct `stack_size` is used.
- **VM:** Adjusts `Closure` to store an `ExecNode` for parameters and
updates `VM::unpack` to accept an `ExecNode`.
Introduces a `StackAllocator` to manage local slot mapping during
lowering. This allows for efficient calculation of stack sizes for
lambdas and the root node. Tail position marking is now handled more
accurately.
Change the `slots` field in `TypeContext` from a `Vec` to a `HashMap`.
This allows for sparse local variable storage and avoids the need to
pre-allocate for all slots, which is more efficient when not all locals
are used.
This commit replaces the `UntypedNode` enum with the more accurately
named `SyntaxNode`. This change is primarily for clarity and better
reflects the role of these nodes as representing the structure of the
source code prior to semantic analysis.
The corresponding enum `UntypedKind` has also been renamed to
`SyntaxKind` to maintain consistency.
No functional changes are introduced by this refactoring; it is purely a
renaming and organizational update.
Simplified the AST architecture by removing the overly complex Node<K,
T> structure and replacing it with specialized types for
each phase:
- Introduced UntypedNode in nodes.rs for the Parser and Macro system,
eliminating redundant ty: () initializations.
- Defined a new generic Node<T> in bound_nodes.rs for all compiler
stages, where T represents the phase-specific metadata.
- Updated the Binder to act as the bridge between UntypedNode and
Node<()>.
- Simplified type aliases: TypedNode, AnalyzedNode, and ExecNode now
leverage the streamlined Node<T> structure.
- Updated Parser, Macros, Type-Checker, Optimizer, and VM to reflect
the architectural changes.
- Fixed import chains and resolved needless borrows via clippy.
This change improves type safety, reduces boilerplate code in the
parser, and makes the compiler stages more idiomatic and
readable.
This commit refactors the purity analysis from using a
`HashMap<GlobalIdx, Purity>` to a `Vec<Purity>` indexed by
`GlobalIdx.0`.
This change simplifies the data structure and makes purity lookups more
efficient. The `Binder`'s `globals` field has also been removed as it
was not being used.
The `Binder` struct now accepts a `Rc<RefCell<HashMap<Symbol,
(GlobalIdx, Identity)>>>` for global names. This allows the binder to
directly access and manage global symbols during the binding process,
simplifying the logic and improving efficiency.
Additionally, `CompilerScope` now implements `Default`, and a type alias
`BindingResult` has been introduced for clarity. The `bind_root`
function signature has been updated to accept the `globals` argument.
The `Binder` and `FunctionCompiler` initialization has been refactored
to accept initial scopes and slot counts. This allows for more flexible
management of the compiler's state, particularly for the root scope.
The `bind_root` function now returns the final scopes and slot count of
the root function compiler, enabling the `Environment` to update its
state with these new bindings.
The global variable handling has been integrated into the scope
management, removing the direct reliance on a shared `HashMap` for
globals. This promotes a more consistent approach to symbol resolution.
The `LocalInfo` struct now includes a `purity` field to track the purity
of local variables and function arguments. This is initialized to
`Purity::Impure` for local variables and arguments within functions and
when registering native functions. Additionally, the `Environment`
struct is updated to include `root_scopes` and `root_slot_count` to
support parallel mode compilation by populating the root scope with
initial function and native function information.
Removes global variables and related logic from the Binder, simplifying
its responsibilities to managing local scopes and function compilers.
This change also streamlines the `define_variable` and `resolve_local`
methods within `FunctionCompiler` and removes unused fields from
`FunctionCompiler` and `Binder`.
Introduces `fixed_scope_idx` to `Binder` and `Environment` to track
immutable scopes.
This allows enforcing that definitions (`def`) cannot occur in scopes
that are considered fixed or immutable, such as the global scope or the
initial bootstrap scope.
The `FunctionCompiler` is updated to use `fixed_scope_idx` and `is_root`
to determine scope immutability.
The TCO (Tail Call Optimization) module has been renamed to `lowering`.
This change better reflects the module's broader responsibility, which
includes not only TCO but also general AST transformations and
preparation for VM execution.
The `optimize` function has been renamed to `lower` to align with the
module's new name.
The `calculate_stack_size` method was duplicated in `bound_nodes.rs` and
`tco.rs`. This commit extracts it into a single standalone function in
`tco.rs` to avoid duplication. The stack size is now calculated and
stored in the `ExecNode`'s `ty` field during the TCO optimization phase.
Store the pre-calculated stack size of lambda bodies in the
RuntimeMetadata. This allows the VM to directly access this information
without recalculating it.
This commit introduces a new mechanism for calculating the required
stack size for closures at bind time, rather than storing it in the AST.
This is achieved by adding a `calculate_stack_size` method to
`BoundNode`.
Key changes include:
- `BoundNode::calculate_stack_size`: Recursively traverses the bound AST
to find the maximum `LocalSlot` index used.
- Recursion blocker for lambdas: Ensures that nested lambdas are not
visited during stack size calculation for the outer closure,
preventing incorrect stack size estimations.
- VM update: The `VM::run` method now accepts and uses the
pre-calculated stack size for setting up call frames.
- `Closure` struct update: Stores the `stack_size` directly within the
`Closure` struct.
- Binder and TypeChecker updates: Minor adjustments to accommodate the
new strategy and error reporting.
- Optimizer enhancements: Constant and pure-lambda propagation for
`Define` statements within blocks to ensure inlinability.
This refactoring addresses issues related to accurate stack size
management and improves the overall robustness of the binder and VM.
The previous implementation of the binder had a slightly complex way of
handling the root scope. This commit simplifies that logic by directly
checking if the current scope is the root scope and the scope stack has
only one element. This makes the code more readable and maintainable.
This commit refactors the Binder implementation to better align with the
updated scope structure.
Key changes include:
- Introducing `ExprContext` to distinguish between statement and
expression contexts.
- Modifying `define_variable` to handle both global and local scope
definitions more robustly.
- Updating `resolve_variable` to correctly handle upvalue captures,
especially for global addresses.
- Adjusting `bind` and related functions to use the new `ExprContext`.
- Updating the refactoring log to reflect completed phases.
Implement strict block scoping and distinguish between statements and
expressions.
This change introduces hierarchical scopes for functions and blocks,
enforces that `def` is a statement with no return value, and prevents
statements from appearing in expression positions or as the last element
of a block. A new example `design-flaw.myc` is added to demonstrate
the uninitialized variable issue solved by these changes.
The `Analyzer` is now responsible for determining the purity of `Define`
bindings. Previously, this responsibility was left to the caller.
Added a new example `design-flaw.myc` to test this change.
Updated `soa_series.myc` due to the purity change.
The 'again' macro is a new addition to the macro system, allowing for
recursive macro expansion. This commit introduces the necessary AST
nodes, macro expansion logic, and an integration test to ensure its
functionality. The `soa_series.myc` example has also been updated to
demonstrate its usage.
The `Parameter` kind in `UntypedKind` was only used for identifiers that
were being declared or referenced. This commit renames it to
`Identifier` and updates all the necessary code to reflect this change.
This simplifies the AST and makes it more consistent.
Additionally, a new macro `repeat` has been added to
`src/ast/system.myc`.
The `PipelineNode` has been refactored into `StreamNode`. This commit
updates the example files to reflect this change and also updates the
`series` constructor to accept a lookback parameter.
The `PipelineNode` was an artifact of a previous implementation and is
no longer needed. The `StreamNode` is the appropriate type for
representing the output of a pipe operation.
The `series` function now requires a `lookback` argument to specify the
maximum number of items to store. This makes the behavior of series more
explicit and prevents accidental unbounded memory growth.
Brummel
Michael Schimmel