This commit updates the benchmark metadata in several example files. The
benchmark runs and repeat counts have been slightly adjusted due to
optimizations or minor variations in execution.
This commit introduces several improvements to the AST optimizer and
dead code elimination (DCE) logic:
- **Address Mapping:** The `Get` and `Set` operations now correctly map
local slots and global indices using the substitution map, ensuring
that remapped variables are handled properly.
- **Parameter Slot Mapping:** Parameters are now explicitly mapped to
their correct slots within the `Lambda` node, preventing potential
slot reassignments in the function body.
- **Pure Function Analysis:** The `is_pure` function has been refined to
accurately identify pure expressions, including calls to pure
functions and stateless closures.
- **Dead Code Elimination:** DCE logic in `BoundKind::Block` has been
enhanced to remove unused local definitions and pure, non-essential
expressions more effectively. Global DCE is also improved.
- **Constant Folding:** `try_fold_pure` is now more robust in folding
pure function calls with constant arguments, reducing redundant
computations.
- **Beta Reduction:** `try_beta_reduce` has been updated to avoid
reducing if the body contains `DefLocal`, ensuring correctness.
- **Global Purity Tracking:** A `global_purity` map is introduced to
track the purity of global values, enabling better optimization of
global accesses.
- **Inlinable Value Check:** The `is_inlinable_value` check now
correctly identifies stateless closures for inlining.
- **Optimization Level:** The default `optimization_level` is set to 2
to enable more aggressive optimizations.
This commit introduces purity inference to the compiler's optimizer.
This allows for more aggressive constant folding and dead code
elimination by tracking which functions and global variables
are free of side effects.
Key changes include:
- Added `global_purity` field to `Optimizer` and `Environment`.
- Modified `Optimizer::is_pure` to recursively determine if an
AST node represents a pure computation.
- Introduced `Optimizer::try_fold_pure` to replace the old
`try_fold_intrinsic`, enabling folding of pure function calls
with constant arguments.
- Updated `Environment::register_native` and
`Environment::register_constant`
to optionally record purity.
- Added purity flags to several built-in functions in `core.rs` and
`datetime.rs`.
- A new example `optimizer_purity.myc` demonstrates the new feature.
Introduce global inlining by allowing the Optimizer to access the
environment's global values. This enables replacing global variable
accesses with their constant values when appropriate.
Additionally, implement Dead Code Elimination (DCE) for global
definitions. A global definition can be removed if it was only used for
inlining within the current script and has no side effects.
Update the Optimizer struct to hold an optional reference to the global
values and modify the `Optimizer::new` constructor to accept this. The
`Environment::specialize` and `Environment::compile_script` methods are
updated to pass the global values to the optimizer.
The `SubstitutionMap` is also updated to track script-local global
substitutions and used global indices, supporting both global inlining
and global DCE.
Integrates dead code elimination (DCE) into the optimizer.
This phase removes expressions that have no side effects and are not the
result of the block.
Also includes several improvements to the existing optimization passes:
- DCE now correctly handles assignments to variables that are never used
or captured.
- Explicitly tracks captured slots in the `SubstitutionMap` to prevent
premature inlining.
- Introduces a `is_side_effect_free` helper function for more robust
purity checks.
- Updates dependencies to include `insta` for snapshot testing.
Add a new example file `examples/def_local_inlining.myc` to demonstrate
and test local inlining.
Refine the optimizer to handle local inlining more robustly by:
- Passing a `SubstitutionMap` to `visit_node` to track local variable
substitutions.
- Enabling constant propagation for local variables by checking
`sub.locals` in `BoundKind::Get`.
- Updating `BoundKind::DefLocal` and `BoundKind::Set` to maintain the
substitution map for local variables.
- Adjusting `try_beta_reduce` to use the optimizer's `visit_node` with
the substitution map for inlining.
- Re-indexing upvalues correctly when inlining captured variables into
lambdas.
The `is_tail` field on the `BoundKind::Call` node was an intermediate
representation for tail-call optimization and is no longer needed as a
distinct field. The TCO pass now embeds this information directly into
the `RuntimeMetadata` of the `ExecNode`, which is the VM's internal AST.
This simplifies the `BoundKind::Call` structure and removes redundant
information.
The `TailCall` variant has been merged into `Call` by adding an
`is_tail` boolean field. This simplifies the AST by reducing the number
of node variants and makes it easier to handle tail call optimization.
The `tco.rs` module is responsible for setting this flag when a call
occurs in tail position.
Implement PartialEq for BoundKind to allow for structural equality
checks during optimization. This enables the optimizer to terminate
early when a node no longer changes.
Also, implement PartialEq for Value to facilitate comparisons between
different Value variants.
The optimizer has been refactored to separate its phases and introduce
more aggressive collapsing capabilities.
Phase 2 (Cracking) now focuses on stateless transformations, making it
easier to reason about and potentially parallelize.
Phase 2.5 (Aggressive Collapsing) has been introduced, enabling
optimizations like:
- Beta-reduction for lambda literals.
- Cracking and inlining for constant closures.
- Folding intrinsics for constant arithmetic.
- Short-circuiting conditional expressions.
These changes aim to improve performance by reducing redundant
computations and code bloat. The maximum number of optimization passes
has also been increased to 5 to allow for more complex transformations.
Introduces a new optimizer pass that can "crack" closures, allowing for
more aggressive specialization. It also enables inlining of upvalues
that point to immutable global variables. This removes overhead for
higher-order functions and currying when arguments are statically
resolvable.
Introduce `ValueList` type alias for `Rc<Vec<Value>>` to reduce
boilerplate and improve readability.
Add `as_slice()` method to `Value` to provide a unified way to access
the underlying values of Tuples and Records.
Update `flatten_value` and `unpack` in `VM` to utilize the new
`as_slice()` method, simplifying logic and improving consistency.
This commit renames `Map` to `Record` and updates all related AST nodes,
binders, type checkers, and runtime values to reflect this change. This
is a semantic change to better align with common programming language
terminology.
The `Value::List` and `Value::Record` variants have been consolidated
into a single `Value::Tuple` variant. This new variant uses a
`TupleData` struct to store values and an optional `Rc<Vec<Keyword>>`
for keys, allowing it to represent both ordered lists/tuples and
key-value records.
This change simplifies the internal representation and improves
performance by allowing schema sharing for records. It also includes
updates to the compiler, runtime, and tests to reflect the new
structure.
Replaces the use of `BTreeMap` and `HashMap` for maps and records with
`Vec<(Keyword, Value)>` and `Vec<(Keyword, StaticType)>`. This
simplifies the data structure and allows for ordered iteration, which is
important for serialization and comparison.
Also updates the `unpack` function in `vm.rs` to handle records as well
as lists when destructuring.
Introduce `BoundKind::Parameter` to represent function parameters.
Modify `Binder` to correctly handle parameters within lambda
definitions, ensuring they are only defined within function scopes.
Update `LambdaCollector` to register the body of lambdas as templates
for global definitions.
Adjust `Dumper` to accurately represent lambda parameters.
Update `Specializer` and `TCO` to handle the new `BoundKind::Parameter`.
Refactor `Call` and `TailCall` to use a single `args` node, often a
tuple.
Adjust type signatures in RTL to use `StaticType::Tuple` for function
parameters.
The type inference for collections (tuples, vectors, matrices, and
records) has been significantly refactored.
This includes:
- Introducing distinct `StaticType` variants for `Tuple`, `Vector`, and
`Matrix`.
- Implementing more robust type checking for these collections, allowing
for homogeneous and heterogeneous structures.
- Enhancing the `is_assignable_from` method to handle type compatibility
between these collection types.
- Updating `Value::static_type()` to correctly infer the types of
literal collections.
- Improving the type inference for map literals to create `Record`
types.
- Adding comprehensive unit tests for tuple, vector, matrix, and record
type inference.
Replaces nested `if let` statements with sequential `if let` bindings,
improving readability. This change is applied to `Dumper::log_bound`,
`LambdaCollector::visit_bound`, `Environment::call_object_method`, and
`Environment::call_lambda`.
This commit updates benchmark values in several example files to reflect
minor performance variations. It also includes adjustments to
integration
and unit tests to align with recent changes in the type checking and VM
logic, specifically concerning closures and TCO handling.
correctly initialize the `TypeChecker` with argument types during
macro expansion.
- Refactor `Specializer::compile` to perform type checking with provided
arguments before specialization and to correctly extract the return
type.
- Enhance the `Dumper` to introspect and display specialized closure
bodies.
- Update `LambdaCollector` to use `BoundNode` consistently.
- Modify `TypeChecker` to accept and inject specialized argument types
for lambdas.
Introduces `LambdaCollector` to gather lambda functions and populate the
function registry. This enables the `Specializer` to work with
user-defined functions.
The `Environment` struct is updated to manage the `function_registry`
and `monomorph_cache`, which are essential for the specialization
process.
The `link` method in `Environment` now incorporates lambda collection
and node specialization before applying TCO optimization. This ensures
that lambdas are properly processed and specialized for potential
performance gains.
The `Specializer`'s `new` constructor has been modified to accept and
initialize the `MonoCache` through an `Rc<RefCell<MonoCache>>`. This
allows the cache to be shared across different specialized functions.
Also includes minor refactoring and type adjustments in `specializer.rs`
for better clarity and consistency.
The specialization logic has been refactored to include a call to
`rtl_lookup` for host functions. This allows the specializer to
recognize and optimize calls to built-in runtime functions by directly
returning their specialized values.
Additionally, the `_rtl_lookup` field in `Specializer` has been renamed
to `rtl_lookup` for clarity. The `visit_call` method has also been
renamed to `specialize_call_logic` to better reflect its purpose, and it
now correctly handles `TailCall` nodes by preserving them.
The `Get`, `DefLocal`, and `DefGlobal` bound kinds now store the symbol
name associated with the variable. This information is useful for
debugging and provides more context in the compiled output.
Update the testing section to clarify when warnings should be eliminated
and tests run.
Add the new `specializer` module to the compiler's public API.
Add the `type_registry` module to the `rtl` module's public API.
The `unless` macro's expansion was updated to use `...` (NOP) instead of
`void`. This change also involved removing the `void` keyword from the
parser, as it is no longer used.
Boolean literals `true` and `false` are now parsed as identifiers,
matching the behavior of other constants like `NaN`. This change aligns
the AST representation of boolean literals with their runtime constants.
This commit introduces a new module `ast::rtl` to house the standard
runtime library functions for the language. It includes implementations
for arithmetic operators (+, -, *, /, //, %), logical operators (and,
or, xor, not), bitwise shifts (<<, >>), and comparison operators (=, <>,
<, >, <=, >=).
Additionally, a `date` function for parsing datetime strings has been
added. This enhances the language's capability to handle basic
mathematical and logical operations.
Add RTL functions for arithmetic and comparisons
This commit introduces the runtime library (RTL) functions for basic
arithmetic operations, logical/bitwise operations, and comparisons. It
also includes a `date` function for parsing datetime strings.
Integration tests have been updated to cover these new functionalities.
This commit introduces the `DateTime` type to the language, enabling
users to work with dates and times. It includes:
- A new `DateTime` variant in the `Value` and `StaticType` enums.
- A `date` function for parsing date strings into `DateTime` values.
- Overloads for `+` and `-` operators to support `DateTime` arithmetic.
- Comparison operators (`>`, `<`) for `DateTime` values.
- Corresponding unit tests for `DateTime` operations.
- Updates to the `gemini.md` documentation to reflect the new
functionality.
Add DateTime type and operations
Introduce a new `DateTime` type to the language, allowing for date and
time manipulation. This includes:
* Parsing dates and datetimes from strings.
* Performing arithmetic operations (addition and subtraction) between
`DateTime` and `Int` (representing milliseconds) and between two
`DateTime` values (resulting in an `Int` duration).
* Enabling comparison operations (`>`, `<`) between `DateTime` values.
This commit refactors the type checking logic for functions and
operators to improve type safety and expressiveness.
Key changes include:
- Introduced `StaticType::FunctionOverloads` to represent functions with
multiple possible signatures, enabling better handling of operator
overloading.
- Updated the `TypeChecker` to correctly infer function types and
resolve calls with overloaded functions.
- Modified the `Environment` to register standard library functions with
specific `FunctionOverloads` signatures, replacing the previous
`StaticType::Any` approach.
- Enhanced the `StaticType::resolve_call` method to intelligently match
argument types against expected parameters, including handling of
overloaded functions.
- Added new tests to verify the correctness of type inference for lambda
returns and operator overloading.
Introduces the `TypeChecker` struct and its associated `TypeContext` for
performing static type analysis on the abstract syntax tree.
This change includes:
- A new `type_checker.rs` module.
- Integration of `TypeChecker` into the `Environment::compile` method.
- Updates to `Environment` to manage global types.
- Renaming `bound_nodes.rs`'s `BoundNode` to `TypedNode` to reflect its
type-checked nature.
- Refinements in binder and macro expansion to accommodate type
information.
The Binder has been refactored to simplify its internal types and reduce
redundancy.
Key changes include:
- Removed StaticType from LocalInfo, as it's not strictly needed for the
binder.
- Simplified `define` in CompilerScope to not take a StaticType.
- Removed StaticType from upvalues in FunctionCompiler.
- Adjusted global mappings to store only the index, not the type.
- Updated BoundKind to use a generic type parameter `T` for metadata,
defaulting to `()` for untyped bound nodes.
- Introduced `BoundNode` and `TypedNode` type aliases for clarity.
- Minor adjustments to dumper and VM to accommodate the new generic
BoundKind.
This commit replaces the `lazy_static` crate with `std::sync::OnceLock`.
This removes an external dependency and utilizes a standard library
feature for lazy initialization.
Additionally, a benchmark regression test has been added to
`src/ast/tester.rs`.
Introduces a `Symbol` struct to represent identifiers, incorporating
macro hygiene context. Updates various parts of the AST compiler and
environment to use `Symbol` instead of raw `Rc<str>` for identifiers,
improving robustness for macro expansions.
Also includes:
- Adds a new example `macro_hygiene.myc`.
- Updates `.gitignore`.
- Refactors `MacroExpander` to use `ExpansionState` for cleaner template
expansion.
- Adjusts `VM::eval_observed` to ensure `after_eval` is called
correctly.
- Resets `Environment` for each test run and compilation/dumping in
`main.rs`.
Ensure that variables are not redefined within the same scope, both for
global and function-local variables. This commit modifies the `define`
method in `CompilerScope` to return a `Result` and includes checks for
existing definitions. The `Binder`'s logic for handling global variables
and function parameters is also updated to incorporate these new checks
and return appropriate errors.
Truncate trace logs to improve GUI performance and prevent excessive
display. This change also introduces line length limits for individual
log entries, adding "..." for truncated lines. The display logic for
trace logs has been updated to use `egui::ScrollArea` for better user
experience.
This commit introduces support for AST macros and templates, enabling
users to define and use custom, reusable components within the visual
DSL.
Key changes include:
- A new `MacroRegistry` to manage macro definitions.
- A `MacroExpander` to process macro calls and expand templates.
- A `MacroEvaluator` trait for evaluating expressions during expansion.
- The `Expansion` node in `BoundKind` to preserve the original macro
call and its expanded form for debugging.
- Updates to the `Binder`, `Dumper`, and `UpvalueAnalyzer` to handle the
new macro constructs.
- New examples demonstrating various macro functionalities like
`unless`, splicing, and nested macros.
The `eval` and `eval_observed` methods in the `VM` struct were very
similar, with the primary difference being the call to the `VMObserver`
trait. This commit refactors these methods into a single macro,
`dispatch_eval!`, which reduces code duplication and improves
maintainability.
The `Value::TailCallRequest` variant was also updated to use `Box` for
its contents, which helps keep the `Value` enum size consistent.
Finally, the benchmarks in the `examples` directory have been updated to
reflect minor performance changes resulting from these code
modifications.
This commit introduces several improvements:
- Pre-compiles regular expressions in `ast/tester.rs` to avoid redundant
compilation, improving performance.
- Replaces `.extension().map_or(false, |ext| ext == "myc")` with the
more concise and readable `.extension().is_some_and(|ext| ext ==
"myc")`.
- Simplifies upvalue capture logic by using `.or_default()` instead of
`.or_insert_with(HashSet::new)`.
- Adds a `Default` implementation for `TracingObserver`.
- Optimizes string literal parsing in `highlight_myc` to correctly
handle escape sequences.
- Uses `saturating_sub` for bracket depth to prevent underflow.
Introduce a `debug_mode` flag to the `Environment` and a new `run_debug`
method. This method uses a `TracingObserver` to log the VM's execution
flow, including node evaluation and scope state changes. This allows for
detailed inspection of script execution.
The `BoundKind` enum now also includes a `display_name` method for
better log readability.
The `UpvalueAnalyzer` now correctly identifies which lambdas capture
which variable declarations, rather than just marking declarations that
need boxing. This information is stored in a `capture_map`.
The `Binder` has been updated to use this new `capture_map` instead of a
`HashSet` of boxed declarations. The `DefLocal` node in `bound_nodes.rs`
now stores a `captured_by` field, which is a list of lambda identities
that capture the local variable.
A new `dumper` module has been added to provide a human-readable
representation of the bound AST, including information about captured
variables.
The `VM` has been updated to use the `captured_by` field to determine if
a local variable needs to be stored in a `Cell`, rather than relying on
a boolean `is_boxed` flag.
An example script `extreme_capture.myc` has been added to test deep
nesting and variable capture.
A new "Dump AST" button has been added to the UI, which uses the new
`Dumper` to display the bound AST.
The binder now uses an `UpvalueAnalyzer` to identify local variables
that are captured by nested functions. These identified variables are
marked with `is_boxed: true` during `DefLocal` node creation. The VM
then uses this flag to wrap such variables in a `Value::Cell` (using
`Rc<RefCell<_>>`) to ensure they can be mutated across function calls.
feat: Introduce boxing for captured variables
Add UpvalueAnalyzer to identify variables captured by nested lambdas.
Modify Binder to use the analyzer and mark captured local variables for
boxing.
Update BoundKind::DefLocal to include an `is_boxed` flag.
Update VM to box captured variables when they are defined.
Add tests for upvalue capture detection.
Ensure that a script does not contain any tokens after the main
expression. This prevents accidental trailing expressions and enforces a
clearer script structure.
This commit introduces a new `tester` module to the `ast` crate,
enabling functional tests and performance benchmarks for MYC scripts.
Functional tests are executed by reading `.myc` files from the
`examples` directory, parsing expected output comments, and comparing
them against the actual script execution results.
Benchmarking involves measuring the execution time of scripts,
calculating median durations, and comparing them against stored
baselines. The commit also adds support for updating these baselines.
The `ast.rs` CLI and the `main.rs` GUI application have been updated to
expose these new testing and benchmarking features. The `Cargo.toml` and
`Cargo.lock` files have been updated to include the `regex` dependency
required for parsing benchmark comments.
Michael Schimmel