The Binder and its helper functions have been updated to accept and
propagate a `Diagnostics` struct. This allows for better error reporting
during the binding phase, enabling the compiler to continue processing
even after encountering errors and collect all issues before halting.
The `BoundKind::Error` node and `StaticType::Error` are introduced as
"poison" nodes/types. These nodes indicate that an error occurred during
compilation, preventing further valid processing of that specific AST
fragment but allowing the compiler to continue with other parts of the
code.
The `Parser` has also been updated to return a `Diagnostics` struct,
enabling it to report errors during the initial parsing stage while
still attempting to build a partial AST. This adheres to the same error
recovery strategy.
- Use a `make-random` factory to create isolated random number
generators.
- Remove the global `prng` from the `Environment`.
- Ensure `make-random` can be called with or without a seed.
Adds support for pipelines, allowing streams to be chained together and
transformed.
This includes new types for `PipeStream` and `SourceAdapter`, along with
modifications
to the `Environment` to manage pipeline execution. A new example
`pipeline.myc`
demonstrates its usage.
Introduce a new field to `Environment` to store a list of pipeline
generator functions.
Add a `run_pipeline` method to `Environment` that iterates through and
executes all registered generators until they are exhausted.
Implement `ObservableStream` trait for `RootStream` and `PipeStream`.
Add a `StreamNode` wrapper for `ObservableStream` to be used as a script
object.
Register `create-random-ohlc` as a native function that creates a
`RootStream`, sets up a OHLC record layout, and registers a stateful
generator closure in the environment's pipeline generators.
The `CallFrame` struct and various VM methods now use `Rc<dyn Object>`
to hold closures, allowing for more flexibility and avoiding unnecessary
cloning.
This change also addresses the performance concern regarding deep copies
of closures, preferring `Rc<dyn Trait>` with local `downcast_ref` where
appropriate.
Additionally, a documentation note has been added to `gemini.md`
regarding this performance preference in Rust.
This commit introduces a new prelude file that defines the `while` macro
and registers it during environment bootstrapping. The `again` macro has
been updated to accept multiple arguments for its recursive call, and
several examples have been adjusted to reflect this change.
Additionally, the `MacroRegistry` in the compiler is now cloneable and
can be moved out of the `MacroExpander`.
Introduces a new `RecordLayout` system for efficient and type-safe
record handling.
Key changes:
- `RecordLayout` struct with `O(1)` field lookup using FMap
optimization.
- Field accessors (`.name`) are now first-class callable values.
- Parser recognizes dot-prefixed identifiers as field accessors.
- Binder and TypeChecker handle field accessors.
- Optimizer transforms field accessor calls into specialized `GetField`
nodes.
- VM executes `GetField` efficiently by directly accessing record
values.
- Adds a new `records.myc` example showcasing record features.
- Improves comparison logic for records to use pointer equality for
layout.
This commit introduces newtype wrappers for `LocalSlot`, `UpvalueIdx`,
and `GlobalIdx` to improve type safety and clarity. These wrappers
replace direct use of `u32` for indices, making the code more robust and
easier to understand.
The changes include:
- Defining `LocalSlot`, `UpvalueIdx`, and `GlobalIdx` structs.
- Implementing `Display` for these new types to provide user-friendly
output.
- Updating the `Address` enum to use these new types.
- Modifying various compiler components (analyzer, binder, optimizer,
type checker, environment, VM) to use the new types.
- Adjusting tests to accommodate the changes.
The `NodeIdentity` struct has been refactored to use a unique `id` field
generated by an atomic counter. This ensures that each `NodeIdentity`
instance is distinct, even if it has the same `SourceLocation`. The
`location` field is now optional, allowing for anonymous nodes.
This change improves the reliability of identity comparisons and
provides a more robust way to manage AST node identities.
The `UpvalueAnalyzer` pass is no longer necessary as the binder now
directly tracks captures. This commit removes the `UpvalueAnalyzer`
struct and associated logic from `src/ast/compiler/upvalues.rs`.
The `Binder::bind_root` function now returns the `captures` map, which
is then processed by a new `CapturePass` in `src/ast/environment.rs`
before type checking. This consolidates capture logic within the binder
and its subsequent processing steps.
The Analyzer has been refactored to decorate `TypedNode`s with their
purity and recursion status. This involves creating a new `AnalyzedNode`
type and a `NodeMetrics` struct to hold this information. The `Analyzer`
now returns an `AnalyzedNode` instead of a separate `Analysis` struct.
This change lays the groundwork for future optimizations and analysis
passes.
This commit introduces a new AST analysis pass that identifies function
purity and recursion. This information is then used by the optimizer and
specializer to make more informed decisions, particularly regarding
inlining.
The `Analyzer` struct and its associated `Analysis` struct are
responsible for traversing the AST and collecting this data.
Key changes include:
- A new `analyzer` module is added to `ast::compiler`.
- `Analyzer::analyze` performs a two-pass traversal to collect
global-to-lambda mappings and then analyze purity and recursion.
- The `Optimizer` and `Specializer` are updated to accept and utilize
the `Analysis` data.
- Recursion checks in `Optimizer` and `Specializer` are replaced with
checks against the pre-computed `Analysis.is_recursive` set.
- The `Environment` now stores and passes the `Analysis` results to the
compiler stages.
Introduces `register_native` for direct registration of `NativeFunction`
and `register_native_fn` for convenience from closures. The
`Environment::run`
method is removed, and its functionality is now handled by
`Environment::instantiate`,
which packages the linked AST into an invokable `NativeFunction`. This
streamlines
the execution path and better separates compilation/linking from runtime
execution.
Move `Purity` enum definition from `optimizer.rs` to `types.rs` and
create a `NativeFunction` struct to hold the function and its purity.
Update `Value::Function` to store `Rc<NativeFunction>` and
`Value::make_function`
helper to simplify creation.
This change allows tracking the purity of native functions, which is
useful
for optimization and static analysis.
Replaces the custom LCG implementation with `fastrand` for improved
random number generation.
This commit introduces the `fastrand` crate to the project for robust
and efficient pseudo-random number generation.
The `Environment` struct now includes a `prng` field to hold the random
number generator.
The built-in `random` function now utilizes this PRNG for generating
floating-point random numbers.
A new `seed!` function is added to allow users to seed the PRNG for
deterministic random sequences.
This change enhances the randomness capabilities of the language, making
it suitable for simulations and other applications requiring good
quality random numbers.
Refactor the optimizer to use a Purity enum instead of a boolean for
tracking function purity. This allows for a more granular representation
of purity:
- `Impure`: Functions with side effects.
- `SideEffectFree`: Functions without side effects but may not be
deterministic (e.g., `now()`, `random()`).
- `Pure`: Functions without side effects and are deterministic.
This change enhances the optimizer's ability to perform more aggressive
optimizations by accurately determining function purity.
The Optimizer now accepts a typed lambda registry, allowing it to
perform more aggressive inlining and beta-reduction on globally defined
functions. This is a significant step towards optimizing recursive and
globally defined lambdas more effectively.
The optimizer now uses a simple boolean flag (`optimization`) instead of
an `optimization_level` (u32). This simplifies the optimizer's logic and
makes it easier to enable or disable optimizations.
The command-line interface and internal testing have been updated to
reflect this change.
This commit introduces several improvements to the AST optimizer and
dead code elimination (DCE) logic:
- **Address Mapping:** The `Get` and `Set` operations now correctly map
local slots and global indices using the substitution map, ensuring
that remapped variables are handled properly.
- **Parameter Slot Mapping:** Parameters are now explicitly mapped to
their correct slots within the `Lambda` node, preventing potential
slot reassignments in the function body.
- **Pure Function Analysis:** The `is_pure` function has been refined to
accurately identify pure expressions, including calls to pure
functions and stateless closures.
- **Dead Code Elimination:** DCE logic in `BoundKind::Block` has been
enhanced to remove unused local definitions and pure, non-essential
expressions more effectively. Global DCE is also improved.
- **Constant Folding:** `try_fold_pure` is now more robust in folding
pure function calls with constant arguments, reducing redundant
computations.
- **Beta Reduction:** `try_beta_reduce` has been updated to avoid
reducing if the body contains `DefLocal`, ensuring correctness.
- **Global Purity Tracking:** A `global_purity` map is introduced to
track the purity of global values, enabling better optimization of
global accesses.
- **Inlinable Value Check:** The `is_inlinable_value` check now
correctly identifies stateless closures for inlining.
- **Optimization Level:** The default `optimization_level` is set to 2
to enable more aggressive optimizations.
This commit introduces purity inference to the compiler's optimizer.
This allows for more aggressive constant folding and dead code
elimination by tracking which functions and global variables
are free of side effects.
Key changes include:
- Added `global_purity` field to `Optimizer` and `Environment`.
- Modified `Optimizer::is_pure` to recursively determine if an
AST node represents a pure computation.
- Introduced `Optimizer::try_fold_pure` to replace the old
`try_fold_intrinsic`, enabling folding of pure function calls
with constant arguments.
- Updated `Environment::register_native` and
`Environment::register_constant`
to optionally record purity.
- Added purity flags to several built-in functions in `core.rs` and
`datetime.rs`.
- A new example `optimizer_purity.myc` demonstrates the new feature.
Introduce global inlining by allowing the Optimizer to access the
environment's global values. This enables replacing global variable
accesses with their constant values when appropriate.
Additionally, implement Dead Code Elimination (DCE) for global
definitions. A global definition can be removed if it was only used for
inlining within the current script and has no side effects.
Update the Optimizer struct to hold an optional reference to the global
values and modify the `Optimizer::new` constructor to accept this. The
`Environment::specialize` and `Environment::compile_script` methods are
updated to pass the global values to the optimizer.
The `SubstitutionMap` is also updated to track script-local global
substitutions and used global indices, supporting both global inlining
and global DCE.
The `is_tail` field on the `BoundKind::Call` node was an intermediate
representation for tail-call optimization and is no longer needed as a
distinct field. The TCO pass now embeds this information directly into
the `RuntimeMetadata` of the `ExecNode`, which is the VM's internal AST.
This simplifies the `BoundKind::Call` structure and removes redundant
information.
The optimizer has been refactored to separate its phases and introduce
more aggressive collapsing capabilities.
Phase 2 (Cracking) now focuses on stateless transformations, making it
easier to reason about and potentially parallelize.
Phase 2.5 (Aggressive Collapsing) has been introduced, enabling
optimizations like:
- Beta-reduction for lambda literals.
- Cracking and inlining for constant closures.
- Folding intrinsics for constant arithmetic.
- Short-circuiting conditional expressions.
These changes aim to improve performance by reducing redundant
computations and code bloat. The maximum number of optimization passes
has also been increased to 5 to allow for more complex transformations.
Introduces a new optimizer pass that can "crack" closures, allowing for
more aggressive specialization. It also enables inlining of upvalues
that point to immutable global variables. This removes overhead for
higher-order functions and currying when arguments are statically
resolvable.
Replaces nested `if let` statements with sequential `if let` bindings,
improving readability. This change is applied to `Dumper::log_bound`,
`LambdaCollector::visit_bound`, `Environment::call_object_method`, and
`Environment::call_lambda`.
This commit updates benchmark values in several example files to reflect
minor performance variations. It also includes adjustments to
integration
and unit tests to align with recent changes in the type checking and VM
logic, specifically concerning closures and TCO handling.
correctly initialize the `TypeChecker` with argument types during
macro expansion.
- Refactor `Specializer::compile` to perform type checking with provided
arguments before specialization and to correctly extract the return
type.
- Enhance the `Dumper` to introspect and display specialized closure
bodies.
- Update `LambdaCollector` to use `BoundNode` consistently.
- Modify `TypeChecker` to accept and inject specialized argument types
for lambdas.
Introduces `LambdaCollector` to gather lambda functions and populate the
function registry. This enables the `Specializer` to work with
user-defined functions.
The `Environment` struct is updated to manage the `function_registry`
and `monomorph_cache`, which are essential for the specialization
process.
The `link` method in `Environment` now incorporates lambda collection
and node specialization before applying TCO optimization. This ensures
that lambdas are properly processed and specialized for potential
performance gains.
The `Specializer`'s `new` constructor has been modified to accept and
initialize the `MonoCache` through an `Rc<RefCell<MonoCache>>`. This
allows the cache to be shared across different specialized functions.
Also includes minor refactoring and type adjustments in `specializer.rs`
for better clarity and consistency.
This commit introduces a new module `ast::rtl` to house the standard
runtime library functions for the language. It includes implementations
for arithmetic operators (+, -, *, /, //, %), logical operators (and,
or, xor, not), bitwise shifts (<<, >>), and comparison operators (=, <>,
<, >, <=, >=).
Additionally, a `date` function for parsing datetime strings has been
added. This enhances the language's capability to handle basic
mathematical and logical operations.
Add RTL functions for arithmetic and comparisons
This commit introduces the runtime library (RTL) functions for basic
arithmetic operations, logical/bitwise operations, and comparisons. It
also includes a `date` function for parsing datetime strings.
Integration tests have been updated to cover these new functionalities.
This commit introduces the `DateTime` type to the language, enabling
users to work with dates and times. It includes:
- A new `DateTime` variant in the `Value` and `StaticType` enums.
- A `date` function for parsing date strings into `DateTime` values.
- Overloads for `+` and `-` operators to support `DateTime` arithmetic.
- Comparison operators (`>`, `<`) for `DateTime` values.
- Corresponding unit tests for `DateTime` operations.
- Updates to the `gemini.md` documentation to reflect the new
functionality.
Add DateTime type and operations
Introduce a new `DateTime` type to the language, allowing for date and
time manipulation. This includes:
* Parsing dates and datetimes from strings.
* Performing arithmetic operations (addition and subtraction) between
`DateTime` and `Int` (representing milliseconds) and between two
`DateTime` values (resulting in an `Int` duration).
* Enabling comparison operations (`>`, `<`) between `DateTime` values.
This commit refactors the type checking logic for functions and
operators to improve type safety and expressiveness.
Key changes include:
- Introduced `StaticType::FunctionOverloads` to represent functions with
multiple possible signatures, enabling better handling of operator
overloading.
- Updated the `TypeChecker` to correctly infer function types and
resolve calls with overloaded functions.
- Modified the `Environment` to register standard library functions with
specific `FunctionOverloads` signatures, replacing the previous
`StaticType::Any` approach.
- Enhanced the `StaticType::resolve_call` method to intelligently match
argument types against expected parameters, including handling of
overloaded functions.
- Added new tests to verify the correctness of type inference for lambda
returns and operator overloading.
Introduces the `TypeChecker` struct and its associated `TypeContext` for
performing static type analysis on the abstract syntax tree.
This change includes:
- A new `type_checker.rs` module.
- Integration of `TypeChecker` into the `Environment::compile` method.
- Updates to `Environment` to manage global types.
- Renaming `bound_nodes.rs`'s `BoundNode` to `TypedNode` to reflect its
type-checked nature.
- Refinements in binder and macro expansion to accommodate type
information.
Introduces a `Symbol` struct to represent identifiers, incorporating
macro hygiene context. Updates various parts of the AST compiler and
environment to use `Symbol` instead of raw `Rc<str>` for identifiers,
improving robustness for macro expansions.
Also includes:
- Adds a new example `macro_hygiene.myc`.
- Updates `.gitignore`.
- Refactors `MacroExpander` to use `ExpansionState` for cleaner template
expansion.
- Adjusts `VM::eval_observed` to ensure `after_eval` is called
correctly.
- Resets `Environment` for each test run and compilation/dumping in
`main.rs`.
This commit introduces support for AST macros and templates, enabling
users to define and use custom, reusable components within the visual
DSL.
Key changes include:
- A new `MacroRegistry` to manage macro definitions.
- A `MacroExpander` to process macro calls and expand templates.
- A `MacroEvaluator` trait for evaluating expressions during expansion.
- The `Expansion` node in `BoundKind` to preserve the original macro
call and its expanded form for debugging.
- Updates to the `Binder`, `Dumper`, and `UpvalueAnalyzer` to handle the
new macro constructs.
- New examples demonstrating various macro functionalities like
`unless`, splicing, and nested macros.
Introduce a `debug_mode` flag to the `Environment` and a new `run_debug`
method. This method uses a `TracingObserver` to log the VM's execution
flow, including node evaluation and scope state changes. This allows for
detailed inspection of script execution.
The `BoundKind` enum now also includes a `display_name` method for
better log readability.
The `UpvalueAnalyzer` now correctly identifies which lambdas capture
which variable declarations, rather than just marking declarations that
need boxing. This information is stored in a `capture_map`.
The `Binder` has been updated to use this new `capture_map` instead of a
`HashSet` of boxed declarations. The `DefLocal` node in `bound_nodes.rs`
now stores a `captured_by` field, which is a list of lambda identities
that capture the local variable.
A new `dumper` module has been added to provide a human-readable
representation of the bound AST, including information about captured
variables.
The `VM` has been updated to use the `captured_by` field to determine if
a local variable needs to be stored in a `Cell`, rather than relying on
a boolean `is_boxed` flag.
An example script `extreme_capture.myc` has been added to test deep
nesting and variable capture.
A new "Dump AST" button has been added to the UI, which uses the new
`Dumper` to display the bound AST.
The binder now uses an `UpvalueAnalyzer` to identify local variables
that are captured by nested functions. These identified variables are
marked with `is_boxed: true` during `DefLocal` node creation. The VM
then uses this flag to wrap such variables in a `Value::Cell` (using
`Rc<RefCell<_>>`) to ensure they can be mutated across function calls.
feat: Introduce boxing for captured variables
Add UpvalueAnalyzer to identify variables captured by nested lambdas.
Modify Binder to use the analyzer and mark captured local variables for
boxing.
Update BoundKind::DefLocal to include an `is_boxed` flag.
Update VM to box captured variables when they are defined.
Add tests for upvalue capture detection.
Ensure that a script does not contain any tokens after the main
expression. This prevents accidental trailing expressions and enforces a
clearer script structure.
This commit introduces two main changes:
1. **Binder Refactoring**: The `Binder` has been refactored to support
recursive definitions by pre-declaring names in the scope before
binding their values. This ensures that a name is visible to itself
when its definition is being processed.
2. **Stdlib Operator Implementation**: Several standard library
operators, including `+`, `-`, `*`, `/`, `>`, and `<`, have been
implemented. These operators now correctly handle both `Int` and
`Float` types for numerical operations and comparisons.
Additionally, the display formatting for `Value::List` and
`Value::Record` has been improved for better readability. The
default source code in `main.rs` has also been updated to include a
Fibonacci example, demonstrating the new recursive capabilities.
This commit replaces `Arc<Mutex<T>>` with `Rc<RefCell<T>>` for managing
shared mutable state within the AST and VM. This change is primarily an
internal refactoring to leverage Rust's standard library more
effectively for single-threaded scenarios, improving performance by
avoiding the overhead of mutexes.
The following types and their usage have been updated:
- `Environment.global_names` and `Environment.global_values`
- `Binder.globals`
- `bound_nodes::BoundKind::Lambda.body` (now `Rc<Node>`)
- `ast::types::NodeIdentity` (now `Rc<NodeIdentity>`)
- `ast::types::Value::List`, `Value::Record`, `Value::Function`,
`Value::Object`, `Value::Cell`
- `ast::nodes::Scope` and `ast::nodes::Context`
- `ast::vm::Closure.function_node` and `Closure.upvalues`
- `ast::vm::VM.globals`
This change does not alter the external behavior of the library but
streamlines internal data management.
This commit introduces `StaticType` and enhances the `Binder` to perform
basic type checking during the binding phase.
Key changes include:
- **`StaticType` enum:** Represents static types such as `Any`, `Void`,
`Bool`, `Int`, `Float`, `Text`, `List`, `Record`, and `Function`.
- **`Node<K, T>`:** The generic `Node` now includes a `ty` field to
store its inferred static type.
- **Binder enhancements:**
- `CompilerScope` now stores `LocalInfo` containing the variable's
slot and `StaticType`.
- `FunctionCompiler` stores upvalues with their associated
`StaticType`.
- `Binder::bind` now returns `Node<BoundKind, StaticType>`,
propagating type information.
- Type checking is added for `If` conditions and `Assign`
operations.
- `Def` nodes now infer and store the type of the defined variable.
- **`Value::static_type()`:** A new method to determine the `StaticType`
of a `Value`.
- **Environment modifications:** `global_names` now stores `(u32,
StaticType)` to associate global variables with their types.
- **Parser modifications:** The `ty` field of nodes is initialized to
`()` by default.
- **VM modifications:** `VM::run` and `VM::eval` now operate on
`Node<BoundKind, StaticType>`.
- **Tests:** Added basic evaluation and type error tests.
feat: Add StaticType and type checking
Introduces `StaticType` enum and integrates it into the AST. The binder
now performs type checking during compilation, ensuring that expressions
conform to expected types, especially for control flow structures like
`if`. This lays the groundwork for static type analysis and error
reporting.
Adds a new `Binder` struct that traverses the AST and resolves variable
references to concrete `Address` types (Local, Upvalue, Global). This
information is crucial for the Virtual Machine's execution phase.
Introduces the `BoundKind` enum to represent the AST after binding.
The `VM` is updated to handle `BoundKind` nodes and execute the bound
AST.
It now manages a call stack, local variables, and closures for function
calls and upvalue capturing.
The `Environment` struct is enhanced to manage global variables and
provide
a unified interface for parsing, binding, and executing scripts. It also
includes basic standard library functions.
Michael Schimmel