Implement strict block scoping and distinguish between statements and
expressions.
This change introduces hierarchical scopes for functions and blocks,
enforces that `def` is a statement with no return value, and prevents
statements from appearing in expression positions or as the last element
of a block. A new example `design-flaw.myc` is added to demonstrate
the uninitialized variable issue solved by these changes.
The `Parameter` kind in `UntypedKind` was only used for identifiers that
were being declared or referenced. This commit renames it to
`Identifier` and updates all the necessary code to reflect this change.
This simplifies the AST and makes it more consistent.
Additionally, a new macro `repeat` has been added to
`src/ast/system.myc`.
Introduce new indicators for Simple Moving Average (SMA) and Weighted
Moving Average (WMA), along with their Hull Moving Average (HMA)
derivative.
Also refactors series implementation to use `RefCell` for interior
mutability and adds `SeriesMember` trait for better dynamic series
access. Updates `soa_series.myc` example to reflect new series creation
and push syntax.
This commit refactors various AST node types to use `Rc` (Reference
Counting) instead of `Box`. This change is primarily driven by the need
for shared ownership and more efficient handling of potentially
recursive or shared data structures within the AST.
The main benefits of this change include:
- **Reduced cloning:** `Rc` allows multiple parts of the AST to refer to
the same node without needing to clone the entire node. This is
particularly beneficial for optimization passes where nodes might be
visited multiple times.
- **Enabling sharing:** It makes it easier to represent scenarios where
a single AST node might be a child of multiple parent nodes.
- **Performance improvements:** By avoiding unnecessary deep copies of
AST nodes, this change can lead to performance gains, especially
during complex compilation phases.
The Binder and its helper functions have been updated to accept and
propagate a `Diagnostics` struct. This allows for better error reporting
during the binding phase, enabling the compiler to continue processing
even after encountering errors and collect all issues before halting.
The `BoundKind::Error` node and `StaticType::Error` are introduced as
"poison" nodes/types. These nodes indicate that an error occurred during
compilation, preventing further valid processing of that specific AST
fragment but allowing the compiler to continue with other parts of the
code.
The `Parser` has also been updated to return a `Diagnostics` struct,
enabling it to report errors during the initial parsing stage while
still attempting to build a partial AST. This adheres to the same error
recovery strategy.
Introduces type specialization for reactive pipeline nodes and their
data buffers. This eliminates the overhead of generic `Value` enums and
`SharedValueSeries` when dealing with known scalar or record types.
The architecture shifts buffer instantiation from the VM to the runtime
(RTL), leveraging type information from the AST. A new `out_type` field
is added to `BoundKind::Pipe` to store the static type of the pipeline's
output, determined by the type checker.
This enables the creation of specialized `RingBuffer<T>` and
`SharedRecordSeries` (for Struct-of-Arrays layout) when the output type
is known, significantly improving memory usage and processing speed for
time-series data, especially in financial analysis.
Adds support for pipelines, allowing streams to be chained together and
transformed.
This includes new types for `PipeStream` and `SourceAdapter`, along with
modifications
to the `Environment` to manage pipeline execution. A new example
`pipeline.myc`
demonstrates its usage.
Introduces a new `Pipe` node to the Abstract Syntax Tree (AST).
This node represents a functional composition pattern, allowing
for chaining of operations.
The changes include:
- Defining the `Pipe` variant in `BoundKind` and `UntypedKind`.
- Implementing parsing logic for the `pipe` keyword.
- Adding handling for the `Pipe` node in various compiler passes
(analyzer, binder, captures, dumper, macros, optimizer, TCO,
type checker).
- Including a placeholder implementation for `Pipe` execution in the
VM.
- Updating integration tests to use the new `pipe` syntax.
This commit refactors the internal representation of `BoundKind::Record`
to store a `RecordLayout` and a `Vec` of values, rather than a `Vec` of
key-value pairs. This change simplifies the representation and improves
efficiency by decoupling the record's structure from its specific values
during compilation and analysis.
The `RecordLayout` now defines the structure of the record, and the
values are stored in a separate vector, ordered according to the layout.
This allows for better optimization and type checking, as the record's
shape is explicitly defined and immutable once created.
Introduces a new `RecordLayout` system for efficient and type-safe
record handling.
Key changes:
- `RecordLayout` struct with `O(1)` field lookup using FMap
optimization.
- Field accessors (`.name`) are now first-class callable values.
- Parser recognizes dot-prefixed identifiers as field accessors.
- Binder and TypeChecker handle field accessors.
- Optimizer transforms field accessor calls into specialized `GetField`
nodes.
- VM executes `GetField` efficiently by directly accessing record
values.
- Adds a new `records.myc` example showcasing record features.
- Improves comparison logic for records to use pointer equality for
layout.
This commit introduces newtype wrappers for `LocalSlot`, `UpvalueIdx`,
and `GlobalIdx` to improve type safety and clarity. These wrappers
replace direct use of `u32` for indices, making the code more robust and
easier to understand.
The changes include:
- Defining `LocalSlot`, `UpvalueIdx`, and `GlobalIdx` structs.
- Implementing `Display` for these new types to provide user-friendly
output.
- Updating the `Address` enum to use these new types.
- Modifying various compiler components (analyzer, binder, optimizer,
type checker, environment, VM) to use the new types.
- Adjusting tests to accommodate the changes.
The `NodeIdentity` struct has been refactored to use a unique `id` field
generated by an atomic counter. This ensures that each `NodeIdentity`
instance is distinct, even if it has the same `SourceLocation`. The
`location` field is now optional, allowing for anonymous nodes.
This change improves the reliability of identity comparisons and
provides a more robust way to manage AST node identities.
The `UpvalueAnalyzer` pass is no longer necessary as the binder now
directly tracks captures. This commit removes the `UpvalueAnalyzer`
struct and associated logic from `src/ast/compiler/upvalues.rs`.
The `Binder::bind_root` function now returns the `captures` map, which
is then processed by a new `CapturePass` in `src/ast/environment.rs`
before type checking. This consolidates capture logic within the binder
and its subsequent processing steps.
Introduce a `ScopeKind` enum to distinguish between root and local
scopes.
This allows the `FunctionCompiler` to correctly handle global variable
declarations in the root scope and local variables in other scopes.
The `define_variable` method is introduced to encapsulate this logic.
This commit refactors the `UsageInfo` struct to use a single
`HashSet<Address>` for both used and assigned addresses, simplifying the
logic. It also updates the `SubstitutionMap` to use a similar approach
for tracking assigned values.
Key changes include:
- `UsageInfo` now has `used` and `assigned` fields of type
`HashSet<Address>`.
- `SubstitutionMap`'s `add_local`, `add_global`, `add_upvalue`,
`remove_local`, `remove_global`, and `remove_upvalue` methods have
been replaced with a generic `add_value` and `remove_value` that
operate on `Address`.
- The `Optimizer`'s `collect_pattern_usage` and `visit_node` methods
have been updated to use the new `UsageInfo` and `SubstitutionMap`
APIs.
- `Get` and `Set` nodes now use `sub.map_address` to transform addresses
based on the current substitution.
This commit consolidates `DefLocal` and `DefGlobal` into a single
`Define` bound kind. This simplifies the AST and makes it more
consistent.
It also introduces `DeclarationKind` to differentiate between variable
and parameter definitions.
Renames `DefDestructure` to `Destructure` to better reflect its use in
both definitions and assignments.
Introduces `bind_assign_pattern` to handle assignment destructuring in
the binder.
Adds `test_assign_destructuring` to verify assignment destructuring
functionality.
This commit introduces the `DefDestructure` bound kind and modifies the
binder, analyzer, type checker, and VM to support destructuring in `def`
statements. This allows for pattern matching on the right-hand side of a
`def` to bind multiple variables.
The parser has been updated to accept patterns in `def` statements. The
binder now handles `UntypedKind::Def` with a `target` pattern, rather
than a simple `name`. This enables destructuring.
The `Gemini.md` documentation has been updated to include a new rule for
incremental development.
The `again` keyword is introduced to facilitate explicit recursive
function calls.
It is restricted to tail-call positions to prevent dead code and ensure
TCO
optimization. Type checking is enhanced to validate argument types
against
function parameters.
This commit introduces optimizations for nested destructuring, allowing
tuples and records to be flattened and matched directly against function
arguments. This significantly improves performance by enabling more
constant folding and reducing intermediate allocations.
The changes include:
- Modifying the `Binder` to correctly count nested parameters.
- Enhancing `flatten_tuple` in the `Optimizer` to handle records and NOP
nodes.
- Updating `map_params_to_args` to recursively destructure nested
compound arguments.
- Adding integration tests to verify the correctness of tuple-to-tuple
and record-to-tuple destructuring optimizations.
The `is_tail` field on the `BoundKind::Call` node was an intermediate
representation for tail-call optimization and is no longer needed as a
distinct field. The TCO pass now embeds this information directly into
the `RuntimeMetadata` of the `ExecNode`, which is the VM's internal AST.
This simplifies the `BoundKind::Call` structure and removes redundant
information.
The `TailCall` variant has been merged into `Call` by adding an
`is_tail` boolean field. This simplifies the AST by reducing the number
of node variants and makes it easier to handle tail call optimization.
The `tco.rs` module is responsible for setting this flag when a call
occurs in tail position.
This commit renames `Map` to `Record` and updates all related AST nodes,
binders, type checkers, and runtime values to reflect this change. This
is a semantic change to better align with common programming language
terminology.
Introduce `BoundKind::Parameter` to represent function parameters.
Modify `Binder` to correctly handle parameters within lambda
definitions, ensuring they are only defined within function scopes.
Update `LambdaCollector` to register the body of lambdas as templates
for global definitions.
Adjust `Dumper` to accurately represent lambda parameters.
Update `Specializer` and `TCO` to handle the new `BoundKind::Parameter`.
Refactor `Call` and `TailCall` to use a single `args` node, often a
tuple.
Adjust type signatures in RTL to use `StaticType::Tuple` for function
parameters.
The `Get`, `DefLocal`, and `DefGlobal` bound kinds now store the symbol
name associated with the variable. This information is useful for
debugging and provides more context in the compiled output.
Introduces the `TypeChecker` struct and its associated `TypeContext` for
performing static type analysis on the abstract syntax tree.
This change includes:
- A new `type_checker.rs` module.
- Integration of `TypeChecker` into the `Environment::compile` method.
- Updates to `Environment` to manage global types.
- Renaming `bound_nodes.rs`'s `BoundNode` to `TypedNode` to reflect its
type-checked nature.
- Refinements in binder and macro expansion to accommodate type
information.
The Binder has been refactored to simplify its internal types and reduce
redundancy.
Key changes include:
- Removed StaticType from LocalInfo, as it's not strictly needed for the
binder.
- Simplified `define` in CompilerScope to not take a StaticType.
- Removed StaticType from upvalues in FunctionCompiler.
- Adjusted global mappings to store only the index, not the type.
- Updated BoundKind to use a generic type parameter `T` for metadata,
defaulting to `()` for untyped bound nodes.
- Introduced `BoundNode` and `TypedNode` type aliases for clarity.
- Minor adjustments to dumper and VM to accommodate the new generic
BoundKind.
Introduces a `Symbol` struct to represent identifiers, incorporating
macro hygiene context. Updates various parts of the AST compiler and
environment to use `Symbol` instead of raw `Rc<str>` for identifiers,
improving robustness for macro expansions.
Also includes:
- Adds a new example `macro_hygiene.myc`.
- Updates `.gitignore`.
- Refactors `MacroExpander` to use `ExpansionState` for cleaner template
expansion.
- Adjusts `VM::eval_observed` to ensure `after_eval` is called
correctly.
- Resets `Environment` for each test run and compilation/dumping in
`main.rs`.
Ensure that variables are not redefined within the same scope, both for
global and function-local variables. This commit modifies the `define`
method in `CompilerScope` to return a `Result` and includes checks for
existing definitions. The `Binder`'s logic for handling global variables
and function parameters is also updated to incorporate these new checks
and return appropriate errors.
This commit introduces support for AST macros and templates, enabling
users to define and use custom, reusable components within the visual
DSL.
Key changes include:
- A new `MacroRegistry` to manage macro definitions.
- A `MacroExpander` to process macro calls and expand templates.
- A `MacroEvaluator` trait for evaluating expressions during expansion.
- The `Expansion` node in `BoundKind` to preserve the original macro
call and its expanded form for debugging.
- Updates to the `Binder`, `Dumper`, and `UpvalueAnalyzer` to handle the
new macro constructs.
- New examples demonstrating various macro functionalities like
`unless`, splicing, and nested macros.
The `UpvalueAnalyzer` now correctly identifies which lambdas capture
which variable declarations, rather than just marking declarations that
need boxing. This information is stored in a `capture_map`.
The `Binder` has been updated to use this new `capture_map` instead of a
`HashSet` of boxed declarations. The `DefLocal` node in `bound_nodes.rs`
now stores a `captured_by` field, which is a list of lambda identities
that capture the local variable.
A new `dumper` module has been added to provide a human-readable
representation of the bound AST, including information about captured
variables.
The `VM` has been updated to use the `captured_by` field to determine if
a local variable needs to be stored in a `Cell`, rather than relying on
a boolean `is_boxed` flag.
An example script `extreme_capture.myc` has been added to test deep
nesting and variable capture.
A new "Dump AST" button has been added to the UI, which uses the new
`Dumper` to display the bound AST.
The binder now uses an `UpvalueAnalyzer` to identify local variables
that are captured by nested functions. These identified variables are
marked with `is_boxed: true` during `DefLocal` node creation. The VM
then uses this flag to wrap such variables in a `Value::Cell` (using
`Rc<RefCell<_>>`) to ensure they can be mutated across function calls.
feat: Introduce boxing for captured variables
Add UpvalueAnalyzer to identify variables captured by nested lambdas.
Modify Binder to use the analyzer and mark captured local variables for
boxing.
Update BoundKind::DefLocal to include an `is_boxed` flag.
Update VM to box captured variables when they are defined.
Add tests for upvalue capture detection.
This commit introduces two main changes:
1. **Binder Refactoring**: The `Binder` has been refactored to support
recursive definitions by pre-declaring names in the scope before
binding their values. This ensures that a name is visible to itself
when its definition is being processed.
2. **Stdlib Operator Implementation**: Several standard library
operators, including `+`, `-`, `*`, `/`, `>`, and `<`, have been
implemented. These operators now correctly handle both `Int` and
`Float` types for numerical operations and comparisons.
Additionally, the display formatting for `Value::List` and
`Value::Record` has been improved for better readability. The
default source code in `main.rs` has also been updated to include a
Fibonacci example, demonstrating the new recursive capabilities.
This commit introduces support for tuple and map literals in the AST.
Tuples are now represented by `UntypedKind::Tuple` and maps by
`UntypedKind::Map`.
The `Binder` has been updated to correctly handle these new node types
and infer their types.
The `VM` now also supports evaluating tuple and map literals.
This commit replaces `Arc<Mutex<T>>` with `Rc<RefCell<T>>` for managing
shared mutable state within the AST and VM. This change is primarily an
internal refactoring to leverage Rust's standard library more
effectively for single-threaded scenarios, improving performance by
avoiding the overhead of mutexes.
The following types and their usage have been updated:
- `Environment.global_names` and `Environment.global_values`
- `Binder.globals`
- `bound_nodes::BoundKind::Lambda.body` (now `Rc<Node>`)
- `ast::types::NodeIdentity` (now `Rc<NodeIdentity>`)
- `ast::types::Value::List`, `Value::Record`, `Value::Function`,
`Value::Object`, `Value::Cell`
- `ast::nodes::Scope` and `ast::nodes::Context`
- `ast::vm::Closure.function_node` and `Closure.upvalues`
- `ast::vm::VM.globals`
This change does not alter the external behavior of the library but
streamlines internal data management.
This commit introduces `StaticType` and enhances the `Binder` to perform
basic type checking during the binding phase.
Key changes include:
- **`StaticType` enum:** Represents static types such as `Any`, `Void`,
`Bool`, `Int`, `Float`, `Text`, `List`, `Record`, and `Function`.
- **`Node<K, T>`:** The generic `Node` now includes a `ty` field to
store its inferred static type.
- **Binder enhancements:**
- `CompilerScope` now stores `LocalInfo` containing the variable's
slot and `StaticType`.
- `FunctionCompiler` stores upvalues with their associated
`StaticType`.
- `Binder::bind` now returns `Node<BoundKind, StaticType>`,
propagating type information.
- Type checking is added for `If` conditions and `Assign`
operations.
- `Def` nodes now infer and store the type of the defined variable.
- **`Value::static_type()`:** A new method to determine the `StaticType`
of a `Value`.
- **Environment modifications:** `global_names` now stores `(u32,
StaticType)` to associate global variables with their types.
- **Parser modifications:** The `ty` field of nodes is initialized to
`()` by default.
- **VM modifications:** `VM::run` and `VM::eval` now operate on
`Node<BoundKind, StaticType>`.
- **Tests:** Added basic evaluation and type error tests.
feat: Add StaticType and type checking
Introduces `StaticType` enum and integrates it into the AST. The binder
now performs type checking during compilation, ensuring that expressions
conform to expected types, especially for control flow structures like
`if`. This lays the groundwork for static type analysis and error
reporting.
Adds a new `Binder` struct that traverses the AST and resolves variable
references to concrete `Address` types (Local, Upvalue, Global). This
information is crucial for the Virtual Machine's execution phase.
Introduces the `BoundKind` enum to represent the AST after binding.
The `VM` is updated to handle `BoundKind` nodes and execute the bound
AST.
It now manages a call stack, local variables, and closures for function
calls and upvalue capturing.
The `Environment` struct is enhanced to manage global variables and
provide
a unified interface for parsing, binding, and executing scripts. It also
includes basic standard library functions.
Michael Schimmel