Implement strict block scoping and distinguish between statements and
expressions.
This change introduces hierarchical scopes for functions and blocks,
enforces that `def` is a statement with no return value, and prevents
statements from appearing in expression positions or as the last element
of a block. A new example `design-flaw.myc` is added to demonstrate
the uninitialized variable issue solved by these changes.
The `Analyzer` is now responsible for determining the purity of `Define`
bindings. Previously, this responsibility was left to the caller.
Added a new example `design-flaw.myc` to test this change.
Updated `soa_series.myc` due to the purity change.
The 'again' macro is a new addition to the macro system, allowing for
recursive macro expansion. This commit introduces the necessary AST
nodes, macro expansion logic, and an integration test to ensure its
functionality. The `soa_series.myc` example has also been updated to
demonstrate its usage.
The `Parameter` kind in `UntypedKind` was only used for identifiers that
were being declared or referenced. This commit renames it to
`Identifier` and updates all the necessary code to reflect this change.
This simplifies the AST and makes it more consistent.
Additionally, a new macro `repeat` has been added to
`src/ast/system.myc`.
The `PipelineNode` has been refactored into `StreamNode`. This commit
updates the example files to reflect this change and also updates the
`series` constructor to accept a lookback parameter.
The `PipelineNode` was an artifact of a previous implementation and is
no longer needed. The `StreamNode` is the appropriate type for
representing the output of a pipe operation.
The `series` function now requires a `lookback` argument to specify the
maximum number of items to store. This makes the behavior of series more
explicit and prevents accidental unbounded memory growth.
This commit extracts the logic for inlining function calls into a new
private
method, `try_inline`. This improves code organization and reduces
duplication in the `Optimizer`'s `visit_node` method.
The extracted method handles the preparation of substitutions and the
recursive visiting of the inlined node. It also ensures that the
substitution map state is correctly synchronized back to the calling
context.
The type checker now correctly handles unwrapping `Stream(T)` types for
lambda arguments.
A new `build_map_stream` function has been added to `rtl/streams.rs` to
facilitate field access on streams.
The `create-random-ohlc` function now returns a `Stream` instead of a
`Series`.
Examples have been updated to reflect these changes and demonstrate
stream usage.
The type checker's fallback logic for non-lambda nodes was unnecessarily
complex. This commit simplifies it by directly checking the node within
a new, basic type context.
Additionally, the environment now explicitly wraps non-lambda AST nodes
in a lambda before type checking, ensuring consistent handling. This
aligns the type checking process for all top-level expressions.
The `LambdaCollector` was cloning nodes and wrapping them in `Rc`.
However, the `registry` itself is already holding `Rc` clones of the
nodes. This commit removes the redundant `Rc::new` to simplify the code.
This change also addresses a performance regression observed in
benchmarks due to unnecessary atomic overhead from `Arc` in the
`Keyword` type.
order
- Add new_for_inlining to SubstitutionMap to preserve the next_slot
counter, preventing local slot collisions when merging
scopes.
- Update Inliner to request fresh slots for parameters during beta
reduction instead of forcing identity mappings.
- Fix execution order in Optimizer::visit_node for closure inlining
to run prepare_beta_reduction before visiting the function
body, preventing corrupted upvalue captures.
- Fix CLI --no-opt flag to correctly disable the optimizer in the ast
binary.
- Add integration test to prevent future slot clash regressions.
The specializer now correctly identifies when a compiled value is a
scalar and avoids folding it into a constant node. Instead, it updates
the callee to the specialized version if it's an object. This ensures
that scalar folding does not interfere with the program's execution.
Introduce new indicators for Simple Moving Average (SMA) and Weighted
Moving Average (WMA), along with their Hull Moving Average (HMA)
derivative.
Also refactors series implementation to use `RefCell` for interior
mutability and adds `SeriesMember` trait for better dynamic series
access. Updates `soa_series.myc` example to reflect new series creation
and push syntax.
This commit refactors various AST node types to use `Rc` (Reference
Counting) instead of `Box`. This change is primarily driven by the need
for shared ownership and more efficient handling of potentially
recursive or shared data structures within the AST.
The main benefits of this change include:
- **Reduced cloning:** `Rc` allows multiple parts of the AST to refer to
the same node without needing to clone the entire node. This is
particularly beneficial for optimization passes where nodes might be
visited multiple times.
- **Enabling sharing:** It makes it easier to represent scenarios where
a single AST node might be a child of multiple parent nodes.
- **Performance improvements:** By avoiding unnecessary deep copies of
AST nodes, this change can lead to performance gains, especially
during complex compilation phases.
This change introduces Rc<MacroMap> to allow for copy-on-write semantics
when defining macros in nested scopes. When a macro is defined in a
scope that is already shared, the MacroMap is cloned before
modification, ensuring that changes to one scope do not affect others
that share the same map.
Introduces `GlobalFunctionRegistry` and `GlobalAnalyzedRegistry` type
aliases to clarify the usage of `HashMap`s for storing global functions
and their analyzed versions. This change also refactors the
`LambdaCollector` and `Optimizer` to use these new aliases, improving
code readability and maintainability.
The Binder and its helper functions have been updated to accept and
propagate a `Diagnostics` struct. This allows for better error reporting
during the binding phase, enabling the compiler to continue processing
even after encountering errors and collect all issues before halting.
The `BoundKind::Error` node and `StaticType::Error` are introduced as
"poison" nodes/types. These nodes indicate that an error occurred during
compilation, preventing further valid processing of that specific AST
fragment but allowing the compiler to continue with other parts of the
code.
The `Parser` has also been updated to return a `Diagnostics` struct,
enabling it to report errors during the initial parsing stage while
still attempting to build a partial AST. This adheres to the same error
recovery strategy.
Introduces type specialization for reactive pipeline nodes and their
data buffers. This eliminates the overhead of generic `Value` enums and
`SharedValueSeries` when dealing with known scalar or record types.
The architecture shifts buffer instantiation from the VM to the runtime
(RTL), leveraging type information from the AST. A new `out_type` field
is added to `BoundKind::Pipe` to store the static type of the pipeline's
output, determined by the type checker.
This enables the creation of specialized `RingBuffer<T>` and
`SharedRecordSeries` (for Struct-of-Arrays layout) when the output type
is known, significantly improving memory usage and processing speed for
time-series data, especially in financial analysis.
Adds support for pipelines, allowing streams to be chained together and
transformed.
This includes new types for `PipeStream` and `SourceAdapter`, along with
modifications
to the `Environment` to manage pipeline execution. A new example
`pipeline.myc`
demonstrates its usage.
Introduces `StaticType::Optional` to represent values that can be either
of a specific type `T` or `Void`. This is crucial for handling
situations in filter pipes where an expression might not always produce
a value.
The type checker now correctly deduces and propagates this `Optional`
type, and the pipeline operator (`pipe`) is updated to unwrap
`Optional(T)` results, yielding `T`. This ensures that the type system
accurately reflects the potential absence of values in intermediate
pipeline steps.
Also includes a minor rename in the tuple-struct example from `pipe` to
`p` for clarity, and registers the `streams` module.
Introduces a new `Pipe` node to the Abstract Syntax Tree (AST).
This node represents a functional composition pattern, allowing
for chaining of operations.
The changes include:
- Defining the `Pipe` variant in `BoundKind` and `UntypedKind`.
- Implementing parsing logic for the `pipe` keyword.
- Adding handling for the `Pipe` node in various compiler passes
(analyzer, binder, captures, dumper, macros, optimizer, TCO,
type checker).
- Including a placeholder implementation for `Pipe` execution in the
VM.
- Updating integration tests to use the new `pipe` syntax.
This commit refactors the internal representation of `BoundKind::Record`
to store a `RecordLayout` and a `Vec` of values, rather than a `Vec` of
key-value pairs. This change simplifies the representation and improves
efficiency by decoupling the record's structure from its specific values
during compilation and analysis.
The `RecordLayout` now defines the structure of the record, and the
values are stored in a separate vector, ordered according to the layout.
This allows for better optimization and type checking, as the record's
shape is explicitly defined and immutable once created.
The `SubstitutionMap::new_inner()` method provides a more robust way to
create nested substitution maps, ensuring that only global substitutions
are inherited. This prevents potential issues with overlapping local and
upvalue addresses in nested scopes, such as during lambda inlining.
This change also includes:
- A new `try_fold_record` method in `folder.rs` to optimize record
literals into constant values.
- Updates to `inliner.rs` to recognize `Value::Record` for inlining.
- Various test cases to verify record optimization and constant folding.
This commit introduces a new prelude file that defines the `while` macro
and registers it during environment bootstrapping. The `again` macro has
been updated to accept multiple arguments for its recursive call, and
several examples have been adjusted to reflect this change.
Additionally, the `MacroRegistry` in the compiler is now cloneable and
can be moved out of the `MacroExpander`.
Introduces a new `RecordLayout` system for efficient and type-safe
record handling.
Key changes:
- `RecordLayout` struct with `O(1)` field lookup using FMap
optimization.
- Field accessors (`.name`) are now first-class callable values.
- Parser recognizes dot-prefixed identifiers as field accessors.
- Binder and TypeChecker handle field accessors.
- Optimizer transforms field accessor calls into specialized `GetField`
nodes.
- VM executes `GetField` efficiently by directly accessing record
values.
- Adds a new `records.myc` example showcasing record features.
- Improves comparison logic for records to use pointer equality for
layout.
These tests were for record destructuring, which has been removed.
The code for record destructuring was also removed from the type checker
and types modules.
Removes unused `flatten_tuple` function from optimizer utilities. The
functionality was moved to the `Folder` struct, making it more
contextually appropriate. This change streamlines the utility module and
improves code organization.
Remove redundant tuple flattening logic in `map_params_to_args`. The
existing logic for iterating through tuple elements already correctly
handles nested tuples and records by recursively calling
`map_params_to_args`.
Introduces the `Inliner` struct to encapsulate logic related to value
and function inlining. This improves code organization and readability
within the optimizer engine.
Key changes include:
- Moving `is_inlinable_value` to the `Inliner` struct.
- Extracting beta-reduction preparation logic into
`Inliner::prepare_beta_reduction`.
- Moving parameter slot collection to
`Inliner::collect_parameter_slots`.
- Introducing `flatten_tuple` to a new `utils` module, used by both
`Folder` and `Inliner`.
Extract common AST manipulation and folding logic into a new Folder
struct. This improves code organization and reusability. The Optimizer
now delegates these tasks to the Folder.
This commit reorganizes the optimizer code by creating a new `optimizer`
module. The `Optimizer` struct and its related logic remain in
`optimizer/optimizer.rs`, while `SubstitutionMap` and `UsageInfo` are
moved to `optimizer/substitution_map.rs`.
This change improves code organization and makes it easier to manage the
optimizer's components.
This commit introduces newtype wrappers for `LocalSlot`, `UpvalueIdx`,
and `GlobalIdx` to improve type safety and clarity. These wrappers
replace direct use of `u32` for indices, making the code more robust and
easier to understand.
The changes include:
- Defining `LocalSlot`, `UpvalueIdx`, and `GlobalIdx` structs.
- Implementing `Display` for these new types to provide user-friendly
output.
- Updating the `Address` enum to use these new types.
- Modifying various compiler components (analyzer, binder, optimizer,
type checker, environment, VM) to use the new types.
- Adjusting tests to accommodate the changes.
Recursively traverse through `BoundKind::Expansion` nodes when
collecting lambdas to ensure that macros correctly expanded into lambdas
are registered. This also updates assignments to global variables to
correctly track lambdas that have been assigned after macro expansion.
Added a new integration test to verify that macro-wrapped function calls
are correctly inlined and optimized, and that unused definitions are
removed by dead code elimination.
The `NodeIdentity` struct has been refactored to use a unique `id` field
generated by an atomic counter. This ensures that each `NodeIdentity`
instance is distinct, even if it has the same `SourceLocation`. The
`location` field is now optional, allowing for anonymous nodes.
This change improves the reliability of identity comparisons and
provides a more robust way to manage AST node identities.
The `UpvalueAnalyzer` pass is no longer necessary as the binder now
directly tracks captures. This commit removes the `UpvalueAnalyzer`
struct and associated logic from `src/ast/compiler/upvalues.rs`.
The `Binder::bind_root` function now returns the `captures` map, which
is then processed by a new `CapturePass` in `src/ast/environment.rs`
before type checking. This consolidates capture logic within the binder
and its subsequent processing steps.
Introduce a `ScopeKind` enum to distinguish between root and local
scopes.
This allows the `FunctionCompiler` to correctly handle global variable
declarations in the root scope and local variables in other scopes.
The `define_variable` method is introduced to encapsulate this logic.
Introduces a `global_types` field to `TypeContext` and updates the
`TypeChecker` to pass it down. This allows type checking to correctly
infer and set types for global variables, removing the previous
hardcoded `StaticType::Any` for global addresses.
Rename `ast_locals` to `ast_substitutions` for clarity.
Update `is_inlinable_value` to accept `Address` and check global purity
separately.
Simplify inlining logic in `optimize_node`.
This commit refactors the `UsageInfo` struct to use a single
`HashSet<Address>` for both used and assigned addresses, simplifying the
logic. It also updates the `SubstitutionMap` to use a similar approach
for tracking assigned values.
Key changes include:
- `UsageInfo` now has `used` and `assigned` fields of type
`HashSet<Address>`.
- `SubstitutionMap`'s `add_local`, `add_global`, `add_upvalue`,
`remove_local`, `remove_global`, and `remove_upvalue` methods have
been replaced with a generic `add_value` and `remove_value` that
operate on `Address`.
- The `Optimizer`'s `collect_pattern_usage` and `visit_node` methods
have been updated to use the new `UsageInfo` and `SubstitutionMap`
APIs.
- `Get` and `Set` nodes now use `sub.map_address` to transform addresses
based on the current substitution.
This commit consolidates `DefLocal` and `DefGlobal` into a single
`Define` bound kind. This simplifies the AST and makes it more
consistent.
It also introduces `DeclarationKind` to differentiate between variable
and parameter definitions.
The `UsageInfo` struct has been updated to include tracking for
`used_upvalues` and `assigned_upvalues`. The `collect_usage` function
has been modified to handle these new fields when encountering
`Address::Upvalue`.
The `map_params_to_args` function now takes `body_usage` as an argument
to ensure that parameters assigned to or used in the body are correctly
substituted. A new helper function, `collect_parameter_slots_set`, has
been added to gather all parameter slots within a pattern.
The inlining logic in `Optimizer::inline_call` has been enhanced to
prevent inlining if a parameter is used or assigned in the function body
but cannot be substituted. This addresses a bug where aggressive
inlining could lead to incorrect code generation when parameters were
reassigned.
A new integration test, `test_closure_reassignment_optimization_bug`,
has been added to specifically target and verify the fix for this
inlining issue.
Refine the type checker to correctly handle destructuring of various
collection types (tuples, vectors, matrices, lists, records).
Additionally, prevent implicit type coercion for function arguments,
ensuring that only tuples are passed to functions expecting tuple
arguments. This avoids unexpected behavior where vectors or matrices
might be treated as tuples.
Add a new example file demonstrating pattern matching and destructuring
rules.
The `flatten_tuple` function was unnecessarily recursive. It can now
simply return the elements of the tuple directly. The VM's destructuring
logic has been updated to handle nested destructuring more efficiently.
A new integration test case for multi-level destructuring has been
added.
Michael Schimmel