The `Program` node kind is introduced to represent top-level code
blocks, distinguishing them from `Block` nodes which introduce new
scopes. This commit updates various compiler passes to handle the
`Program` node, ensuring correct AST traversal and processing.
The `Program` node is similar to `Block`, but its definitions propagate
to the enclosing scope, unlike `Block` which creates a new, isolated
scope. This distinction is important for how variables and functions are
resolved within the compiled code.
The `analyzer.rs` and `lambda_collector.rs` files have been updated to
correctly traverse and process new node kinds in the Abstract Syntax
Tree (AST). This ensures that these new node types are properly included
in the analysis and lambda collection phases of the compiler.
Introduces `CallHooks` to enable extensions to the type inference and
AST finalization process. This allows for more dynamic and configurable
behavior without hardcoding symbol names in the compiler core.
Specifically, this commit adds:
- **`series_post_call`**: A hook for the `(series n)` function. It
ensures that the return type is a fresh `TypeVar` when the element
type is `Any`, allowing for proper type inference from subsequent
`push` calls.
- **`series_finalize`**: A hook for `(series n)` that rewrites the AST
node. It replaces the generic `series` call with a pre-configured
factory closure that allocates the correct series storage type (e.g.,
`ScalarSeries<f64>` for floats, `RecordSeries` for records). This
avoids runtime dispatch and relies on type information captured during
compilation.
- **`push_post_call`**: A hook for the `(push series val)` function. It
unifies the series element `TypeVar` with the pushed value's type. It
also handles numeric and record field promotion and updates the
`TypeVar` binding if a promotion occurs.
This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.
Specifically, the following changes were made:
- In `analyzer.rs`, `crate::ast::types::Identity` and
`crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
`crate::ast::types::SourceLocation`,
`crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
`SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
`RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
`AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
`StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
`PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
`build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
`Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
`build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.
The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
The Binder and related types have been refactored to use the new
`NodeKind` enum instead of the previous `BoundKind`. This commit updates
all references to use the new structure, ensuring consistency across the
compiler's AST representation.
Key changes include:
- Replacing `BoundKind` with `NodeKind` in the Binder's `bind` and
`visit` methods.
- Updating pattern matching and field access to reflect the new enum
variants (e.g., `NodeKind::Def` instead of `BoundKind::Define`).
- Adjusting identifier bindings to use the new `IdentifierBinding` enum.
- Reflecting changes in `DefBinding`, `AssignBinding`, and
`LambdaBinding` structures.
- Ensuring all newly created nodes use `NodeKind` and the appropriate
metadata.
Introduce generic `CompilerPhase` trait to unify different stages of the
bound AST. Rename `LocalSlot` to `VirtualId` and introduce `StackOffset`
for clearer distinction between compile-time and run-time addressing.
Update type aliases and implementations to reflect these changes.
Simplified the AST architecture by removing the overly complex Node<K,
T> structure and replacing it with specialized types for
each phase:
- Introduced UntypedNode in nodes.rs for the Parser and Macro system,
eliminating redundant ty: () initializations.
- Defined a new generic Node<T> in bound_nodes.rs for all compiler
stages, where T represents the phase-specific metadata.
- Updated the Binder to act as the bridge between UntypedNode and
Node<()>.
- Simplified type aliases: TypedNode, AnalyzedNode, and ExecNode now
leverage the streamlined Node<T> structure.
- Updated Parser, Macros, Type-Checker, Optimizer, and VM to reflect
the architectural changes.
- Fixed import chains and resolved needless borrows via clippy.
This change improves type safety, reduces boilerplate code in the
parser, and makes the compiler stages more idiomatic and
readable.
This commit refactors the purity analysis from using a
`HashMap<GlobalIdx, Purity>` to a `Vec<Purity>` indexed by
`GlobalIdx.0`.
This change simplifies the data structure and makes purity lookups more
efficient. The `Binder`'s `globals` field has also been removed as it
was not being used.
The `Analyzer` is now responsible for determining the purity of `Define`
bindings. Previously, this responsibility was left to the caller.
Added a new example `design-flaw.myc` to test this change.
Updated `soa_series.myc` due to the purity change.
This commit refactors various AST node types to use `Rc` (Reference
Counting) instead of `Box`. This change is primarily driven by the need
for shared ownership and more efficient handling of potentially
recursive or shared data structures within the AST.
The main benefits of this change include:
- **Reduced cloning:** `Rc` allows multiple parts of the AST to refer to
the same node without needing to clone the entire node. This is
particularly beneficial for optimization passes where nodes might be
visited multiple times.
- **Enabling sharing:** It makes it easier to represent scenarios where
a single AST node might be a child of multiple parent nodes.
- **Performance improvements:** By avoiding unnecessary deep copies of
AST nodes, this change can lead to performance gains, especially
during complex compilation phases.
The Binder and its helper functions have been updated to accept and
propagate a `Diagnostics` struct. This allows for better error reporting
during the binding phase, enabling the compiler to continue processing
even after encountering errors and collect all issues before halting.
The `BoundKind::Error` node and `StaticType::Error` are introduced as
"poison" nodes/types. These nodes indicate that an error occurred during
compilation, preventing further valid processing of that specific AST
fragment but allowing the compiler to continue with other parts of the
code.
The `Parser` has also been updated to return a `Diagnostics` struct,
enabling it to report errors during the initial parsing stage while
still attempting to build a partial AST. This adheres to the same error
recovery strategy.
Introduces type specialization for reactive pipeline nodes and their
data buffers. This eliminates the overhead of generic `Value` enums and
`SharedValueSeries` when dealing with known scalar or record types.
The architecture shifts buffer instantiation from the VM to the runtime
(RTL), leveraging type information from the AST. A new `out_type` field
is added to `BoundKind::Pipe` to store the static type of the pipeline's
output, determined by the type checker.
This enables the creation of specialized `RingBuffer<T>` and
`SharedRecordSeries` (for Struct-of-Arrays layout) when the output type
is known, significantly improving memory usage and processing speed for
time-series data, especially in financial analysis.
Introduces a new `Pipe` node to the Abstract Syntax Tree (AST).
This node represents a functional composition pattern, allowing
for chaining of operations.
The changes include:
- Defining the `Pipe` variant in `BoundKind` and `UntypedKind`.
- Implementing parsing logic for the `pipe` keyword.
- Adding handling for the `Pipe` node in various compiler passes
(analyzer, binder, captures, dumper, macros, optimizer, TCO,
type checker).
- Including a placeholder implementation for `Pipe` execution in the
VM.
- Updating integration tests to use the new `pipe` syntax.
This commit refactors the internal representation of `BoundKind::Record`
to store a `RecordLayout` and a `Vec` of values, rather than a `Vec` of
key-value pairs. This change simplifies the representation and improves
efficiency by decoupling the record's structure from its specific values
during compilation and analysis.
The `RecordLayout` now defines the structure of the record, and the
values are stored in a separate vector, ordered according to the layout.
This allows for better optimization and type checking, as the record's
shape is explicitly defined and immutable once created.
Introduces a new `RecordLayout` system for efficient and type-safe
record handling.
Key changes:
- `RecordLayout` struct with `O(1)` field lookup using FMap
optimization.
- Field accessors (`.name`) are now first-class callable values.
- Parser recognizes dot-prefixed identifiers as field accessors.
- Binder and TypeChecker handle field accessors.
- Optimizer transforms field accessor calls into specialized `GetField`
nodes.
- VM executes `GetField` efficiently by directly accessing record
values.
- Adds a new `records.myc` example showcasing record features.
- Improves comparison logic for records to use pointer equality for
layout.
This commit introduces newtype wrappers for `LocalSlot`, `UpvalueIdx`,
and `GlobalIdx` to improve type safety and clarity. These wrappers
replace direct use of `u32` for indices, making the code more robust and
easier to understand.
The changes include:
- Defining `LocalSlot`, `UpvalueIdx`, and `GlobalIdx` structs.
- Implementing `Display` for these new types to provide user-friendly
output.
- Updating the `Address` enum to use these new types.
- Modifying various compiler components (analyzer, binder, optimizer,
type checker, environment, VM) to use the new types.
- Adjusting tests to accommodate the changes.
This commit consolidates `DefLocal` and `DefGlobal` into a single
`Define` bound kind. This simplifies the AST and makes it more
consistent.
It also introduces `DeclarationKind` to differentiate between variable
and parameter definitions.
Renames `DefDestructure` to `Destructure` to better reflect its use in
both definitions and assignments.
Introduces `bind_assign_pattern` to handle assignment destructuring in
the binder.
Adds `test_assign_destructuring` to verify assignment destructuring
functionality.
This commit introduces the `DefDestructure` bound kind and modifies the
binder, analyzer, type checker, and VM to support destructuring in `def`
statements. This allows for pattern matching on the right-hand side of a
`def` to bind multiple variables.
The parser has been updated to accept patterns in `def` statements. The
binder now handles `UntypedKind::Def` with a `target` pattern, rather
than a simple `name`. This enables destructuring.
The `Gemini.md` documentation has been updated to include a new rule for
incremental development.
The `again` keyword is introduced to facilitate explicit recursive
function calls.
It is restricted to tail-call positions to prevent dead code and ensure
TCO
optimization. Type checking is enhanced to validate argument types
against
function parameters.
The Analyzer has been refactored to decorate `TypedNode`s with their
purity and recursion status. This involves creating a new `AnalyzedNode`
type and a `NodeMetrics` struct to hold this information. The `Analyzer`
now returns an `AnalyzedNode` instead of a separate `Analysis` struct.
This change lays the groundwork for future optimizations and analysis
passes.
This commit introduces a new AST analysis pass that identifies function
purity and recursion. This information is then used by the optimizer and
specializer to make more informed decisions, particularly regarding
inlining.
The `Analyzer` struct and its associated `Analysis` struct are
responsible for traversing the AST and collecting this data.
Key changes include:
- A new `analyzer` module is added to `ast::compiler`.
- `Analyzer::analyze` performs a two-pass traversal to collect
global-to-lambda mappings and then analyze purity and recursion.
- The `Optimizer` and `Specializer` are updated to accept and utilize
the `Analysis` data.
- Recursion checks in `Optimizer` and `Specializer` are replaced with
checks against the pre-computed `Analysis.is_recursive` set.
- The `Environment` now stores and passes the `Analysis` results to the
compiler stages.
Brummel
Michael Schimmel