This commit introduces support for unhygienic macro expansion using the
`~ident` syntax. When an identifier is prefixed with a tilde (`~`)
within a macro template, it is no longer subject to hygienic renaming.
Instead, it directly refers to a binding in the call site's environment.
This allows macros to interact with variables defined outside the
macro's scope, such as:
- Modifying call-site variables.
- Capturing call-site variables as upvalues in closures.
- Defining new variables that are visible in the call site.
A new test case `test_macro_tilde_nonparam_assign` has been added to
verify this functionality. Previously, unhygienic escapes were
implicitly handled by `SyntaxKind::Identifier` if they were not macro
parameters. This change makes this behavior explicit and correctly
handles non-parameter identifiers by returning a bare symbol.
Introduces a new AST emitter module responsible for serializing AST
nodes back into Myc source code. This emitter preserves leading comments
and whitespace, enabling a round-trip guarantee for the parser and
emitter.
The documentation in `docs/BNF.md` has been updated to reflect the new
syntax rules for comments, including single-line comments (`;`) and
documentation comments (`;;`). The rules clarify that comments must be
whole lines and that inline comments are disallowed.
The lexer and AST node structures have been updated to accommodate
`CommentLine` variants, allowing for different types of comments
(regular, documentation, and blank lines) to be stored and processed.
A new test suite `tests/comment_roundtrip.rs` has been added to verify
the round-trip property of the parser and emitter across all example
`.myc` files. This test ensures that parsing source code, emitting it,
and then parsing the emitted code again results in the same output
string.
The `ast.rs` binary now includes a `--emit` flag that uses the new
emitter to output the parsed source code, facilitating debugging and
testing of the emitter itself.
This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
Replaces `Value::Object(Rc<dyn Object>)` for SyntaxNodes with a
dedicated `Value::Quote(Rc<SyntaxNode>)`. This provides better type
safety and clarity when dealing with AST nodes as values.
Refactor Value enum for Quote node
This commit introduces a new `Value::Quote` variant to represent quoted
AST nodes directly, replacing the generic `Value::Object` for this
purpose.
This change simplifies the handling of quoted nodes by:
- Directly storing an `Rc<SyntaxNode>` instead of relying on dynamic
downcasting from `Rc<dyn Object>`.
- Streamlining macro expansion logic by removing the need to check for
`SyntaxNode` within `Value::Object`.
- Improving type safety and explicitness in the `Value` enum.
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.
Specifically, the following changes were made:
- In `analyzer.rs`, `crate::ast::types::Identity` and
`crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
`crate::ast::types::SourceLocation`,
`crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
`SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
`RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
`AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
`StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
`PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
`build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
`Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
`build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.
The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
The Binder and related types have been refactored to use the new
`NodeKind` enum instead of the previous `BoundKind`. This commit updates
all references to use the new structure, ensuring consistency across the
compiler's AST representation.
Key changes include:
- Replacing `BoundKind` with `NodeKind` in the Binder's `bind` and
`visit` methods.
- Updating pattern matching and field access to reflect the new enum
variants (e.g., `NodeKind::Def` instead of `BoundKind::Define`).
- Adjusting identifier bindings to use the new `IdentifierBinding` enum.
- Reflecting changes in `DefBinding`, `AssignBinding`, and
`LambdaBinding` structures.
- Ensuring all newly created nodes use `NodeKind` and the appropriate
metadata.
Introduce generic `CompilerPhase` trait to unify different stages of the
bound AST. Rename `LocalSlot` to `VirtualId` and introduce `StackOffset`
for clearer distinction between compile-time and run-time addressing.
Update type aliases and implementations to reflect these changes.
This commit replaces the `UntypedNode` enum with the more accurately
named `SyntaxNode`. This change is primarily for clarity and better
reflects the role of these nodes as representing the structure of the
source code prior to semantic analysis.
The corresponding enum `UntypedKind` has also been renamed to
`SyntaxKind` to maintain consistency.
No functional changes are introduced by this refactoring; it is purely a
renaming and organizational update.
Simplified the AST architecture by removing the overly complex Node<K,
T> structure and replacing it with specialized types for
each phase:
- Introduced UntypedNode in nodes.rs for the Parser and Macro system,
eliminating redundant ty: () initializations.
- Defined a new generic Node<T> in bound_nodes.rs for all compiler
stages, where T represents the phase-specific metadata.
- Updated the Binder to act as the bridge between UntypedNode and
Node<()>.
- Simplified type aliases: TypedNode, AnalyzedNode, and ExecNode now
leverage the streamlined Node<T> structure.
- Updated Parser, Macros, Type-Checker, Optimizer, and VM to reflect
the architectural changes.
- Fixed import chains and resolved needless borrows via clippy.
This change improves type safety, reduces boilerplate code in the
parser, and makes the compiler stages more idiomatic and
readable.
This commit refactors the purity analysis from using a
`HashMap<GlobalIdx, Purity>` to a `Vec<Purity>` indexed by
`GlobalIdx.0`.
This change simplifies the data structure and makes purity lookups more
efficient. The `Binder`'s `globals` field has also been removed as it
was not being used.
The `Binder` struct now accepts a `Rc<RefCell<HashMap<Symbol,
(GlobalIdx, Identity)>>>` for global names. This allows the binder to
directly access and manage global symbols during the binding process,
simplifying the logic and improving efficiency.
Additionally, `CompilerScope` now implements `Default`, and a type alias
`BindingResult` has been introduced for clarity. The `bind_root`
function signature has been updated to accept the `globals` argument.
The `Binder` and `FunctionCompiler` initialization has been refactored
to accept initial scopes and slot counts. This allows for more flexible
management of the compiler's state, particularly for the root scope.
The `bind_root` function now returns the final scopes and slot count of
the root function compiler, enabling the `Environment` to update its
state with these new bindings.
The global variable handling has been integrated into the scope
management, removing the direct reliance on a shared `HashMap` for
globals. This promotes a more consistent approach to symbol resolution.
Introduces `fixed_scope_idx` to `Binder` and `Environment` to track
immutable scopes.
This allows enforcing that definitions (`def`) cannot occur in scopes
that are considered fixed or immutable, such as the global scope or the
initial bootstrap scope.
The `FunctionCompiler` is updated to use `fixed_scope_idx` and `is_root`
to determine scope immutability.
The 'again' macro is a new addition to the macro system, allowing for
recursive macro expansion. This commit introduces the necessary AST
nodes, macro expansion logic, and an integration test to ensure its
functionality. The `soa_series.myc` example has also been updated to
demonstrate its usage.
The `Parameter` kind in `UntypedKind` was only used for identifiers that
were being declared or referenced. This commit renames it to
`Identifier` and updates all the necessary code to reflect this change.
This simplifies the AST and makes it more consistent.
Additionally, a new macro `repeat` has been added to
`src/ast/system.myc`.
This change introduces Rc<MacroMap> to allow for copy-on-write semantics
when defining macros in nested scopes. When a macro is defined in a
scope that is already shared, the MacroMap is cloned before
modification, ensuring that changes to one scope do not affect others
that share the same map.
The Binder and its helper functions have been updated to accept and
propagate a `Diagnostics` struct. This allows for better error reporting
during the binding phase, enabling the compiler to continue processing
even after encountering errors and collect all issues before halting.
The `BoundKind::Error` node and `StaticType::Error` are introduced as
"poison" nodes/types. These nodes indicate that an error occurred during
compilation, preventing further valid processing of that specific AST
fragment but allowing the compiler to continue with other parts of the
code.
The `Parser` has also been updated to return a `Diagnostics` struct,
enabling it to report errors during the initial parsing stage while
still attempting to build a partial AST. This adheres to the same error
recovery strategy.
Introduces a new `Pipe` node to the Abstract Syntax Tree (AST).
This node represents a functional composition pattern, allowing
for chaining of operations.
The changes include:
- Defining the `Pipe` variant in `BoundKind` and `UntypedKind`.
- Implementing parsing logic for the `pipe` keyword.
- Adding handling for the `Pipe` node in various compiler passes
(analyzer, binder, captures, dumper, macros, optimizer, TCO,
type checker).
- Including a placeholder implementation for `Pipe` execution in the
VM.
- Updating integration tests to use the new `pipe` syntax.
This commit introduces a new prelude file that defines the `while` macro
and registers it during environment bootstrapping. The `again` macro has
been updated to accept multiple arguments for its recursive call, and
several examples have been adjusted to reflect this change.
Additionally, the `MacroRegistry` in the compiler is now cloneable and
can be moved out of the `MacroExpander`.
This commit introduces newtype wrappers for `LocalSlot`, `UpvalueIdx`,
and `GlobalIdx` to improve type safety and clarity. These wrappers
replace direct use of `u32` for indices, making the code more robust and
easier to understand.
The changes include:
- Defining `LocalSlot`, `UpvalueIdx`, and `GlobalIdx` structs.
- Implementing `Display` for these new types to provide user-friendly
output.
- Updating the `Address` enum to use these new types.
- Modifying various compiler components (analyzer, binder, optimizer,
type checker, environment, VM) to use the new types.
- Adjusting tests to accommodate the changes.
The `NodeIdentity` struct has been refactored to use a unique `id` field
generated by an atomic counter. This ensures that each `NodeIdentity`
instance is distinct, even if it has the same `SourceLocation`. The
`location` field is now optional, allowing for anonymous nodes.
This change improves the reliability of identity comparisons and
provides a more robust way to manage AST node identities.
The `UpvalueAnalyzer` pass is no longer necessary as the binder now
directly tracks captures. This commit removes the `UpvalueAnalyzer`
struct and associated logic from `src/ast/compiler/upvalues.rs`.
The `Binder::bind_root` function now returns the `captures` map, which
is then processed by a new `CapturePass` in `src/ast/environment.rs`
before type checking. This consolidates capture logic within the binder
and its subsequent processing steps.
This commit introduces the `DefDestructure` bound kind and modifies the
binder, analyzer, type checker, and VM to support destructuring in `def`
statements. This allows for pattern matching on the right-hand side of a
`def` to bind multiple variables.
The parser has been updated to accept patterns in `def` statements. The
binder now handles `UntypedKind::Def` with a `target` pattern, rather
than a simple `name`. This enables destructuring.
The `Gemini.md` documentation has been updated to include a new rule for
incremental development.
This commit renames `Map` to `Record` and updates all related AST nodes,
binders, type checkers, and runtime values to reflect this change. This
is a semantic change to better align with common programming language
terminology.
Introduce `BoundKind::Parameter` to represent function parameters.
Modify `Binder` to correctly handle parameters within lambda
definitions, ensuring they are only defined within function scopes.
Update `LambdaCollector` to register the body of lambdas as templates
for global definitions.
Adjust `Dumper` to accurately represent lambda parameters.
Update `Specializer` and `TCO` to handle the new `BoundKind::Parameter`.
Refactor `Call` and `TailCall` to use a single `args` node, often a
tuple.
Adjust type signatures in RTL to use `StaticType::Tuple` for function
parameters.
The type inference for collections (tuples, vectors, matrices, and
records) has been significantly refactored.
This includes:
- Introducing distinct `StaticType` variants for `Tuple`, `Vector`, and
`Matrix`.
- Implementing more robust type checking for these collections, allowing
for homogeneous and heterogeneous structures.
- Enhancing the `is_assignable_from` method to handle type compatibility
between these collection types.
- Updating `Value::static_type()` to correctly infer the types of
literal collections.
- Improving the type inference for map literals to create `Record`
types.
- Adding comprehensive unit tests for tuple, vector, matrix, and record
type inference.
The `unless` macro's expansion was updated to use `...` (NOP) instead of
`void`. This change also involved removing the `void` keyword from the
parser, as it is no longer used.
Boolean literals `true` and `false` are now parsed as identifiers,
matching the behavior of other constants like `NaN`. This change aligns
the AST representation of boolean literals with their runtime constants.
Introduces the `TypeChecker` struct and its associated `TypeContext` for
performing static type analysis on the abstract syntax tree.
This change includes:
- A new `type_checker.rs` module.
- Integration of `TypeChecker` into the `Environment::compile` method.
- Updates to `Environment` to manage global types.
- Renaming `bound_nodes.rs`'s `BoundNode` to `TypedNode` to reflect its
type-checked nature.
- Refinements in binder and macro expansion to accommodate type
information.
The Binder has been refactored to simplify its internal types and reduce
redundancy.
Key changes include:
- Removed StaticType from LocalInfo, as it's not strictly needed for the
binder.
- Simplified `define` in CompilerScope to not take a StaticType.
- Removed StaticType from upvalues in FunctionCompiler.
- Adjusted global mappings to store only the index, not the type.
- Updated BoundKind to use a generic type parameter `T` for metadata,
defaulting to `()` for untyped bound nodes.
- Introduced `BoundNode` and `TypedNode` type aliases for clarity.
- Minor adjustments to dumper and VM to accommodate the new generic
BoundKind.
Introduces a `Symbol` struct to represent identifiers, incorporating
macro hygiene context. Updates various parts of the AST compiler and
environment to use `Symbol` instead of raw `Rc<str>` for identifiers,
improving robustness for macro expansions.
Also includes:
- Adds a new example `macro_hygiene.myc`.
- Updates `.gitignore`.
- Refactors `MacroExpander` to use `ExpansionState` for cleaner template
expansion.
- Adjusts `VM::eval_observed` to ensure `after_eval` is called
correctly.
- Resets `Environment` for each test run and compilation/dumping in
`main.rs`.
This commit introduces support for AST macros and templates, enabling
users to define and use custom, reusable components within the visual
DSL.
Key changes include:
- A new `MacroRegistry` to manage macro definitions.
- A `MacroExpander` to process macro calls and expand templates.
- A `MacroEvaluator` trait for evaluating expressions during expansion.
- The `Expansion` node in `BoundKind` to preserve the original macro
call and its expanded form for debugging.
- Updates to the `Binder`, `Dumper`, and `UpvalueAnalyzer` to handle the
new macro constructs.
- New examples demonstrating various macro functionalities like
`unless`, splicing, and nested macros.
Brummel
Michael Schimmel