Compare commits

135 Commits

Author SHA1 Message Date
Brummel 817f7f24c8 Remove data_server integration test (moved to data-server crate)
The /mnt/tickdata integration test now lives in the standalone
data-server crate (tests/data_server.rs there) so it travels with
the code on reuse. The inline unit tests already moved with the
sources during the earlier extraction.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 11:23:02 +02:00
Brummel fad8bc3471 Extract data_server into external data-server crate
The data_server module had no dependency on the rest of myc (only
chrono/regex/zip), so it was moved verbatim into a standalone leaf
crate at its own Gitea repo and is now consumed as a git dependency.
A 'pub use ::data_server;' re-export in src/ast/mod.rs keeps all
existing myc::ast::data_server::… paths valid; no call sites changed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 11:04:03 +02:00
Brummel 9b38907cd5 Add float division and integer/float coercion
This commit introduces support for floating-point division, including
cases where an integer is divided by a float or vice versa. This allows
for more flexible arithmetic operations within the DSL.

Additionally, the `Value::Float` display implementation is adjusted to
show `.0` for whole numbers, improving readability and consistency in
output.
2026-03-31 23:09:02 +02:00
Brummel ed1b5bae7a Remove unused VirtualId import
The `VirtualId` enum was imported but not used in the `macros.rs` file.
This commit removes the unused import to clean up the code.

Fix Zero Width Space handling in lexer

The lexer incorrectly handled the Zero Width Space character (U+200B)
when it appeared inside identifiers. This commit corrects the lexer to
ignore this character in such contexts, ensuring that identifiers
containing it are parsed correctly.

Add REPL functionality to MCP server

This commit introduces a REPL (Read-Eval-Print Loop) mode to the MCP
server, allowing for interactive evaluation of Myc code with persistent
state between calls.

Key additions:
- `REPL_ENV`: A thread-local static variable to hold the persistent
  `Environment` for the REPL session. It's necessary because
  `Environment` uses `Rc<RefCell<_>>` internally and is not
  `Send`/`Sync`, but the `rmcp` server requires its handler to be `Send
  + Sync`. This is safe as the underlying Tokio runtime is
  single-threaded.
- `repl_eval`: A new tool that evaluates a given code string within the
  persistent REPL environment. Bindings (variables, functions, macros)
  are preserved across invocations.
- `repl_reset`: A tool to reset the REPL session, clearing all
  accumulated bindings and starting with a fresh environment.
- The main tool description in `mcp_server.rs` is updated to mention the
  new REPL functionalities.
2026-03-31 21:56:01 +02:00
Brummel 1a4952f055 Refactor type variable counter to be shared 2026-03-31 18:06:47 +02:00
Brummel 7d6e040721 Enforce exhaustive matches for AST nodes
This commit addresses the critical rule regarding exhaustive `match`
expressions on AST node kinds. By explicitly listing all variants
instead of using wildcards (`_ =>`), we ensure that new AST node types
are consciously handled throughout the compilation pipeline. This
practice prevents subtle bugs and makes the compiler more robust against
future changes.

The `CLAUDE.md` file has been updated to reflect this critical rule.

The deleted `BUG_program_node_typeinference.md` file indicates that a
previously identified bug related to type inference with `Program` nodes
has been resolved, likely as a consequence of enforcing these exhaustive
matches.
2026-03-31 17:33:45 +02:00
Brummel 8d14da82c5 Persist macros after expansion
When expanding macros, the expander may encounter new macro
declarations.
These need to be stored back into the environment's macro registry so
that they are available for subsequent compilation passes or other
environments.
Persist macros after expansion

This commit modifies the `expand` method in `src/ast/environment.rs` to
persist macro declarations encountered during expansion. When a macro is
expanded, the `Expander` updates its internal registry. This change
ensures that the updated registry is then stored back into the
`Environment`'s `macro_registry`. This allows subsequent compilations or
expansions within the same environment to correctly recognize and
utilize these newly defined macros.

Additionally, a `NodeKind::Program` case was added to the
`CapturePass::transform` method in `src/ast/compiler/captures.rs`. This
ensures that expressions within a program node are also recursively
transformed, maintaining consistency in the AST processing pipeline.
2026-03-31 17:11:57 +02:00
Brummel 90f9afa0fd Fix: Simplify type checker and optimizer
Removes unnecessary lambda wrapping in the type checking pipeline.
Adds optimization logic for `NodeKind::Program` to handle sequences of
expressions.
Removes debug print statements from the type checker.
2026-03-31 16:42:13 +02:00
Brummel 19008b5d3e BUG-TAG: Type inference for lambda parameters
The type checker incorrectly inferred `Any` for lambda parameters when a
`Program` node was involved, preventing optimizations like constant
folding. This was because the `check_params_tuple` function was
resolving `TypeVar` to `StaticType::Any` instead of propagating the type
variable.

This commit addresses the issue by:
- Explicitly wrapping the bound AST in a parameterless lambda within
  `compile_pipeline`. This ensures that the type checker always receives
  a `Lambda` node, even if the original input was a `Program` node.
- Adding debug logging to the type checker to help diagnose similar
  issues in the future.

Additionally, the commit fixes a bug where `parser.parse_program()` was
used instead of `parser.parse_expression()`, which would consume the
entire input and prevent checking for trailing expressions.
2026-03-31 16:17:47 +02:00
Brummel af8fb5cb7f Refactor binder to use scope depth
Introduced `scope_depth` to the `Binder` to track block nesting. This
allows for correctly determining whether a `def` binding should be
`Address::Global` (at scope depth 0) or `Address::Local`.

The `type_checker` was also updated to handle `Program` and `Block`
nodes by creating a new `TypeContext` for them.

Removed the `wrap_as_lambda` method from `Environment` as the compiler
now always produces a `Program` node, effectively handling the wrapping
at the AST level. Correspondingly, `instantiate` and
`run_script_compiled` were simplified to directly run the `Program`
node.
2026-03-31 15:22:51 +02:00
Brummel 8fd2f2d113 Refactor environment loading and program execution
The `preload_dependencies` function has been refactored to directly run
parsed programs instead of collecting all forms into a single `Program`
node. This simplifies the compilation pipeline by allowing each
dependency to be processed individually.

Additionally, the `VM::run_with_observer` method has been updated to
manage its stack and frames more cleanly, avoiding the need for
`eval_in_frame`. The main loop now correctly extracts expressions from
the `Program` node for source code emission.
2026-03-31 13:59:28 +02:00
Brummel 02ea2f0d80 Add Program node kind
The `Program` node kind is introduced to represent top-level code
blocks, distinguishing them from `Block` nodes which introduce new
scopes. This commit updates various compiler passes to handle the
`Program` node, ensuring correct AST traversal and processing.

The `Program` node is similar to `Block`, but its definitions propagate
to the enclosing scope, unlike `Block` which creates a new, isolated
scope. This distinction is important for how variables and functions are
resolved within the compiled code.
2026-03-31 13:53:51 +02:00
Brummel 35f5ea0db3 Refactor Binder and Environment, remove fixed_scope_idx
The `fixed_scope_idx` field has been removed from `Binder` and
`Environment`. This field was used to enforce immutability on certain
scopes during binding.

The logic for handling the immutability of the root scope (scope 0) has
been moved and refactored. Now, during the `Environment::new` bootstrap
phase, the addresses within the RTL scope (scope 0) are directly
translated from `Address::Local` to `Address::Global`. This ensures that
the Binder inherently treats these as global and immutable without
needing an explicit `fixed_scope_idx` check.

The `Binder::bind_root` signature has been updated to reflect this
change by removing the `fixed_scope_idx` parameter. Consequently, tests
and other usages of `bind_root` have been adjusted.

This change simplifies the binding process by centralizing the
immutability handling to the environment setup phase, making the
Binder's logic cleaner.
2026-03-31 11:19:56 +02:00
Brummel 2cf48566f7 Remove SMA macro
The `sma.myc` file, which defined the Simple Moving Average macro, has
been removed. This functionality is no longer needed.
2026-03-31 10:07:31 +02:00
Brummel dd81482acb Add parse_program to parser
The `Parser` can now parse a sequence of top-level expressions,
representing a complete program. This change modifies the `roundtrip`
function to iterate over these expressions and emit them, allowing for
multi-line program parsing.
2026-03-31 07:55:20 +02:00
Brummel 9013f262ce Add egui_extras dependency
Adds `egui_extras` to Cargo.toml and Cargo.lock to enable enhanced table
widgets.
Also introduces a new `StateTab` enum and integrates the
`egui_extras::TableBuilder` to display global variables in a structured
table within the application's state panel. This enhances the debugging
and introspection capabilities of the compiler UI by providing a clear
view of the environment's global state.
2026-03-30 15:41:25 +02:00
Brummel 3628802fc8 Refactor streams and hooks logic
This commit reorganizes the stream-related logic by splitting the
`streams.rs` file into smaller, more manageable modules: `nodes.rs`,
`hooks.rs`, and `register.rs`.

The `nodes.rs` module now contains the core stream implementations like
`RootStream`, `PipeStream`, and the associated observer patterns.

The `hooks.rs` module encapsulates the compiler hook logic for `pipe`
and `pipe-series`, including type resolution and argument hinting. It
also includes helper functions for building pipe executors and
extracting stream information.

The `register.rs` module handles the registration of stream-related
built-in functions (`create-random-ohlc`, `create-ticker`, `pipe`,
`pipe-series`) within the environment.

This refactoring improves code organization, maintainability, and
testability by separating concerns into dedicated modules.
2026-03-30 15:06:02 +02:00
Brummel 8ef93e2af5 Refactor TypeChecker into modules
The TypeChecker implementation was becoming quite large, so it has been
refactored into several modules:
- `context`: Handles type checking contexts and inference access.
- `inference`: Contains the core Hindley-Milner inference logic
  (unification, generalization, etc.).
- `finalize`: Manages the finalization step, applying substitutions and
  dispatching hooks.
- `check`: Implements the main type checking logic for AST nodes.

This modularization improves code organization and maintainability.
2026-03-30 14:47:38 +02:00
Brummel 2a5b87f45f Add handling for new node kinds in visitors
The `analyzer.rs` and `lambda_collector.rs` files have been updated to
correctly traverse and process new node kinds in the Abstract Syntax
Tree (AST). This ensures that these new node types are properly included
in the analysis and lambda collection phases of the compiler.
2026-03-30 14:30:07 +02:00
Brummel 90df8bf788 Fix type-of and comparison for new Value variants 2026-03-30 14:23:48 +02:00
Brummel 3676eb51e5 Rename pipe-buffered to pipe-series
The `pipe-buffered` function has been renamed to `pipe-series` to better
reflect its functionality. This change involves updating the function
name in the example file `examples/data_stream_pipe_buffered.myc` and
across the Rust source code in `src/ast/rtl/streams.rs`. Additionally,
test cases and documentation have been updated accordingly.
2026-03-30 14:16:04 +02:00
Brummel 24d36a08a6 Refactor 'again' type check to use unify
This change replaces a manual type check for the `again` function with a
call to the generic `unify` function. This improves consistency and
leverages existing type-checking logic.

A new test case `test_again_type_mismatch` has been added to
specifically verify that type mismatches in `again` calls are caught at
compile time, preventing potential infinite loops.
2026-03-30 11:50:08 +02:00
Brummel c403c80f5b Refine type error reporting for readability
The `display_compact` method on `StaticType` has been enhanced to
provide more concise representations of types within error messages.
This change aims to improve the clarity of type-related diagnostics
generated by the compiler.
2026-03-30 11:45:31 +02:00
Brummel 8aa2a67b5b Introduce StaticType::Variadic for variadic parameters
This commit introduces `StaticType::Variadic` to represent variadic
parameter types in function signatures. This allows for more precise
type checking of functions that accept a variable number of arguments,
such as arithmetic operators.

Previously, variadic functions were handled using `StaticType::Any`,
which was too permissive and could lead to runtime errors. The new
`Variadic` type enforces that all arguments must conform to a specified
inner type.

Changes include:

- Adding `StaticType::Variadic` to `src/ast/types.rs`.
- Updating the type checker to handle `Variadic` types correctly when
  unifying with tuples or single arguments.
- Modifying built-in functions like arithmetic operators to use
  `Variadic` for their parameter types.
- Improving error messages for function call mismatches to be more
  specific.
- Adding a new test case to ensure arithmetic type mismatches are caught
  at compile time.
- A new example `data_stream_pipe_buffered.myc` is added.
- The `HMA.myc` example now has a `Skip: output` directive.
2026-03-30 11:36:26 +02:00
Brummel f1621ed1ac Update data_stream_print.myc 2026-03-30 10:54:48 +02:00
Brummel 6d24c5b64c Add pipe-buffered function
Implements `pipe-buffered`, a new primitive that allows accumulating
values from multiple input streams into internal series buffers before
executing a user-provided lambda. This is useful for implementing
lookback logic and ensuring that the lambda only executes when
sufficient data is available in all input streams.

The function takes a `lookback` integer, input streams, and a lambda.
The lambda receives `Series` objects for each input, allowing indexed
access to past values. The `pipe-buffered` function manages the
buffering and fill-gate logic.
2026-03-30 10:54:32 +02:00
Brummel 141ae36ede Refactor: Create typed series factory function
Introduces `create_typed_series` to centralize series creation logic.
This function dispatches on `StaticType` to select the appropriate
`SeriesStorage` implementation,
improving code reuse and maintainability within the series RTL.

The `SeriesHook` compiler hook is updated to use this new factory
function.
Additionally, new tests have been added to `rtl/streams.rs` and
`tests/pipeline.rs`
to specifically cover the behavior of `pipe-buffered`, focusing on its
fill gate
mechanism and multi-input handling.
2026-03-30 10:38:28 +02:00
Brummel 6628941a50 Create propa.myc 2026-03-30 10:31:51 +02:00
Brummel 0ef5faed91 Remove unused Propa example
The `Propa.myc` example was not being used and has been removed to clean
up the examples directory.
2026-03-30 10:31:45 +02:00
Brummel 9b13546609 feat: Add pipe parameter validation and fix VM call frame
Adds a check within the `PipeHook` to ensure the number of input streams
provided to a `pipe` operation matches the number of parameters expected
by the associated lambda function. This prevents runtime errors and
provides a clearer compile-time diagnostic.

Additionally, this commit corrects the order in which a `CallFrame` is
pushed onto the VM's frame stack. By pushing the frame earlier, it
ensures that slow-path `unpack` operations have access to a valid frame,
preventing "No call frame" errors.
2026-03-30 09:43:58 +02:00
Brummel e82d48d1cb Refactor data streams to support time windows
This commit introduces time windowing functionality to the M1 and tick
data streams.

Key changes:

- **`DataServer::stream_m1_windowed` and
  `DataServer::stream_tick_windowed`**: New methods added to
  `DataServer` to allow streaming data within a specified `from_ms` and
  `to_ms` range.
- **`SymbolChunkIter` modifications**: The `SymbolChunkIter` struct now
  includes `from_ms`, `to_ms`, and `exhausted` fields to manage time
  window filtering.
- **`filter_chunk` helper function**: A new utility function to
  efficiently filter individual data chunks based on the time window.
  This function also handles early termination when `to_ms` is exceeded.
- **`HasTimestamp` trait**: Introduced to provide a common interface for
  accessing timestamps across different record types (`M1Parsed`,
  `TickParsed`), enabling generic filtering.
- **`filter_files` method**: Added to `DataServer` to pre-filter file
  keys based on the time window before creating iterators. This
  optimizes file loading by skipping files that fall entirely outside
  the desired range.
- **`create-m1-stream` and `create-tick-stream`**: The native functions
  in `data_streams.rs` are updated to accept optional `DateTime`
  arguments for `from` and `to` timestamps. They now also handle unknown
  symbols by returning `Value::Void` and return an empty stream if the
  time window results in no data.
- **Documentation**: Updated doc comments for the stream creation
  functions to reflect the new time windowing capabilities and to
  clarify behavior for unknown symbols and empty time windows.

These changes significantly enhance the flexibility of data streaming,
allowing users to query specific time ranges of M1 and tick data more
efficiently.
2026-03-29 22:03:33 +02:00
Brummel 1f68371920 Add symbol-based cache eviction
Introduces a reference counting mechanism for symbols within the
`FileCache`. When a `SymbolChunkIter` is created, it increments the
reference count for its symbol. When the iterator is dropped, the
reference count is decremented. If a symbol's reference count drops to
zero, all cached data associated with that symbol is evicted from the
cache, freeing up memory. This prevents stale data from being held
indefinitely when no active iterators are using it.
2026-03-29 21:40:30 +02:00
Brummel c32cc0f49c Refactor FileCache to use Condvar for efficient waits
The `FileCache` now uses `std::sync::Condvar` to signal when files have
finished loading. This replaces the previous spin-wait mechanism,
significantly improving efficiency by allowing waiting threads to sleep
until explicitly woken up.

The `wait_for_load` method now blocks on the `Condvar`, and the
`insert_m1` and `insert_tick` methods call `notify_all` after updating
the cache and removing the loading guard. This ensures that all waiting
threads are woken up and can re-check the cache.

Additionally, `ensure_m1_loaded` and `ensure_tick_loaded` in
`DataServer` have been simplified to directly call `wait_for_load` when
another thread is already loading the file, removing the redundant
synchronous loading logic.
Refactor FileCache to use Condvar
2026-03-29 20:47:49 +02:00
Brummel 38f657076b feat: Add data server for market data loading
This commit introduces a new data server module (`src/ast/data_server`)
designed for high-performance, thread-safe loading and caching of market
data.

Key components:
- `DataServer`: Manages symbol indexing and delegates loading/caching.
- `SymbolIndex`: Scans the data directory and maintains a sorted index
  of data files per symbol.
- `FileCache`: Implements a thread-safe cache using `RwLock` and a
  loading guard to prevent duplicate work.
- `loader`: Handles ZIP decompression and binary record parsing from
  `.bin` files.
- `records`: Defines raw and parsed data structures for M1 and tick
  data.
- `SymbolChunkIter`: Provides an iterator over pre-parsed data chunks,
  prefetching subsequent files.

This architecture allows multiple VM threads to access market data
concurrently without redundant I/O, mirroring the strategy used in the
original Delphi `TDataServer`.
2026-03-29 20:28:55 +02:00
Brummel 1f3fb40bcc Update record type computation
This commit refactors the way record types are computed within the
optimizer. Previously, the optimizer would often reuse the existing
`NodeMetrics` for records, which could lead to stale or polymorphic
types.

The changes introduce a new function, `compute_record_type`, which
explicitly recalculates the `StaticType` for a `Record` node based on
the concrete types of its child fields. This ensures that after
optimizations like inlining and folding, record types are as concrete as
possible, eliminating unresolved `TypeVars`.

Additionally, the `folder.rs` module is updated to recompute the
`RecordLayout` when creating constant record values, further improving
type accuracy. Two new tests are added to verify that inlined records,
including those with multiple fields, correctly result in concrete
types.
2026-03-29 18:59:38 +02:00
Brummel 6c8e2df45c Refactor: Improve type propagation in optimizer
Refactor the `updated_metrics` function to `refreshed_metrics`. This
function now clones the `NodeMetrics` if the new type is identical to
the original, avoiding unnecessary allocations.

Also, adjust the `Block` and `Lambda` node kinds to correctly propagate
concrete types after inlining, especially when dealing with polymorphic
functions. This ensures that the return types of blocks and lambdas
reflect the actual computed types rather than stale type variables.

This change improves type inference accuracy and prepares for more
advanced optimizations by ensuring type information is correctly
propagated through the AST.
2026-03-29 18:50:05 +02:00
Brummel 03862d50c6 Add tuple type recomputation after inlining
Introduce helper functions to recompute the `StaticType` of a tuple node
based on its children's types. This ensures that after inlining, tuples
within the AST have concrete types, rather than unresolved type
variables, even when derived from polymorphic functions.

Also adds tests to verify this behavior for both homogeneous and
heterogeneous tuples.
2026-03-29 18:44:48 +02:00
Brummel 0e806b9e5d feat: Add compact AST dump
Introduce a new `dump_ast_compact` function for the environment and
Dumper. This function provides a simplified, human-readable
representation of the AST, focusing on inferred types and purity markers
instead of detailed debug information.

This change adds:
- A `compact` flag to the `Dumper` struct.
- The `dump_compact` method to the `Dumper`.
- The `dump_ast_compact` method to the `Environment`.
- A new `dump_ast_compact` command-line argument for the `ast` binary.
- A new MCP server endpoint `dump_ast_compact`.
- A `CompactMetadata` trait to abstract over different metadata types
  for compact display.
- Updates to documentation and CLI help messages.
2026-03-29 18:28:10 +02:00
Brummel 10a7fcb576 Refactor: Replace Object with StreamStorage for streams
The `Object` trait is too generic and has been causing confusion. This
commit introduces `StreamStorage` as a dedicated trait for reactive
stream types.

This change involves:

- Renaming `Object` to `StreamStorage` in relevant places.
- Updating the `Value::Object` enum variant to `Value::Stream`.
- Modifying how streams are handled in the compiler, VM, and tests to
  use the new `StreamStorage` trait.
- Adjusting example scripts and tests to reflect the change in type
  representation.
- The `StreamNode` struct now explicitly holds its `element_type`.
2026-03-29 18:01:46 +02:00
Brummel a665e2c1a5 Refactor call hooks to use trait objects
This commit refactors the call hook mechanism in the compiler. Instead
of using function pointer types (`PostCallHook`, `FinalizeHook`) and a
struct to hold them (`CallHooks`), it introduces a trait
`RtlCompilerHook` with default implementations for `post_call` and
`finalize`.

This change aligns with Rust's idiomatic way of handling extension
points and allows for more flexible and extensible compiler behavior.
The `TypeChecker` now holds a map of `Rc<dyn RtlCompilerHook>`, enabling
different RTL functions to register their specific compiler hooks. The
`SeriesHook` and `PushHook` structs are introduced to demonstrate this
new pattern.
2026-03-29 14:52:58 +02:00
Brummel 916460ab03 Remove IndexElementFn and series_element_type
The `IndexElementFn` trait and its associated `series_element_type`
function have been removed. The `TypeChecker` now directly inspects the
`StaticType::Series` variant to extract the element type when
propagating index constraints. This simplifies the type-checking logic
by removing an unnecessary layer of indirection and adhering to the
project rule of keeping the AST simple and pushing complexity into the
system.
2026-03-29 14:00:48 +02:00
Brummel 0292ead7a5 Refactor InferenceAccess for call hooks
The `TypeInferenceAccess` and `TypeSlotAccess` traits have been
consolidated into a single `InferenceAccess` trait. This simplifies the
call hook interface by providing a unified way to access type inference
operations and scope slot types.

A new struct, `CheckerInferenceAccess`, is introduced to provide a
concrete implementation of `InferenceAccess` for the `TypeChecker`. This
struct wraps both the `TypeChecker` and the `TypeContext`, allowing call
hooks to interact with both without needing to know their specific
types. This avoids potential circular dependencies between modules.

The `get_slot_type` method has been moved to the new `InferenceAccess`
trait. This method is crucial for call hooks that need to recover the
original `TypeVar` ID after substitutions have occurred, particularly in
the `push_post_call` function for series.
2026-03-29 13:55:32 +02:00
Brummel 93a85cd15c Add IndexElementFn for series type probing
Introduce `IndexElementFn` as a type alias for a function that maps a
`StaticType` to an optional `StaticType` representing its element type.
This allows the `TypeChecker` to determine the element type of indexable
types like `Series` without needing to be aware of their specific
implementations.

The `series_element_type` function is created and registered with the
`TypeChecker` to handle `Series` types. This design centralizes
type-specific logic for indexable types, making the `TypeChecker` more
generic and extensible for future indexable types such as `Stream`.
2026-03-29 13:20:49 +02:00
Brummel 9c434fb49d Fix: Unify TypeVars when indexing generic parameters twice
When a generic function indexes the same TypeVar parameter multiple
times,
all resulting TypeVars must share a single element type. This change
ensures
that subsequent indexing operations unify their fresh TypeVar with an
existing
one if the parameter has already been indexed, preventing incorrect type
inference and false negatives.

A new test, `test_let_poly_multi_index_all_results_typed`, is added to
verify this behavior.
2026-03-28 23:03:38 +01:00
Brummel 84abbd9721 Add Forall type for polymorphism
Introduce `StaticType::Forall` to represent universally quantified
types. This enables Hindley-Milner's let-polymorphism, allowing
functions to be generalized and reused with different concrete types.

- **Generalization at `def`:** Function values are now generalized using
  `self.generalize` when they are defined, wrapping their type in
  `Forall`. This occurs at `def` boundaries.
- **Instantiation at use:** Function types are instantiated with fresh
  type variables at each call site using `self.instantiate`. This
  ensures that different uses of the same polymorphic function do not
  interfere with each other's type inference.
- **Value restriction:** Only function-typed definitions are
  generalized. Mutable state like series and scalars remain monomorphic
  to prevent unexpected behavior.
- **Type context awareness:** The `generalize` function considers the
  current type context to avoid quantifying type variables that are
  still in use elsewhere.
  Feat: Add Forall type for polymorphism

Introduces the `Forall` type to support polymorphic functions and enable
let-polymorphism.

- `Forall(vars, body)`: Represents a universally quantified type where
  `vars` are the type variables bound by this quantification, and `body`
  is the type itself.
- `TypeChecker::generalize`: Wraps a type in `Forall` when a `def`
  boundary is encountered for function-typed values. This ensures that
  each use site of a polymorphic function gets its own instantiation.
- `TypeChecker::instantiate`: Replaces the type variables within a
  `Forall` type with fresh ones. This is performed at each call site of
  a polymorphic function.
- Updates `TypeChecker::unify` and `TypeChecker::apply_subst` to
  correctly handle `Forall` types, ensuring that bound variables within
  a `Forall` are not affected by global substitutions and that `Forall`
  types are instantiated before unification.
- Adds a new example script `examples/Propa.myc` to demonstrate basic
  usage.
- Includes a regression test
  `test_let_poly_non_series_callable_no_false_positive` to ensure that
  passing non-series callables to generic functions does not lead to
  type errors.
- Introduces `TypeChecker::bind_var` to handle lazy propagation of
  index-call constraints, crucial for correctly typing series lookups
  like `(s 0)`.
2026-03-28 22:49:44 +01:00
Brummel bf12755905 Add HM Let-Polymorphism tests
This commit introduces a new test suite for HM Let-Polymorphism.
These tests verify that user-defined generic functions work correctly
across multiple series element types without type conflicts. This is
crucial for ensuring that the type inference system can handle
polymorphic functions properly, especially when they are applied to
different types. The tests cover scenarios like generic accessors,
runtime correctness with different series types, record series, and
higher-order functions. They also include a test for the value
restriction to ensure that series bindings are not generalized, which
would lead to incorrect type errors.
2026-03-28 21:40:35 +01:00
Brummel 8efd12add6 Add call hooks for series and push
Introduces `CallHooks` to enable extensions to the type inference and
AST finalization process. This allows for more dynamic and configurable
behavior without hardcoding symbol names in the compiler core.

Specifically, this commit adds:

- **`series_post_call`**: A hook for the `(series n)` function. It
  ensures that the return type is a fresh `TypeVar` when the element
  type is `Any`, allowing for proper type inference from subsequent
  `push` calls.
- **`series_finalize`**: A hook for `(series n)` that rewrites the AST
  node. It replaces the generic `series` call with a pre-configured
  factory closure that allocates the correct series storage type (e.g.,
  `ScalarSeries<f64>` for floats, `RecordSeries` for records). This
  avoids runtime dispatch and relies on type information captured during
  compilation.
- **`push_post_call`**: A hook for the `(push series val)` function. It
  unifies the series element `TypeVar` with the pushed value's type. It
  also handles numeric and record field promotion and updates the
  `TypeVar` binding if a promotion occurs.
2026-03-28 21:13:26 +01:00
Brummel 6cab8f649b Introduce numeric and record widening
This commit introduces implicit numeric widening from Int to Float and a
corresponding record promotion mechanism.

When pushing values into a series, if the series' element type is a
TypeVar,
and the pushed value is of a different numeric type (e.g., Int and
Float),
the TypeVar will be unified with the wider type (Float). This ensures
that
operations on series elements handle type differences gracefully.

The record promotion handles cases where two records are involved, and
their
fields have compatible types, either identical or through numeric
widening.
This allows for more flexible type inference in complex structures.
2026-03-28 14:52:24 +01:00
Brummel 350b7b49b6 Fix: Apply substitutions to upvalue types
When an identifier is used as an upvalue within a nested scope (e.g., a
`while` loop), the type checker needs to apply the global substitution
map to resolve `TypeVar`s that might have been determined in outer
scopes. This ensures that the correct type is used even if
`ctx.set_type` couldn't propagate the change directly through the
upvalue address.

This fix also includes a regression test to specifically cover this
scenario with series and closures.
2026-03-28 14:22:25 +01:00
Brummel 9c85fb605b Update series definition to infer schema
The `series` function no longer requires an explicit schema. The element
type is now inferred at compile time from subsequent `push` calls using
Hindley-Milner type inference. This simplifies series creation and
aligns with the project's goal of making the untyped AST simple while
the system handles complexity.
2026-03-28 14:05:20 +01:00
Brummel d7d1aef8ed Refactor type checker for series and overloads
Introduce `unify_matched_overload` to handle type variable propagation
for overloaded functions. This ensures that when an overload is chosen,
its concrete parameter types are unified with the actual argument types.
This is crucial for resolving type variables nested within closures
called through overloaded functions.

Additionally, this commit:
- Allows `series` to infer schema type from `push` calls, simplifying
  its signature.
- Adds a fallback to `ValueSeries` in `rtl::series` if schema injection
  fails.
- Updates `StaticType::TypeVar` to return `Any` when accessed,
  simplifying field access logic.
- Adjusts tests to reflect the simplified `series` signature.
2026-03-28 14:03:48 +01:00
Brummel d2cba2d55d Refactor: Simplify series creation and infer element type
The `series` constructor now only accepts the `lookback` limit. The
element type is inferred by the type checker and implicitly added as a
schema argument during compilation. This simplifies the API and
centralizes type inference.

This change also includes:
- Updates to example scripts (`HMA.myc`, `sma.myc`, `soa_series.myc`) to
  reflect the new `series` signature.
- Modifications to the `TypeChecker` to handle the inferred schema
  injection.
- An addition of a regression test for type inference through nested
  closures with `series`.
- Removal of `#[allow(dead_code)]` from several `TypeChecker` methods as
  they are now used.
- Update to `rtl/prelude.myc` macro `cache`.
- Update to `rtl/series/mod.rs` to reflect new signature.
- Update to `ast/types.rs` to allow `series` to be indexed with an
  integer to retrieve its element type.
2026-03-28 13:47:15 +01:00
Brummel e595e747d4 Add type inference infrastructure
Introduces a `TypeChecker` struct with logic for Hindley-Milner type
inference.
This includes:
- `var_counter` for generating unique type variable IDs.
- `subst` for the global substitution map.
- `fresh_var` to create new type variables.
- `apply_subst` to resolve type variables recursively.
- `occurs` to check for cycles during unification.
- `unify` to merge types and extend the substitution.

The `StaticType` enum is extended with a `TypeVar(u32)` variant to
represent unresolved type variables.
The `Display` and `is_assignable_from` implementations for `StaticType`
are updated to handle `TypeVar`.
2026-03-28 13:08:15 +01:00
Brummel 982d2239b6 Refactor GlobalStore to use separate RTL and user value stores
The `GlobalStore` was refactored to clearly distinguish between
immutable RTL values and mutable user-defined global slots.

The `Environment` struct now holds:
- `rtl_values`: An immutable `Rc<[Value]>` for pre-defined RTL values.
- `user_values`: An `Rc<RefCell<Vec<Value>>>` for user-defined mutable
  globals.

The `GlobalStore` struct now takes both `rtl` and `user` as parameters
and uses `rtl_len` to determine which store to access. This change
separates concerns and better reflects the immutability of RTL values
after the bootstrap phase, aligning with the project's concurrency
rules.

Documentation in `docs/Analysis_Environment.md` was updated to reflect
these structural changes.
2026-03-28 00:02:34 +01:00
Brummel 0e8cd0832f Refactor global variable access in VM and optimizer
Replaces `Rc<RefCell<Vec<Value>>>` with a new `GlobalStore` struct for
accessing global variables. This provides a more structured way to
manage global values and allows for future optimizations like sharing
immutable RTL portions across environments. The `GlobalStore` also
includes an `rtl_len` field to distinguish between RTL and user global
slots, enabling debug-time checks for writes to immutable RTL slots.
2026-03-27 23:40:42 +01:00
Brummel 893d9936ed Introduce Rtl struct and from_rtl constructor
This commit refactors the `Environment` struct to better manage the Rust
Runtime Library (RTL) bootstrap state.

A new `Rtl` struct is introduced to hold a frozen snapshot of the
initial RTL state, including scopes, types, purity, values, and
documentation. This snapshot is created once after `rtl::register()`
completes.

The `Environment::new()` constructor is updated to perform a two-phase
bootstrap: first, it registers RTL functions, and then it freezes the
scope 0 and captures the RTL state into the new `Rtl` struct.

A new constructor, `Environment::from_rtl()`, is added. This allows
creating independent execution environments that all share the same
underlying RTL state. This is crucial for isolation between different
execution contexts, as user definitions and loaded modules will not leak
between environments created from the same `Rtl` snapshot.

The `Environment::list_bindings` and `Environment::list_rtl_docs`
methods are updated to use the `rtl` field for accessing the RTL scope
and documentation, reinforcing the concept of a shared, immutable RTL
state.

A new test, `test_environments_from_shared_rtl_are_independent`, is
added to verify that environments created from the same `Rtl` snapshot
are indeed independent and do not leak user-defined bindings.
2026-03-27 22:49:03 +01:00
Brummel b6dcdbde8d Refactor instantiate to return Closure
The `instantiate` method in `Environment` has been refactored to return
`Result<Rc<Closure>, String>` instead of `Rc<NativeFunction>`. This
change separates the creation of a `Closure` from its execution,
allowing the caller to manage the `VM` lifecycle and choose the
appropriate execution strategy (e.g., single run vs. repeated benchmark
iterations).

The `NativeFunction` type is now exclusively used for true Rust
intrinsics. The benchmark runner has also been updated to reuse a single
`VM` across iterations, improving performance by avoiding repeated VM
allocations.
2026-03-27 22:04:20 +01:00
Brummel 3d0ea094b0 Fix: Dump AST ignores parse errors
The `dump_ast` method previously bypassed the `compile` method, which
meant it would silently ignore parsing errors. This commit ensures that
`dump_ast` now delegates to `compile` and correctly propagates any
parsing errors. This aligns the behavior of `dump_ast` with other
compilation steps and provides better feedback to the user.
Fix: Dump AST ignores parse errors

The `dump_ast` method now delegates to `compile(source).into_result()?`
to properly handle and report parse errors, ensuring consistency with
the compilation process.
2026-03-27 21:36:01 +01:00
Brummel 1736602b0d Create Analysis_Environment.md 2026-03-27 21:16:19 +01:00
Brummel ddea07666d Monomorphize function templates by re-type-checking
Re-type-checks function templates with concrete argument types to
produce specialized TypedNodes for monomorphization. This process
leverages the `check_node_as_bound` method, which is generic enough to
handle different binding phases, allowing for fresh type inference from
the call site.
2026-03-27 21:16:13 +01:00
Brummel 1379ab366a Refactor module loading to improve clarity
The module loading logic has been refactored to enhance readability and
maintainability. Key changes include:

*   **Path Handling**: Simplified path construction using `join` for
    better clarity and consistency.
*   **Error Reporting**: Improved error messages during parsing by
    utilizing `parser.diagnostics.format_errors()`.
*   **Module Loading Logic**:
    *   The `loaded_modules` set now uses `insert()` to both add new
        modules and efficiently check for duplicates, returning `false`
        if the module was already loaded.
    *   The recursive call to `collect_dependencies` is now placed
        before processing the current file's AST, ensuring a topological
        order of dependency loading.
*   **Helper Function Scope**: `extract_use_directives` is now a static
    method as it doesn't depend on the `ModuleLoader` instance.
2026-03-27 16:16:45 +01:00
Brummel 1f0208af95 Add module loader and related tests
Introduce `ModuleLoader` to handle `#use` directives and dependency
resolution. This involves adding `tempfile` dependency, modifying
`Cargo.toml` and `Cargo.lock`, creating a new `module_loader.rs` file,
and updating `environment.rs` to use the new module.

Also, add comprehensive tests for `Diagnostics::format_errors`,
`CompilationResult`, and the new `ModuleLoader` functionality. These
tests cover various scenarios including empty diagnostics, multiple
errors, error formatting, compilation success/failure, and complex
dependency loading with topological ordering and search path resolution.
2026-03-27 15:08:28 +01:00
Brummel 636f5a1814 Extract RTL documentation types
Move `RtlDocEntry`, `RtlRegistration`, and `PipelineGenerator` to a new
`docs` module within `src/ast/rtl/` and re-export them. This improves
organization and modularity of the RTL component.
2026-03-27 14:54:36 +01:00
Brummel 4f05562e0c Introduce CompilationResult struct
This commit introduces the `CompilationResult` struct to encapsulate the
outcome of a compilation pass. It holds either a successfully compiled
`TypedNode` or a `Diagnostics` object containing errors. Helper methods
for creating success or error results and for converting to a `Result`
are also included. This improves error handling and clarity in the
compilation process.
2026-03-27 13:25:12 +01:00
Brummel 9b7c0b68a4 Refactor global slot allocation
Introduce `allocate_slot` helper function to centralize the logic for
allocating new global slots. This improves code clarity and reduces
duplication by consolidating common operations like updating scopes,
types, purity, and values.
2026-03-27 13:18:59 +01:00
Brummel 325e690cd9 Add format_errors to Diagnostics
Introduces a new method `format_errors` to the `Diagnostics` struct.
This method iterates through all diagnostics, filters for errors, and
formats them into a newline-joined string. This simplifies error
reporting in compilation results.
2026-03-27 13:17:58 +01:00
Brummel 9c5506b83e Implement unhygienic macro expansion for ~ident
This commit introduces support for unhygienic macro expansion using the
`~ident` syntax. When an identifier is prefixed with a tilde (`~`)
within a macro template, it is no longer subject to hygienic renaming.
Instead, it directly refers to a binding in the call site's environment.

This allows macros to interact with variables defined outside the
macro's scope, such as:
- Modifying call-site variables.
- Capturing call-site variables as upvalues in closures.
- Defining new variables that are visible in the call site.

A new test case `test_macro_tilde_nonparam_assign` has been added to
verify this functionality. Previously, unhygienic escapes were
implicitly handled by `SyntaxKind::Identifier` if they were not macro
parameters. This change makes this behavior explicit and correctly
handles non-parameter identifiers by returning a bare symbol.
2026-03-27 10:06:50 +01:00
Brummel adce3a6818 Fix: Simplify scoping tests
Removes outdated tests for macro expansion, destructuring, and if branch
scope leaks. These tests were either superseded by newer, more accurate
tests or are no longer relevant to the current implementation of the
compiler's scoping rules. The remaining test correctly asserts that
defining the same variable in different 'if' branches is functional.
2026-03-27 09:26:30 +01:00
Brummel 3a757b7551 Add tests for series and scoping features 2026-03-27 08:27:40 +01:00
Brummel cb4162e420 Refactor discover_globals to handle patterns
The `discover_globals` function has been refactored to better handle
pattern matching within definitions. It now uses a recursive helper
function `register_pattern` to correctly process identifiers and tuple
patterns. This ensures that all declared variables within a `Def` node
are registered with their corresponding scope and virtual ID.
Additionally, support for `Expansion` nodes has been added to ensure
that any nested global definitions within them are also discovered.
2026-03-27 08:27:31 +01:00
Brummel 903c77a665 Update series syntax and binder scope handling
Refine the BNF for `series` to explicitly include the `lookback_limit`
and `schema`.
Introduce scope management within the `bind` function for `if/else`
branches to ensure correct context handling during compilation.
Add `Again` and `GetField` node kinds to the `Specializer`.
Improve lexer to ignore invisible characters within identifiers,
demonstrated by a new test case.
2026-03-26 16:04:45 +01:00
Brummel 6d0882dd4a Remove example design-flaw.myc
This example demonstrated a known language limitation regarding
conditional `def` statements and has been removed.
2026-03-26 15:10:19 +01:00
Brummel 16af3ae9fc Refactor Examples to use assert_eq
This commit updates all example files to use `assert_eq!` for verifying
output, replacing the previous `Output:` comments. This change makes the
examples more robust and self-testing.

The benchmark results in some examples have also been slightly adjusted,
reflecting minor performance variations after the refactoring.

Additionally, a runtime panic handling mechanism has been introduced in
the VM for function calls, which improves error reporting for unexpected
panics.
2026-03-26 15:07:48 +01:00
Brummel 0088a644eb Refactor stack allocator to reuse freed slots
Introduce `free_slots` in `StackAllocator` to keep track of physical
slots that have been freed. When mapping a new virtual slot, the
allocator first attempts to reuse a slot from the `free_slots` vector
before allocating a new one.

This change also includes a new function `collect_scope_locals` that
identifies non-captured local variables defined within a block. After
these expressions are lowered, the allocator reclaims the physical slots
associated with these locals, making them available for reuse by
subsequent blocks.

A new test `test_slot_reuse_non_overlapping_scopes` is added to verify
that distinct scopes correctly reuse stack slots, ensuring optimal stack
usage.
2026-03-26 14:01:46 +01:00
Brummel 78813e7197 Refactor: Optimize block handling in optimizer
The optimizer now correctly handles empty blocks by collapsing them into
Nop nodes. It also unwraps blocks containing a single expression, making
the AST more concise. This change improves AST simplification by
ensuring that redundant block structures are removed.
2026-03-26 13:44:38 +01:00
Brummel 273dc83d68 Refactor if-else binding scope management
The explicit scope management within the `if`/`else` binding logic was
redundant.
The `bind` function recursively handles scope creation and destruction
for
expressions, making manual scope manipulation here unnecessary and
error-prone.
2026-03-26 12:19:33 +01:00
Brummel c0125216b4 Add HMA example script
This commit introduces a new example script, `HMA.myc`, which
demonstrates the implementation and usage of the Hull Moving Average
(HMA) indicator.

The script includes factory functions for Simple Moving Average
(`make-sma`), Weighted Moving Average (`make-wma`), and the Hull Moving
Average itself (`make-hma`). It also includes a small example loop to
initialize and print HMA values.
2026-03-26 12:00:27 +01:00
Brummel 4338197bc9 Add format (roundtrip) button 2026-03-25 19:33:07 +01:00
Brummel e757b453a4 feat: Add AST emitter and comment support
Introduces a new AST emitter module responsible for serializing AST
nodes back into Myc source code. This emitter preserves leading comments
and whitespace, enabling a round-trip guarantee for the parser and
emitter.

The documentation in `docs/BNF.md` has been updated to reflect the new
syntax rules for comments, including single-line comments (`;`) and
documentation comments (`;;`). The rules clarify that comments must be
whole lines and that inline comments are disallowed.

The lexer and AST node structures have been updated to accommodate
`CommentLine` variants, allowing for different types of comments
(regular, documentation, and blank lines) to be stored and processed.

A new test suite `tests/comment_roundtrip.rs` has been added to verify
the round-trip property of the parser and emitter across all example
`.myc` files. This test ensures that parsing source code, emitting it,
and then parsing the emitted code again results in the same output
string.

The `ast.rs` binary now includes a `--emit` flag that uses the new
emitter to output the parsed source code, facilitating debugging and
testing of the emitter itself.
2026-03-25 19:15:40 +01:00
Brummel ee89d2330d feat: Add comments to AST nodes
This commit introduces support for comments within the AST nodes.
Comments are now parsed by the lexer and stored within the `Token`
struct.
These comments are then propagated through various compiler phases,
including
the `Node` struct, ensuring they are preserved in the Abstract Syntax
Tree.
This change enhances the AST's ability to retain source code
information,
which can be valuable for debugging and analysis.
2026-03-25 18:36:53 +01:00
Brummel b436e09840 Add documentation infrastructure for RTL symbols
Introduces `RtlDocEntry` and `RtlRegistration` structs to store and
manage documentation for native functions and constants. The
`Environment` struct is updated to include a registry for these
documentation entries.

The `register_native_fn` and `register_constant` methods now return an
`RtlRegistration` builder, allowing for optional chaining of `.doc()`,
`.description()`, and `.examples()` methods before the registration is
finalized when the builder is dropped.

Macros `rtl_fn!` and `rtl_const!` are introduced to simplify the
registration process with documentation.

The MCP server is extended with `get_rtl_docs` and `get_symbol_doc`
tools to expose this documentation externally.

Specific RTL functions and constants (e.g., `+`, `-`, `*`, `/`, `NaN`,
`true`, `false`, `date`, `now`, math functions, `series`, `push`, `len`,
`create-random-ohlc`, `create-ticker`, `pipe`) have been updated to
include their respective documentation using the new infrastructure.

The `Signature` and `StaticType` types have gained `to_doc_string`
methods for generating human-readable documentation strings.
2026-03-25 15:02:19 +01:00
Brummel 446bdcd42d Create mcp_server.rs 2026-03-25 14:03:27 +01:00
Brummel 641a19e736 Add MCP server to AST tool
Integrates the MCP server functionality into the AST tool binary. This
allows the tool to act as a server for LLM integration, enabling
communication over stdio. The server can list available bindings and
execute scripts.
2026-03-25 14:03:22 +01:00
Brummel 6042415dfc Refactor series types and value enum
This commit refactors the way series are represented and handled within
the AST.
Key changes include:

- Introducing `SeriesStorage` and `PushableStorage` traits to provide a
  more
  unified and type-safe interface for series data.
- Renaming `Object` trait and its methods to clarify that it's for RTL
  extensions
  other than series (like Streams).
- Updating the `Value` enum to have a distinct `Series` variant,
  separating it
  from `Object`.
- Adjusting various parts of the `VM` and `register` functions to work
  with the
  new series traits and `Value::Series` variant.

This change aims to improve the type system's clarity and safety when
dealing with
series data, aligning with Rust's best practices for trait design.
2026-03-23 16:05:58 +01:00
Brummel a5c3f3da04 feat: Add bidirectional type inference for lambdas
This commit introduces bidirectional type inference, enabling the
compiler to infer lambda parameter types based on the context of their
usage. This is particularly useful for functions like `pipe`, where a
lambda's signature can be deduced from the types of the streams it
operates on.

Key changes include:

- **`extract_lambda_param_hints` function:** This new helper function in
  `type_checker.rs` is responsible for extracting expected parameter
  types for a lambda based on the callee's signature and already known
  argument types.
- **`check_lambda_with_param_hints` function:** This function allows
  type-checking a lambda node using provided parameter type hints,
  preserving upvalue types from the enclosing scope.
- **Call expression type checking:** The `check_call` function now
  phases its argument type checking. Non-lambda arguments are typed
  first, and then lambda arguments are typed using hints derived from
  `extract_lambda_param_hints`.
- **`pipe_arg_hint_resolver`:** This new resolver for the `pipe`
  function's `PolymorphicFn` type allows it to provide specific type
  hints for its lambda argument based on the types of its stream inputs
  (single stream, vector of streams, or tuple of streams).
- **`StaticType::PolymorphicFn` update:** The `PolymorphicFn` enum
  variant has been extended to include `resolve_arg_hints`, enabling
  functions to provide lambda parameter hints during bidirectional type
  inference.
- **New tests:** Added tests to verify the correct inference of lambda
  parameters for `pipe` with vector and tuple inputs, ensuring that
  inner stream types are correctly propagated. Also, a test to confirm
  `pipe` with an optional return still produces `Stream(Float)`.
2026-03-23 10:51:06 +01:00
Brummel 1875cfdc9b Remove unused pipe node definitions
The `Pipe` node and its associated logic have been removed from the AST.
This commit cleans up the code by removing all references to this
defunct node in the analyzer, binder, captures, dumper, lowering,
macros, optimizer, and type checker. The parser no longer recognizes the
`pipe` keyword.
2026-03-23 10:22:12 +01:00
Brummel 99b540c84e Refactor series implementation into its own module
This commit reorganizes the series-related code by moving the
`RingBuffer`, `ScalarSeries`, `ValueSeries`, `RecordSeries`,
`SeriesView`, and `SharedSeries` structs into a new
`src/ast/rtl/series/data.rs` file.

The `src/ast/rtl/series/mod.rs` file now contains the module
declaration, re-exports, and the `register` function for registering
series-related native functions with the environment. This improves code
organization and maintainability.
2026-03-22 21:22:56 +01:00
Brummel 09ae98728f Refactor Value enum for Quote node
Replaces `Value::Object(Rc<dyn Object>)` for SyntaxNodes with a
dedicated `Value::Quote(Rc<SyntaxNode>)`. This provides better type
safety and clarity when dealing with AST nodes as values.
Refactor Value enum for Quote node

This commit introduces a new `Value::Quote` variant to represent quoted
AST nodes directly, replacing the generic `Value::Object` for this
purpose.

This change simplifies the handling of quoted nodes by:
- Directly storing an `Rc<SyntaxNode>` instead of relying on dynamic
  downcasting from `Rc<dyn Object>`.
- Streamlining macro expansion logic by removing the need to check for
  `SyntaxNode` within `Value::Object`.
- Improving type safety and explicitness in the `Value` enum.
2026-03-22 20:53:30 +01:00
Brummel ae1d94e507 Refactor PushableSeries trait
Introduce a new trait `PushableSeries` to distinguish objects that can
have values pushed into them from those that are only readable series.
This clarifies the API and allows for more precise type checking.

The `Object` trait no longer has a default `push_value` implementation.
Instead, `PushableSeries` now provides this functionality.
`SeriesMember` has also been updated to require `PushableSeries` for
RecordSeries field buffers.
2026-03-22 19:45:50 +01:00
Brummel 843ee7dfed Refactor: Use Value::Closure enum variant
Previously, compiled closures were stored as `Value::Object` and then
downcasted. This commit introduces a dedicated `Value::Closure` enum
variant to represent compiled closures. This improves type safety and
simplifies handling of closures throughout the compiler and VM.

This change involves:
- Updating type definitions to use `Value::Closure`.
- Adjusting downcasting logic to directly access the `Closure` struct.
- Modifying `run_with_args` and `run_with_args_observed` to expect
  `Value::Closure` for tail-call targets.
- Refining the handling of closures in the `Optimizer` and `Inliner` for
  better type clarity.
2026-03-22 19:41:23 +01:00
Brummel 6c37b0c64f Introduce closure AST node
The `Closure` struct has been moved from `ast/vm.rs` to a new module
`ast/closure.rs`. This improves the organization of the AST nodes. The
`Closure` struct represents a compiled function body along with its
captured upvalues, which is a key feature for Myc Script.
2026-03-22 19:37:04 +01:00
Brummel 37d7ed3e75 Remove unused TailCallRequest variant
The `TailCallRequest` enum variant and its associated logic have been
removed. Instead, the VM now uses an `Option<(Rc<dyn Object>,
Vec<Value>)>` field named `tail_call` to manage tail-call requests. This
change simplifies the `Value` enum and centralizes tail-call handling
within the `VM` struct.
2026-03-22 19:20:18 +01:00
Brummel f6c963cb41 Add into_rc_any and push_value to Object
This commit introduces the `into_rc_any` and `push_value` methods to the
`Object` trait.

The `into_rc_any` method allows for efficient, zero-cost downcasting of
`Rc<Self>` to `Rc<dyn Any>`. This is a crucial step for enabling more
flexible dynamic dispatch and type manipulation within the VM.

The `push_value` method is implemented for series types (`ScalarSeries`,
`ValueSeries`, `RecordSeries`) to allow pushing values directly onto
them. This simplifies the `push` intrinsic function, removing the need
for manual downcasting within its implementation. The default
implementation for non-series objects panics, enforcing that only
mutable series support this operation.

Additionally, `ScalarValue` now includes a `from_value` method to
facilitate converting `Value` back into concrete scalar types. This
improves type safety and reduces boilerplate code when handling scalar
conversions.
2026-03-22 19:04:32 +01:00
Brummel db443df932 Implement Display for SharedValueSeries
The `Value::Object` variant in `ast/types.rs` has been updated to
provide a more specific display format for `SharedValueSeries`.
Previously, it would generically display the object's type name. This
change allows it to directly show the length of the `SharedValueSeries`
buffer, which is more informative.
2026-03-22 18:40:23 +01:00
Brummel 007446a167 Update various types to use their own definitions
This commit refactors several modules to use the defined types from
`ast::types` and `ast::nodes` directly, rather than using fully
qualified paths. This improves code readability and reduces redundancy.

Specifically, the following changes were made:

- In `analyzer.rs`, `crate::ast::types::Identity` and
  `crate::ast::types::Purity` are now used directly.
- In `binder.rs`, `crate::ast::types::NodeIdentity`,
  `crate::ast::types::SourceLocation`,
  `crate::ast::types::RecordLayout`, and `crate::ast::types::Value` are
  now used directly.
- In `macros.rs`, types like `Address`, `VirtualId`, `NodeIdentity`,
  `SourceLocation`, `StaticType`, and `Purity` are now used directly.
- In `optimizer/engine.rs`, `Address<VirtualId>` is now used directly.
- In `type_checker.rs`, `BoundPhase`, `Signature`, `Keyword`, and
  `RecordLayout` are now used directly.
- In `environment.rs`, `CompilerScope`, `LocalInfo`, `CapturePass`,
  `AnalyzedPhase`, `NativeFunction`, and `Closure` are now used
  directly.
- In `nodes.rs`, `RecordLayout`, `Keyword`, `Purity`, and `StaticType`
  are now used directly.
- In `parser.rs`, `SourceLocation` and `NodeIdentity` are now used
  directly.
- In `rtl/math.rs`, `NativeFunction`, `Purity`, `Signature`,
  `StaticType`, `Value`, `RefCell`, and `Rc` are now used directly.
- In `rtl/streams.rs`, `ScalarValue`, `SeriesMember`, `Object`,
  `PipeFn`, `Value`, `VM`, `RingBuffer`, `RecordSeries`, `SeriesView`,
  `build_map_stream`, and `build_pipeline_node` are now used directly.
- In `rtl/type_registry.rs`, `RecordLayout`, `Keyword`, `Purity`,
  `Signature`, `StaticType`, and `Value` are now used directly.
- In `vm.rs`, `RecordSeries`, `SeriesView`, `build_map_stream`,
  `build_pipeline_node`, `StreamNode`, `Object`, and `PipeFn` are now
  used directly.
- In `utils/tester.rs`, `TypedNode` is now used directly.
2026-03-22 18:30:56 +01:00
Brummel 1cbc656554 Refactor: Move compiler node definitions to src/ast/nodes
The `bound_nodes.rs` file has been removed and its contents have been
moved to `src/ast/nodes.rs`. This consolidates all AST node definitions
into a single module, improving organization and maintainability.

The `compiler` modules now import these definitions from
`crate::ast::nodes` instead of `crate::ast::compiler::bound_nodes`.
2026-03-22 17:58:49 +01:00
Brummel 49ad76e7c3 Refactor: Remove integration tests and organize into modules
This commit refactors the test suite by removing the monolithic
`integration_test.rs` file.
The tests have been categorized and moved into dedicated modules within
the `tests/` directory:

- `tests/parser.rs`: Contains tests for parsing various literal values.
- `tests/destructuring.rs`: Houses tests related to destructuring
  syntax.
- `tests/closures.rs`: Includes tests for closure behavior and related
  optimizations.
- `tests/optimizer.rs`: Contains tests specifically targeting optimizer
  logic and regressions.
- `tests/records.rs`: Holds tests for record syntax and associated
  optimizations.
- `tests/rtl.rs`: Contains tests for runtime library operators and
  functions.
- `tests/pipeline.rs`: Tests for pipeline functionality.
- `tests/macros.rs`: Tests for macro expansion and related issues.
- `tests/error_recovery.rs`: Includes tests for compiler error handling
  and recovery mechanisms.
- `tests/examples.rs`: Verifies the execution of example scripts.

This reorganization improves test discoverability and maintainability.
The `lib.rs` file has also been updated to reflect these changes by
removing the reference to the old `integration_test` module.
The `type_checker.rs` file has had some tests removed, as they are now
covered by the more specific tests in `tests/error_recovery.rs`.
2026-03-21 15:56:07 +01:00
Brummel bc287d27dd Add destructuring support to dead-def elimination
Introduce `collect_pattern_addrs` to gather all bound addresses from a
destructuring pattern. This enables the optimizer to remove unused
destructuring definitions, similar to how simple unused definitions are
handled. Added integration tests to verify this behavior and other
optimizer robustness scenarios.
2026-03-21 15:35:28 +01:00
Brummel 6264e21f24 Remove unused BoundKind enum
The `BoundKind` enum was declared but never used. This commit removes it
to clean up the codebase.
2026-03-21 14:16:18 +01:00
Brummel e65402364d Refactor: Use new NodeKind and clean up Binder definitions
The Binder and related types have been refactored to use the new
`NodeKind` enum instead of the previous `BoundKind`. This commit updates
all references to use the new structure, ensuring consistency across the
compiler's AST representation.

Key changes include:

- Replacing `BoundKind` with `NodeKind` in the Binder's `bind` and
  `visit` methods.
- Updating pattern matching and field access to reflect the new enum
  variants (e.g., `NodeKind::Def` instead of `BoundKind::Define`).
- Adjusting identifier bindings to use the new `IdentifierBinding` enum.
- Reflecting changes in `DefBinding`, `AssignBinding`, and
  `LambdaBinding` structures.
- Ensuring all newly created nodes use `NodeKind` and the appropriate
  metadata.
2026-03-21 14:12:14 +01:00
Brummel 99fef2fc86 Refactor AssignBinding and add GetField node
The `AssignBinding` struct now uses `Option<Address<L>>` to
differentiate between simple assignments and destructuring assignments.
For destructuring, the addresses are managed by the `Identifier` nodes
within the target pattern.

A new `GetField` node kind has been introduced to represent optimized
field access, which is generated during the lowering phase for the
`RuntimePhase`. This change aids in more efficient code generation for
accessing fields.
2026-03-21 12:51:39 +01:00
Brummel 63a474030d Add .claudeignore and bytecode architecture documentation
This commit introduces two new files:
- `.claudeignore`: Specifies files and directories that should be
  ignored by Claude, such as snapshot files and the `target` directory.
- `docs/bytecode-architecture.md`: Documents the architecture for
  bytecode generation, outlining the hybrid execution model, the
  transformation pipeline (Specializer and Lowering), and the role of
  Tail Call Optimization (TCO).
2026-03-21 12:43:53 +01:00
Brummel 72244d9da7 Add generic NodeKind enum
This commit introduces a new `NodeKind` enum that serves as a unified
representation for AST nodes across different compilation phases. It
replaces the separate `SyntaxKind` and `BoundKind` enums, allowing
phase-specific information to be attached through generic type
parameters.

This change simplifies the AST structure and makes it easier to manage
node information consistently throughout the compilation process. The
`NodeKind` enum now holds all possible AST node variants, with
phase-specific data encapsulated within `CompilerPhase` trait
implementations.
2026-03-21 12:43:46 +01:00
Brummel e4be31729e Update benchmark results in examples 2026-03-17 17:33:24 +01:00
Brummel 5ca069d93d Add GEMINI.md 2026-03-15 22:40:45 +01:00
Brummel 19feb26247 Rename GEMINI.md 2026-03-15 22:33:13 +01:00
Michael Schimmel 1a7bb5d3e6 Optimize Clone for BoundKind and VM GetField
This commit reduces unnecessary cloning for fields like `addr`, `kind`,
and `field` within `BoundKind` and `VM::GetField`. This improves
performance by avoiding redundant memory allocations.
2026-03-13 19:55:48 +01:00
Michael Schimmel 0dfcfac7b3 Refactor AST substitution logic
This commit refactors the AST substitution logic to correctly handle
nested expansions and improve the inlining of lambda expressions.

The changes include:
- Extracting the core value from potentially expanded AST nodes before
  checking their kind.
- Ensuring that lambda expressions with empty upvalues are correctly
  substituted.
- Adding support for inlining global get expressions as AST
  substitutions.
- Improving the `UsageInfo` collection to correctly track assigned
  values.
2026-03-13 19:51:56 +01:00
Michael Schimmel e5d82ee2b6 Refactor bound node types for clarity
Introduce generic `CompilerPhase` trait to unify different stages of the
bound AST. Rename `LocalSlot` to `VirtualId` and introduce `StackOffset`
for clearer distinction between compile-time and run-time addressing.
Update type aliases and implementations to reflect these changes.
2026-03-13 19:20:44 +01:00
Michael Schimmel d035da9d91 Refactor VM field access to use Rc clones
The VM's field access logic has been refactored to clone `Rc` pointers
for addresses and field identifiers. This change aims to improve
efficiency by avoiding unnecessary dereferencing and copying of values
when accessing fields within records and objects.

Additionally, the `Value::Object` handling in `BoundKind::GetField` has
been refined to more explicitly manage `RecordSeries` and `StreamNode`
types, ensuring that field access is handled polymorphically and
efficiently without copying underlying data structures.
2026-03-13 18:44:43 +01:00
Michael Schimmel b87a6d7ada Refactor closure instantiation and inlining
This commit makes several changes related to how closures are handled:

- **Dumper:** Removes redundant introspection of closure ASTs, as this
  is no longer relevant.
- **Optimizer:** Updates `try_inline` to accept `ExecNode` for
  parameters and adds a check to ensure the `function_node` is a
  `Lambda` before attempting inlining.
- **Environment:** Simplifies closure creation by passing an `ExecNode`
  for parameters and ensuring the correct `stack_size` is used.
- **VM:** Adjusts `Closure` to store an `ExecNode` for parameters and
  updates `VM::unpack` to accept an `ExecNode`.
2026-03-13 18:00:58 +01:00
Michael Schimmel ef5c2367a7 Remove unused collect_parameter_slots
This function was no longer being called and was removed as part of a
refactoring.
2026-03-13 17:13:06 +01:00
Michael Schimmel 222fe64a1f Refactor stack size calculation
Introduces a `StackAllocator` to manage local slot mapping during
lowering. This allows for efficient calculation of stack sizes for
lambdas and the root node. Tail position marking is now handled more
accurately.
2026-03-13 16:50:57 +01:00
Michael Schimmel 8f3e9cbd72 Refactor global slot registration
Replace manual resizing of global arrays with simple pushes. This
simplifies the logic and removes the need for index calculations within
the registration functions.
2026-03-13 16:42:55 +01:00
Michael Schimmel 5a6cae1866 Refactor TypeContext locals to HashMap
Change the `slots` field in `TypeContext` from a `Vec` to a `HashMap`.
This allows for sparse local variable storage and avoids the need to
pre-allocate for all slots, which is more efficient when not all locals
are used.
2026-03-13 16:37:00 +01:00
Michael Schimmel ba36e2157e Configure Gemini for repomix MCP server
Add configuration for the repomix MCP server to the Gemini settings.json
file. This will allow Gemini to use repomix for code analysis and
understanding.
2026-03-13 14:21:39 +01:00
Michael Schimmel 84226f6a16 Refactor: Replace UntypedNode with SyntaxNode
This commit replaces the `UntypedNode` enum with the more accurately
named `SyntaxNode`. This change is primarily for clarity and better
reflects the role of these nodes as representing the structure of the
source code prior to semantic analysis.

The corresponding enum `UntypedKind` has also been renamed to
`SyntaxKind` to maintain consistency.

No functional changes are introduced by this refactoring; it is purely a
renaming and organizational update.
2026-03-13 14:21:28 +01:00
Michael Schimmel 7d72a99fa1 refactor: separate parser and compiler AST node structures
Simplified the AST architecture by removing the overly complex Node<K,
  T> structure and replacing it with specialized types for
  each phase:

   - Introduced UntypedNode in nodes.rs for the Parser and Macro system,
     eliminating redundant ty: () initializations.
   - Defined a new generic Node<T> in bound_nodes.rs for all compiler
     stages, where T represents the phase-specific metadata.
   - Updated the Binder to act as the bridge between UntypedNode and
     Node<()>.
   - Simplified type aliases: TypedNode, AnalyzedNode, and ExecNode now
     leverage the streamlined Node<T> structure.
   - Updated Parser, Macros, Type-Checker, Optimizer, and VM to reflect
     the architectural changes.
   - Fixed import chains and resolved needless borrows via clippy.

  This change improves type safety, reduces boilerplate code in the
  parser, and makes the compiler stages more idiomatic and
  readable.
2026-03-13 13:16:03 +01:00
Michael Schimmel bf86c76bb6 Refactor: Use root_purity instead of global_purity
This commit refactors the purity analysis from using a
`HashMap<GlobalIdx, Purity>` to a `Vec<Purity>` indexed by
`GlobalIdx.0`.

This change simplifies the data structure and makes purity lookups more
efficient. The `Binder`'s `globals` field has also been removed as it
was not being used.
2026-03-12 16:57:06 +01:00
Michael Schimmel dcb7685d29 Refactor: Pass global names to Binder
The `Binder` struct now accepts a `Rc<RefCell<HashMap<Symbol,
(GlobalIdx, Identity)>>>` for global names. This allows the binder to
directly access and manage global symbols during the binding process,
simplifying the logic and improving efficiency.

Additionally, `CompilerScope` now implements `Default`, and a type alias
`BindingResult` has been introduced for clarity. The `bind_root`
function signature has been updated to accept the `globals` argument.
2026-03-12 16:21:20 +01:00
Michael Schimmel a220815bd6 Refactor Binder initialization and root binding
The `Binder` and `FunctionCompiler` initialization has been refactored
to accept initial scopes and slot counts. This allows for more flexible
management of the compiler's state, particularly for the root scope.

The `bind_root` function now returns the final scopes and slot count of
the root function compiler, enabling the `Environment` to update its
state with these new bindings.

The global variable handling has been integrated into the scope
management, removing the direct reliance on a shared `HashMap` for
globals. This promotes a more consistent approach to symbol resolution.
2026-03-12 14:40:13 +01:00
Michael Schimmel 08b5bba2c4 Add purity to LocalInfo and initialize in binder
The `LocalInfo` struct now includes a `purity` field to track the purity
of local variables and function arguments. This is initialized to
`Purity::Impure` for local variables and arguments within functions and
when registering native functions. Additionally, the `Environment`
struct is updated to include `root_scopes` and `root_slot_count` to
support parallel mode compilation by populating the root scope with
initial function and native function information.
2026-03-12 14:07:27 +01:00
Michael Schimmel 3780674eb1 Refactor binder to use local scopes
Removes global variables and related logic from the Binder, simplifying
its responsibilities to managing local scopes and function compilers.
This change also streamlines the `define_variable` and `resolve_local`
methods within `FunctionCompiler` and removes unused fields from
`FunctionCompiler` and `Binder`.
2026-03-12 13:50:37 +01:00
Michael Schimmel bc13ce7abf Make LocalInfo and CompilerScope fields public 2026-03-12 12:57:04 +01:00
Michael Schimmel be2bef373f Refactor scope handling and immutability
Introduces `fixed_scope_idx` to `Binder` and `Environment` to track
immutable scopes.
This allows enforcing that definitions (`def`) cannot occur in scopes
that are considered fixed or immutable, such as the global scope or the
initial bootstrap scope.
The `FunctionCompiler` is updated to use `fixed_scope_idx` and `is_root`
to determine scope immutability.
2026-03-12 11:22:40 +01:00
Michael Schimmel bb77caf1af Refactor: Rename TCO module to lowering
The TCO (Tail Call Optimization) module has been renamed to `lowering`.
This change better reflects the module's broader responsibility, which
includes not only TCO but also general AST transformations and
preparation for VM execution.

The `optimize` function has been renamed to `lower` to align with the
module's new name.
2026-03-11 10:49:24 +01:00
Michael Schimmel db26719cad Extract calculate_stack_size to a standalone function
The `calculate_stack_size` method was duplicated in `bound_nodes.rs` and
`tco.rs`. This commit extracts it into a single standalone function in
`tco.rs` to avoid duplication. The stack size is now calculated and
stored in the `ExecNode`'s `ty` field during the TCO optimization phase.
2026-03-11 10:35:18 +01:00
Michael Schimmel 3657f19047 Refactor stack management to use resize
Replaces manual iteration for pushing `Value::Void` with `Vec::resize`
for more efficient stack management. This change improves performance by
reducing redundant operations and simplifies the code.
2026-03-10 20:43:46 +01:00
Michael Schimmel efcb4e3685 feat: Add stack size to runtime metadata
Store the pre-calculated stack size of lambda bodies in the
RuntimeMetadata. This allows the VM to directly access this information
without recalculating it.
2026-03-10 20:25:04 +01:00
Michael Schimmel 8c865681ff Refactor: Implement late stack size calculation
This commit introduces a new mechanism for calculating the required
stack size for closures at bind time, rather than storing it in the AST.
This is achieved by adding a `calculate_stack_size` method to
`BoundNode`.

Key changes include:
- `BoundNode::calculate_stack_size`: Recursively traverses the bound AST
  to find the maximum `LocalSlot` index used.
- Recursion blocker for lambdas: Ensures that nested lambdas are not
  visited during stack size calculation for the outer closure,
  preventing incorrect stack size estimations.
- VM update: The `VM::run` method now accepts and uses the
  pre-calculated stack size for setting up call frames.
- `Closure` struct update: Stores the `stack_size` directly within the
  `Closure` struct.
- Binder and TypeChecker updates: Minor adjustments to accommodate the
  new strategy and error reporting.
- Optimizer enhancements: Constant and pure-lambda propagation for
  `Define` statements within blocks to ensure inlinability.

This refactoring addresses issues related to accurate stack size
management and improves the overall robustness of the binder and VM.
2026-03-10 20:05:32 +01:00
Michael Schimmel cb4d3525c2 Refactor binder to simplify root scope logic
The previous implementation of the binder had a slightly complex way of
handling the root scope. This commit simplifies that logic by directly
checking if the current scope is the root scope and the scope stack has
only one element. This makes the code more readable and maintainable.
2026-03-10 17:25:27 +01:00
Michael Schimmel a78e72d074 Refactor Binder to use new scope structure
This commit refactors the Binder implementation to better align with the
updated scope structure.
Key changes include:

- Introducing `ExprContext` to distinguish between statement and
  expression contexts.
- Modifying `define_variable` to handle both global and local scope
  definitions more robustly.
- Updating `resolve_variable` to correctly handle upvalue captures,
  especially for global addresses.
- Adjusting `bind` and related functions to use the new `ExprContext`.
- Updating the refactoring log to reflect completed phases.
2026-03-10 17:20:47 +01:00
Michael Schimmel 3b063dc2c9 Add block scoping and statement/expression distinction
Implement strict block scoping and distinguish between statements and
expressions.
This change introduces hierarchical scopes for functions and blocks,
enforces that `def` is a statement with no return value, and prevents
statements from appearing in expression positions or as the last element
of a block. A new example `design-flaw.myc` is added to demonstrate
the uninitialized variable issue solved by these changes.
2026-03-10 17:03:46 +01:00
Michael Schimmel 961168f3b6 Refactor analyzer to return impurity
The `Analyzer` is now responsible for determining the purity of `Define`
bindings. Previously, this responsibility was left to the caller.

Added a new example `design-flaw.myc` to test this change.
Updated `soa_series.myc` due to the purity change.
2026-03-10 16:13:39 +01:00
Michael Schimmel 348b1686f2 Refactor soa_series example to use a constant for ticks 2026-03-10 15:34:10 +01:00
128 changed files with 18439 additions and 10725 deletions
+4
View File
@@ -0,0 +1,4 @@
*.snap
*.snap.new
target/
docs/delphi/
+10
View File
@@ -3,5 +3,15 @@
"fileFiltering": {
"respectGitIgnore": false
}
},
"mcpServers": {
"repomix": {
"command": "repomix",
"args": [
"--ignore",
".*/**,.*,docs,examples,tests,repomix*,gemini*,**.snap",
"--mcp"
]
}
}
}
+1
View File
@@ -1,3 +1,4 @@
/target
Delphi
*.snap.new
repomix-output.xml
+74
View File
@@ -0,0 +1,74 @@
# Projekt: AST-Compiler
* Dieses Repository enthält einen unfertigen Compiler für eine DSL, die zur Finanzanalyse konzipiert ist.
### Design
* Der AST soll 1st class citizen sein. Das Skript ist nur eine mögliche Art, ihn darzustellen.
* Eine geplante Darstellung des AST ist vollständige grafische Visualisierung.
* Es wird eine DSL für Finanzanalyse.
* Aufgrund der geforderten Visualisierbarkeit muss der untyped AST (das, was der User bearbeiten kann) möglichst simpel sein. Komplexität muss vom System übernommen werden.
* Closures sind Key-Feature.
### Concurrency
* Die Root-Scopes sind die Grenze für Multithreading. Nichts verlässt den Root-Scope und alles innerhalb des Scope ist Single-Threaded.
* Globals sind nach dem Bootstrapping immutable.
* Da das Skript single-threaded ist, kann auf atomare Operationen (Mutex, Arc) verzichtet werden!
### Spezielle Datentypen
* Serien sind "unendliche" Queues mit maximaler Länge (Lookback). Serie[0] ist das zuletzt gepushte Item.
* Streams sind virtuelle Konfigurationen aus Serien, die in der Lage sind einen neuen Item-Record zu propagieren.
* Pipes sind Streams mit einer weiteren Funktion: Sie können einen Stream als Input akzeptieren und einen anderen Stream als Output produzieren.
## Portierungsregeln
* Wir wollen die Sprachfeatures von Rust nutzen.
* Der Code soll aber so gut wie möglich nach Rust-Style-Guidelines geschrieben werden.
* Dokumentation nach Rust-Regeln.
* Im Code und den Kommentaren muss alles auf englisch sein.
## Rust
* Performance: Bevorzuge `Rc<dyn Trait>` + lokales `downcast_ref` gegenüber Deep Copies via `Rc::new(obj.clone())`.
* CRITICAL: Keine Wildcards (`_ =>`, `other => other`) in `match`-Ausdrücken auf `NodeKind` oder `SyntaxKind`. Alle Varianten müssen explizit aufgeführt werden. Exhaustive Matches erzwingen, dass neue Varianten an jeder Stelle bewusst behandelt werden.
## Interaktion
* Ich kann Rust noch nicht so gut. Erkläre mir, was du machst und welche Konzepte du nutzt. Ich will was dazulernen. Wenn erforderlich, nutze zur Veranschaulichung Delphi aus Vergleichssprache.
* Wenn clippy etwas vorschlägt, von dem du glaubst, dass es nicht stabil ist, dann überprüfe die aktuelle Version. Wahrscheinlich ist das Feature mittlerweile im trunk. Wir gehen immer davon aus, dass clippy recht hat.
* Keine Interaktion mit GIT. Kein Commit, oder ähnliches. Schlage auch nichts dergleichen vor.
* Gehe immer schrittweise vor. Zeige mir, was du vor hast, bevor du Code erzeugst.
* Sprich im Chat deutsch mit mir.
* Nutze den Repomix-MCP-Server um den Code zu lesen.
## Testing & Debugging
* Das Projekt enthält eine Testsuite für Skripte. Logik in src/utils/tester.rs. Diese Tests werden automatisch in "cargo test" eingebunden.
* Nach Fertigstellung einer Aufgabe, oder wenn ich zum Testen auffordere: Warnungen sind zu eliminieren. Tests müssen durchlaufen. Wenn alles funktioniert, muss auch clippy ohne Beanstandungen durchlaufen.
* Du kannst Skripte selbst testen! Nutze das ast-tool:
> ./target/release/ast.exe --help
MYC AST Compiler & Benchmarker
Usage: ast.exe [OPTIONS] [FILE]
Arguments:
[FILE] The script file to run or benchmark
Options:
-e, --eval <EVAL> Run a script string directly
-b, --bench Run benchmarks (Only allowed in Release mode)
-u, --update-bench Update the benchmark baseline (Only allowed in Release mode)
-d, --dump Dump the compiled AST
-t, --trace Run with TracingObserver enabled
--no-opt Disable optimization
-h, --help Print help
-V, --version Print version
# Sprachspezifikation
* Dieses Dokument sollte aktuell gehalten werden:
@docs/BNF.md
Generated
+728 -1
View File
File diff suppressed because it is too large Load Diff
+7
View File
@@ -6,10 +6,17 @@ default-run = "myc"
[dependencies]
eframe = "0.33.3"
egui_extras = "0.33"
clap = { version = "4.5", features = ["derive"] }
chrono = "0.4"
regex = "1.10"
fastrand = "2.3"
rmcp = { version = "1.2", features = ["server", "transport-io"] }
tokio = { version = "1", features = ["rt", "io-std", "macros"] }
serde = { version = "1", features = ["derive"] }
zip = "2"
data-server = { git = "http://192.168.178.103:3000/Brummel/data-server.git", branch = "main" }
[dev-dependencies]
insta = { version = "1.39", features = ["yaml"] }
tempfile = "3"
+5 -8
View File
@@ -1,10 +1,6 @@
# Rust-Portierung des Delphi-AST-Compilers
# Projekt: AST-Compiler
* Dieses Repository enthält den Rust-Port des Delphi-Compilers.
* Das Ziel ist, die alte Delphi-Codebase nach Rust zu portieren.
* Die alte Delphi-Codebase ist nicht zu verändern.
Wichtig: zentraler Einstiegspunkt ist Delphi/Myc.Ast.Environment.pas und die dort referenzierten Units.
* Dieses Repository enthält einen unfertigen Compiler für eine DSL, die zur Finanzanalyse konzipiert ist.
### Design
@@ -39,11 +35,12 @@ Wichtig: zentraler Einstiegspunkt ist Delphi/Myc.Ast.Environment.pas und die dor
## Interaktion
* Ich kann Rust noch nicht so gut. Erkläre mir, was du machst und welche Konzepte du nutzt. Ich will was dazulernen.
* Ich kann Rust noch nicht so gut. Erkläre mir, was du machst und welche Konzepte du nutzt. Ich will was dazulernen. Wenn erforderlich, nutze zur Veranschaulichung Delphi aus Vergleichssprache.
* Wenn clippy etwas vorschlägt, von dem du glaubst, dass es nicht stabil ist, dann überprüfe die aktuelle Version. Wahrscheinlich ist das Feature mittlerweile im trunk. Wir gehen immer davon aus, dass clippy recht hat.
* Keine Interaktion mit GIT. Kein Commit, oder ähnliches. Schlage auch nichts dergleichen vor.
* Gehe immer schrittweise vor. Zeige mir, was du vor hast, bevor du Code erzeugst.
* Sprich im Chat deutsch mit mir.
* Nutze den Repomix-MCP-Server um den Code zu lesen.
## Testing & Debugging
@@ -71,6 +68,6 @@ Options:
# Sprachspezifikation
* Dieses Dokument sollte aktuell gehalten werden.
* Dieses Dokument sollte aktuell gehalten werden:
@docs/BNF.md
+27
View File
@@ -0,0 +1,27 @@
# Analyse von `src/ast/environment.rs`
## 1. Struktur von `Environment`
Die `Environment`-Struktur fungiert im Projekt als zentraler "State-Manager" und Orchestrator. Sie hält den globalen Zustand für den gesamten Lebenszyklus eines Skripts, von der Quelle bis zur Ausführung:
* **Zustandsverwaltung (State Container):** Beinhaltet den globalen Programmzustand in Form von Registern und Caches. Nahezu alles ist in `Rc<RefCell<T>>` gekapselt (z.B. `root_types`, `root_values`, `root_scopes`, Caches für Optimierung und Monomorphisierung).
* **Modulladung & Abhängigkeiten:** `preload_dependencies` und `discover_globals` lesen `#use`-Abhängigkeiten, durchsuchen den Code vorab nach globalen Definitionen (`def`) und Makros und laden die Standardbibliothek (`prelude.myc`).
* **Kompilierungs-Pipeline:** Die Methoden `compile`, `compile_syntax`, `compile_pipeline` und `link` steuern den Code durch alle Compiler-Phasen: Macro-Expansion -> Binding -> Type-Checking -> Analysis -> Specialization -> Optimization -> Lowering.
* **Makro-Evaluierung:** Die interne Struktur `RuntimeMacroEvaluator` wird genutzt, um AST-Knoten zur Compile-Zeit an eine VM zu übergeben und den Code für Makros auszuführen.
* **Laufzeit-Ausführung & RTL (Runtime Library):** Methoden wie `run_script`, `run_debug` und `instantiate` starten die `VM`. Zudem gibt es Methoden (`register_native`, `allocate_slot`), um native Rust-Funktionen (Intrinsics) im globalen RTL-Scope (Scope 0) zu registrieren.
* **Dokumentations-Registry:** Es speichert sowohl RTL-Dokumentation als auch aus dem Source-Code extrahierte Kommentare (`myc_docs`).
## 2. Prüfung auf Boilerplate
Der Code weist an mehreren Stellen typischen Rust-Boilerplate für Single-Threaded-Interpreter auf:
* **Das `Rc<RefCell>`-Muster:** Um denselben globalen Zustand zwischen Parser, Compiler, Pipeline und VM zu teilen, bestehen 14 der 19 Felder aus `Rc<RefCell<...>>`. Das zwingt überall im Code zu redundantem `.borrow()`, `.borrow_mut()` und `.clone()` (z. B. beim Klonen in `RuntimeMacroEvaluator` über `get_expander`). Ausnahme: `rtl_values: Rc<[Value]>` ist bewusst **ohne** `RefCell` — die eingefrorenen RTL-Werte sind nach dem Bootstrap-Freeze immutable und werden im VM-Hotpath direkt gelesen.
* **Fehler-Mapping:** Beim Modulladen wird oft repetitiv mit `.map_err(|e: String| format!("...", e))` Boilerplate geschrieben, anstatt einen zentralen `Error`-Typen mit `thiserror` oder `anyhow` zu verwenden.
* **Manuelle AST-Traversierung:** In `discover_globals` und `collect_doc_comments` wird der AST per Hand (mittels `match` und Rekursion) durchlaufen, anstatt ein zentrales "Visitor-Pattern" wiederzuverwenden.
## 3. Prüfung auf "unscharfe Interfaces" (Fuzzy Interfaces)
Der Code enthält einige starke Indikatoren für unscharfe Systemgrenzen ("Leaky Abstractions") und Vermischung von Zuständigkeiten (God Object Smell):
* **God Object:** `Environment` weiß zu viel. Es parst Dateien, wertet Makros aus, instanziiert den Typ-Prüfer (`TypeChecker`), betreibt Caching für Optimierer, startet die `VM` und sammelt nebenbei Doc-Strings. Eine klarere Trennung zwischen `CompilerEnvironment` (statische Phasen) und `RuntimeEnvironment` (VM-Zustand/Values) fehlt hier.
* ~~**Verwaschene Kompilierungsschritte:**~~ **✓ Behoben (Design korrekt, ein Detail bereinigt).** Die drei Methoden bilden eine saubere Adapter-Hierarchie: `compile_pipeline` (Kern, Diagnostics-Sink) ← `compile` (public API, merged Parse+Compile-Fehler in `CompilationResult`) / `compile_syntax` (privater Adapter für Modul-Loader, konvertiert zu `Result<_, String>`). Das unterschiedliche Error-Reporting ist situations-appropriat, nicht inkonsistent. **Behobener Design-Geruch:** `dump_ast` umging zuvor `compile()` und verschluckte Parse-Fehler still. Behoben: `dump_ast` delegiert jetzt an `compile(source).into_result()?`.
* ~~**Die `specialize_node`-Closure:**~~ **✓ Kein Problem (Design korrekt verstanden).** Die `compiler`-Closure ist kein Zeichen unsauberer Kapselung — im Gegenteil. `Specializer` ist bewusst von TypeChecker, Optimizer und VM entkoppelt; der `CompileFunc`-Typ (`specializer.rs:15`) ist der explizite **Dependency-Injection-Punkt** (Strategy Pattern). Die innere Pipeline (TypeCheck → Analyze → sub-Specialize → Optimize → Lower → VM.run) **dupliziert nicht** die äußere — ihr Zweck ist fundamental verschieden: sie erzeugt einen gecachten `Value` (compile-and-evaluate), während die äußere Pipeline einen `ExecNode` für die Skriptausführung erzeugt. Die Closure captured 6 `Rc<RefCell<T>>`-Handles statt `&self`, weil Rust keine `self`-Referenz in eine `'static`-Closure erlaubt — das ist der korrekte Rust-Weg. **Design-Limitation (dokumentiert, kein Bug):** Der `sub_specializer` innerhalb der Closure hat `compiler: None`, was Endlos-Rekursion verhindert, aber bedeutet, dass nur eine Spezialisierungsebene pro Aufruf expandiert wird.
* ~~**Type-Checker Hack:**~~ **✓ Kein Problem (Kommentar bereinigt).** Der ursprüngliche Kommentar war veraltet und irreführend. Das `BoundLike`-Trait vereinigt alle Phasen mit identischer Binding-Struktur (`BoundPhase`, `TypedPhase`, `AnalyzedPhase`). `TypeChecker::check_node_as_bound<P: BoundLike>()` ist korrekte Generic-Programmierung — kein Hack. Die Phasen passen sauber ineinander. Der Kommentar wurde durch eine korrekte Erklärung des Monomorphisierungs-Designs ersetzt.
* ~~**Konzeptioneller Bruch bei `instantiate`:**~~ **✓ Behoben.** `instantiate` gibt jetzt `Result<Rc<Closure>, String>` zurück statt `Rc<NativeFunction>`. Die VM-Erzeugung und -Ausführung liegt beim Aufrufer (`run_script_compiled`, Benchmark-Runner), der die passende Strategie kennt (einmaliger Run vs. N Iterationen mit VM-Reuse). `NativeFunction` bleibt ausschließlich für echte Rust-Intrinsics (RTL). Bonus: der Benchmark-Runner erstellt jetzt eine VM für N Iterationen statt N VMs — `run_with_args` resettet `stack` und `frames` am Anfang jedes Aufrufs, VM-Reuse ist sicher.
+158 -6
View File
@@ -66,6 +66,19 @@ This document defines the syntax (BNF) and semantics of the Myc language, a Lisp
```
*(Lexical Note: `<ident-start>` consists of alphabetic characters or `+-*/<=>!$%&_?.`. `<ident-char>` includes digits as well. Identifiers evaluating to a field accessor begin with a `.` followed by one or more valid identifier characters.)*
### Comments
```ebnf
<comment> ::= ";" " "? <line-text> <newline>
<doc-comment> ::= ";;" " "? <line-text> <newline>
```
**Rules:**
- Comments must be **whole lines** — they start at the beginning of a line. Inline comments (after code on the same line) are a compile error.
- Comment lines directly preceding an expression form a **comment block** attached to that expression.
- Blank lines between comment lines are preserved as separators within the block. The first and last line of a block must not be blank.
- `;;` (doc comment) before a `def` or `macro` registers the text in the symbol documentation registry (available via `--dump-docs`). Before other expressions it is allowed but has no registry effect.
## 2. Core Semantics & Data Types
- **Integers and Floats:** Standard 64-bit numeric types.
@@ -79,7 +92,8 @@ This document defines the syntax (BNF) and semantics of the Myc language, a Lisp
- **Vectors (Tuples):** Enclosed in square brackets `[1 2 3]`. Evaluates to a vector/tuple structure.
- **Records:** Enclosed in curly braces with keyword keys and any expression as values `{:id 101 :name "Alice"}`. They provide O(1) field access using internal memory layouts. Structural equality `(= {:a 1} {:a 1})` is `true`.
- **Series:** A core concept for financial analysis. Series are "infinite" queues with a maximum length (lookback).
- They are created via the `(series ...)` function specifying a record layout (e.g., `(series {:price :float :volume :int})`).
- They are created via `(series lookback_limit)`. The element type is **inferred at compile time** from subsequent `push` calls (Hindley-Milner type inference) — no explicit schema is needed.
- Example: `(series 100)` — the compiler infers the element type from usage.
- **Indexing:** You access items in a series by calling it or an extracted field like a function with an integer index: `(my_series 0)`. **Crucially, index `0` represents the most recently pushed item.** Index `1` is the second most recent, and so on (lookback indexing).
## 4. Special Forms and Evaluation logic
@@ -123,7 +137,7 @@ The Myc runtime environment provides a collection of built-in functions and macr
- `now`: Returns the current timestamp.
### Series & Streaming
- `series`: Creates a new series with a defined layout.
- `series`: Creates a new series. Takes a lookback limit; the element type is inferred at compile time from `push` calls.
- `push`: Pushes a new item into a series.
- `len`: Returns the current number of elements in a series.
- `create-random-ohlc`: Generates a mock Open-High-Low-Close data stream.
@@ -138,15 +152,16 @@ The Myc runtime environment provides a collection of built-in functions and macr
```clojure
(do
(def user {:id 101 :name "Alice" :role :admin})
(def role (.role user)) ; Evaluates to :admin
; Evaluates to :admin
(def role (.role user))
)
```
**Financial Pipelines and Series (with Indexing):**
```clojure
(do
;; Create a series with a typed layout (Struct-of-Arrays under the hood)
(def my_ticks (series {:price :float :volume :int :msg :text}))
;; No schema needed — the compiler infers the element type from push calls.
(def my_ticks (series 100))
(push my_ticks {:price 10.5 :volume 100 :msg "A"})
(push my_ticks {:price 11.2 :volume 200 :msg "B"})
@@ -156,6 +171,143 @@ The Myc runtime environment provides a collection of built-in functions and macr
(def prices (.price my_ticks))
;; Series indexing: 0 is the newest (15.5), 1 is the previous (11.2)
(+ (prices 0) (prices 1)) ;; Output: 26.7
;; Output: 26.7
(+ (prices 0) (prices 1))
)
```
## 8. Common Mistakes & Pitfalls
This section is especially relevant for LLMs generating Myc code.
### Wrong keywords from other Lisps
| Wrong (Clojure/Scheme) | Correct Myc |
|---|---|
| `(let [x 1] ...)` | `(do (def x 1) ...)` |
| `(begin ...)` | `(do ...)` |
| `(lambda [x] ...)` | `(fn [x] ...)` |
| `(define x 1)` | `(def x 1)` |
| `(set! x 2)` | `(assign x 2)` |
| `(cond ...)` | nested `(if ...)` |
| `(not= a b)` | `(<> a b)` |
### Series indexing is reversed
Index `0` is the **most recently pushed** item, not the oldest. This is the opposite of normal array indexing:
```clojure
(push s {:v 1})
(push s {:v 2})
(push s {:v 3})
;; ((.v s) 0) => 3 (newest)
;; ((.v s) 1) => 2
;; ((.v s) 2) => 1 (oldest)
```
### Record keys must be keywords, not strings
```clojure
;; WRONG:
{"price" 10.5}
;; CORRECT:
{:price 10.5}
```
### `again` is only valid inside `fn`
`(again ...)` jumps back to the enclosing `fn`. Using it outside a function is a compile error.
```clojure
;; WRONG — top-level again:
(again 1 2)
;; CORRECT — inside fn:
(fn [n] (if (= n 0) "done" (again (- n 1))))
```
### `check_syntax` accepts only a single expression
The `check_syntax` tool uses the single-expression compiler pass. Wrap multiple expressions in `(do ...)`:
```clojure
;; WRONG:
(def x 1) (+ x 2)
;; CORRECT:
(do (def x 1) (+ x 2))
```
### Inline comments are not allowed
Comments must be **whole lines** starting with `;` or `;;`. Placing a comment after code on the same line is a compile error.
```clojure
;; WRONG — inline comment causes a compile error:
(def x 42) ; this is x
;; CORRECT — comment on its own line before the expression:
; this is x
(def x 42)
```
### Field accessors are functions, not property syntax
```clojure
;; WRONG — not valid syntax:
user.name
;; CORRECT — field accessor called as a function:
(.name user)
;; Also correct — extract accessor first, then call:
(def get-name .name)
(get-name user)
```
**Recursive Function with Tail-Call Optimization:**
```clojure
(do
;; Factorial using explicit TCO via (again ...) — like Clojure's recur
(def factorial
(fn [n acc]
(if (= n 0)
acc
(again (- n 1) (* acc n)))))
;; Output: 3628800
(factorial 10 1)
)
```
**Macro Definition:**
```clojure
(do
;; Define a macro that inverts a condition
(macro unless [c body]
`(if ~c ... ~body))
;; Output: "executed"
(unless false "executed")
)
```
**Series with while Loop:**
```clojure
(do
(def ticks (series 100))
(def i 0)
;; Push 5 prices into the series
(while (< i 5)
(do
(push ticks {:price (* i 1.5)})
(assign i (+ i 1))))
;; Read back the two most recent prices
(def p (.price ticks))
;; newest + second-newest
(+ (p 0) (p 1))
)
```
+108
View File
@@ -0,0 +1,108 @@
using System;
using System.IO;
using System.IO.Compression;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using cAlgo.API;
namespace cAlgo.Robots
{
public enum ExportMode { M1, Tick }
[Robot(AccessRights = AccessRights.FullAccess)]
public class MS_SimpleExport : Robot
{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private struct M1Record {
public double Time;
public double O, H, L, C;
public float S;
public int V;
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
private struct TickRecord {
public double Time;
public double Ask, Bid;
}
[Parameter("Export Mode", DefaultValue = ExportMode.M1)]
public ExportMode Mode { get; set; }
[Parameter("Output File (ZIP)", DefaultValue = "")]
public string OutputFile { get; set; }
[Parameter("Is Probing", DefaultValue = false)]
public bool IsProbing { get; set; }
private List<M1Record> _m1Buffer = new List<M1Record>();
private List<TickRecord> _tickBuffer = new List<TickRecord>();
protected override void OnStart() {
// No price factor needed for double/float export
}
protected override void OnTick() {
if (Mode != ExportMode.Tick) return;
if (IsProbing) {
Print("DATA_FOUND");
Stop();
return;
}
_tickBuffer.Add(new TickRecord {
Time = Server.TimeInUtc.ToOADate(),
Ask = Symbol.Ask,
Bid = Symbol.Bid
});
}
protected override void OnBar() {
if (Mode != ExportMode.M1) return;
if (IsProbing) {
Print("DATA_FOUND");
Stop();
return;
}
var bar = Bars.Last(1);
_m1Buffer.Add(new M1Record {
Time = bar.OpenTime.ToOADate(),
O = bar.Open,
H = bar.High,
L = bar.Low,
C = bar.Close,
S = (float)Symbol.Spread,
V = (int)bar.TickVolume
});
}
protected override void OnStop() {
if (IsProbing || string.IsNullOrEmpty(OutputFile)) return;
try {
using var fileStream = new FileStream(OutputFile, FileMode.Create);
using var archive = new ZipArchive(fileStream, ZipArchiveMode.Create);
var entry = archive.CreateEntry($"{Symbol.Name}.bin", CompressionLevel.Fastest);
using var entryStream = entry.Open();
if (Mode == ExportMode.M1) {
ReadOnlySpan<M1Record> span = CollectionsMarshal.AsSpan(_m1Buffer);
ReadOnlySpan<byte> byteSpan = MemoryMarshal.Cast<M1Record, byte>(span);
if (!byteSpan.IsEmpty) entryStream.Write(byteSpan);
} else {
ReadOnlySpan<TickRecord> span = CollectionsMarshal.AsSpan(_tickBuffer);
ReadOnlySpan<byte> byteSpan = MemoryMarshal.Cast<TickRecord, byte>(span);
if (!byteSpan.IsEmpty) entryStream.Write(byteSpan);
}
Print("EXPORT_SUCCESS");
} catch (Exception ex) {
Print("ERROR: {0}", ex.Message);
}
}
}
}
+77
View File
@@ -0,0 +1,77 @@
# Development Log: Binder Refactoring & Block Scoping
## Status Quo
- Myc currently uses a flat scope per function.
- `def` is treated as a side-effect expression that "leaks" into the function scope regardless of nesting (e.g., inside `if` or `do`).
- This leads to uninitialized variables at runtime if the definition is skipped by control flow.
## Goal
Implement strict **Block Scoping** and distinguish between **Statements** and **Expressions** to eliminate undefined variable states.
### Rules
1. **Block Scoping:** Every `(do ...)` and `if` branch creates a new lexical scope.
2. **Statements vs. Expressions:**
- `def` is a **Statement**.
- Statements have no return value (Void).
- Statements **cannot** be used as expressions (e.g., as arguments, RHS of assignments, or conditions).
- Statements **cannot** be the last element of a block (since the block's value is determined by its last expression).
3. **No Shadowing:** Redefining a symbol within the **same** scope level is an error. Shadowing from outer scopes is allowed (standard lexical scoping).
4. **Self-Reference:** A variable is only available in its scope **after** its definition is complete (RHS of `def` sees outer scope).
## Implementation Plan
### Phase 1: Binder Infrastructure (Completed)
1. [x] Update `CompilerScope` to support hierarchical nesting via a stack in `FunctionCompiler`.
2. [x] Implement `push_scope` and `pop_scope` in `Binder`.
3. [x] Refactor `resolve_variable` to traverse the scope stack (inner-to-outer).
4. [x] Update `define_variable` to enforce "No Shadowing" at the current scope level.
### Phase 2: Statement/Expression Validation (Completed)
1. [x] Introduce a mechanism to track "Context" (Expression vs. Statement) during binding.
2. [x] Validate that `def` is only used in statement positions.
3. [x] Enforce that the last node in a `BoundKind::Block` is an expression.
### Phase 3: VM Compatibility
1. [ ] Ensure the VM handles "Void" results from statements correctly.
2. [ ] (Optional) Optimize stack allocation based on scope depth.
---
## Log Entries
### 2024-05-22: Phase 2 Completed & Strategy Adjustment
- Successfully implemented strict **Block Scoping** and **Statement vs. Expression** validation.
- Repaired the Root-Scope "leaking" problem: `def` only creates Globals at the top-most level of the script.
- **Challenge:** Many existing tests fail because they use `def` in expression positions or as the last element of a block.
- **Strategy Change for Stack Size:** Instead of storing `slot_count` in the AST during Binding, we will implement **Late Counting**. Just before execution, we will determine the required stack size by finding the maximum `LocalSlot` index used in each lambda. This avoids propagating metadata through all compiler passes and remains accurate after optimizations.
## Phase 4: Late Stack Size Calculation & VM Robustness (Completed)
1. [x] Implement a `calculate_stack_size()` method on `BoundNode` to find the maximum `LocalSlot` used in a frame.
2. [x] Recursion stops at lambda boundaries to ensure each closure gets its own independent stack size.
3. [x] Update VM to pre-allocate the stack with `Value::Void` for each call frame. This eliminates "Stack Underflow" when control flow skips a variable definition.
4. [x] Fixed "Nested Cell Bug": `Define` now checks if a slot is already a `Cell` (due to forward capture) before wrapping, preventing `Cell(Cell(Value))` corruption.
5. [x] Fixed "Root Closure Execution": Root scripts are now correctly executed as parameterless lambdas, returning the actual script result instead of the closure object.
### Phase 5: Optimizer Enhancements (Completed)
1. [x] Enabled constant and pure-lambda propagation for `Define` statements within blocks.
2. [x] This ensures that macro expansion and record field access remain fully inlinable even with strict statement rules.
---
## Final Status: 100% Green Tests
- All 64 tests (Unit + Integration + Examples) are passing.
- Strict **Block Scoping** is enforced everywhere.
- `def` is a pure **Statement** (no return value, not allowed at end of blocks).
- Design flaw "Uninitialized variables" is completely eliminated via pre-allocation and binder visibility rules.
## Key Learnings (Rust Context)
- **Late Counting vs. AST Metadata:** Keeping the AST clean and calculating runtime needs (stack size) just before execution is more robust against optimizer changes.
- **Nested Ownership (Value::Cell):** Explicitly handling the state of stack slots (raw value vs. heap-allocated cell) is crucial for correct closure captures.
- **Root Scope Specialization:** Treating the initial script as a "Function with special global-promotion rules" keeps the compiler logic consistent.
### Phase 1 & 2 Completed
- Introduced scope stack into `FunctionCompiler`.
- Introduced `ExprContext` (`Expression` / `Statement`) in the `Binder`.
- `def` now acts strictly as a statement.
- Variables defined inside nested scopes (e.g. inside `if` or `do`) do not leak into outer local scopes anymore.
- Outstanding behavior to discuss: `def` inside nested scopes at the *root* level still leaks into globals.
+81
View File
@@ -0,0 +1,81 @@
# Bytecode Generation Architecture
This document describes the transition from a pure tree-walking AST interpreter to a hybrid execution model using specialized bytecode.
## 1. Overview: The Hybrid Model
The MYC VM operates in a hybrid mode. While the Abstract Syntax Tree (AST) is the first-class citizen and primary representation, "hot" paths—specifically monomorphized functions and financial pipeline lambdas—are lowered into a linear bytecode representation.
---
## 2. The Integrated Transformation Pipeline
To ensure maximum performance and correct Tail Call Optimization (TCO), bytecode generation is integrated into the structural optimization phase.
### Step 1: Specializer (Structural Optimization & Emission)
The **Specializer** is the core engine for code improvement.
- **Inlining & Monomorphization:** It first performs all structural changes (e.g., inlining function bodies). This is critical because inlining changes what constitutes a "tail position."
- **Symbolic Emission:** Once the structure of a function/block is final, the Specializer converts it into **Symbolic Bytecode**.
- **Integrated TCO Analysis:** During emission, the Specializer tracks the tail-call state locally. It emits `OP_RECUR` or `OP_TAIL_CALL` based on the final, inlined structure.
- **Variable Mapping:** Instructions still use `VirtualId` (e.g., `OP_GET_VIRTUAL(V102)`).
### Step 2: Lowering (Address Resolution & Linking)
The **Lowering** pass acts as the "Linker". It does not change the structure anymore.
- **Address Resolution:** It walks the symbolic bytecode and replaces `VirtualId` with concrete `StackOffset` values.
- **Stack Allocation:** It calculates the final `stack_size` for the bytecode block.
- **Finalization:** It "freezes" the instruction vector for the `ExecNode`.
---
## 3. Why This Order? (Inlining vs. TCO)
TCO analysis must occur **after** structural optimizations like inlining.
If TCO were analyzed before inlining, a tail-call in a small function might lose its tail position once embedded into a larger caller. By performing TCO analysis **during** bytecode emission (after inlining), we guarantee that the `is_tail` flags are always semantically correct.
---
## 4. Instruction Set Architecture (ISA) & TCO
The ISA uses specialized instructions to bypass the heavy tree-walking logic:
1. **OP_RECUR:**
- Used for the `again` keyword.
- Triggers an immediate jump to PC 0 within the same bytecode executor.
- Repacks the current stack frame with new arguments.
2. **OP_TAIL_CALL(Callee):**
- Used for function calls in tail positions.
- If the callee is bytecode, it performs a fast-swap of the instruction vector.
- If the callee is native/tree, it returns a `TailCallRequest` to the VM dispatcher.
---
## 5. The Hybrid VM Dispatcher
The VM's `eval_internal` remains the master controller. It dispatches to the Bytecode Executor when encountering a `BoundKind::Bytecode` node. The executor can "escape" back to the tree-walker for any operation it cannot handle natively (e.g., calling an opaque extension).
```rust
fn run_bytecode(instructions: &[Instruction], pc: &mut usize) -> Value {
loop {
match instructions[*pc] {
Instruction::PushConst(v) => stack.push(v),
Instruction::AddInt => {
let b = stack.pop();
let a = stack.pop();
stack.push(a + b);
}
Instruction::TailCall(callee) => {
// Return to dispatcher if it's a cross-world call
return self.handle_tail_call(callee);
}
Instruction::CallTree(node) => {
// Re-enter tree-walking for specific node
let res = self.eval_internal(node)?;
stack.push(res);
}
// ...
}
pc += 1;
}
}
```
+71
View File
@@ -0,0 +1,71 @@
;; Benchmark: 98.9us
;; Benchmark-Repeat: 22
;; Skip: output
(do
;; make-sma Factory (O(1))
(def make-sma
(fn [lookback]
(do
(def s (series lookback))
(def running-sum 0.0)
(def count 0)
(fn [x]
(do
(if (< count lookback)
(do (assign running-sum (+ running-sum x)) (assign count (+ count 1)))
(do (assign running-sum (+ (- running-sum (s (- lookback 1))) x))))
(push s x)
(/ running-sum count))))))
;; make-wma Factory (O(1))
(def make-wma
(fn [lookback]
(do
;; Sicherstellen, dass lookback mindestens 1 ist
(def n (if (< lookback 1) 1 lookback))
(def s (series n))
(def price-sum 0.0)
(def weighted-sum 0.0)
(def count 0)
(def weight-total (/ (* n (+ n 1)) 2))
(fn [x]
(do
(if (< count n)
(do
(assign count (+ count 1))
(assign price-sum (+ price-sum x))
(assign weighted-sum (+ weighted-sum (* x count)))
(push s x)
(/ weighted-sum (/ (* count (+ count 1)) 2)))
(do
(def oldest (s (- n 1)))
(assign weighted-sum (+ (- weighted-sum price-sum) (* x n)))
(assign price-sum (+ (- price-sum oldest) x))
(push s x)
(/ weighted-sum weight-total))))))))
;; make-hma Factory (O(1))
;; Hull Moving Average: WMA(sqrt(n), 2*WMA(n/2) - WMA(n))
(def make-hma
(fn [lookback]
(do
(def n lookback)
(def wma1 (make-wma (trunc (/ n 2))))
(def wma2 (make-wma n))
(def wma-final (make-wma (trunc (sqrt n))))
(fn [x]
(do
(def v1 (wma1 x))
(def v2 (wma2 x))
(wma-final (- (* 2.0 v1) v2)))))))
;; Beispiele & Test
(def hma16 (make-hma 16))
(print "HMA (16) Initialisierung gestartet...")
(def i 1.0)
(while (<= i 20.0)
(do
(print "HMA Step" i ":" (hma16 (* i 10.0)))
(assign i (+ i 1))))
)
+3 -4
View File
@@ -1,10 +1,9 @@
;; Benchmark: 1.8us
;; Benchmark-Repeat: 1122
;; Output: 120
;; Benchmark: 1.5us
;; Benchmark-Repeat: 1306
(do
(def factorial (fn [n acc]
(if (= n 0)
acc
(again (- n 1) (* acc n)))))
(factorial 5 1))
(assert-eq 120 (factorial 5 1)))
+3 -4
View File
@@ -1,12 +1,11 @@
;; Closure Scope Test
;; Benchmark: 416ns
;; Benchmark-Repeat: 4848
;; Output: 15
;; Benchmark: 116ns
;; Benchmark-Repeat: 18852
(do
(def make-adder (fn [x]
(fn [y] (+ x y))))
(def add5 (make-adder 5))
(add5 10)
(assert-eq 15 (add5 10))
)
+3 -4
View File
@@ -1,4 +1,3 @@
;; Benchmark: 89ns
;; Benchmark-Repeat: 23053
;; Output: 5
(((fn [x] (fn [] x)) 5))
;; Benchmark: 110ns
;; Benchmark-Repeat: 18341
(assert-eq 5 (((fn [x] (fn [] x)) 5)))
+3 -4
View File
@@ -1,7 +1,6 @@
;; Benchmark: 89ns
;; Benchmark-Repeat: 23311
;; Output: 5
;; Benchmark: 112ns
;; Benchmark-Repeat: 18321
(do
(def f (fn [x] (fn [] x)))
((f 5))
(assert-eq 5 ((f 5)))
)
+3 -4
View File
@@ -1,7 +1,6 @@
;; Benchmark: 87ns
;; Benchmark-Repeat: 23196
;; Output: 42
;; Benchmark: 111ns
;; Benchmark-Repeat: 18551
(do
(def f (fn [a] (fn [b] (fn [c] (+ a (+ b c))))))
(((f 10) 12) 20)
(assert-eq 42 (((f 10) 12) 20))
)
+3 -4
View File
@@ -1,7 +1,6 @@
;; Complex Data Structure Test
;; Benchmark: 32.9us
;; Benchmark-Repeat: 61
;; Output: {:name "Fibonacci", :input 10, :output 55}
;; Benchmark: 36.6us
;; Benchmark-Repeat: 55
(do
(def fib (fn [n]
(if (< n 2)
@@ -16,5 +15,5 @@
:output result
})
data
(assert-eq {:name "Fibonacci" :input 10 :output 55} data)
)
+18
View File
@@ -0,0 +1,18 @@
;; Skip: data output to stdout
(do
(def EURUSD (create-m1-stream "EURUSD" (date "2020-01-03 09:30:00") (date "2020-01-03 10:00:00")))
(def GER40 (create-m1-stream "GER40" (date "2020-01-03 09:30:00") (date "2020-01-03 10:00:00")))
(def cnt 1)
(pipe-series 2 [EURUSD GER40]
(fn [eu ge]
(do
(def dax (.close (ge 0)))
(def eur (- dax (.close (ge 1))))
(print cnt ": dax=" eur " (" (/ eur (.close (eu 0))) "$)" )
(assign cnt (+ cnt 1))
)
)
)
)
+22
View File
@@ -0,0 +1,22 @@
;; Skip: data output to stdout
(do
(def EURUSD (create-m1-stream "EURUSD" (date "2020-01-03 09:30:00") (date "2020-01-03 13:30:00")))
(def GER40 (create-m1-stream "GER40" (date "2020-01-03 09:30:00") (date "2020-01-03 13:30:00")))
;; Sink: print returns void, so nothing is propagated downstream
(def cnt 1)
(def gs (series 2))
(pipe [EURUSD GER40]
(fn [eu ge]
(do
(def dax (.close ge))
(push gs dax)
(def eur (- dax (gs 1)))
(print cnt ": dax=" eur " (" (/ eur (.close eu)) "$)" )
(assign cnt (+ cnt 1))
)
)
)
)
+2 -2
View File
@@ -1,5 +1,5 @@
;; Benchmark: 300ns
;; Benchmark-Repeat: 6698
;; Benchmark: 211ns
;; Benchmark-Repeat: 9558
;; examples/def_local_inlining.myc
;; Demonstrates potential for DefLocal-Inlining (Phase 2.5)
+5 -12
View File
@@ -1,5 +1,5 @@
;; Benchmark: 773ns
;; Benchmark-Repeat: 2598
;; Benchmark: 776ns
;; Benchmark-Repeat: 2744
;; Comprehensive Destructuring Test
;; Covers: Nested tuples, mixed params, dynamic passing
@@ -15,13 +15,6 @@
(def data [10 [20 30]])
;; Validation
(if (= (deep [1 [[2 3] 4]]) 10)
(if (= (mixed [1 2] 3) 6)
(if (= (call-dynamic (fn [[a [b c]]] (+ a (+ b c))) data) 60)
"PASS"
"FAIL-DYNAMIC")
"FAIL-MIXED")
"FAIL-DEEP"))
;; Output: "PASS"
(assert-eq 10 (deep [1 [[2 3] 4]]))
(assert-eq 6 (mixed [1 2] 3))
(assert-eq 60 (call-dynamic (fn [[a [b c]]] (+ a (+ b c))) data)))
+3 -4
View File
@@ -1,6 +1,5 @@
;; Benchmark: 85ns
;; Benchmark-Repeat: 23630
;; Output: 36
;; Benchmark: 110ns
;; Benchmark-Repeat: 18390
(do
;; Excessive capture test
;; This script creates deep nesting where the inner-most lambda
@@ -23,5 +22,5 @@
(+ v1 v2 v3 v4 v5 v6 v7 v8))))))))))
;; Execution: (1 + 2 + 3 + 4 + 5 + 6 + 7 + 8) = 36
((((excessive 1) 3) 5) 7)
(assert-eq 36 ((((excessive 1) 3) 5) 7))
)
+3 -4
View File
@@ -1,12 +1,11 @@
;; Fibonacci Recursive
;; Benchmark: 32.8us
;; Benchmark-Repeat: 62
;; Output: 55
;; Benchmark: 37.1us
;; Benchmark-Repeat: 55
(do
(def fib (fn [n]
(if (< n 2)
n
(+ (fib (- n 1)) (fib (- n 2))))))
(fib 10)
(assert-eq 55 (fib 10))
)
+3 -4
View File
@@ -1,4 +1,3 @@
;; Benchmark: 87ns
;; Benchmark-Repeat: 23361
;; Output: 30
(+ 10 20)
;; Benchmark: 112ns
;; Benchmark-Repeat: 18385
(assert-eq 30 (+ 10 20))
+3 -4
View File
@@ -1,10 +1,9 @@
;; Higher-Order Function Example
;; Benchmark: 86ns
;; Benchmark-Repeat: 23023
;; Output: 25
;; Benchmark: 112ns
;; Benchmark-Repeat: 17678
(do
(def apply (fn [f x] (f x)))
(def square (fn [x] (* x x)))
(apply square 5)
(assert-eq 25 (apply square 5))
)
+3 -4
View File
@@ -1,9 +1,8 @@
;; Benchmark: 88ns
;; Benchmark-Repeat: 22522
;; Benchmark: 137ns
;; Benchmark-Repeat: 14903
;; Financial DSL Macro example
;; Demonstrates generating complex records from simple parameters.
;; Output: {:price 100, :size 2, :total 200}
(do
(macro trade [p s] `{:price ~p :size ~s :total (* ~p ~s)})
(trade 100 2))
(assert-eq {:price 100 :size 2 :total 200} (trade 100 2)))
+3 -4
View File
@@ -1,9 +1,8 @@
;; Benchmark: 246ns
;; Benchmark-Repeat: 8261
;; Output: 5
;; Benchmark: 268ns
;; Benchmark-Repeat: 7910
(do
(def y 1)
(macro m1 [x] `(do (def y 10) (assign y 4)))
(m1 8)
(+ y (m1 8))
(assert-eq 5 (+ y (m1 8)))
)
+3 -2
View File
@@ -1,5 +1,6 @@
;; Output: 42
;; Benchmark: 107ns
;; Benchmark-Repeat: 18414
(do
(macro t [x] `(fn [~x] ~x))
(def id (t a))
(id 42))
(assert-eq 42 (id 42)))
+3 -4
View File
@@ -1,9 +1,8 @@
;; Benchmark: 194ns
;; Benchmark-Repeat: 10337
;; Benchmark: 159ns
;; Benchmark-Repeat: 12580
;; Nested Macro and Arithmetic example
;; Output: 81
(do
(macro square [x] `(* ~x ~x))
(macro power4 [x] `(square (square ~x)))
(power4 3))
(assert-eq 81 (power4 3)))
+3 -4
View File
@@ -1,9 +1,8 @@
;; Benchmark: 195ns
;; Benchmark-Repeat: 10279
;; Benchmark: 252ns
;; Benchmark-Repeat: 7955
;; Macro Splicing example
;; Demonstrates unrolling a list into another list.
;; Output: [0 1 2 3 4]
(do
(macro wrap [items] `[0 ~@items 4])
(wrap [1 2 3]))
(assert-eq [0 1 2 3 4] (wrap [1 2 3])))
+3 -4
View File
@@ -1,9 +1,8 @@
;; Benchmark: 87ns
;; Benchmark-Repeat: 23129
;; Benchmark: 108ns
;; Benchmark-Repeat: 18646
;; Classic "unless" macro example
;; Demonstrates simple template substitution.
;; Output: 42
(do
(macro unless [c b] `(if ~c ... ~b))
(unless false 42))
(assert-eq 42 (unless false 42)))
+12 -12
View File
@@ -1,6 +1,5 @@
;; Benchmark: 1.4us
;; Benchmark-Repeat: 1473
;; Output: [150 130 "Insufficient funds" 130]
;; Benchmark: 1.3us
;; Benchmark-Repeat: 1528
; ---------------------------------------------------------
; Object-Oriented Records Example
@@ -34,19 +33,20 @@
; 1. Create an instance
(def my-acc (make-account 100))
; 2. Call 'deposit' method
; Note the double parens: (.deposit my-acc) gets the function, then we call it
(def b1 ((.deposit my-acc) 50)) ; balance is now 150
; 2. deposit 50 -> balance is now 150
(def b1 ((.deposit my-acc) 50))
; 3. Call 'withdraw' method
(def b2 ((.withdraw my-acc) 20)) ; balance is now 130
; 3. withdraw 20 -> balance is now 130
(def b2 ((.withdraw my-acc) 20))
; 4. Call 'withdraw' with invalid amount
; 4. withdraw 1000 -> insufficient funds
(def error-msg ((.withdraw my-acc) 1000))
; 5. Call 'balance' getter
; 5. balance getter
(def final-balance ((.balance my-acc)))
; Return summary of operations
[b1 b2 error-msg final-balance]
(assert-eq 150 b1)
(assert-eq 130 b2)
(assert-eq "Insufficient funds" error-msg)
(assert-eq 130 final-balance)
)
+3 -4
View File
@@ -1,6 +1,5 @@
;; Benchmark: 86ns
;; Benchmark-Repeat: 23328
;; Output: 13
;; Benchmark: 110ns
;; Benchmark-Repeat: 19224
(do
(macro wrap [f] `(fn [x] (~f x)))
@@ -10,4 +9,4 @@
(def w1 (wrap add1))
(def w2 (wrap add2))
(w1 (w2 10)))
(assert-eq 13 (w1 (w2 10))))
+3 -4
View File
@@ -1,7 +1,6 @@
;; Benchmark: 87ns
;; Benchmark-Repeat: 23072
;; Output: 31.41592653589793
;; Benchmark: 111ns
;; Benchmark-Repeat: 18959
(do
(def area (fn [x] (* PI x)))
(area 10)
(assert-eq 31.41592653589793 (area 10))
)
+7 -6
View File
@@ -1,13 +1,14 @@
;; Benchmark: 167ns
;; Benchmark-Repeat: 12120
;; Output: [42 150]
;; Benchmark: 196ns
;; Benchmark-Repeat: 10532
(do
;; 1. Provokation Stack-Gap (bei Optimization Level 2)
(def result
(fn []
(do
(def unused 10) ;; Wird entfernt, Slot 0 ist dann "leer"
(def used 42) ;; Bleibt auf Slot 1 -> Lücke!
;; Wird entfernt, Slot 0 ist dann "leer"
(def unused 10)
;; Bleibt auf Slot 1 -> Lücke!
(def used 42)
used)))
;; 2. Provokation Inlining-Blockade
@@ -18,5 +19,5 @@
(def b 100)
(+ a b))))
[(result) (complex-add 50)]
(assert-eq [42 150] [(result) (complex-add 50)])
)
+3 -4
View File
@@ -1,6 +1,5 @@
;; Benchmark: 2.1us
;; Benchmark-Repeat: 921
;; Output: <StreamNode>
;; Benchmark: 1.8us
;; Benchmark-Repeat: 1150
(do
(def src1 (create-random-ohlc 42 3))
@@ -21,5 +20,5 @@
)
)
my_indicator
(assert-eq :stream (type-of my_indicator))
)
+5 -5
View File
@@ -1,6 +1,5 @@
;; Benchmark: 2.3us
;; Benchmark-Repeat: 870
;; Output: <StreamNode>
;; Benchmark: 2.7us
;; Benchmark-Repeat: 749
(do
;; Set the random seed to match the existing test output exactly
@@ -17,7 +16,8 @@
(pipe [ticker]
(fn [_]
(do
(def change (* (- (random) 0.5) 2.0)) ; Matches Rust: (rng.f64() - 0.5) * 2.0
; Matches Rust: (rng.f64() - 0.5) * 2.0
(def change (* (- (random) 0.5) 2.0))
(def open last-close)
(def high (+ open (abs (* (random) 2.0))))
(def low (- open (abs (* (random) 2.0))))
@@ -44,5 +44,5 @@
)
)
my_indicator
(assert-eq :stream (type-of my_indicator))
)
+11
View File
@@ -0,0 +1,11 @@
(do
(def last (fn [s] (s 0)))
(def f (fn [n] n))
(def r (series 5))
(push r 4)
[(last f) (last r)]
)
+3 -4
View File
@@ -1,12 +1,11 @@
;; Benchmark: 1.0ms
;; Benchmark-Repeat: 3
;; Benchmark: 868.2us
;; Benchmark-Repeat: 4
;; Tests the effect of record inlining and field lookup optimization
;; Output: 10000
(do
(def config {:start 0 :limit 10000 :step 2})
(def x (.start config))
(while (< x (.limit config))
(assign x (+ x (.step config))))
x
(assert-eq 10000 x)
)
+2 -4
View File
@@ -1,11 +1,9 @@
;; Benchmark: 1.1us
;; Benchmark-Repeat: 1895
;; Benchmark-Repeat: 1783
;; Test Record SoA specialization in Pipelines
;; Dank der neuen Spezialisierung wird hierfür im Hintergrund
;; eine SharedRecordSeries mit SoA-Layout (Float-Puffer für mid und range) erstellt.
;; Output: <StreamNode>
(do
(def ohlc_stream (create-random-ohlc 42 10))
@@ -21,5 +19,5 @@
)
)
indicator
(assert-eq :stream (type-of indicator))
)
+8 -5
View File
@@ -1,6 +1,5 @@
;; Benchmark: 1.2us
;; Benchmark-Repeat: 1634
;; Output: ["Alice" 101 :admin "Zürich" ["Alice" "Bob"] true]
;; Benchmark: 1.3us
;; Benchmark-Repeat: 1597
; ---------------------------------------------------------
; Records & Optimized Field Access Showcase
@@ -44,6 +43,10 @@
; Records with the same layout share memory; equality checks values.
(def is-equal (= {:x 1 :y 2} {:x 1 :y 2}))
; Return a summary of all tested features
[name id role city names is-equal]
(assert-eq "Alice" name)
(assert-eq 101 id)
(assert-eq :admin role)
(assert-eq "Zürich" city)
(assert-eq ["Alice" "Bob"] names)
(assert is-equal)
)
+2
View File
@@ -1,3 +1,5 @@
;; Benchmark: 18.8us
;; Benchmark-Repeat: 113
(do
(repeat n 10 (print n)
)
+4 -3
View File
@@ -1,3 +1,4 @@
;; Skip: work in progress — requires external RTL module (SMA not yet implemented)
#use rtl
(do
@@ -10,11 +11,11 @@
(pipe [(pipe [(.close src)] (SMA 5))] (printx "sma5="))
(pipe [(pipe [(.close src)] (SMA 100))] (printx "sma100="))
(macro cache [lookback type src] `(do (def data (series ~lookback ~type)) (pipe [~src] (fn [s] (do (push data s)))) data))
(macro cache [lookback src] `(do (def data (series ~lookback)) (pipe [~src] (fn [s] (do (push data s)))) data))
; (def bars (series 20 :float))
; (def bars (series 20))
; (pipe [src] (fn [s] (do (push bars (.close s)))))
(cache 20 :float src)
(cache 20 src)
)
+5 -7
View File
@@ -1,16 +1,14 @@
;; Output: 228
;; Benchmark: 1.4us
;; Benchmark-Repeat: 1406
;; Benchmark: 1.1us
;; Benchmark-Repeat: 1877
(do
(def my_ticks (series 100 {:price :float :volume :int :msg :text}))
(def my_ticks (series 100))
(push my_ticks {:price 10.5 :volume 100 :msg "A"})
(push my_ticks {:price 11.2 :volume 200 :msg "B"})
(push my_ticks {:price 15.5 :volume 300 :msg "C"})
(repeat n 100 (push my_ticks {:price (+ n 15.5) :volume (ceil (/ n 300)) :msg "??"})
)
(def prices (.price my_ticks))
;; prices 0 = 15.5 (newest), prices 1 = 11.2
(assert-eq 26.7 (+ (prices 0) (prices 1)))
)
+1
View File
@@ -1,3 +1,4 @@
;; Skip: known macro hygiene issue — 'stream' is renamed inside template expansion
(do
(def stream (create-random-ohlc 42 200))
+3 -4
View File
@@ -1,9 +1,8 @@
;; Takeuchi Function Benchmark
;; heavily recursive, benefits significantly from specialization of integer arithmetic
;; and direct function calls (skipping dynamic dispatch).
;; Output: 5
;; Benchmark: 312.6us
;; Benchmark: 339.7us
;; Benchmark-Repeat: 7
(do
(def tak (fn [x y z]
@@ -13,5 +12,5 @@
(tak (- y 1) z x)
(tak (- z 1) x y)))))
(assert-eq 5 (tak 12 8 4))
)
+3 -3
View File
@@ -1,10 +1,10 @@
;; Benchmark: 2.0ms
;; Output: "done"
;; Benchmark: 1.8ms
;; Benchmark-Repeat: 3
(do
(def count_down (fn [n]
(if (< n 1)
"done"
(count_down (- n 1)))))
(count_down 10000)
(assert-eq "done" (count_down 10000))
)
+4 -4
View File
@@ -1,6 +1,5 @@
;; Benchmark: 855ns
;; Benchmark-Repeat: 2366
;; Output: ["Symbol:" "btc" "field:" :close "id:" "cls"]
;; Benchmark: 664ns
;; Benchmark-Repeat: 3026
(do
(def p (fn [conf]
(do
@@ -11,5 +10,6 @@
)
)
(p ["btc" [:close "cls"]])
(assert-eq ["Symbol:" "btc" "field:" :close "id:" "cls"]
(p ["btc" [:close "cls"]]))
)
+4 -5
View File
@@ -1,12 +1,10 @@
;; Benchmark: 1.7us
;; Benchmark-Repeat: 1175
;; Benchmark: 2.4us
;; Benchmark-Repeat: 785
;; Demonstration of N-dimensional Tuples, Vectors, and Matrices
;; Based on docs/Tupel 1.md
;;
;; This test verifies that the literals are correctly parsed and represented.
;; Type inference is verified via internal compiler unit tests.
;;
;; Output: [[1 3.14 "text"] [10 20 30 40] [[1 2] [3 4]] [[[1 2] [3 4]] [[5 6] [7 8]]] [[1 2] [3 4 5]] {:x 1, :y 0.3}]
(do
(def tpl [1, 3.14, "text"])
@@ -16,5 +14,6 @@
(def mixed [[1 2] [3 4 5]])
(def rec {:x 1, :y 0.3})
[tpl v mat2d mat3d mixed rec]
(assert-eq [[1 3.14 "text"] [10 20 30 40] [[1 2] [3 4]] [[[1 2] [3 4]] [[5 6] [7 8]]] [[1 2] [3 4 5]] {:x 1 :y 0.3}]
[tpl v mat2d mat3d mixed rec])
)
-20
View File
@@ -1,20 +0,0 @@
(do
(def SMA
(fn [length]
(do
(def history (series length :float))
(def sum 0.0)
(fn [val]
(do
(def hist_len (len history))
(if (>= hist_len length)
(assign sum (- sum (history (- length 1)))))
(push history val)
(assign sum (+ sum val))
(if (>= hist_len length)
(/ sum length))
)))))
)
+46
View File
@@ -0,0 +1,46 @@
use crate::ast::nodes::{AnalyzedNode, ExecNode};
use crate::ast::types::Value;
use std::cell::RefCell;
use std::rc::Rc;
/// A compiled closure: a function body together with its captured upvalues.
///
/// Closures are the primary callable value in Myc Script. They are created by
/// `fn`-expressions and stored as `Value::Closure`.
#[derive(Debug, Clone)]
pub struct Closure {
/// The executable parameter pattern.
pub parameter_node: Rc<ExecNode>,
/// The analyzed body (before TCO transformation).
pub function_node: Rc<AnalyzedNode>,
/// The executable node (after TCO transformation).
pub exec_node: Rc<ExecNode>,
/// Captured variables from enclosing scopes.
pub upvalues: Vec<Rc<RefCell<Value>>>,
/// Number of positional parameters, if known statically.
pub positional_count: Option<u32>,
/// Number of stack slots required for this closure (set during compilation).
pub stack_size: u32,
}
impl Closure {
#[inline]
pub fn new(
params: Rc<ExecNode>,
body: Rc<AnalyzedNode>,
exec: Rc<ExecNode>,
upvalues: Vec<Rc<RefCell<Value>>>,
positional_count: Option<u32>,
stack_size: u32,
) -> Self {
Self {
parameter_node: params,
function_node: body,
exec_node: exec,
upvalues,
positional_count,
stack_size,
}
}
}
+152 -168
View File
@@ -1,27 +1,28 @@
use crate::ast::compiler::bound_nodes::{
Address, AnalyzedNode, BoundKind, GlobalIdx, NodeMetrics, TypedNode,
use crate::ast::nodes::{
Address, AnalyzedNode, GlobalIdx, IdentifierBinding, Node,
NodeKind, NodeMetrics, TypedNode,
};
use crate::ast::types::Purity;
use crate::ast::types::{Identity, Purity, Value};
use std::collections::{HashMap, HashSet};
use std::rc::Rc;
pub struct Analyzer<'a> {
global_purity: &'a HashMap<GlobalIdx, Purity>,
root_purity: &'a [Purity],
/// Stack of currently visiting lambdas to detect direct recursion.
lambda_stack: Vec<crate::ast::types::Identity>,
lambda_stack: Vec<Identity>,
/// Map of global index to its Lambda identity if known.
globals_to_lambdas: HashMap<GlobalIdx, crate::ast::types::Identity>,
globals_to_lambdas: HashMap<GlobalIdx, Identity>,
/// Set of identities that were found to be recursive.
recursive_identities: HashSet<crate::ast::types::Identity>,
recursive_identities: HashSet<Identity>,
}
impl<'a> Analyzer<'a> {
pub fn analyze(
node: &TypedNode,
global_purity: &'a HashMap<GlobalIdx, Purity>,
root_purity: &'a [Purity],
) -> AnalyzedNode {
let mut analyzer = Self {
global_purity,
root_purity,
lambda_stack: Vec::new(),
globals_to_lambdas: HashMap::new(),
recursive_identities: HashSet::new(),
@@ -36,27 +37,78 @@ impl<'a> Analyzer<'a> {
fn collect_globals(&mut self, node: &TypedNode) {
match &node.kind {
BoundKind::Define {
addr: Address::Global(global_index),
NodeKind::Def {
pattern,
value,
..
} => {
if let BoundKind::Lambda { .. } = &value.kind {
if let NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr: Address::Global(global_index), .. },
..
} = &pattern.kind
&& let NodeKind::Lambda { .. } = &value.kind
{
self.globals_to_lambdas
.insert(*global_index, value.identity.clone());
}
self.collect_globals(value);
}
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } | NodeKind::Program { exprs } => {
for e in exprs {
self.collect_globals(e);
}
}
_ => {
node.kind
.for_each_child(|child| self.collect_globals(child));
NodeKind::If { cond, then_br, else_br } => {
self.collect_globals(cond);
self.collect_globals(then_br);
if let Some(e) = else_br {
self.collect_globals(e);
}
}
NodeKind::Lambda { params, body, .. } => {
self.collect_globals(params);
self.collect_globals(body);
}
NodeKind::Call { callee, args } => {
self.collect_globals(callee);
self.collect_globals(args);
}
NodeKind::Again { args } => {
self.collect_globals(args);
}
NodeKind::Assign { target, value, .. } => {
self.collect_globals(target);
self.collect_globals(value);
}
NodeKind::Tuple { elements } => {
for e in elements {
self.collect_globals(e);
}
}
NodeKind::Record { fields, .. } => {
for (k, v) in fields {
self.collect_globals(k);
self.collect_globals(v);
}
}
NodeKind::GetField { rec, .. } => {
self.collect_globals(rec);
}
NodeKind::Expansion { expanded, .. } => {
self.collect_globals(expanded);
}
// Leaf nodes — no children.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::Identifier { .. }
| NodeKind::FieldAccessor(_)
| NodeKind::Error
| NodeKind::Extension(_)
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
fn visit(&mut self, node_rc: Rc<TypedNode>) -> AnalyzedNode {
@@ -64,32 +116,36 @@ impl<'a> Analyzer<'a> {
let mut is_recursive = false;
let (new_kind, purity) = match &node.kind {
BoundKind::Constant(v) => (BoundKind::Constant(v.clone()), Purity::Pure),
BoundKind::Nop => (BoundKind::Nop, Purity::Pure),
BoundKind::Get { addr, name } => {
let p = match addr {
Address::Global(idx) => {
self.global_purity.get(idx).cloned().unwrap_or(Purity::Pure)
// Propagate the declared purity of function constants so the constant
// folder does not evaluate impure factory closures at compile time.
NodeKind::Constant(v) => {
let purity = if let Value::Function(f) = v { f.purity } else { Purity::Pure };
(NodeKind::Constant(v.clone()), purity)
}
_ => Purity::Pure,
NodeKind::Nop => (NodeKind::Nop, Purity::Pure),
NodeKind::Identifier { symbol, binding } => {
let p = if let IdentifierBinding::Reference(Address::Global(idx)) = binding {
self.root_purity.get(idx.0 as usize).cloned().unwrap_or(Purity::Pure)
} else {
Purity::Pure
};
(
BoundKind::Get {
addr: *addr,
name: name.clone(),
NodeKind::Identifier {
symbol: symbol.clone(),
binding: binding.clone(),
},
p,
)
}
BoundKind::FieldAccessor(k) => (BoundKind::FieldAccessor(*k), Purity::Pure),
NodeKind::FieldAccessor(k) => (NodeKind::FieldAccessor(*k), Purity::Pure),
BoundKind::GetField { rec, field } => {
NodeKind::GetField { rec, field } => {
let rec_m = self.visit(rec.clone());
let p = rec_m.ty.purity;
(
BoundKind::GetField {
NodeKind::GetField {
rec: Rc::new(rec_m),
field: *field,
},
@@ -97,39 +153,37 @@ impl<'a> Analyzer<'a> {
)
}
BoundKind::Set { addr, value } => {
NodeKind::Assign { target, value, info } => {
let target_m = self.visit(target.clone());
let val_m = self.visit(value.clone());
(
BoundKind::Set {
addr: *addr,
NodeKind::Assign {
target: Rc::new(target_m),
value: Rc::new(val_m),
info: info.clone(),
},
Purity::Impure,
)
}
BoundKind::Define {
name,
addr,
kind,
NodeKind::Def {
pattern,
value,
captured_by,
info,
} => {
let pat_m = self.visit(pattern.clone());
let val_m = self.visit(value.clone());
let p = val_m.ty.purity;
(
BoundKind::Define {
name: name.clone(),
addr: *addr,
kind: *kind,
NodeKind::Def {
pattern: Rc::new(pat_m),
value: Rc::new(val_m),
captured_by: captured_by.clone(),
info: info.clone(),
},
p,
Purity::Impure,
)
}
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -143,7 +197,7 @@ impl<'a> Analyzer<'a> {
p = p.min(em.ty.purity);
}
(
BoundKind::If {
NodeKind::If {
cond: Rc::new(cond_m),
then_br: Rc::new(then_m),
else_br: else_m.map(Rc::new),
@@ -152,11 +206,10 @@ impl<'a> Analyzer<'a> {
)
}
BoundKind::Lambda {
NodeKind::Lambda {
params,
upvalues,
body,
positional_count,
info,
} => {
self.lambda_stack.push(node.identity.clone());
let params_m = self.visit(params.clone());
@@ -165,34 +218,21 @@ impl<'a> Analyzer<'a> {
is_recursive = self.recursive_identities.contains(&node.identity);
(
BoundKind::Lambda {
NodeKind::Lambda {
params: Rc::new(params_m),
upvalues: upvalues.clone(),
body: Rc::new(body_m),
positional_count: *positional_count,
info: info.clone(),
},
Purity::Pure,
)
}
BoundKind::Destructure { pattern, value } => {
let pat_m = self.visit(pattern.clone());
let val_m = self.visit(value.clone());
(
BoundKind::Destructure {
pattern: Rc::new(pat_m),
value: Rc::new(val_m),
},
Purity::Impure,
)
}
BoundKind::Call { callee, args } => {
NodeKind::Call { callee, args } => {
let callee_m = self.visit(callee.clone());
let args_m = self.visit(args.clone());
if let BoundKind::Get {
addr: Address::Global(idx),
if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(idx)),
..
} = &callee.kind
&& let Some(lambda_id) = self.globals_to_lambdas.get(idx)
@@ -202,13 +242,13 @@ impl<'a> Analyzer<'a> {
is_recursive = true;
}
let p_func = if let BoundKind::Get {
addr: Address::Global(idx),
let p_func = if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(idx)),
..
} = &callee.kind
{
self.global_purity
.get(idx)
self.root_purity
.get(idx.0 as usize)
.cloned()
.unwrap_or(Purity::Impure)
} else {
@@ -216,7 +256,7 @@ impl<'a> Analyzer<'a> {
};
let p = callee_m.ty.purity.min(args_m.ty.purity).min(p_func);
(
BoundKind::Call {
NodeKind::Call {
callee: Rc::new(callee_m),
args: Rc::new(args_m),
},
@@ -224,40 +264,20 @@ impl<'a> Analyzer<'a> {
)
}
BoundKind::Again { args } => {
NodeKind::Again { args } => {
let args_m = self.visit(args.clone());
if let Some(lambda_id) = self.lambda_stack.last() {
self.recursive_identities.insert(lambda_id.clone());
is_recursive = true;
}
(
BoundKind::Again {
NodeKind::Again {
args: Rc::new(args_m),
},
Purity::Impure,
)
}
BoundKind::Pipe {
inputs,
lambda,
out_type,
} => {
let mut analyzed_inputs = Vec::with_capacity(inputs.len());
for input in inputs {
analyzed_inputs.push(Rc::new(self.visit(input.clone())));
}
let a_lambda = Rc::new(self.visit(lambda.clone()));
(
BoundKind::Pipe {
inputs: analyzed_inputs,
lambda: a_lambda,
out_type: out_type.clone(),
},
Purity::Impure,
)
}
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } => {
let mut new_exprs = Vec::with_capacity(exprs.len());
let mut p = Purity::Pure;
for e in exprs {
@@ -265,10 +285,20 @@ impl<'a> Analyzer<'a> {
p = p.min(em.ty.purity);
new_exprs.push(Rc::new(em));
}
(BoundKind::Block { exprs: new_exprs }, p)
(NodeKind::Block { exprs: new_exprs }, p)
}
NodeKind::Program { exprs } => {
let mut new_exprs = Vec::with_capacity(exprs.len());
let mut p = Purity::Pure;
for e in exprs {
let em = self.visit(e.clone());
p = p.min(em.ty.purity);
new_exprs.push(Rc::new(em));
}
(NodeKind::Program { exprs: new_exprs }, p)
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
let mut new_elements = Vec::with_capacity(elements.len());
let mut p = Purity::Pure;
for e in elements {
@@ -277,51 +307,59 @@ impl<'a> Analyzer<'a> {
new_elements.push(Rc::new(em));
}
(
BoundKind::Tuple {
NodeKind::Tuple {
elements: new_elements,
},
p,
)
}
BoundKind::Record { layout, values } => {
let mut new_values = Vec::with_capacity(values.len());
NodeKind::Record { fields, layout } => {
let mut new_fields = Vec::with_capacity(fields.len());
let mut p = Purity::Pure;
for v in values {
let vm = self.visit(v.clone());
for (key_node, val_node) in fields {
let km = self.visit(key_node.clone());
let vm = self.visit(val_node.clone());
p = p.min(vm.ty.purity);
new_values.push(Rc::new(vm));
new_fields.push((Rc::new(km), Rc::new(vm)));
}
(
BoundKind::Record {
NodeKind::Record {
fields: new_fields,
layout: layout.clone(),
values: new_values,
},
p,
)
}
BoundKind::Expansion {
NodeKind::Expansion {
original_call,
bound_expanded,
expanded,
} => {
let expanded_m = self.visit(bound_expanded.clone());
let expanded_m = self.visit(expanded.clone());
(
BoundKind::Expansion {
NodeKind::Expansion {
original_call: original_call.clone(),
bound_expanded: Rc::new(expanded_m.clone()),
expanded: Rc::new(expanded_m.clone()),
},
expanded_m.ty.purity,
)
}
BoundKind::Extension(_) => (BoundKind::Nop, Purity::Impure),
BoundKind::Error => (BoundKind::Error, Purity::Impure),
NodeKind::Extension(_) => (NodeKind::Nop, Purity::Impure),
NodeKind::Error => (NodeKind::Error, Purity::Impure),
// Syntax-only variants should not appear in typed phases
NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => (NodeKind::Error, Purity::Impure),
};
crate::ast::nodes::Node {
Node {
identity: node.identity.clone(),
kind: new_kind,
comments: node.comments.clone(),
ty: NodeMetrics {
original: node_rc,
purity,
@@ -331,57 +369,3 @@ impl<'a> Analyzer<'a> {
}
}
trait NodeExt {
fn for_each_child<F: FnMut(&TypedNode)>(&self, f: F);
}
impl NodeExt for BoundKind<crate::ast::types::StaticType> {
fn for_each_child<F: FnMut(&TypedNode)>(&self, mut f: F) {
match self {
BoundKind::If {
cond,
then_br,
else_br,
} => {
f(cond);
f(then_br);
if let Some(e) = else_br {
f(e);
}
}
BoundKind::Define { value, .. } | BoundKind::Set { value, .. } => {
f(value);
}
BoundKind::GetField { rec, .. } => {
f(rec);
}
BoundKind::Lambda { params, body, .. } => {
f(params);
f(body);
}
BoundKind::Call { callee, args } => {
f(callee);
f(args);
}
BoundKind::Block { exprs } => {
for e in exprs {
f(e);
}
}
BoundKind::Tuple { elements } => {
for e in elements {
f(e);
}
}
BoundKind::Record { values, .. } => {
for v in values {
f(v);
}
}
BoundKind::Expansion { bound_expanded, .. } => {
f(bound_expanded);
}
_ => {}
}
}
}
+433 -300
View File
File diff suppressed because it is too large Load Diff
-347
View File
@@ -1,347 +0,0 @@
use crate::ast::nodes::{Node, Symbol};
use crate::ast::types::{Identity, StaticType, Value};
use std::rc::Rc;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct LocalSlot(pub u32);
impl std::fmt::Display for LocalSlot {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "L{}", self.0)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct UpvalueIdx(pub u32);
impl std::fmt::Display for UpvalueIdx {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "U{}", self.0)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct GlobalIdx(pub u32);
impl std::fmt::Display for GlobalIdx {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "G{}", self.0)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Address {
Local(LocalSlot), // Stack-Slot index (relative to frame base)
Upvalue(UpvalueIdx), // Index in the closure's upvalue array
Global(GlobalIdx), // Index in the global environment vector
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum DeclarationKind {
Variable,
Parameter,
}
/// Trait for DSL-specific AST extensions (like Pipe, Series-Access, etc.)
pub trait BoundExtension<T>: std::fmt::Debug {
fn clone_box(&self) -> Box<dyn BoundExtension<T>>;
fn display_name(&self) -> String;
}
impl<T> Clone for Box<dyn BoundExtension<T>> {
fn clone(&self) -> Self {
self.clone_box()
}
}
/// A bound AST node, decorated with type or metric information T.
pub type BoundNode<T = ()> = Node<BoundKind<T>, T>;
/// Type alias for a node that has been fully type-checked.
pub type TypedNode = BoundNode<StaticType>;
/// Type alias for the global function registry.
pub type GlobalFunctionRegistry = std::collections::HashMap<GlobalIdx, Rc<BoundNode>>;
/// Metrics collected during the analysis phase.
#[derive(Debug, Clone, PartialEq)]
pub struct NodeMetrics {
pub original: Rc<TypedNode>,
pub purity: crate::ast::types::Purity,
pub is_recursive: bool,
}
/// Type alias for a node that has been analyzed.
pub type AnalyzedNode = BoundNode<NodeMetrics>;
/// Type alias for the global analyzed function registry.
pub type GlobalAnalyzedRegistry = std::collections::HashMap<GlobalIdx, Rc<AnalyzedNode>>;
#[derive(Debug, Clone)]
pub enum BoundKind<T = ()> {
Nop,
Constant(Value),
// Variable Access (Resolved)
Get {
addr: Address,
name: Symbol,
},
// Variable Update (Assignment)
Set {
addr: Address,
value: Rc<BoundNode<T>>,
},
// Variable Declaration (Unified for Local/Global/Parameter)
Define {
name: Symbol,
addr: Address,
kind: DeclarationKind,
value: Rc<BoundNode<T>>,
captured_by: Vec<Identity>,
},
/// A first-class field accessor (e.g. .name)
FieldAccessor(crate::ast::types::Keyword),
/// Specialized field access (O(1) via RecordLayout)
GetField {
rec: Rc<BoundNode<T>>,
field: crate::ast::types::Keyword,
},
If {
cond: Rc<BoundNode<T>>,
then_br: Rc<BoundNode<T>>,
else_br: Option<Rc<BoundNode<T>>>,
},
/// A destructuring operation (can be a definition or an assignment depending on the pattern)
Destructure {
pattern: Rc<BoundNode<T>>,
value: Rc<BoundNode<T>>,
},
Lambda {
params: Rc<BoundNode<T>>,
// The list of variables captured from enclosing scopes
upvalues: Vec<Address>,
body: Rc<BoundNode<T>>,
/// Static optimization: number of positional parameters if the pattern is flat.
positional_count: Option<u32>,
},
Call {
callee: Rc<BoundNode<T>>,
args: Rc<BoundNode<T>>,
},
Again {
args: Rc<BoundNode<T>>,
},
Pipe {
inputs: Vec<Rc<BoundNode<T>>>,
lambda: Rc<BoundNode<T>>,
out_type: crate::ast::types::StaticType,
},
Block {
exprs: Vec<Rc<BoundNode<T>>>,
},
Tuple {
elements: Vec<Rc<BoundNode<T>>>,
},
Record {
layout: std::sync::Arc<crate::ast::types::RecordLayout>,
values: Vec<Rc<BoundNode<T>>>,
},
/// An expanded macro call, preserving the original call for debugging and UI.
Expansion {
/// The original call from the untyped AST.
original_call: Rc<Node<crate::ast::nodes::UntypedKind>>,
/// The result of binding the expanded AST.
bound_expanded: Rc<BoundNode<T>>,
},
/// A diagnostic poison node, allowing compilation to continue after an error.
Error,
/// DSL-specific extension slot
Extension(Box<dyn BoundExtension<T>>),
}
impl<T> PartialEq for BoundKind<T>
where
T: PartialEq,
{
fn eq(&self, other: &Self) -> bool {
match (self, other) {
(BoundKind::Nop, BoundKind::Nop) => true,
(BoundKind::Constant(a), BoundKind::Constant(b)) => a == b,
(BoundKind::Get { addr: aa, name: na }, BoundKind::Get { addr: ab, name: nb }) => {
aa == ab && na == nb
}
(
BoundKind::Set {
addr: aa,
value: va,
},
BoundKind::Set {
addr: ab,
value: vb,
},
) => aa == ab && Rc::ptr_eq(va, vb),
(
BoundKind::Define {
name: na,
addr: aa,
kind: ka,
value: va,
captured_by: ca,
},
BoundKind::Define {
name: nb,
addr: ab,
kind: kb,
value: vb,
captured_by: cb,
},
) => na == nb && aa == ab && ka == kb && Rc::ptr_eq(va, vb) && ca == cb,
(BoundKind::FieldAccessor(a), BoundKind::FieldAccessor(b)) => a == b,
(
BoundKind::GetField { rec: ra, field: fa },
BoundKind::GetField { rec: rb, field: fb },
) => Rc::ptr_eq(ra, rb) && fa == fb,
(
BoundKind::If {
cond: ca,
then_br: ta,
else_br: ea,
},
BoundKind::If {
cond: cb,
then_br: tb,
else_br: eb,
},
) => Rc::ptr_eq(ca, cb) && Rc::ptr_eq(ta, tb) && match (ea, eb) {
(Some(a), Some(b)) => Rc::ptr_eq(a, b),
(None, None) => true,
_ => false,
},
(
BoundKind::Destructure {
pattern: pa,
value: va,
},
BoundKind::Destructure {
pattern: pb,
value: vb,
},
) => Rc::ptr_eq(pa, pb) && Rc::ptr_eq(va, vb),
(
BoundKind::Lambda {
params: pa,
upvalues: ua,
body: ba,
positional_count: pca,
},
BoundKind::Lambda {
params: pb,
upvalues: ub,
body: bb,
positional_count: pcb,
},
) => Rc::ptr_eq(pa, pb) && ua == ub && Rc::ptr_eq(ba, bb) && pca == pcb,
(
BoundKind::Call {
callee: ca,
args: aa,
},
BoundKind::Call {
callee: cb,
args: ab,
},
) => Rc::ptr_eq(ca, cb) && Rc::ptr_eq(aa, ab),
(BoundKind::Again { args: aa }, BoundKind::Again { args: ab }) => Rc::ptr_eq(aa, ab),
(BoundKind::Block { exprs: ea }, BoundKind::Block { exprs: eb }) => {
ea.len() == eb.len() && ea.iter().zip(eb.iter()).all(|(a, b)| Rc::ptr_eq(a, b))
}
(BoundKind::Tuple { elements: ea }, BoundKind::Tuple { elements: eb }) => {
ea.len() == eb.len() && ea.iter().zip(eb.iter()).all(|(a, b)| Rc::ptr_eq(a, b))
}
(
BoundKind::Record {
layout: la,
values: va,
},
BoundKind::Record {
layout: lb,
values: vb,
},
) => std::sync::Arc::ptr_eq(la, lb) && va.len() == vb.len() && va.iter().zip(vb.iter()).all(|(a, b)| Rc::ptr_eq(a, b)),
(
BoundKind::Expansion {
original_call: ca,
bound_expanded: ea,
},
BoundKind::Expansion {
original_call: cb,
bound_expanded: eb,
},
) => Rc::ptr_eq(ca, cb) && Rc::ptr_eq(ea, eb),
(BoundKind::Error, BoundKind::Error) => true,
(BoundKind::Extension(_), BoundKind::Extension(_)) => false,
_ => false,
}
}
}
/// A single field in a Record literal (Key-Value pair)
pub type RecordField<T> = (BoundNode<T>, BoundNode<T>);
impl<T> BoundKind<T> {
pub fn display_name(&self) -> String {
match self {
BoundKind::Nop => "NOP".to_string(),
BoundKind::Constant(v) => format!("CONST({})", v),
BoundKind::Get { addr, name } => format!("GET({}, {:?})", name.name, addr),
BoundKind::Set { addr, .. } => format!("SET({:?})", addr),
BoundKind::Define {
name, addr, kind, ..
} => {
let k_str = match kind {
DeclarationKind::Variable => "VAR",
DeclarationKind::Parameter => "PARAM",
};
format!("DEF_{}({}, {:?})", k_str, name.name, addr)
}
BoundKind::FieldAccessor(k) => format!("FIELD_ACCESSOR(.{})", k.name()),
BoundKind::GetField { field, .. } => format!("GET_FIELD(.{})", field.name()),
BoundKind::If { .. } => "IF".to_string(),
BoundKind::Destructure { .. } => "DESTRUCTURE".to_string(),
BoundKind::Lambda {
params, upvalues, ..
} => {
let p_str = match &params.kind {
BoundKind::Tuple { elements } => format!("p:{}", elements.len()),
_ => "p:1".to_string(),
};
format!("LAMBDA({}, Captures:{})", p_str, upvalues.len())
}
BoundKind::Call { .. } => "CALL".to_string(),
BoundKind::Again { .. } => "AGAIN".to_string(),
BoundKind::Pipe { .. } => "PIPE".to_string(),
BoundKind::Block { .. } => "BLOCK".to_string(),
BoundKind::Tuple { elements } => format!("TUPLE({})", elements.len()),
BoundKind::Record { values, .. } => format!("RECORD({})", values.len()),
BoundKind::Expansion { .. } => "EXPANSION".to_string(),
BoundKind::Extension(ext) => ext.display_name(),
BoundKind::Error => "ERROR".to_string(),
}
}
}
+72
View File
@@ -0,0 +1,72 @@
use std::collections::HashMap;
use std::rc::Rc;
use crate::ast::diagnostics::Diagnostics;
use crate::ast::nodes::{Address, NodeKind, TypedNode, TypedPhase, VirtualId};
use crate::ast::types::StaticType;
/// Gives a compiler hook access to the type-checker's core inference operations
/// and scope slot types without depending on the concrete `TypeChecker` type
/// (avoids a circular module dependency).
pub trait InferenceAccess {
/// Allocate a fresh unique `TypeVar`.
fn fresh_var(&self) -> StaticType;
/// Unify two types, recording the substitution. Emits a diagnostic on conflict.
fn unify(&self, a: StaticType, b: StaticType, diag: &mut Diagnostics);
/// Directly bind a `TypeVar` ID to a resolved type in the substitution map.
fn bind_typevar(&self, id: u32, ty: StaticType);
/// Apply the current substitution to a type, resolving all known `TypeVar`s.
fn apply_subst_ty(&self, ty: StaticType) -> StaticType;
/// Int/Float numeric-widening rule. Returns the promoted type, or `None` if
/// the combination is not a widening pair.
fn try_numeric_widen(&self, a: &StaticType, b: &StaticType) -> Option<StaticType>;
/// Record-field promotion rule. Returns the promoted record type when one record
/// is a strict widening of the other, or `None` otherwise.
fn try_record_promote(&self, a: &StaticType, b: &StaticType) -> Option<StaticType>;
/// Read the raw (unsubstituted) type stored in a scope slot.
/// Used by hooks that need to recover the original `TypeVar` ID after
/// the substitution has already resolved it to a concrete type.
fn get_slot_type(&self, addr: Address<VirtualId>) -> StaticType;
}
/// Extension point that lets RTL functions participate in the compiler's
/// type-inference and AST-finalization passes without hardcoding symbol
/// names anywhere in the compiler core.
///
/// Each RTL function that needs custom type-level behaviour (e.g. `series`,
/// `push`) implements this trait. Hooks are registered by global slot index
/// during RTL bootstrap and dispatched automatically by the type checker.
///
/// Both methods have no-op defaults so that implementors only override what
/// they need.
pub trait RtlCompilerHook {
/// Called after the call's return type is resolved during type inference.
/// May replace the return type and/or perform unification side-effects.
fn post_call(
&self,
_args: &TypedNode,
ret_ty: StaticType,
_ctx: &dyn InferenceAccess,
_diag: &mut Diagnostics,
) -> StaticType {
ret_ty
}
/// Called during AST finalization.
/// Returns `Some(new_kind)` to rewrite the node, or `None` to keep the original.
fn finalize(
&self,
_callee: Rc<TypedNode>,
_args: Rc<TypedNode>,
_node_ty: &StaticType,
_subst: &HashMap<u32, StaticType>,
) -> Option<NodeKind<TypedPhase>> {
None
}
}
+86 -113
View File
@@ -1,154 +1,127 @@
use crate::ast::compiler::bound_nodes::{BoundKind, BoundNode};
use crate::ast::nodes::{CompilerPhase, DefBinding, Node, NodeKind};
use crate::ast::types::Identity;
use std::collections::HashMap;
use std::rc::Rc;
pub struct CapturePass;
impl CapturePass {
pub fn apply(node: BoundNode, capture_map: &HashMap<Identity, Vec<Identity>>) -> BoundNode {
pub fn apply<P: CompilerPhase<DefInfo = DefBinding>>(node: Node<P>, capture_map: &HashMap<Identity, Vec<Identity>>) -> Node<P> {
Self::transform(node, capture_map)
}
fn transform(mut node: BoundNode, capture_map: &HashMap<Identity, Vec<Identity>>) -> BoundNode {
use std::rc::Rc;
match node.kind {
BoundKind::Define {
name,
addr,
kind,
fn transform<P: CompilerPhase<DefInfo = DefBinding>>(mut node: Node<P>, capture_map: &HashMap<Identity, Vec<Identity>>) -> Node<P> {
node.kind = match node.kind {
NodeKind::Def {
pattern,
value,
..
mut info,
} => {
let captured_by = capture_map.get(&node.identity).cloned().unwrap_or_default();
node.kind = BoundKind::Define {
name,
addr,
kind,
if let Some(capturers) = capture_map.get(&node.identity) {
info.captured_by.extend(capturers.iter().cloned());
}
NodeKind::Def {
pattern: Rc::new(Self::transform(pattern.as_ref().clone(), capture_map)),
value: Rc::new(Self::transform(value.as_ref().clone(), capture_map)),
captured_by,
};
info,
}
}
BoundKind::If {
NodeKind::Lambda {
params,
body,
info,
} => NodeKind::Lambda {
params: Rc::new(Self::transform(params.as_ref().clone(), capture_map)),
body: Rc::new(Self::transform(body.as_ref().clone(), capture_map)),
info,
},
NodeKind::Call { callee, args } => NodeKind::Call {
callee: Rc::new(Self::transform(callee.as_ref().clone(), capture_map)),
args: Rc::new(Self::transform(args.as_ref().clone(), capture_map)),
},
NodeKind::Again { args } => NodeKind::Again {
args: Rc::new(Self::transform(args.as_ref().clone(), capture_map)),
},
NodeKind::If {
cond,
then_br,
else_br,
} => {
node.kind = BoundKind::If {
} => NodeKind::If {
cond: Rc::new(Self::transform(cond.as_ref().clone(), capture_map)),
then_br: Rc::new(Self::transform(then_br.as_ref().clone(), capture_map)),
else_br: else_br.map(|e| Rc::new(Self::transform(e.as_ref().clone(), capture_map))),
};
}
else_br: else_br
.map(|e| Rc::new(Self::transform(e.as_ref().clone(), capture_map))),
},
BoundKind::Set { addr, value } => {
node.kind = BoundKind::Set {
addr,
NodeKind::Assign { target, value, info } => NodeKind::Assign {
target: Rc::new(Self::transform(target.as_ref().clone(), capture_map)),
value: Rc::new(Self::transform(value.as_ref().clone(), capture_map)),
};
}
info,
},
BoundKind::FieldAccessor(_) => {}
BoundKind::GetField { rec, field } => {
node.kind = BoundKind::GetField {
rec: Rc::new(Self::transform(rec.as_ref().clone(), capture_map)),
field,
};
}
BoundKind::Destructure { pattern, value } => {
node.kind = BoundKind::Destructure {
pattern: Rc::new(Self::transform(pattern.as_ref().clone(), capture_map)),
value: Rc::new(Self::transform(value.as_ref().clone(), capture_map)),
};
}
BoundKind::Lambda {
params,
upvalues,
body,
positional_count,
} => {
node.kind = BoundKind::Lambda {
params: Rc::new(Self::transform(params.as_ref().clone(), capture_map)),
upvalues,
body: Rc::new(Self::transform(body.as_ref().clone(), capture_map)),
positional_count,
};
}
BoundKind::Call { callee, args } => {
node.kind = BoundKind::Call {
callee: Rc::new(Self::transform(callee.as_ref().clone(), capture_map)),
args: Rc::new(Self::transform(args.as_ref().clone(), capture_map)),
};
}
BoundKind::Again { args } => {
node.kind = BoundKind::Again {
args: Rc::new(Self::transform(args.as_ref().clone(), capture_map)),
};
}
BoundKind::Pipe {
inputs,
lambda,
out_type,
} => {
let mut t_inputs = Vec::with_capacity(inputs.len());
for input in inputs {
t_inputs.push(Rc::new(Self::transform(input.as_ref().clone(), capture_map)));
}
node.kind = BoundKind::Pipe {
inputs: t_inputs,
lambda: Rc::new(Self::transform(lambda.as_ref().clone(), capture_map)),
out_type: out_type.clone(),
};
}
BoundKind::Block { exprs } => {
node.kind = BoundKind::Block {
NodeKind::Block { exprs } => NodeKind::Block {
exprs: exprs
.into_iter()
.map(|e| Rc::new(Self::transform(e.as_ref().clone(), capture_map)))
.collect(),
};
}
},
BoundKind::Tuple { elements } => {
node.kind = BoundKind::Tuple {
NodeKind::Program { exprs } => NodeKind::Program {
exprs: exprs
.into_iter()
.map(|e| Rc::new(Self::transform(e.as_ref().clone(), capture_map)))
.collect(),
},
NodeKind::Tuple { elements } => NodeKind::Tuple {
elements: elements
.into_iter()
.map(|e| Rc::new(Self::transform(e.as_ref().clone(), capture_map)))
.collect(),
};
}
},
BoundKind::Record { layout, values } => {
node.kind = BoundKind::Record {
layout,
values: values
NodeKind::Record { fields, layout } => NodeKind::Record {
fields: fields
.into_iter()
.map(|v| Rc::new(Self::transform(v.as_ref().clone(), capture_map)))
.map(|(k, v)| (k, Rc::new(Self::transform(v.as_ref().clone(), capture_map))))
.collect(),
};
}
layout,
},
BoundKind::Expansion {
NodeKind::Expansion {
original_call,
bound_expanded,
} => {
node.kind = BoundKind::Expansion {
expanded,
} => NodeKind::Expansion {
original_call,
bound_expanded: Rc::new(Self::transform(bound_expanded.as_ref().clone(), capture_map)),
};
}
expanded: Rc::new(Self::transform(
expanded.as_ref().clone(),
capture_map,
)),
},
BoundKind::Nop
| BoundKind::Constant(_)
| BoundKind::Get { .. }
| BoundKind::Extension(_)
| BoundKind::Error => {}
}
NodeKind::GetField { rec, field } => NodeKind::GetField {
rec: Rc::new(Self::transform(rec.as_ref().clone(), capture_map)),
field,
},
// Leaf nodes — no children to recurse into.
kind @ (NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::Identifier { .. }
| NodeKind::FieldAccessor(_)
| NodeKind::Error
| NodeKind::Extension(_)) => kind,
// Syntax-only variants should not appear after macro expansion.
kind @ (NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_)) => kind,
};
node
}
}
+142 -121
View File
@@ -1,21 +1,33 @@
use crate::ast::compiler::bound_nodes::BoundKind;
use crate::ast::nodes::Node;
use crate::ast::types::Value;
use crate::ast::vm::Closure;
use std::fmt::Debug;
use crate::ast::nodes::{CompactMetadata, CompilerPhase, Node, NodeKind};
/// Human-readable AST dumper for the bound AST.
pub struct Dumper {
output: String,
indent: usize,
/// When `true`, emits a compact format suitable for GUI display:
/// type annotations via `Display` and purity symbols instead of raw debug structs.
compact: bool,
}
impl Dumper {
/// Produces a formatted string representation of the given bound AST node and its children.
pub fn dump<T: Debug>(node: &Node<BoundKind<T>, T>) -> String {
/// Produces a verbose debug representation of the given AST node and its children.
pub fn dump<P: CompilerPhase>(node: &Node<P>) -> String {
let mut dumper = Self {
output: String::new(),
indent: 0,
compact: false,
};
dumper.visit(node);
dumper.output
}
/// Produces a compact, human-readable representation suitable for GUI display.
/// Shows inferred types and purity (`!` / `~`) instead of raw debug structs.
pub fn dump_compact<P: CompilerPhase>(node: &Node<P>) -> String {
let mut dumper = Self {
output: String::new(),
indent: 0,
compact: true,
};
dumper.visit(node);
dumper.output
@@ -27,104 +39,73 @@ impl Dumper {
}
}
fn log<T: Debug>(&mut self, label: &str, node: &Node<BoundKind<T>, T>) {
fn log<P: CompilerPhase>(&mut self, label: &str, node: &Node<P>) {
self.write_indent();
self.output.push_str(label);
self.output
.push_str(&format!(" <Metadata: {:?}>\n", node.ty));
if self.compact {
if let Some(annotation) = node.ty.compact_label() {
self.output.push_str("");
self.output.push_str(&annotation);
}
} else {
self.output
.push_str(&format!(" <Metadata: {:?}>", node.ty));
}
fn visit<T: Debug>(&mut self, node: &Node<BoundKind<T>, T>) {
match &node.kind {
BoundKind::Nop => self.log("Nop", node),
BoundKind::Constant(v) => {
self.log(&format!("Constant: {}", v), node);
// Introspect Closure AST if possible
if let Value::Object(obj) = v
&& let Some(closure) = obj.as_any().downcast_ref::<Closure>()
{
self.indent += 1;
self.write_indent();
self.output.push_str("--- Specialized Body ---\n");
// We need to cast the inner TypedNode to the generic T required by visit.
// Since Dumper is generic over T, but Closure stores TypedNode (where T = StaticType),
// we can only fully dump if T is StaticType.
// However, we can hack it by creating a new Dumper for the inner AST string.
// We can't call self.visit because types mismatch if T != StaticType.
// So we just recursively dump to string and append.
let inner_dump = Dumper::dump(&closure.function_node);
for line in inner_dump.lines() {
self.write_indent();
self.output.push_str(line);
self.output.push('\n');
}
self.indent -= 1;
fn visit<P: CompilerPhase>(&mut self, node: &Node<P>) {
match &node.kind {
NodeKind::Nop => self.log("Nop", node),
NodeKind::Constant(v) => {
self.log(&format!("Constant: {}", v), node);
}
}
BoundKind::Get { addr, name } => {
self.log(&format!("Get: {} ({:?})", name.name, addr), node)
NodeKind::Identifier { symbol, binding } => {
let label = if self.compact {
format!("Identifier: {}", symbol.name)
} else {
format!("Identifier: {} ({:?})", symbol.name, binding)
};
self.log(&label, node);
}
BoundKind::FieldAccessor(k) => self.log(&format!("FieldAccessor: .{}", k.name()), node),
NodeKind::FieldAccessor(k) => self.log(&format!("FieldAccessor: .{}", k.name()), node),
BoundKind::GetField { rec, field } => {
NodeKind::GetField { rec, field } => {
self.log(&format!("GetField: .{}", field.name()), node);
self.indent += 1;
self.visit(rec);
self.indent -= 1;
}
BoundKind::Set { addr, value } => {
self.log(&format!("Set: {:?}", addr), node);
self.indent += 1;
self.visit(value);
self.indent -= 1;
}
BoundKind::Define {
name,
addr,
kind,
value,
captured_by,
} => {
let k_str = match kind {
crate::ast::compiler::bound_nodes::DeclarationKind::Variable => "Variable",
crate::ast::compiler::bound_nodes::DeclarationKind::Parameter => "Parameter",
};
let capture_info = if captured_by.is_empty() {
String::from("not captured")
NodeKind::Assign { target, value, info } => {
let label = if self.compact {
"Assign".to_string()
} else {
format!("captured by {} lambdas", captured_by.len())
format!("Assign: {:?}", info)
};
self.log(
&format!(
"Define {} (Name: '{}', Address: {:?}, {})",
k_str, name.name, addr, capture_info
),
node,
);
self.log(&label, node);
self.indent += 1;
if !captured_by.is_empty() {
for capturer in captured_by {
self.write_indent();
let loc = capturer
.location
.unwrap_or(crate::ast::types::SourceLocation { line: 0, col: 0 });
self.output.push_str(&format!(
"- Capturer: Lambda at line {}, col {}\n",
loc.line, loc.col
));
}
}
self.output.push_str("Target:\n");
self.visit(target);
self.write_indent();
self.output.push_str("Value:\n");
self.visit(value);
self.indent -= 1;
}
BoundKind::Destructure { pattern, value } => {
self.log("Destructure", node);
NodeKind::Def {
pattern,
value,
info,
} => {
let label = if self.compact {
"Def".to_string()
} else {
format!("Def ({:?})", info)
};
self.log(&label, node);
self.indent += 1;
self.write_indent();
self.output.push_str("Pattern:\n");
@@ -135,7 +116,7 @@ impl Dumper {
self.indent -= 1;
}
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -159,28 +140,28 @@ impl Dumper {
self.indent -= 1;
}
BoundKind::Lambda {
NodeKind::Lambda {
params,
upvalues,
body,
..
info,
} => {
self.log(&format!("Lambda (Upvalues: {})", upvalues.len()), node);
let label = if self.compact {
"Lambda".to_string()
} else {
format!("Lambda ({:?})", info)
};
self.log(&label, node);
self.indent += 1;
self.write_indent();
self.output.push_str("Parameters:\n");
self.visit(params);
if !upvalues.is_empty() {
self.write_indent();
self.output.push_str(&format!("Upvalues: {:?}\n", upvalues));
}
self.visit(body);
self.indent -= 1;
}
BoundKind::Call { callee, args } => {
NodeKind::Call { callee, args } => {
self.log("Call", node);
self.indent += 1;
@@ -195,23 +176,13 @@ impl Dumper {
self.indent -= 1;
}
BoundKind::Again { args } => {
NodeKind::Again { args } => {
self.log("Again", node);
self.indent += 1;
self.visit(args);
self.indent -= 1;
}
BoundKind::Pipe { inputs, lambda, .. } => {
self.log("Pipe", node);
self.indent += 1;
for input in inputs {
self.visit(input);
}
self.visit(lambda);
self.indent -= 1;
}
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } => {
self.log("Block", node);
self.indent += 1;
for expr in exprs {
@@ -219,8 +190,16 @@ impl Dumper {
}
self.indent -= 1;
}
NodeKind::Program { exprs } => {
self.log("Program", node);
self.indent += 1;
for expr in exprs {
self.visit(expr);
}
self.indent -= 1;
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
self.log("Tuple", node);
self.indent += 1;
for el in elements {
@@ -229,35 +208,77 @@ impl Dumper {
self.indent -= 1;
}
BoundKind::Record { layout, values } => {
self.log(
&format!("Record (Layout: {} fields)", layout.fields.len()),
node,
);
NodeKind::Record { fields, layout } => {
let label = if self.compact {
"Record".to_string()
} else {
format!("Record (Layout: {:?})", layout)
};
self.log(&label, node);
self.indent += 1;
for v in values {
self.visit(v);
for (key, val) in fields {
self.write_indent();
self.output.push_str("Key:\n");
self.indent += 1;
self.visit(key);
self.indent -= 1;
self.write_indent();
self.output.push_str("Value:\n");
self.indent += 1;
self.visit(val);
self.indent -= 1;
}
self.indent -= 1;
}
BoundKind::Expansion {
NodeKind::Expansion {
original_call,
bound_expanded,
expanded,
} => {
self.log(
&format!("Expansion (Original: {:?})", original_call.kind),
node,
);
let label = if self.compact {
"Expansion".to_string()
} else {
format!("Expansion (Original: {:?})", original_call.kind)
};
self.log(&label, node);
self.indent += 1;
self.visit(bound_expanded);
self.visit(expanded);
self.indent -= 1;
}
BoundKind::Extension(ext) => {
NodeKind::MacroDecl { name, params, body } => {
self.log(&format!("MacroDecl: {}", name.name), node);
self.indent += 1;
self.visit(params);
self.visit(body);
self.indent -= 1;
}
NodeKind::Template(inner) => {
self.log("Template", node);
self.indent += 1;
self.visit(inner);
self.indent -= 1;
}
NodeKind::Placeholder(inner) => {
self.log("Placeholder", node);
self.indent += 1;
self.visit(inner);
self.indent -= 1;
}
NodeKind::Splice(inner) => {
self.log("Splice", node);
self.indent += 1;
self.visit(inner);
self.indent -= 1;
}
NodeKind::Extension(ext) => {
self.log(&ext.display_name(), node);
}
BoundKind::Error => {
NodeKind::Error => {
self.log("ERROR_NODE", node);
}
}
+267
View File
@@ -0,0 +1,267 @@
use crate::ast::nodes::{NodeKind, SyntaxNode};
use crate::ast::types::{CommentLine, Value};
use std::rc::Rc;
/// Emits a [`SyntaxNode`] back to Myc source code, preserving leading comments.
///
/// Round-trip guarantee: `emit(parse(emit(parse(source)))) == emit(parse(source))`.
/// In other words, the emitter is idempotent after the first pass.
pub fn emit(node: &SyntaxNode) -> String {
let mut out = String::new();
// The top-level node is always at block level (owns its own line).
emit_block(node, 0, &mut out);
out
}
// ── Block-level emission ────────────────────────────────────────────────────
/// Emits a node in **block-level** context: writes leading comments and
/// the appropriate indentation prefix, then the code.
///
/// Callers must only write a `\n` separator before calling this — never `\n + pad`.
fn emit_block(node: &SyntaxNode, indent: usize, out: &mut String) {
emit_comments(&node.comments, indent, out);
out.push_str(&" ".repeat(indent));
emit_kind(node, indent, out);
}
/// Emits a node in **inline** context: no indentation, no comments.
/// Used for sub-expressions that are part of a larger expression on the same line.
fn emit_inline(node: &SyntaxNode, indent: usize, out: &mut String) {
emit_kind(node, indent, out);
}
// ── Comment emission ────────────────────────────────────────────────────────
fn emit_comments(comments: &Rc<[CommentLine]>, indent: usize, out: &mut String) {
if comments.is_empty() {
return;
}
let pad = " ".repeat(indent);
for line in comments.iter() {
match line {
CommentLine::Comment(text) => {
out.push_str(&pad);
out.push(';');
if !text.is_empty() {
out.push(' ');
out.push_str(text);
}
out.push('\n');
}
CommentLine::Doc(text) => {
out.push_str(&pad);
out.push_str(";;");
if !text.is_empty() {
out.push(' ');
out.push_str(text);
}
out.push('\n');
}
CommentLine::Blank => {
out.push('\n');
}
}
}
}
// ── Kind emission ───────────────────────────────────────────────────────────
fn emit_kind(node: &SyntaxNode, indent: usize, out: &mut String) {
match &node.kind {
NodeKind::Nop => out.push_str("..."),
NodeKind::Constant(val) => emit_value(val, out),
NodeKind::Identifier { symbol, .. } => out.push_str(&symbol.name),
NodeKind::FieldAccessor(k) => {
out.push('.');
out.push_str(&k.name());
}
NodeKind::If { cond, then_br, else_br } => {
out.push_str("(if ");
emit_inline(cond, indent + 1, out);
out.push('\n');
emit_block(then_br, indent + 1, out);
if let Some(else_node) = else_br {
out.push('\n');
emit_block(else_node, indent + 1, out);
}
out.push(')');
}
NodeKind::Def { pattern, value, .. } => {
out.push_str("(def ");
emit_inline(pattern, indent, out);
out.push('\n');
emit_block(value, indent + 1, out);
out.push(')');
}
NodeKind::Assign { target, value, .. } => {
out.push_str("(assign ");
emit_inline(target, indent, out);
out.push(' ');
emit_inline(value, indent + 1, out);
out.push(')');
}
NodeKind::Lambda { params, body, .. } => {
out.push_str("(fn ");
emit_inline(params, indent, out);
out.push('\n');
emit_block(body, indent + 1, out);
out.push(')');
}
NodeKind::Call { callee, args } => {
out.push('(');
emit_inline(callee, indent, out);
if let NodeKind::Tuple { elements } = &args.kind {
for arg in elements {
out.push(' ');
emit_inline(arg, indent + 1, out);
}
}
out.push(')');
}
NodeKind::Again { args } => {
out.push_str("(again");
if let NodeKind::Tuple { elements } = &args.kind {
for arg in elements {
out.push(' ');
emit_inline(arg, indent + 1, out);
}
}
out.push(')');
}
NodeKind::Block { exprs } => {
out.push_str("(do");
for expr in exprs {
out.push('\n');
emit_block(expr, indent + 1, out);
}
out.push(')');
}
NodeKind::Program { exprs } => {
out.push_str("(do");
for expr in exprs {
out.push('\n');
emit_block(expr, indent + 1, out);
}
out.push(')');
}
NodeKind::Tuple { elements } => {
out.push('[');
for (i, el) in elements.iter().enumerate() {
if i > 0 {
out.push(' ');
}
emit_inline(el, indent, out);
}
out.push(']');
}
NodeKind::Record { fields, .. } => {
if fields.is_empty() {
out.push_str("{}");
return;
}
// Always multi-line so that any leading comments on keys are valid.
out.push('{');
for (key, val) in fields.iter() {
out.push('\n');
emit_block(key, indent + 1, out);
out.push(' ');
emit_inline(val, indent + 1, out);
}
out.push('\n');
out.push_str(&" ".repeat(indent));
out.push('}');
}
NodeKind::MacroDecl { name, params, body } => {
out.push_str("(macro ");
out.push_str(&name.name);
out.push(' ');
emit_inline(params, indent, out);
out.push('\n');
emit_block(body, indent + 1, out);
out.push(')');
}
NodeKind::Template(inner) => {
out.push('`');
emit_inline(inner, indent, out);
}
NodeKind::Placeholder(inner) => {
out.push('~');
emit_inline(inner, indent, out);
}
NodeKind::Splice(inner) => {
out.push_str("~@");
emit_inline(inner, indent, out);
}
NodeKind::GetField { rec, field } => {
out.push_str("(.");
out.push_str(&field.name());
out.push(' ');
emit_inline(rec, indent, out);
out.push(')');
}
NodeKind::Expansion { original_call, .. } => {
// Round-trip uses the original pre-expansion form.
emit_inline(original_call, indent, out);
}
NodeKind::Error | NodeKind::Extension(_) => {
out.push_str("<error>");
}
}
}
fn emit_value(val: &Value, out: &mut String) {
match val {
Value::Int(n) => out.push_str(&n.to_string()),
Value::Float(f) => {
let s = format!("{}", f);
// Ensure a decimal point is present so it round-trips as a float.
if s.contains('.') || s.contains('e') {
out.push_str(&s);
} else {
out.push_str(&s);
out.push_str(".0");
}
}
Value::Text(t) => {
out.push('"');
for c in t.chars() {
match c {
'"' => out.push_str("\\\""),
'\\' => out.push_str("\\\\"),
'\n' => out.push_str("\\n"),
'\r' => out.push_str("\\r"),
'\t' => out.push_str("\\t"),
c => out.push(c),
}
}
out.push('"');
}
Value::Keyword(k) => {
out.push(':');
out.push_str(&k.name());
}
Value::Bool(b) => out.push_str(if *b { "true" } else { "false" }),
other => out.push_str(&other.to_string()),
}
}
+59 -27
View File
@@ -1,37 +1,49 @@
use crate::ast::compiler::bound_nodes::{Address, BoundKind, BoundNode, GlobalIdx};
use crate::ast::nodes::{
Address, BoundLike, GlobalIdx, IdentifierBinding, Node, NodeKind,
};
use std::collections::HashMap;
use std::rc::Rc;
/// A pass that collects all global function definitions (lambdas) into a registry.
/// This allows the Specializer to retrieve the original AST of a function for monomorphization.
pub struct LambdaCollector<'a, T> {
registry: &'a mut HashMap<GlobalIdx, Rc<BoundNode<T>>>,
pub struct LambdaCollector<'a, P: BoundLike> {
registry: &'a mut HashMap<GlobalIdx, Rc<Node<P>>>,
}
impl<'a, T: Clone> LambdaCollector<'a, T> {
impl<'a, P> LambdaCollector<'a, P>
where
P: BoundLike,
{
/// Performs a full traversal of the AST and populates the provided registry.
pub fn collect(node: &BoundNode<T>, registry: &'a mut HashMap<GlobalIdx, Rc<BoundNode<T>>>) {
pub fn collect(node: &Node<P>, registry: &'a mut HashMap<GlobalIdx, Rc<Node<P>>>) {
let mut collector = Self { registry };
collector.visit(node);
}
fn visit(&mut self, node: &BoundNode<T>) {
fn visit(&mut self, node: &Node<P>) {
match &node.kind {
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } | NodeKind::Program { exprs } => {
for expr in exprs {
self.visit(expr);
}
}
BoundKind::Define { addr, value, .. } => {
NodeKind::Def { pattern, value, .. } => {
// Register global function definitions (lambdas)
if let Address::Global(global_index) = addr {
if let NodeKind::Identifier {
binding: IdentifierBinding::Declaration {
addr: Address::Global(global_index),
..
},
..
} = &pattern.kind
{
let mut current = value;
while let BoundKind::Expansion { bound_expanded, .. } = &current.kind {
current = bound_expanded;
while let NodeKind::Expansion { expanded, .. } = &current.kind {
current = expanded;
}
if let BoundKind::Lambda { .. } = &current.kind {
if let NodeKind::Lambda { .. } = &current.kind {
self.registry
.insert(*global_index, (*current).clone());
}
@@ -39,15 +51,15 @@ impl<'a, T: Clone> LambdaCollector<'a, T> {
self.visit(value);
}
BoundKind::Set { addr, value } => {
NodeKind::Assign { value, info, .. } => {
// Also track assignments to globals if they hold lambdas.
if let Address::Global(global_index) = addr {
if let Some(Address::Global(global_index)) = &info.addr {
let mut current = value;
while let BoundKind::Expansion { bound_expanded, .. } = &current.kind {
current = bound_expanded;
while let NodeKind::Expansion { expanded, .. } = &current.kind {
current = expanded;
}
if let BoundKind::Lambda { .. } = &current.kind {
if let NodeKind::Lambda { .. } = &current.kind {
self.registry
.insert(*global_index, (*current).clone());
}
@@ -55,7 +67,7 @@ impl<'a, T: Clone> LambdaCollector<'a, T> {
self.visit(value);
}
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -67,33 +79,53 @@ impl<'a, T: Clone> LambdaCollector<'a, T> {
}
}
BoundKind::Lambda { params, body, .. } => {
NodeKind::Lambda { params, body, .. } => {
self.visit(params);
self.visit(body);
}
BoundKind::Call { callee, args } => {
NodeKind::Call { callee, args } => {
self.visit(callee);
self.visit(args);
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
for el in elements {
self.visit(el);
}
}
BoundKind::Record { values, .. } => {
for v in values {
NodeKind::Record { fields, .. } => {
for (_, v) in fields {
self.visit(v);
}
}
BoundKind::Expansion { bound_expanded, .. } => {
self.visit(bound_expanded);
NodeKind::Expansion { expanded, .. } => {
self.visit(expanded);
}
_ => {} // Leaf nodes
NodeKind::GetField { rec, .. } => {
self.visit(rec);
}
NodeKind::Again { args } => {
self.visit(args);
}
NodeKind::MacroDecl { params, body, .. } => {
self.visit(params);
self.visit(body);
}
NodeKind::Template(inner)
| NodeKind::Placeholder(inner)
| NodeKind::Splice(inner) => {
self.visit(inner);
}
// Leaf nodes — no children to visit.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::Identifier { .. }
| NodeKind::FieldAccessor(_)
| NodeKind::Error
| NodeKind::Extension(_) => {}
}
}
}
+366
View File
@@ -0,0 +1,366 @@
use crate::ast::nodes::{
Address, AnalyzedNode, AssignBinding, ExecNode,
IdentifierBinding, LambdaBinding, Node, NodeKind, RuntimeMetadata,
StackOffset, VirtualId,
};
use std::collections::HashMap;
use std::rc::Rc;
struct StackAllocator {
mapping: HashMap<u32, u32>,
/// Physical slots that have been freed and can be reused.
free_slots: Vec<u32>,
next_slot: u32,
}
impl StackAllocator {
fn new() -> Self {
Self {
mapping: HashMap::new(),
free_slots: Vec::new(),
next_slot: 0,
}
}
fn map_slot(&mut self, slot: VirtualId) -> StackOffset {
let entry = self.mapping.entry(slot.0).or_insert_with(|| {
// Reuse a freed slot before growing the frame.
if let Some(recycled) = self.free_slots.pop() {
recycled
} else {
let s = self.next_slot;
self.next_slot += 1;
s
}
});
StackOffset(*entry)
}
fn map_address(&mut self, addr: Address<VirtualId>) -> Address<StackOffset> {
match addr {
Address::Local(slot) => Address::Local(self.map_slot(slot)),
Address::Upvalue(idx) => Address::Upvalue(idx),
Address::Global(idx) => Address::Global(idx),
}
}
/// Releases the physical slot of `vid` back to the free-list.
/// If `vid` was never mapped (e.g. a dead def removed by the optimizer)
/// this is a no-op.
fn free_slot(&mut self, vid: VirtualId) {
if let Some(&physical) = self.mapping.get(&vid.0) {
self.free_slots.push(physical);
}
}
}
pub struct Lowering;
impl Lowering {
/// Lowers an AnalyzedNode to an ExecNode, marking tail positions and calculating stack sizes.
pub fn lower(node: AnalyzedNode) -> ExecNode {
let mut allocator = StackAllocator::new();
let mut exec_node = Self::transform(Rc::new(node), true, &mut allocator);
// If the top-level node is a Lambda, it already has its internal stack_size
// calculated during transform(). For non-lambdas (like raw expressions),
// we use the allocator's next_slot to determine the required root stack size.
if !matches!(exec_node.kind, NodeKind::Lambda { .. }) {
exec_node.ty.stack_size = allocator.next_slot;
}
exec_node
}
fn transform(
node_rc: Rc<AnalyzedNode>,
is_tail_position: bool,
allocator: &mut StackAllocator,
) -> ExecNode {
let node = &*node_rc;
let mut lambda_stack_size = 0;
let new_kind = match &node.kind {
NodeKind::Call { callee, args } => {
// Optimized Field Access: Call { callee: FieldAccessor, args: Tuple[1] } → GetField
if let NodeKind::FieldAccessor(k) = &callee.kind
&& let NodeKind::Tuple { elements } = &args.kind
&& elements.len() == 1
{
let rec = Rc::new(Self::transform(elements[0].clone(), false, allocator));
return Node {
identity: node.identity.clone(),
kind: NodeKind::GetField {
rec,
field: *k,
},
ty: RuntimeMetadata {
ty: node.ty.original.ty.clone(),
is_tail: is_tail_position,
original: node_rc.clone(),
stack_size: 0,
},
comments: node.comments.clone(),
};
}
NodeKind::Call {
callee: Rc::new(Self::transform(callee.clone(), false, allocator)),
args: Rc::new(Self::transform(args.clone(), false, allocator)),
}
}
NodeKind::Again { args } => {
if !is_tail_position {
panic!("'again' is only allowed in tail position to avoid dead code.");
}
NodeKind::Again {
args: Rc::new(Self::transform(args.clone(), false, allocator)),
}
}
NodeKind::If {
cond,
then_br,
else_br,
} => NodeKind::If {
cond: Rc::new(Self::transform(cond.clone(), false, allocator)),
then_br: Rc::new(Self::transform(then_br.clone(), is_tail_position, allocator)),
else_br: else_br
.as_ref()
.map(|e| Rc::new(Self::transform(e.clone(), is_tail_position, allocator))),
},
NodeKind::Block { exprs } | NodeKind::Program { exprs } => {
let is_program = matches!(&node.kind, NodeKind::Program { .. });
if exprs.is_empty() {
if is_program {
NodeKind::Program { exprs: vec![] }
} else {
NodeKind::Block { exprs: vec![] }
}
} else {
// Collect VirtualIds of non-captured locals before
// transforming. After all exprs are lowered their
// physical slots can be returned to the free-list,
// allowing subsequent blocks to reuse them.
let scope_locals = Self::collect_scope_locals(exprs);
let last_idx = exprs.len() - 1;
let mut new_exprs = Vec::with_capacity(exprs.len());
for (i, expr) in exprs.iter().enumerate() {
let is_last = i == last_idx;
new_exprs.push(Rc::new(Self::transform(
expr.clone(),
is_tail_position && is_last,
allocator,
)));
}
// Release slots — variables defined here are no longer
// live after this block ends.
for vid in scope_locals {
allocator.free_slot(vid);
}
if is_program {
NodeKind::Program { exprs: new_exprs }
} else {
NodeKind::Block { exprs: new_exprs }
}
}
}
NodeKind::Lambda {
params,
body,
info: lambda_info,
} => {
// Upvalues refer to the PARENT scope's addresses.
let mapped_upvalues = lambda_info
.upvalues
.iter()
.map(|a| allocator.map_address(*a))
.collect();
// New allocator for the lambda's own stack frame
let mut lambda_allocator = StackAllocator::new();
let t_params = Rc::new(Self::transform(params.clone(), false, &mut lambda_allocator));
let t_body = Rc::new(Self::transform(body.clone(), true, &mut lambda_allocator));
lambda_stack_size = lambda_allocator.next_slot;
NodeKind::Lambda {
params: t_params,
body: t_body,
info: LambdaBinding {
upvalues: mapped_upvalues,
positional_count: lambda_info.positional_count,
},
}
}
NodeKind::Assign { target, value, info } => {
let new_info = if let Some(addr) = info.addr {
AssignBinding {
addr: Some(allocator.map_address(addr)),
}
} else {
AssignBinding { addr: None }
};
NodeKind::Assign {
target: Rc::new(Self::transform(target.clone(), false, allocator)),
value: Rc::new(Self::transform(value.clone(), false, allocator)),
info: new_info,
}
}
NodeKind::Def {
pattern,
value,
info,
} => NodeKind::Def {
pattern: Rc::new(Self::transform(pattern.clone(), false, allocator)),
value: Rc::new(Self::transform(value.clone(), false, allocator)),
info: info.clone(),
},
NodeKind::Identifier { symbol, binding } => {
let new_binding = match binding {
IdentifierBinding::Reference(addr) => {
IdentifierBinding::Reference(allocator.map_address(*addr))
}
IdentifierBinding::Declaration { addr, kind } => {
IdentifierBinding::Declaration {
addr: allocator.map_address(*addr),
kind: *kind,
}
}
};
NodeKind::Identifier {
symbol: symbol.clone(),
binding: new_binding,
}
}
NodeKind::Record { fields, layout } => {
let new_fields = fields
.iter()
.map(|(k, v)| {
(
Rc::new(Self::transform(k.clone(), false, allocator)),
Rc::new(Self::transform(v.clone(), false, allocator)),
)
})
.collect();
NodeKind::Record {
fields: new_fields,
layout: layout.clone(),
}
}
NodeKind::Tuple { elements } => {
let new_elements = elements
.iter()
.map(|e| Rc::new(Self::transform(e.clone(), false, allocator)))
.collect();
NodeKind::Tuple {
elements: new_elements,
}
}
NodeKind::Constant(v) => NodeKind::Constant(v.clone()),
NodeKind::FieldAccessor(k) => NodeKind::FieldAccessor(*k),
NodeKind::GetField { rec, field } => NodeKind::GetField {
rec: Rc::new(Self::transform(rec.clone(), false, allocator)),
field: *field,
},
NodeKind::Nop => NodeKind::Nop,
NodeKind::Expansion {
original_call,
expanded,
} => NodeKind::Expansion {
original_call: original_call.clone(),
expanded: Rc::new(Self::transform(
expanded.clone(),
is_tail_position,
allocator,
)),
},
NodeKind::Extension(_) => NodeKind::Nop,
NodeKind::Error => NodeKind::Error,
// Syntax-only variants should not appear in AnalyzedPhase
NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => NodeKind::Nop,
};
let stack_size = if let NodeKind::Lambda { .. } = &new_kind {
lambda_stack_size
} else {
0
};
Node {
identity: node.identity.clone(),
kind: new_kind,
ty: RuntimeMetadata {
ty: node.ty.original.ty.clone(),
is_tail: is_tail_position,
original: node_rc.clone(),
stack_size,
},
comments: node.comments.clone(),
}
}
/// Collects the `VirtualId`s of all non-captured locals defined at the
/// top level of `exprs`. These slots are safe to reclaim once the block
/// ends. Captured locals (upvalues) are excluded — closures that outlive
/// the block still hold live references to those physical slots.
fn collect_scope_locals(exprs: &[Rc<AnalyzedNode>]) -> Vec<VirtualId> {
let mut locals = Vec::new();
for expr in exprs {
if let NodeKind::Def { pattern, info, .. } = &expr.kind
&& info.captured_by.is_empty()
{
Self::collect_pattern_vids(pattern, &mut locals);
}
}
locals
}
/// Recursively extracts `VirtualId`s from declaration identifiers in a
/// binding pattern. Handles flat (`x`) and destructuring (`[x y]`) forms.
fn collect_pattern_vids(pattern: &AnalyzedNode, out: &mut Vec<VirtualId>) {
match &pattern.kind {
NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr: Address::Local(vid), .. },
..
} => out.push(*vid),
// Non-local declarations (Global, Upvalue) are not stack-allocated.
NodeKind::Identifier { .. } => {}
NodeKind::Tuple { elements } => {
for e in elements {
Self::collect_pattern_vids(e, out);
}
}
// Patterns should only contain Identifier and Tuple nodes.
// Other variants indicate a compiler bug upstream.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Def { .. }
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
}
+373 -281
View File
File diff suppressed because it is too large Load Diff
+42 -4
View File
@@ -1,21 +1,59 @@
pub mod analyzer;
pub mod binder;
pub mod bound_nodes;
pub mod call_hooks;
pub mod module_loader;
pub mod captures;
pub mod dumper;
pub mod emitter;
pub mod lambda_collector;
pub mod macros;
pub mod optimizer;
pub mod specializer;
pub mod tco;
pub mod lowering;
pub mod type_checker;
pub use crate::ast::nodes::*;
pub use binder::*;
pub use bound_nodes::*;
pub use captures::*;
pub use dumper::*;
pub use macros::*;
pub use optimizer::*;
pub use specializer::*;
pub use tco::*;
pub use lowering::*;
pub use type_checker::*;
use crate::ast::diagnostics::Diagnostics;
/// The result of a compilation pass: either a typed AST or a set of diagnostics.
pub struct CompilationResult {
pub ast: Option<TypedNode>,
pub diagnostics: Diagnostics,
}
impl CompilationResult {
pub fn success(ast: TypedNode) -> Self {
Self {
ast: Some(ast),
diagnostics: Diagnostics::new(),
}
}
pub fn error(msg: impl Into<String>) -> Self {
let mut diag = Diagnostics::new();
diag.push_error(msg, None);
Self {
ast: None,
diagnostics: diag,
}
}
pub fn into_result(self) -> Result<TypedNode, String> {
if self.diagnostics.has_errors() {
Err(self.diagnostics.format_errors())
} else if let Some(ast) = self.ast {
Ok(ast)
} else {
Err("Compilation failed without diagnostics".to_string())
}
}
}
+163
View File
@@ -0,0 +1,163 @@
use std::cell::RefCell;
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::rc::Rc;
use crate::ast::nodes::SyntaxNode;
use crate::ast::parser::Parser;
/// Resolves `#use` directives and collects `.myc` dependency files in topological order.
/// File I/O and dependency graph traversal are fully encapsulated here;
/// compilation and execution remain the caller's responsibility.
pub struct ModuleLoader {
search_paths: Rc<RefCell<Vec<PathBuf>>>,
loaded_modules: Rc<RefCell<HashSet<PathBuf>>>,
}
impl ModuleLoader {
pub fn new(
search_paths: Rc<RefCell<Vec<PathBuf>>>,
loaded_modules: Rc<RefCell<HashSet<PathBuf>>>,
) -> Self {
Self {
search_paths,
loaded_modules,
}
}
/// Collects all dependency files for `source` in topological order (dependencies before
/// dependents). Already-loaded modules (tracked in `loaded_modules`) are skipped.
pub fn collect_dependency_files(
&self,
source: &str,
base_path: &Path,
) -> Result<Vec<(PathBuf, SyntaxNode)>, String> {
let mut files = Vec::new();
self.collect_dependencies(source, base_path, &mut files)?;
Ok(files)
}
/// Returns a list of all matching `.myc` files for `module_path`
/// (resolves to single file or all files in a directory, alphabetically sorted).
fn resolve_module_paths(
&self,
module_path: &str,
base_path: &Path,
) -> Result<Vec<PathBuf>, String> {
let relative_path = module_path.replace("->", std::path::MAIN_SEPARATOR_STR);
let file_name = format!("{}.myc", relative_path);
let check_path = |base: &Path| -> Option<Vec<PathBuf>> {
let file_p = base.join(&file_name);
if file_p.is_file() && let Ok(canon) = file_p.canonicalize() {
return Some(vec![canon]);
}
let dir_p = base.join(&relative_path);
if dir_p.is_dir() {
let mut files = Vec::new();
if let Ok(entries) = std::fs::read_dir(&dir_p) {
let mut valid_entries: Vec<_> = entries.filter_map(|e| e.ok()).collect();
valid_entries.sort_by_key(|e| e.path());
for entry in valid_entries {
let Ok(file_type) = entry.file_type() else { continue };
if file_type.is_file() {
let p = entry.path();
if p.extension().is_some_and(|ext| ext == "myc")
&& let Ok(canon) = p.canonicalize()
{
files.push(canon);
}
}
}
}
if !files.is_empty() {
return Some(files);
}
}
None
};
if let Some(paths) = check_path(base_path) {
return Ok(paths);
}
for sp in self.search_paths.borrow().iter() {
if let Some(paths) = check_path(sp) {
return Ok(paths);
}
}
Err(format!(
"Could not find module or directory '{}'",
module_path
))
}
/// Recursively collects dependencies into `all_files` in topological order.
fn collect_dependencies(
&self,
source: &str,
base_path: &Path,
all_files: &mut Vec<(PathBuf, SyntaxNode)>,
) -> Result<(), String> {
let directives = Self::extract_use_directives(source);
for module_path in directives {
let abs_paths = self.resolve_module_paths(&module_path, base_path)?;
for abs_path in abs_paths {
// insert() returns false if the path was already present — skip in that case.
if !self.loaded_modules.borrow_mut().insert(abs_path.clone()) {
continue;
}
let lib_source = std::fs::read_to_string(&abs_path)
.map_err(|e| format!("Failed to read module {:?}: {}", abs_path, e))?;
let lib_base = abs_path.parent().unwrap_or(Path::new("."));
self.collect_dependencies(&lib_source, lib_base, all_files)?;
let mut parser = Parser::new(&lib_source);
let syntax_ast = parser.parse_expression();
if parser.diagnostics.has_errors() {
return Err(format!(
"Parser error in module {:?}:\n{}",
abs_path,
parser.diagnostics.format_errors()
));
}
all_files.push((abs_path, syntax_ast));
}
}
Ok(())
}
/// Extracts `#use <path>` directives from the leading lines of `source`.
/// Stops at the first non-directive, non-comment line.
fn extract_use_directives(source: &str) -> Vec<String> {
let mut paths = Vec::new();
for line in source.lines() {
let trimmed = line.trim();
if trimmed.is_empty() || trimmed.starts_with(';') {
continue;
}
if let Some(stripped) = trimmed.strip_prefix("#use ") {
let path = stripped.trim();
let clean_path = if (path.starts_with('"') && path.ends_with('"'))
|| (path.starts_with('\'') && path.ends_with('\''))
{
&path[1..path.len() - 1]
} else {
path
};
paths.push(clean_path.to_string());
} else {
break;
}
}
paths
}
}
File diff suppressed because it is too large Load Diff
+52 -17
View File
@@ -1,15 +1,16 @@
use crate::ast::compiler::bound_nodes::{Address, AnalyzedNode, BoundKind, NodeMetrics};
use crate::ast::nodes::Node;
use crate::ast::nodes::{
Address, AnalyzedNode, IdentifierBinding, Node, NodeKind, NodeMetrics,
};
use crate::ast::types::{Purity, RecordLayout, StaticType, Value};
use std::cell::RefCell;
use crate::ast::vm::GlobalStore;
use std::rc::Rc;
pub struct Folder<'a> {
pub globals: &'a Option<Rc<RefCell<Vec<Value>>>>,
pub globals: &'a Option<GlobalStore>,
}
impl<'a> Folder<'a> {
pub fn new(globals: &'a Option<Rc<RefCell<Vec<Value>>>>) -> Self {
pub fn new(globals: &'a Option<GlobalStore>) -> Self {
Self { globals }
}
@@ -17,12 +18,14 @@ impl<'a> Folder<'a> {
let ty = val.static_type();
let typed_original = Rc::new(Node {
identity: template.identity.clone(),
kind: BoundKind::Constant(val.clone()),
kind: NodeKind::Constant(val.clone()),
comments: template.comments.clone(),
ty: ty.clone(),
});
Node {
identity: template.identity.clone(),
kind: BoundKind::Constant(val),
kind: NodeKind::Constant(val),
comments: template.comments.clone(),
ty: NodeMetrics {
original: typed_original,
purity: Purity::Pure,
@@ -34,12 +37,14 @@ impl<'a> Folder<'a> {
pub fn make_nop_node(&self, template: &AnalyzedNode) -> AnalyzedNode {
let typed_original = Rc::new(Node {
identity: template.identity.clone(),
kind: BoundKind::Nop,
kind: NodeKind::Nop,
comments: template.comments.clone(),
ty: StaticType::Void,
});
Node {
identity: template.identity.clone(),
kind: BoundKind::Nop,
kind: NodeKind::Nop,
comments: template.comments.clone(),
ty: NodeMetrics {
original: typed_original,
purity: Purity::Pure,
@@ -57,14 +62,23 @@ impl<'a> Folder<'a> {
let mut constant_values = Vec::with_capacity(values.len());
for v_node in values {
if let BoundKind::Constant(val) = &v_node.kind {
if let NodeKind::Constant(val) = &v_node.kind {
constant_values.push(val.clone());
} else {
return None;
}
}
let record_val = Value::Record(layout.clone(), Rc::new(constant_values));
// Recompute layout from concrete value types to eliminate stale TypeVars.
let new_fields: Vec<_> = layout
.fields
.iter()
.zip(values.iter())
.map(|((keyword, _), v_node)| (*keyword, v_node.ty.original.ty.clone()))
.collect();
let new_layout = RecordLayout::get_or_create(new_fields);
let record_val = Value::Record(new_layout, Rc::new(constant_values));
Some(self.make_constant_node(record_val, template))
}
@@ -79,19 +93,40 @@ impl<'a> Folder<'a> {
let mut arg_values = Vec::with_capacity(arg_nodes.len());
for node in arg_nodes {
if let BoundKind::Constant(val) = &node.kind {
if let NodeKind::Constant(val) = &node.kind {
arg_values.push(val.clone());
} else {
return None;
}
}
let func_val = match &callee.kind {
BoundKind::Get {
addr: Address::Global(idx),
NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(idx)),
..
} => self.globals.as_ref()?.borrow().get(idx.0 as usize)?.clone(),
BoundKind::Constant(val) => val.clone(),
_ => return None,
} => self.globals.as_ref()?.get(idx.0 as usize)?,
NodeKind::Constant(val) => val.clone(),
// Only global identifiers and constants can be folded at compile time.
NodeKind::Identifier { .. }
| NodeKind::Nop
| NodeKind::FieldAccessor(_)
| NodeKind::Def { .. }
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Tuple { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => return None,
};
let result = match func_val {
Value::Function(f) => (f.func)(&arg_values),
+140 -57
View File
@@ -1,30 +1,33 @@
use crate::ast::compiler::bound_nodes::{Address, AnalyzedNode, BoundKind, GlobalIdx, LocalSlot};
use crate::ast::nodes::{
Address, AnalyzedNode, IdentifierBinding, NodeKind, VirtualId,
};
use crate::ast::types::{Purity, Value};
use crate::ast::vm::Closure;
use crate::ast::vm::GlobalStore;
use std::cell::RefCell;
use std::collections::{HashMap, HashSet};
use std::collections::HashSet;
use std::rc::Rc;
use super::substitution_map::SubstitutionMap;
use super::utils::UsageInfo;
pub struct Inliner<'a> {
pub globals: &'a Option<Rc<RefCell<Vec<Value>>>>,
pub global_purity: &'a Option<Rc<RefCell<HashMap<GlobalIdx, Purity>>>>,
pub globals: &'a Option<GlobalStore>,
pub root_purity: &'a Option<Rc<RefCell<Vec<Purity>>>>,
}
impl<'a> Inliner<'a> {
pub fn new(
globals: &'a Option<Rc<RefCell<Vec<Value>>>>,
global_purity: &'a Option<Rc<RefCell<HashMap<GlobalIdx, Purity>>>>,
globals: &'a Option<GlobalStore>,
root_purity: &'a Option<Rc<RefCell<Vec<Purity>>>>,
) -> Self {
Self {
globals,
global_purity,
root_purity,
}
}
pub fn is_inlinable_value(&self, val: &Value, addr: Address) -> bool {
pub fn is_inlinable_value(&self, val: &Value, addr: Address<VirtualId>) -> bool {
let type_ok = match val {
Value::Int(_)
| Value::Float(_)
@@ -33,12 +36,8 @@ impl<'a> Inliner<'a> {
| Value::Keyword(_)
| Value::Record(_, _)
| Value::DateTime(_) => true,
Value::Object(obj) => {
if let Some(closure) = obj.as_any().downcast_ref::<Closure>() {
closure.upvalues.is_empty() && !closure.function_node.ty.is_recursive
} else {
false
}
Value::Closure(closure_rc) => {
closure_rc.upvalues.is_empty() && !closure_rc.function_node.ty.is_recursive
}
_ => false,
};
@@ -48,10 +47,10 @@ impl<'a> Inliner<'a> {
}
if let Address::Global(idx) = addr {
if let Some(purity_rc) = &self.global_purity {
if let Some(purity_rc) = &self.root_purity {
let purity = purity_rc
.borrow()
.get(&idx)
.get(idx.0 as usize)
.cloned()
.unwrap_or(Purity::Impure);
return purity >= Purity::Pure;
@@ -108,71 +107,155 @@ impl<'a> Inliner<'a> {
body_usage: &UsageInfo,
) {
match &pattern.kind {
BoundKind::Define { addr, .. } => {
if let Some(arg) = args.get(*offset)
&& !body_usage.is_assigned(addr)
{
if let BoundKind::Constant(val) = &arg.kind {
sub.add_value(*addr, val.clone());
} else if let BoundKind::Lambda { upvalues, .. } = &arg.kind
&& upvalues.is_empty()
{
sub.add_ast_substitution(*addr, (**arg).clone());
} else if let BoundKind::Get {
addr: Address::Global(_),
NodeKind::Def { pattern: inner_pattern, .. } => {
// Extract addr from the pattern's Identifier binding
let addr = if let NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr, .. },
..
} = &arg.kind
} = &inner_pattern.kind
{
sub.add_ast_substitution(*addr, (**arg).clone());
Some(*addr)
} else {
None
};
if let Some(addr) = addr {
if let Some(arg) = args.get(*offset)
&& !body_usage.is_assigned(&addr)
{
let mut core_arg = arg.as_ref();
while let NodeKind::Expansion { expanded, .. } = &core_arg.kind {
core_arg = expanded.as_ref();
}
if let NodeKind::Constant(val) = &core_arg.kind {
sub.add_value(addr, val.clone());
} else if let NodeKind::Lambda { info: lambda_info, .. } = &core_arg.kind
&& lambda_info.upvalues.is_empty()
{
sub.add_ast_substitution(addr, core_arg.clone());
} else if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(_)),
..
} = &core_arg.kind
{
sub.add_ast_substitution(addr, core_arg.clone());
}
}
if let Address::Local(slot) = addr {
sub.map_slot(*slot);
sub.map_slot(slot);
}
}
*offset += 1;
}
BoundKind::Tuple { elements } => {
NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr, .. },
..
} => {
let addr = *addr;
if let Some(arg) = args.get(*offset)
&& !body_usage.is_assigned(&addr)
{
let mut core_arg = arg.as_ref();
while let NodeKind::Expansion { expanded, .. } = &core_arg.kind {
core_arg = expanded.as_ref();
}
if let NodeKind::Constant(val) = &core_arg.kind {
sub.add_value(addr, val.clone());
} else if let NodeKind::Lambda { info: lambda_info, .. } = &core_arg.kind
&& lambda_info.upvalues.is_empty()
{
sub.add_ast_substitution(addr, core_arg.clone());
} else if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(_)),
..
} = &core_arg.kind
{
sub.add_ast_substitution(addr, core_arg.clone());
}
}
if let Address::Local(slot) = addr {
sub.map_slot(slot);
}
*offset += 1;
}
NodeKind::Tuple { elements } => {
for el in elements {
self.map_params_to_args(el, args, offset, sub, body_usage);
}
}
_ => {}
// Parameters should only contain Def, Identifier (Declaration), and Tuple.
// Reference identifiers don't appear in parameter patterns.
NodeKind::Identifier { .. }
| NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
pub fn collect_parameter_slots_set(&self, node: &AnalyzedNode, slots: &mut HashSet<LocalSlot>) {
pub fn collect_parameter_slots_set(&self, node: &AnalyzedNode, slots: &mut HashSet<VirtualId>) {
match &node.kind {
BoundKind::Define {
addr: Address::Local(slot),
NodeKind::Def { pattern, .. } => {
if let NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr: Address::Local(slot), .. },
..
} = &pattern.kind
{
slots.insert(*slot);
}
}
NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr: Address::Local(slot), .. },
..
} => {
slots.insert(*slot);
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
for el in elements {
self.collect_parameter_slots_set(el, slots);
}
}
_ => {}
}
}
pub fn collect_parameter_slots(&self, node: &AnalyzedNode, sub: &mut SubstitutionMap) {
match &node.kind {
BoundKind::Define {
addr: Address::Local(slot),
..
} => {
sub.slot_mapping.insert(*slot, *slot);
sub.next_slot = sub.next_slot.max(slot.0 + 1);
}
BoundKind::Tuple { elements } => {
for el in elements {
self.collect_parameter_slots(el, sub);
}
}
_ => {}
// Parameters should only contain Def, Identifier (Declaration), and Tuple.
// Reference identifiers don't appear in parameter patterns.
NodeKind::Identifier { .. }
| NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
}
+95 -68
View File
@@ -1,18 +1,20 @@
use crate::ast::compiler::bound_nodes::{Address, AnalyzedNode, BoundKind, LocalSlot, UpvalueIdx};
use crate::ast::nodes::Node;
use crate::ast::nodes::{
Address, AnalyzedNode, AssignBinding, IdentifierBinding,
LambdaBinding, Node, NodeKind, UpvalueIdx, VirtualId,
};
use crate::ast::types::Value;
use std::collections::{HashMap, HashSet};
use std::rc::Rc;
#[derive(Default)]
pub struct SubstitutionMap {
pub values: HashMap<Address, Value>,
pub ast_substitutions: HashMap<Address, Rc<AnalyzedNode>>,
pub slot_mapping: HashMap<LocalSlot, LocalSlot>,
pub assigned: HashSet<Address>,
pub values: HashMap<Address<VirtualId>, Value>,
pub ast_substitutions: HashMap<Address<VirtualId>, Rc<AnalyzedNode>>,
pub slot_mapping: HashMap<VirtualId, VirtualId>,
pub assigned: HashSet<Address<VirtualId>>,
pub next_slot: u32,
pub used: HashSet<Address>,
pub captured_slots: HashSet<LocalSlot>,
pub used: HashSet<Address<VirtualId>>,
pub captured_slots: HashSet<VirtualId>,
}
impl SubstitutionMap {
@@ -44,40 +46,40 @@ impl SubstitutionMap {
inner
}
pub fn add_ast_substitution(&mut self, addr: Address, node: AnalyzedNode) {
pub fn add_ast_substitution(&mut self, addr: Address<VirtualId>, node: AnalyzedNode) {
self.ast_substitutions.insert(addr, Rc::new(node));
}
pub fn map_slot(&mut self, old_slot: LocalSlot) -> LocalSlot {
pub fn map_slot(&mut self, old_slot: VirtualId) -> VirtualId {
if let Some(&new_slot) = self.slot_mapping.get(&old_slot) {
return new_slot;
}
let new_slot = LocalSlot(self.next_slot);
let new_slot = VirtualId(self.next_slot);
self.slot_mapping.insert(old_slot, new_slot);
self.next_slot += 1;
new_slot
}
pub fn map_address(&mut self, addr: Address) -> Address {
pub fn map_address(&mut self, addr: Address<VirtualId>) -> Address<VirtualId> {
match addr {
Address::Local(slot) => Address::Local(self.map_slot(slot)),
other => other,
}
}
pub fn add_value(&mut self, addr: Address, val: Value) {
pub fn add_value(&mut self, addr: Address<VirtualId>, val: Value) {
self.values.insert(addr, val);
}
pub fn get_value(&self, addr: &Address) -> Option<&Value> {
pub fn get_value(&self, addr: &Address<VirtualId>) -> Option<&Value> {
self.values.get(addr)
}
pub fn remove_value(&mut self, addr: &Address) {
pub fn remove_value(&mut self, addr: &Address<VirtualId>) {
self.values.remove(addr);
}
fn reindex_addr(&self, addr: Address, mapping: &[Option<u32>]) -> Address {
fn reindex_addr(&self, addr: Address<VirtualId>, mapping: &[Option<u32>]) -> Address<VirtualId> {
if let Address::Upvalue(idx) = addr
&& let Some(res) = mapping.get(idx.0 as usize)
&& let Some(new_idx) = res
@@ -91,34 +93,48 @@ impl SubstitutionMap {
pub fn reindex_upvalues(&self, node_rc: Rc<AnalyzedNode>, mapping: &[Option<u32>]) -> Rc<AnalyzedNode> {
let node = &*node_rc;
let (new_kind, metrics) = match &node.kind {
BoundKind::Get { addr, name } => (
BoundKind::Get {
addr: self.reindex_addr(*addr, mapping),
name: name.clone(),
},
node.ty.clone(),
),
BoundKind::Lambda {
params,
upvalues,
body,
positional_count,
} => {
let mut next_upvalues = Vec::new();
for addr in upvalues {
next_upvalues.push(self.reindex_addr(*addr, mapping));
NodeKind::Identifier { symbol, binding } => {
let new_binding = match binding {
IdentifierBinding::Reference(addr) => {
IdentifierBinding::Reference(self.reindex_addr(*addr, mapping))
}
IdentifierBinding::Declaration { addr, kind } => {
IdentifierBinding::Declaration {
addr: self.reindex_addr(*addr, mapping),
kind: *kind,
}
}
};
(
BoundKind::Lambda {
params: params.clone(),
upvalues: next_upvalues,
body: body.clone(),
positional_count: *positional_count,
NodeKind::Identifier {
symbol: symbol.clone(),
binding: new_binding,
},
node.ty.clone(),
)
}
BoundKind::If {
NodeKind::Lambda {
params,
body,
info: lambda_info,
} => {
let mut next_upvalues = Vec::new();
for addr in &lambda_info.upvalues {
next_upvalues.push(self.reindex_addr(*addr, mapping));
}
(
NodeKind::Lambda {
params: params.clone(),
body: body.clone(),
info: LambdaBinding {
upvalues: next_upvalues,
positional_count: lambda_info.positional_count,
},
},
node.ty.clone(),
)
}
NodeKind::If {
cond,
then_br,
else_br,
@@ -127,7 +143,7 @@ impl SubstitutionMap {
let then_br = self.reindex_upvalues(then_br.clone(), mapping);
let else_br = else_br.as_ref().map(|e| self.reindex_upvalues(e.clone(), mapping));
(
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -135,67 +151,77 @@ impl SubstitutionMap {
node.ty.clone(),
)
}
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } => {
let exprs = exprs
.iter()
.map(|e| self.reindex_upvalues(e.clone(), mapping))
.collect();
(BoundKind::Block { exprs }, node.ty.clone())
(NodeKind::Block { exprs }, node.ty.clone())
}
BoundKind::Call { callee, args } => {
NodeKind::Call { callee, args } => {
let callee = self.reindex_upvalues(callee.clone(), mapping);
let args = self.reindex_upvalues(args.clone(), mapping);
(BoundKind::Call { callee, args }, node.ty.clone())
(NodeKind::Call { callee, args }, node.ty.clone())
}
BoundKind::Define {
name,
addr,
kind,
NodeKind::Def {
pattern,
value,
captured_by,
info,
} => {
let pattern = self.reindex_upvalues(pattern.clone(), mapping);
let value = self.reindex_upvalues(value.clone(), mapping);
(
BoundKind::Define {
name: name.clone(),
addr: *addr,
kind: *kind,
NodeKind::Def {
pattern,
value,
captured_by: captured_by.clone(),
info: info.clone(),
},
node.ty.clone(),
)
}
BoundKind::Set { addr, value } => {
let value = self.reindex_upvalues(value.clone(), mapping);
(
BoundKind::Set {
addr: self.reindex_addr(*addr, mapping),
NodeKind::Assign {
target,
value,
info: assign_info,
} => {
let target = self.reindex_upvalues(target.clone(), mapping);
let value = self.reindex_upvalues(value.clone(), mapping);
let new_info = if let Some(addr) = assign_info.addr {
AssignBinding {
addr: Some(self.reindex_addr(addr, mapping)),
}
} else {
assign_info.clone()
};
(
NodeKind::Assign {
target,
value,
info: new_info,
},
node.ty.clone(),
)
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
let elements = elements
.iter()
.map(|e| self.reindex_upvalues(e.clone(), mapping))
.collect();
(BoundKind::Tuple { elements }, node.ty.clone())
(NodeKind::Tuple { elements }, node.ty.clone())
}
BoundKind::Record { layout, values } => {
let values = values
NodeKind::Record { fields, layout } => {
let fields = fields
.iter()
.map(|v| self.reindex_upvalues(v.clone(), mapping))
.map(|(k, v)| (k.clone(), self.reindex_upvalues(v.clone(), mapping)))
.collect();
(BoundKind::Record { layout: layout.clone(), values }, node.ty.clone())
(NodeKind::Record { fields, layout: layout.clone() }, node.ty.clone())
}
BoundKind::Expansion { original_call, bound_expanded } => {
let bound_expanded = self.reindex_upvalues(bound_expanded.clone(), mapping);
NodeKind::Expansion { original_call, expanded } => {
let expanded = self.reindex_upvalues(expanded.clone(), mapping);
(
BoundKind::Expansion {
NodeKind::Expansion {
original_call: original_call.clone(),
bound_expanded,
expanded,
},
node.ty.clone(),
)
@@ -205,6 +231,7 @@ impl SubstitutionMap {
Rc::new(Node {
identity: node.identity.clone(),
kind: new_kind,
comments: node.comments.clone(),
ty: metrics,
})
}
+164 -50
View File
@@ -1,6 +1,9 @@
use crate::ast::compiler::bound_nodes::{Address, AnalyzedNode, BoundKind, GlobalIdx};
use crate::ast::nodes::{
Address, AnalyzedNode, GlobalIdx, IdentifierBinding,
NodeKind, VirtualId,
};
use crate::ast::types::{Identity, Value};
use crate::ast::vm::Closure;
use std::collections::HashSet;
// --- PathTracker ---
@@ -32,50 +35,56 @@ impl PathTracker {
// --- UsageInfo ---
#[derive(Default)]
pub struct UsageInfo {
pub used: HashSet<Address>,
pub assigned: HashSet<Address>,
pub used: HashSet<Address<VirtualId>>,
pub assigned: HashSet<Address<VirtualId>>,
pub used_identities: HashSet<Identity>,
}
impl UsageInfo {
pub fn is_assigned(&self, addr: &Address) -> bool {
pub fn is_assigned(&self, addr: &Address<VirtualId>) -> bool {
self.assigned.contains(addr)
}
pub fn is_used(&self, addr: &Address) -> bool {
pub fn is_used(&self, addr: &Address<VirtualId>) -> bool {
self.used.contains(addr)
}
pub fn collect(&mut self, node: &AnalyzedNode) {
match &node.kind {
BoundKind::Destructure { pattern, value } => {
NodeKind::Def { pattern, value, .. } => {
self.collect_pattern(pattern);
self.collect(value);
}
BoundKind::Constant(v) => {
if let Value::Object(obj) = v
&& let Some(closure) = obj.as_any().downcast_ref::<Closure>()
{
NodeKind::Constant(v) => {
if let Value::Closure(closure_rc) = v {
self.used_identities
.insert(closure.function_node.identity.clone());
self.collect(&closure.function_node);
.insert(closure_rc.function_node.identity.clone());
self.collect(&closure_rc.function_node);
}
}
BoundKind::Get { addr, .. } => {
self.used.insert(*addr);
NodeKind::Identifier { binding, .. } => {
let addr = match binding {
IdentifierBinding::Reference(addr) => *addr,
IdentifierBinding::Declaration { addr, .. } => *addr,
};
self.used.insert(addr);
}
NodeKind::Assign { target, value, info, .. } => {
if let Some(addr) = info.addr {
self.assigned.insert(addr);
} else {
// Destructuring assign: collect assigned addresses from target pattern
self.collect_assigned_from_target(target);
}
BoundKind::Set { addr, value } => {
self.assigned.insert(*addr);
self.collect(value);
}
BoundKind::GetField { rec, .. } => {
NodeKind::GetField { rec, .. } => {
self.collect(rec);
}
BoundKind::Lambda {
NodeKind::Lambda {
params,
body,
upvalues,
..
info: lambda_info,
} => {
self.used_identities.insert(node.identity.clone());
@@ -99,7 +108,7 @@ impl UsageInfo {
// Map used upvalues to parent scope
for addr in &inner_info.used {
if let Address::Upvalue(idx) = addr
&& let Some(parent_addr) = upvalues.get(idx.0 as usize)
&& let Some(parent_addr) = lambda_info.upvalues.get(idx.0 as usize)
{
self.used.insert(*parent_addr);
}
@@ -107,18 +116,18 @@ impl UsageInfo {
// Map assigned upvalues to parent scope
for addr in &inner_info.assigned {
if let Address::Upvalue(idx) = addr
&& let Some(parent_addr) = upvalues.get(idx.0 as usize)
&& let Some(parent_addr) = lambda_info.upvalues.get(idx.0 as usize)
{
self.assigned.insert(*parent_addr);
}
}
}
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } | NodeKind::Program { exprs } => {
for e in exprs {
self.collect(e);
}
}
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -129,54 +138,159 @@ impl UsageInfo {
self.collect(e);
}
}
BoundKind::Call { callee, args } => {
NodeKind::Call { callee, args } => {
self.collect(callee);
self.collect(args);
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
for e in elements {
self.collect(e);
}
}
BoundKind::Record { values, .. } => {
for v in values {
NodeKind::Record { fields, .. } => {
for (_, v) in fields {
self.collect(v);
}
}
BoundKind::Define { value, .. } => {
self.collect(value);
NodeKind::Expansion { expanded, .. } => {
self.collect(expanded);
}
BoundKind::Expansion { bound_expanded, .. } => {
self.collect(bound_expanded);
}
BoundKind::Again { args } => {
NodeKind::Again { args } => {
self.collect(args);
}
BoundKind::Pipe { inputs, lambda, .. } => {
for input in inputs {
self.collect(input);
}
self.collect(lambda);
}
BoundKind::Nop
| BoundKind::FieldAccessor(_)
| BoundKind::Extension(_)
| BoundKind::Error => {}
NodeKind::Nop
| NodeKind::FieldAccessor(_)
| NodeKind::Extension(_)
| NodeKind::Error => {}
// Syntax-only variants that should not appear in AnalyzedPhase
NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
pub fn collect_pattern(&mut self, node: &AnalyzedNode) {
match &node.kind {
BoundKind::Define { .. } => {}
BoundKind::Set { addr, .. } => {
self.assigned.insert(*addr);
NodeKind::Identifier {
binding: IdentifierBinding::Declaration { .. },
..
} => {}
NodeKind::Identifier { .. } => {}
NodeKind::Def { .. } => {}
NodeKind::Assign { info, .. } => {
if let Some(addr) = info.addr {
self.assigned.insert(addr);
}
BoundKind::Tuple { elements } => {
}
NodeKind::Tuple { elements } => {
for el in elements {
self.collect_pattern(el);
}
}
_ => {}
// Patterns should only contain Identifier, Def, Assign, and Tuple.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
pub fn collect_assigned_from_target(&mut self, node: &AnalyzedNode) {
match &node.kind {
NodeKind::Identifier { binding, .. } => {
let addr = match binding {
IdentifierBinding::Reference(addr)
| IdentifierBinding::Declaration { addr, .. } => *addr,
};
self.assigned.insert(addr);
}
NodeKind::Tuple { elements } => {
for el in elements {
self.collect_assigned_from_target(el);
}
}
// Assignment targets should only contain Identifier and Tuple.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Def { .. }
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
}
/// Collects all addresses declared in a binding pattern (Identifier Declaration or nested Tuple).
///
/// Used by the optimizer to enumerate all slots bound by a destructuring `def` pattern,
/// enabling dead-def elimination for patterns that `extract_def_addr` cannot handle
/// (i.e. anything other than a plain `Identifier`).
pub fn collect_pattern_addrs(pattern: &AnalyzedNode, out: &mut Vec<Address<VirtualId>>) {
match &pattern.kind {
NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr, .. },
..
} => {
out.push(*addr);
}
// Non-declaration identifiers (references) don't bind new slots.
NodeKind::Identifier { .. } => {}
NodeKind::Def { pattern: inner, .. } => {
collect_pattern_addrs(inner, out);
}
NodeKind::Tuple { elements } => {
for el in elements {
collect_pattern_addrs(el, out);
}
}
// Patterns should only contain Identifier, Def, and Tuple.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {}
}
}
+56 -46
View File
@@ -1,5 +1,6 @@
use crate::ast::compiler::bound_nodes::{Address, AnalyzedNode, BoundKind, BoundNode, NodeMetrics};
use crate::ast::nodes::Node;
use crate::ast::nodes::{
Address, AnalyzedNode, AnalyzedPhase, IdentifierBinding, Node, NodeKind, NodeMetrics, VirtualId,
};
use crate::ast::types::{Purity, Signature, StaticType, Value};
use std::cell::RefCell;
use std::collections::HashMap;
@@ -7,16 +8,16 @@ use std::rc::Rc;
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct MonoCacheKey {
pub address: Address,
pub address: Address<VirtualId>,
pub arg_types: Vec<StaticType>,
}
pub type CompileFunc = Rc<dyn Fn(Rc<BoundNode>, &[StaticType]) -> Result<(Value, StaticType), String>>;
pub type CompileFunc = Rc<dyn Fn(Rc<Node<AnalyzedPhase>>, &[StaticType]) -> Result<(Value, StaticType), String>>;
pub type RtlLookupFunc = Rc<dyn Fn(&str, &[StaticType]) -> Option<(Value, StaticType)>>;
pub trait FunctionRegistry {
fn resolve(&self, addr: Address) -> Option<Rc<BoundNode>>;
fn resolve_analyzed(&self, _addr: Address) -> Option<Rc<AnalyzedNode>> {
fn resolve(&self, addr: Address<VirtualId>) -> Option<Rc<Node<AnalyzedPhase>>>;
fn resolve_analyzed(&self, _addr: Address<VirtualId>) -> Option<Rc<AnalyzedNode>> {
None
}
}
@@ -51,13 +52,13 @@ impl Specializer {
fn visit_node(&self, node: AnalyzedNode) -> AnalyzedNode {
let (new_kind, metrics) = match node.kind {
BoundKind::Call { callee, args } => {
NodeKind::Call { callee, args } => {
let (new_callee, new_args, _ret_ty) =
self.specialize_call_logic(callee, args, node.ty.original.ty.clone());
let new_metrics = node.ty.clone();
(
BoundKind::Call {
NodeKind::Call {
callee: Rc::new(new_callee),
args: Rc::new(new_args),
},
@@ -65,7 +66,7 @@ impl Specializer {
)
}
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -74,7 +75,7 @@ impl Specializer {
let then_br = Rc::new(self.visit_node(then_br.as_ref().clone()));
let else_br = else_br.map(|e| Rc::new(self.visit_node(e.as_ref().clone())));
(
BoundKind::If {
NodeKind::If {
cond,
then_br,
else_br,
@@ -82,72 +83,74 @@ impl Specializer {
node.ty.clone(),
)
}
BoundKind::Block { exprs } => {
NodeKind::Block { exprs } => {
let exprs = exprs.into_iter().map(|e| Rc::new(self.visit_node(e.as_ref().clone()))).collect();
(BoundKind::Block { exprs }, node.ty.clone())
(NodeKind::Block { exprs }, node.ty.clone())
}
BoundKind::Lambda {
NodeKind::Lambda {
params,
upvalues,
body,
positional_count,
info,
} => {
let params = Rc::new(self.visit_node(params.as_ref().clone()));
let body = Rc::new(self.visit_node(body.as_ref().clone()));
(
BoundKind::Lambda {
NodeKind::Lambda {
params,
upvalues,
body,
positional_count,
info,
},
node.ty.clone(),
)
}
BoundKind::Define {
name,
addr,
kind,
NodeKind::Def {
pattern,
value,
captured_by,
info,
} => {
let value = Rc::new(self.visit_node(value.as_ref().clone()));
(
BoundKind::Define {
name: name.clone(),
addr,
kind,
NodeKind::Def {
pattern,
value,
captured_by: captured_by.clone(),
info,
},
node.ty.clone(),
)
}
BoundKind::Set { addr, value } => {
NodeKind::Assign { target, value, info } => {
let value = Rc::new(self.visit_node(value.as_ref().clone()));
(BoundKind::Set { addr, value }, node.ty.clone())
(NodeKind::Assign { target, value, info }, node.ty.clone())
}
BoundKind::Tuple { elements } => {
NodeKind::Tuple { elements } => {
let elements = elements.into_iter().map(|e| Rc::new(self.visit_node(e.as_ref().clone()))).collect();
(BoundKind::Tuple { elements }, node.ty.clone())
(NodeKind::Tuple { elements }, node.ty.clone())
}
BoundKind::Record { layout, values } => {
let values = values.into_iter().map(|v| Rc::new(self.visit_node(v.as_ref().clone()))).collect();
(BoundKind::Record { layout, values }, node.ty.clone())
NodeKind::Record { fields, layout } => {
let fields = fields.into_iter().map(|(k, v)| (k, Rc::new(self.visit_node(v.as_ref().clone())))).collect();
(NodeKind::Record { fields, layout }, node.ty.clone())
}
BoundKind::Expansion {
NodeKind::Expansion {
original_call,
bound_expanded,
expanded,
} => {
let bound_expanded = Rc::new(self.visit_node(bound_expanded.as_ref().clone()));
let expanded = Rc::new(self.visit_node(expanded.as_ref().clone()));
(
BoundKind::Expansion {
NodeKind::Expansion {
original_call,
bound_expanded,
expanded,
},
node.ty.clone(),
)
}
NodeKind::Again { args } => {
let args = Rc::new(self.visit_node(args.as_ref().clone()));
(NodeKind::Again { args }, node.ty.clone())
}
NodeKind::GetField { rec, field } => {
let rec = Rc::new(self.visit_node(rec.as_ref().clone()));
(NodeKind::GetField { rec, field }, node.ty.clone())
}
k => (k, node.ty.clone()),
};
@@ -155,6 +158,7 @@ impl Specializer {
identity: node.identity,
kind: new_kind,
ty: metrics,
comments: node.comments.clone(),
}
}
@@ -167,7 +171,11 @@ impl Specializer {
let new_callee = self.visit_node(callee.as_ref().clone());
let new_args = self.visit_node(args.as_ref().clone());
let address = if let BoundKind::Get { addr, .. } = &new_callee.kind {
let address = if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(addr),
..
} = &new_callee.kind
{
*addr
} else {
return (new_callee, new_args, original_ty);
@@ -202,8 +210,8 @@ impl Specializer {
}
if let Some(rtl_lookup) = &self.rtl_lookup
&& let BoundKind::Get { name, .. } = &new_callee.kind
&& let Some((val, ret_ty)) = rtl_lookup(&name.name, &arg_types)
&& let NodeKind::Identifier { symbol, .. } = &new_callee.kind
&& let Some((val, ret_ty)) = rtl_lookup(&symbol.name, &arg_types)
{
self.cache
.borrow_mut()
@@ -237,7 +245,7 @@ impl Specializer {
// Only replace the callee if the compiled value is actually a function/object.
// If it's a scalar (like 30 from folding), we DON'T fold here.
// We keep the Call but update the callee to the specialized version if it's an object.
if let Value::Object(_) | Value::Function(_) = &compiled_val {
if let Value::Stream(_) | Value::Function(_) = &compiled_val {
let specialized_callee = self.make_constant_node(
compiled_val,
StaticType::Function(Box::new(Signature {
@@ -261,17 +269,19 @@ impl Specializer {
) -> AnalyzedNode {
let typed_original = Rc::new(Node {
identity: template.identity.clone(),
kind: BoundKind::Constant(val.clone()),
kind: NodeKind::Constant(val.clone()),
ty: ty.clone(),
comments: template.comments.clone(),
});
Node {
identity: template.identity.clone(),
kind: BoundKind::Constant(val),
kind: NodeKind::Constant(val),
ty: NodeMetrics {
original: typed_original,
purity: Purity::Pure,
is_recursive: false,
},
comments: template.comments.clone(),
}
}
}
-187
View File
@@ -1,187 +0,0 @@
use crate::ast::compiler::bound_nodes::{AnalyzedNode, BoundKind};
use crate::ast::nodes::Node;
use crate::ast::types::StaticType;
use std::fmt::Debug;
use std::rc::Rc;
#[derive(Clone)]
pub struct RuntimeMetadata {
pub ty: StaticType,
pub is_tail: bool,
/// The analyzed node, containing metrics and a link to the original TypedNode.
pub original: Rc<AnalyzedNode>,
}
impl Debug for RuntimeMetadata {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Metadata")
.field("ty", &self.ty)
.field("is_tail", &self.is_tail)
.finish()
}
}
/// The ExecNode is the AST used by the VM. It carries TCO flags and links to metrics.
pub type ExecNode = Node<BoundKind<RuntimeMetadata>, RuntimeMetadata>;
pub struct TCO;
impl TCO {
/// Lowers an AnalyzedNode to an ExecNode and marks tail positions.
pub fn optimize(node: AnalyzedNode) -> ExecNode {
Self::transform(Rc::new(node), true)
}
fn transform(node_rc: Rc<AnalyzedNode>, is_tail_position: bool) -> ExecNode {
let node = &*node_rc;
let new_kind = match &node.kind {
BoundKind::Call { callee, args } => BoundKind::Call {
callee: Rc::new(Self::transform(callee.clone(), false)),
args: Rc::new(Self::transform(args.clone(), false)),
},
BoundKind::Again { args } => {
if !is_tail_position {
panic!("'again' is only allowed in tail position to avoid dead code.");
}
BoundKind::Again {
args: Rc::new(Self::transform(args.clone(), false)),
}
}
BoundKind::Pipe {
inputs,
lambda,
out_type,
} => {
let mut t_inputs = Vec::with_capacity(inputs.len());
for input in inputs {
t_inputs.push(Rc::new(Self::transform(input.clone(), false)));
}
BoundKind::Pipe {
inputs: t_inputs,
lambda: Rc::new(Self::transform(lambda.clone(), false)),
out_type: out_type.clone(),
}
}
BoundKind::If {
cond,
then_br,
else_br,
} => BoundKind::If {
cond: Rc::new(Self::transform(cond.clone(), false)),
then_br: Rc::new(Self::transform(
then_br.clone(),
is_tail_position,
)),
else_br: else_br
.as_ref()
.map(|e| Rc::new(Self::transform(e.clone(), is_tail_position))),
},
BoundKind::Block { exprs } => {
if exprs.is_empty() {
BoundKind::Block { exprs: vec![] }
} else {
let last_idx = exprs.len() - 1;
let mut new_exprs = Vec::with_capacity(exprs.len());
for (i, expr) in exprs.iter().enumerate() {
let is_last = i == last_idx;
new_exprs.push(Rc::new(Self::transform(
expr.clone(),
is_tail_position && is_last,
)));
}
BoundKind::Block { exprs: new_exprs }
}
}
BoundKind::Lambda {
params,
upvalues,
body,
positional_count,
} => BoundKind::Lambda {
params: Rc::new(Self::transform(params.clone(), false)),
upvalues: upvalues.clone(),
body: Rc::new(Self::transform(body.clone(), true)),
positional_count: *positional_count,
},
BoundKind::Set { addr, value } => BoundKind::Set {
addr: *addr,
value: Rc::new(Self::transform(value.clone(), false)),
},
BoundKind::Define {
name,
addr,
kind,
value,
captured_by,
} => BoundKind::Define {
name: name.clone(),
addr: *addr,
kind: *kind,
value: Rc::new(Self::transform(value.clone(), false)),
captured_by: captured_by.clone(),
},
BoundKind::Destructure { pattern, value } => BoundKind::Destructure {
pattern: Rc::new(Self::transform(pattern.clone(), false)),
value: Rc::new(Self::transform(value.clone(), false)),
},
BoundKind::Record { layout, values } => {
let new_values = values
.iter()
.map(|v| Rc::new(Self::transform(v.clone(), false)))
.collect();
BoundKind::Record {
layout: layout.clone(),
values: new_values,
}
}
BoundKind::Tuple { elements } => {
let new_elements = elements
.iter()
.map(|e| Rc::new(Self::transform(e.clone(), false)))
.collect();
BoundKind::Tuple {
elements: new_elements,
}
}
BoundKind::Constant(v) => BoundKind::Constant(v.clone()),
BoundKind::Get { addr, name } => BoundKind::Get {
addr: *addr,
name: name.clone(),
},
BoundKind::FieldAccessor(k) => BoundKind::FieldAccessor(*k),
BoundKind::GetField { rec, field } => BoundKind::GetField {
rec: Rc::new(Self::transform(rec.clone(), false)),
field: *field,
},
BoundKind::Nop => BoundKind::Nop,
BoundKind::Expansion {
original_call,
bound_expanded,
} => BoundKind::Expansion {
original_call: original_call.clone(),
bound_expanded: Rc::new(Self::transform(
bound_expanded.clone(),
is_tail_position,
)),
},
BoundKind::Extension(_) => BoundKind::Nop,
BoundKind::Error => BoundKind::Error,
};
Node {
identity: node.identity.clone(),
kind: new_kind,
ty: RuntimeMetadata {
ty: node.ty.original.ty.clone(),
is_tail: is_tail_position,
original: node_rc,
},
}
}
}
-930
View File
@@ -1,930 +0,0 @@
use crate::ast::compiler::bound_nodes::{Address, BoundKind, BoundNode, GlobalIdx, TypedNode};
use crate::ast::diagnostics::Diagnostics;
use crate::ast::nodes::Node;
use crate::ast::types::StaticType;
use std::collections::HashMap;
use std::rc::Rc;
/// Manages the types of locals and upvalues during a single type-checking pass.
struct TypeContext<'a> {
_parent: Option<&'a TypeContext<'a>>,
/// Maps slot index -> Inferred Type
slots: Vec<StaticType>,
/// Types of captured variables (passed from outer scope)
upvalue_types: Vec<StaticType>,
/// Access to global types for unified resolution
global_types: &'a std::cell::RefCell<HashMap<GlobalIdx, StaticType>>,
/// The expected parameters of the current function (for 'again' validation)
current_params_ty: Option<StaticType>,
}
impl<'a> TypeContext<'a> {
fn new(
slot_count: u32,
upvalue_types: Vec<StaticType>,
global_types: &'a std::cell::RefCell<HashMap<GlobalIdx, StaticType>>,
parent: Option<&'a TypeContext<'a>>,
) -> Self {
Self {
_parent: parent,
slots: vec![StaticType::Any; slot_count as usize],
upvalue_types,
global_types,
current_params_ty: None,
}
}
fn get_type(&self, addr: Address) -> StaticType {
match addr {
Address::Local(slot) => self
.slots
.get(slot.0 as usize)
.cloned()
.unwrap_or(StaticType::Any),
Address::Global(idx) => self
.global_types
.borrow()
.get(&idx)
.cloned()
.unwrap_or(StaticType::Any),
Address::Upvalue(idx) => self
.upvalue_types
.get(idx.0 as usize)
.cloned()
.unwrap_or(StaticType::Any),
}
}
fn set_type(&mut self, addr: Address, ty: StaticType) {
match addr {
Address::Local(slot) => {
if let Some(entry) = self.slots.get_mut(slot.0 as usize) {
*entry = ty;
}
}
Address::Global(idx) => {
self.global_types.borrow_mut().insert(idx, ty);
}
_ => {}
}
}
}
pub struct TypeChecker {
global_types: Rc<std::cell::RefCell<HashMap<GlobalIdx, StaticType>>>,
}
impl TypeChecker {
pub fn new(global_types: Rc<std::cell::RefCell<HashMap<GlobalIdx, StaticType>>>) -> Self {
Self { global_types }
}
pub fn check(
&self,
node: &BoundNode,
arg_types: &[StaticType],
diag: &mut Diagnostics,
) -> TypedNode {
match &node.kind {
BoundKind::Lambda {
params,
upvalues,
body,
positional_count,
} => {
// 1. Determine types of captured variables (Root lambdas have none)
let mut upvalue_types = Vec::with_capacity(upvalues.len());
for _ in upvalues {
upvalue_types.push(StaticType::Any);
}
// 2. Create the specialized context
let root_ctx = TypeContext::new(0, vec![], &self.global_types, None);
let mut lambda_ctx =
TypeContext::new(64, upvalue_types, &self.global_types, Some(&root_ctx));
// 3. INJECT specialized argument types into slots
let arg_tuple_ty = if arg_types.is_empty() {
StaticType::Any // No specialized info
} else {
StaticType::Tuple(arg_types.to_vec())
};
let params_typed = self.check_params(
params.as_ref(),
&arg_tuple_ty,
&mut lambda_ctx,
diag,
);
// 4. Check body with the new types
let body_typed = self.check_node(body, &mut lambda_ctx, diag);
let ret_ty = body_typed.ty.clone();
// 5. Construct specialized function type
let final_params_ty = params_typed.ty.clone();
let fn_ty = StaticType::Function(Box::new(crate::ast::types::Signature {
params: final_params_ty,
ret: ret_ty,
}));
Node {
identity: node.identity.clone(),
kind: BoundKind::Lambda {
params: Rc::new(params_typed),
upvalues: upvalues.clone(),
body: Rc::new(body_typed),
positional_count: *positional_count,
},
ty: fn_ty,
}
}
_ => {
let mut root_ctx = TypeContext::new(0, vec![], &self.global_types, None);
self.check_node(node, &mut root_ctx, diag)
}
}
}
fn check_params(
&self,
node: &BoundNode,
specialized_ty: &StaticType,
ctx: &mut TypeContext,
diag: &mut Diagnostics,
) -> TypedNode {
let (kind, ty) = match &node.kind {
BoundKind::Define {
name,
addr,
kind: decl_kind,
captured_by,
..
} => {
ctx.set_type(*addr, specialized_ty.clone());
(
BoundKind::Define {
name: name.clone(),
addr: *addr,
kind: *decl_kind,
value: Rc::new(Node {
identity: node.identity.clone(),
kind: BoundKind::Nop,
ty: specialized_ty.clone(),
}),
captured_by: captured_by.clone(),
},
specialized_ty.clone(),
)
}
BoundKind::Set {
addr,
value: _value,
} => {
// In an assignment pattern, 'value' is just a Nop placeholder from the Binder.
// We update the type of the address (if it's a local or global) to match the specialized type.
// Note: For now, we assume assignments are compatible if types were already inferred,
// or we could add a check here against existing type.
(
BoundKind::Set {
addr: *addr,
value: Rc::new(Node {
identity: node.identity.clone(),
kind: BoundKind::Nop,
ty: specialized_ty.clone(),
}),
},
specialized_ty.clone(),
)
}
BoundKind::Tuple { elements } => {
match specialized_ty {
StaticType::Any
| StaticType::Tuple(_)
| StaticType::Vector(_, _)
| StaticType::Matrix(_, _)
| StaticType::List(_)
| StaticType::Record(_)
| StaticType::Error => {}
_ => {
diag.push_error(
format!(
"Cannot destructure type {} as a tuple/vector",
specialized_ty
),
Some(node.identity.clone()),
);
return Node {
identity: node.identity.clone(),
kind: BoundKind::Error,
ty: StaticType::Error,
};
}
}
let mut typed_elements = Vec::new();
let mut elem_types = Vec::new();
for (i, el) in elements.iter().enumerate() {
let sub_ty = match specialized_ty {
StaticType::Tuple(t) => t.get(i).cloned().unwrap_or(StaticType::Any),
StaticType::Vector(inner, _) => (**inner).clone(),
StaticType::Matrix(inner, _) => (**inner).clone(),
StaticType::List(inner) => (**inner).clone(),
StaticType::Record(layout) => layout
.fields
.get(i)
.map(|(_, ty): &(crate::ast::types::Keyword, StaticType)| ty.clone())
.unwrap_or(StaticType::Any),
StaticType::Error => StaticType::Error,
_ => StaticType::Any,
};
let t = self.check_params(el, &sub_ty, ctx, diag);
elem_types.push(t.ty.clone());
typed_elements.push(Rc::new(t));
}
(
BoundKind::Tuple {
elements: typed_elements,
},
StaticType::Tuple(elem_types),
)
}
BoundKind::Error => (BoundKind::Error, StaticType::Error),
_ => {
diag.push_error(
"Invalid node in parameter list",
Some(node.identity.clone()),
);
(BoundKind::Error, StaticType::Error)
}
};
Node {
identity: node.identity.clone(),
kind,
ty,
}
}
fn check_node(
&self,
node: &BoundNode,
ctx: &mut TypeContext,
diag: &mut Diagnostics,
) -> TypedNode {
let (kind, ty) = match &node.kind {
BoundKind::Nop => (BoundKind::Nop, StaticType::Void),
BoundKind::Constant(v) => {
let ty = v.static_type();
(BoundKind::Constant(v.clone()), ty)
}
BoundKind::Define {
name,
addr,
kind: decl_kind,
value,
captured_by,
} => {
let val_typed = self.check_node(value, ctx, diag);
let ty = val_typed.ty.clone();
ctx.set_type(*addr, ty.clone());
(
BoundKind::Define {
name: name.clone(),
addr: *addr,
kind: *decl_kind,
value: Rc::new(val_typed),
captured_by: captured_by.clone(),
},
ty,
)
}
BoundKind::Get { addr, name } => {
let ty = ctx.get_type(*addr);
(
BoundKind::Get {
addr: *addr,
name: name.clone(),
},
ty,
)
}
BoundKind::FieldAccessor(k) => {
(BoundKind::FieldAccessor(*k), StaticType::FieldAccessor(*k))
}
BoundKind::GetField { rec, field } => {
let rec_typed = self.check_node(rec, ctx, diag);
let field_ty = match &rec_typed.ty {
StaticType::Record(layout) => {
if let Some(idx) = layout.index_of(*field) {
layout.fields[idx].1.clone()
} else {
diag.push_error(
format!("Record does not have field :{}", field.name()),
Some(rec_typed.identity.clone()),
);
StaticType::Error
}
}
StaticType::Any => StaticType::Any,
StaticType::Error => StaticType::Error,
_ => {
diag.push_error(
format!(
"Cannot access field :{} on non-record type {}",
field.name(),
rec_typed.ty
),
Some(rec_typed.identity.clone()),
);
StaticType::Error
}
};
(
BoundKind::GetField {
rec: Rc::new(rec_typed),
field: *field,
},
field_ty,
)
}
BoundKind::Set { addr, value } => {
let val_typed = self.check_node(value, ctx, diag);
let ty = val_typed.ty.clone();
ctx.set_type(*addr, ty.clone());
(
BoundKind::Set {
addr: *addr,
value: Rc::new(val_typed),
},
ty,
)
}
BoundKind::Destructure { pattern, value } => {
let val_typed = self.check_node(value, ctx, diag);
let pat_typed = self.check_params(pattern, &val_typed.ty, ctx, diag);
let ty = val_typed.ty.clone();
(
BoundKind::Destructure {
pattern: Rc::new(pat_typed),
value: Rc::new(val_typed),
},
ty,
)
}
BoundKind::If {
cond,
then_br,
else_br,
} => {
let cond_typed = self.check_node(cond, ctx, diag);
let then_typed = self.check_node(then_br, ctx, diag);
let mut else_typed = None;
let mut final_ty = then_typed.ty.clone();
if let Some(e) = else_br {
let et = self.check_node(e, ctx, diag);
// Basic type promotion: if types differ, fall back to Any
if et.ty != final_ty {
final_ty = StaticType::Any;
}
else_typed = Some(Rc::new(et));
} else {
// If without else returns Optional(Then) (T | Void)
final_ty = StaticType::Optional(Box::new(then_typed.ty.clone()));
}
(
BoundKind::If {
cond: Rc::new(cond_typed),
then_br: Rc::new(then_typed),
else_br: else_typed,
},
final_ty,
)
}
BoundKind::Pipe { inputs, lambda, .. } => {
let mut typed_inputs = Vec::with_capacity(inputs.len());
let mut arg_types = Vec::with_capacity(inputs.len());
for input in inputs {
let typed_input = self.check_node(input, ctx, diag);
// Unwrap Series(T) or Stream(T) to T for the lambda argument
let arg_ty = if let StaticType::Series(inner) = &typed_input.ty {
*inner.clone()
} else if let StaticType::Stream(inner) = &typed_input.ty {
*inner.clone()
} else {
StaticType::Any
};
arg_types.push(arg_ty);
typed_inputs.push(Rc::new(typed_input));
}
// Specialized check for the lambda using the input types!
let typed_lambda = self.check(lambda, &arg_types, diag);
let ret_ty = if let StaticType::Function(sig) = &typed_lambda.ty {
// If the lambda returns an Optional(T), the pipeline filters Void and stores T!
if let StaticType::Optional(inner) = &sig.ret {
*inner.clone()
} else {
sig.ret.clone()
}
} else {
StaticType::Any
};
(
BoundKind::Pipe {
inputs: typed_inputs,
lambda: Rc::new(typed_lambda),
out_type: ret_ty.clone(),
},
StaticType::Stream(Box::new(ret_ty)),
)
}
BoundKind::Block { exprs } => {
let mut typed_exprs = Vec::new();
let mut last_ty = StaticType::Void;
for e in exprs {
let t = self.check_node(e, ctx, diag);
last_ty = t.ty.clone();
typed_exprs.push(Rc::new(t));
}
(BoundKind::Block { exprs: typed_exprs }, last_ty)
}
BoundKind::Lambda {
params,
upvalues,
body,
positional_count,
} => {
// 1. Determine types of captured variables
let mut upvalue_types = Vec::with_capacity(upvalues.len());
for &addr in upvalues {
upvalue_types.push(ctx.get_type(addr));
}
// 2. Create nested context for lambda body
let mut lambda_ctx =
TypeContext::new(64, upvalue_types, ctx.global_types, Some(ctx));
// 3. Check parameters and body
let params_typed = self.check_params(
params.as_ref(),
&StaticType::Any,
&mut lambda_ctx,
diag,
);
// Set current params type for 'again' validation
lambda_ctx.current_params_ty = Some(params_typed.ty.clone());
let body_typed = self.check_node(body, &mut lambda_ctx, diag);
let ret_ty = body_typed.ty.clone();
// 4. Construct function type
let fn_ty = StaticType::Function(Box::new(crate::ast::types::Signature {
params: params_typed.ty.clone(),
ret: ret_ty,
}));
(
BoundKind::Lambda {
params: Rc::new(params_typed),
upvalues: upvalues.clone(),
body: Rc::new(body_typed),
positional_count: *positional_count,
},
fn_ty,
)
}
BoundKind::Call { callee, args } => {
let callee_typed = self.check_node(callee, ctx, diag);
// Manually check args (Tuple) to prevent Vector/Matrix promotion
let args_typed = if let BoundKind::Tuple { elements } = &args.kind {
let mut typed_elements = Vec::new();
let mut elem_types = Vec::new();
for e in elements {
let t = self.check_node(e, ctx, diag);
elem_types.push(t.ty.clone());
typed_elements.push(Rc::new(t));
}
Node {
identity: args.identity.clone(),
kind: BoundKind::Tuple {
elements: typed_elements,
},
ty: StaticType::Tuple(elem_types),
}
} else {
// Should not happen as parser wraps args in Tuple, but fallback safely
self.check_node(args, ctx, diag)
};
let ret_ty = match callee_typed.ty.resolve_call(&args_typed.ty) {
Some(ty) => ty,
None => {
diag.push_error(
format!(
"Invalid arguments for function call. Expected {}, got {}",
callee_typed.ty, args_typed.ty
),
Some(node.identity.clone()),
);
StaticType::Error
}
};
(
BoundKind::Call {
callee: Rc::new(callee_typed),
args: Rc::new(args_typed),
},
ret_ty,
)
}
BoundKind::Again { args } => {
// Manually check args (Tuple) to prevent Vector/Matrix promotion
let args_typed = if let BoundKind::Tuple { elements } = &args.kind {
let mut typed_elements = Vec::new();
let mut elem_types = Vec::new();
for e in elements {
let t = self.check_node(e, ctx, diag);
elem_types.push(t.ty.clone());
typed_elements.push(Rc::new(t));
}
Node {
identity: args.identity.clone(),
kind: BoundKind::Tuple {
elements: typed_elements,
},
ty: StaticType::Tuple(elem_types),
}
} else {
self.check_node(args, ctx, diag)
};
if let Some(expected_ty) = &ctx.current_params_ty
&& !expected_ty.is_assignable_from(&args_typed.ty)
{
diag.push_error(
format!(
"Type mismatch in 'again' call: expected {}, but got {}",
expected_ty, args_typed.ty
),
Some(node.identity.clone()),
);
}
(
BoundKind::Again {
args: Rc::new(args_typed),
},
StaticType::Any,
)
}
BoundKind::Tuple { elements } => {
let mut typed_elements = Vec::new();
for e in elements {
typed_elements.push(Rc::new(self.check_node(e, ctx, diag)));
}
let ty = if typed_elements.is_empty() {
StaticType::Vector(Box::new(StaticType::Any), 0)
} else {
let first_ty = &typed_elements[0].ty;
let all_same = typed_elements.iter().all(|e| e.ty == *first_ty);
if all_same {
match first_ty {
StaticType::Vector(inner, len) => {
StaticType::Matrix(inner.clone(), vec![typed_elements.len(), *len])
}
StaticType::Matrix(inner, shape) => {
let mut new_shape = vec![typed_elements.len()];
new_shape.extend(shape);
StaticType::Matrix(inner.clone(), new_shape)
}
_ => {
StaticType::Vector(Box::new(first_ty.clone()), typed_elements.len())
}
}
} else {
StaticType::Tuple(typed_elements.iter().map(|e| e.ty.clone()).collect())
}
};
(
BoundKind::Tuple {
elements: typed_elements,
},
ty,
)
}
BoundKind::Record { layout, values } => {
let mut typed_values = Vec::with_capacity(values.len());
let mut fields_ty = Vec::with_capacity(values.len());
for (i, v) in values.iter().enumerate() {
let vt = self.check_node(v, ctx, diag);
fields_ty.push((layout.fields[i].0, vt.ty.clone()));
typed_values.push(Rc::new(vt));
}
let new_layout = crate::ast::types::RecordLayout::get_or_create(fields_ty);
(
BoundKind::Record {
layout: new_layout.clone(),
values: typed_values,
},
StaticType::Record(new_layout),
)
}
BoundKind::Expansion {
original_call,
bound_expanded,
} => {
let expanded_typed = self.check_node(bound_expanded, ctx, diag);
let ty = expanded_typed.ty.clone();
(
BoundKind::Expansion {
original_call: original_call.clone(),
bound_expanded: Rc::new(expanded_typed),
},
ty,
)
}
BoundKind::Extension(_ext) => {
diag.push_error(
format!(
"TypeChecking for extension '{}' not implemented",
_ext.display_name()
),
Some(node.identity.clone()),
);
(BoundKind::Error, StaticType::Error)
}
BoundKind::Error => (BoundKind::Error, StaticType::Error),
};
Node {
identity: node.identity.clone(),
kind,
ty,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ast::types::StaticType;
fn check_source(source: &str) -> TypedNode {
let env = crate::ast::environment::Environment::new();
env.compile(source).into_result().unwrap()
}
#[test]
fn test_destructuring_scalar_error() {
let env = crate::ast::environment::Environment::new();
let result = env.compile("(def [x] 5)").into_result();
assert!(result.is_err());
assert_eq!(
result.unwrap_err(),
"Cannot destructure type int as a tuple/vector"
);
}
#[test]
fn test_call_argument_mismatch() {
let env = crate::ast::environment::Environment::new();
// fn takes 1 arg, called with 0
let result = env.compile("(do (def f (fn [x] x)) (f))").into_result();
assert!(result.is_err());
assert!(
result
.unwrap_err()
.contains("Invalid arguments for function call")
);
}
#[test]
fn test_destructuring_vector_args() {
let env = crate::ast::environment::Environment::new();
// ((fn [[x y]] (+ x y)) [10 20]) -> 30
let result = env.compile("((fn [[x y]] (+ x y)) [10 20])").into_result();
assert!(result.is_ok());
assert_eq!(get_ret_type(&result.unwrap()), StaticType::Int);
}
fn get_ret_type(node: &TypedNode) -> StaticType {
if let StaticType::Function(sig) = &node.ty {
sig.ret.clone()
} else {
node.ty.clone()
}
}
#[test]
fn test_inference_constants() {
assert_eq!(get_ret_type(&check_source("10")), StaticType::Int);
assert_eq!(get_ret_type(&check_source("10.5")), StaticType::Float);
assert_eq!(get_ret_type(&check_source("true")), StaticType::Bool);
assert_eq!(get_ret_type(&check_source("\"hello\"")), StaticType::Text);
}
#[test]
fn test_inference_variable_propagation() {
// (do (def x 10) x) -> The last 'x' must be Int
let typed = check_source("(do (def x 10) x)");
// Outer is Lambda, Body is Block
if let BoundKind::Lambda { body, .. } = &typed.kind {
if let BoundKind::Block { exprs } = &body.kind {
let last_expr = exprs.last().unwrap();
assert_eq!(
last_expr.ty,
StaticType::Int,
"Variable 'x' should be inferred as Int"
);
} else {
panic!("Expected block in lambda body");
}
} else {
panic!("Expected Lambda wrapper");
}
}
#[test]
fn test_inference_block_type() {
// Block type = last expression type
assert_eq!(get_ret_type(&check_source("(do 1 2.5)")), StaticType::Float);
assert_eq!(
get_ret_type(&check_source("(do 1.5 \"test\")")),
StaticType::Text
);
}
#[test]
fn test_inference_lambda_return() {
// (fn [a] 10) -> fn(any) -> Int
// Since it's already a Lambda, it's NOT wrapped further.
let typed = check_source("(fn [a] 10)");
if let StaticType::Function(sig) = &typed.ty {
assert_eq!(sig.ret, StaticType::Int);
} else {
panic!("Expected function type, got {:?}", typed.ty);
}
// Nested: (fn [] (do 1 2.5)) -> fn() -> Float
let typed_nested = check_source("(fn [] (do 1 2.5))");
if let StaticType::Function(sig) = &typed_nested.ty {
assert_eq!(sig.ret, StaticType::Float);
} else {
panic!("Expected function type");
}
}
#[test]
fn test_inference_assignment_updates_type() {
// (do (def x 10) (assign x 20.5) x) -> x becomes Float after assignment
let typed = check_source("(do (def x 10) (assign x 20.5) x)");
if let BoundKind::Lambda { body, .. } = &typed.kind {
if let BoundKind::Block { exprs } = &body.kind {
let last_expr = exprs.last().unwrap();
assert_eq!(
last_expr.ty,
StaticType::Float,
"Variable 'x' should be specialized to Float after assignment"
);
} else {
panic!("Expected block");
}
} else {
panic!("Expected Lambda");
}
}
#[test]
fn test_operator_overloading_inference() {
assert_eq!(get_ret_type(&check_source("(+ 1 2)")), StaticType::Int);
assert_eq!(
get_ret_type(&check_source("(+ 1.0 2.0)")),
StaticType::Float
);
assert_eq!(
get_ret_type(&check_source("(+ \"a\" \"b\")")),
StaticType::Text
);
assert_eq!(get_ret_type(&check_source("(/ 1 2)")), StaticType::Float);
}
#[test]
fn test_datetime_inference() {
// date("2023-01-01") -> DateTime
assert_eq!(
get_ret_type(&check_source("(date \"2023-01-01\")")),
StaticType::DateTime
);
// DateTime + Int -> DateTime
assert_eq!(
get_ret_type(&check_source("(+ (date \"2023-01-01\") 86400000)")),
StaticType::DateTime
);
// DateTime - DateTime -> Int (Duration)
assert_eq!(
get_ret_type(&check_source(
"(- (date \"2023-01-02\") (date \"2023-01-01\"))"
)),
StaticType::Int
);
// DateTime comparison -> Bool
assert_eq!(
get_ret_type(&check_source(
"(> (date \"2023-01-02\") (date \"2023-01-01\"))"
)),
StaticType::Bool
);
}
#[test]
fn test_inference_tuple_vector_matrix() {
// Heterogeneous -> Tuple
assert_eq!(
get_ret_type(&check_source("[1 3.14 \"text\"]")),
StaticType::Tuple(vec![StaticType::Int, StaticType::Float, StaticType::Text])
);
// Homogeneous -> Vector
assert_eq!(
get_ret_type(&check_source("[10 20 30]")),
StaticType::Vector(Box::new(StaticType::Int), 3)
);
// Nested Homogeneous -> Matrix
assert_eq!(
get_ret_type(&check_source("[[1 2] [3 4]]")),
StaticType::Matrix(Box::new(StaticType::Int), vec![2, 2])
);
// Deep Matrix
assert_eq!(
get_ret_type(&check_source("[[[1 2]] [[3 4]]]")),
StaticType::Matrix(Box::new(StaticType::Int), vec![2, 1, 2])
);
// Shape mismatch -> Tuple of Vectors
let mixed = get_ret_type(&check_source("[[1 2] [3 4 5]]"));
if let StaticType::Tuple(elements) = mixed {
assert_eq!(
elements[0],
StaticType::Vector(Box::new(StaticType::Int), 2)
);
assert_eq!(
elements[1],
StaticType::Vector(Box::new(StaticType::Int), 3)
);
} else {
panic!("Expected Tuple for shape mismatch, got {:?}", mixed);
}
}
#[test]
fn test_inference_record() {
use crate::ast::types::Keyword;
let typed = check_source("{:x 1 :y 0.3}");
let ty = get_ret_type(&typed);
if let StaticType::Record(layout) = ty {
assert_eq!(layout.fields.len(), 2);
assert_eq!(layout.fields[0], (Keyword::intern("x"), StaticType::Int));
assert_eq!(layout.fields[1], (Keyword::intern("y"), StaticType::Float));
} else {
panic!("Expected Record, got {:?}", ty);
}
}
}
+886
View File
@@ -0,0 +1,886 @@
use crate::ast::diagnostics::Diagnostics;
use crate::ast::nodes::{
Address, AssignBinding, BoundLike, DefBinding, IdentifierBinding, LambdaBinding,
Node, NodeKind, TypedNode, TypedPhase,
};
use crate::ast::types::{Keyword, RecordLayout, Signature, StaticType};
use std::rc::Rc;
use super::context::{CheckerInferenceAccess, TypeContext, extract_lambda_param_hints};
use super::TypeChecker;
impl TypeChecker {
/// Types a lambda node using externally provided parameter type hints,
/// while preserving the current scope's upvalue types.
/// Unlike `check_node_as_bound`, this keeps the enclosing `TypeContext` as parent,
/// so captured variables retain their inferred types.
pub(super) fn check_lambda_with_param_hints<P: BoundLike>(
&self,
node: &Node<P>,
param_hints: &[StaticType],
ctx: &mut TypeContext,
diag: &mut Diagnostics,
) -> TypedNode {
let NodeKind::Lambda {
params,
body,
info,
} = &node.kind
else {
return self.check_node(node, ctx, diag);
};
let upvalues = &info.upvalues;
let positional_count = info.positional_count;
let mut upvalue_types = Vec::with_capacity(upvalues.len());
for &addr in upvalues {
upvalue_types.push(ctx.get_type(addr));
}
let mut lambda_ctx = TypeContext::new(64, upvalue_types, ctx.root_types, Some(ctx));
let hint_ty = StaticType::Tuple(param_hints.to_vec());
let params_typed = self.check_params(params.as_ref(), &hint_ty, &mut lambda_ctx, diag);
lambda_ctx.current_params_ty = Some(params_typed.ty.clone());
let body_typed = self.check_node(body, &mut lambda_ctx, diag);
let ret_ty = body_typed.ty.clone();
let fn_ty = StaticType::Function(Box::new(Signature {
params: params_typed.ty.clone(),
ret: ret_ty,
}));
Node {
identity: node.identity.clone(),
kind: NodeKind::Lambda {
params: Rc::new(params_typed),
body: Rc::new(body_typed),
info: LambdaBinding {
upvalues: upvalues.clone(),
positional_count,
},
},
ty: fn_ty,
comments: node.comments.clone(),
}
}
pub(super) fn check_params<P: BoundLike>(
&self,
node: &Node<P>,
specialized_ty: &StaticType,
ctx: &mut TypeContext,
diag: &mut Diagnostics,
) -> TypedNode {
let (kind, ty): (NodeKind<TypedPhase>, StaticType) = match &node.kind {
NodeKind::Def {
pattern,
info,
..
} => {
if let NodeKind::Identifier {
symbol,
binding: IdentifierBinding::Declaration { addr, kind: decl_kind },
} = &pattern.kind
{
ctx.set_type(*addr, specialized_ty.clone());
(
NodeKind::Def {
pattern: Rc::new(Node {
identity: pattern.identity.clone(),
kind: NodeKind::Identifier {
symbol: symbol.clone(),
binding: IdentifierBinding::Declaration {
addr: *addr,
kind: *decl_kind,
},
},
ty: specialized_ty.clone(),
comments: pattern.comments.clone(),
}),
value: Rc::new(Node {
identity: node.identity.clone(),
kind: NodeKind::Nop,
ty: specialized_ty.clone(),
comments: Rc::from([]),
}),
info: DefBinding {
captured_by: info.captured_by.clone(),
},
},
specialized_ty.clone(),
)
} else {
// Destructuring def in params — fall through to tuple handling
// if pattern is a Tuple, handle elements
if let NodeKind::Tuple { elements } = &pattern.kind {
return self.check_params_tuple(node, elements, specialized_ty, ctx, diag);
}
diag.push_error(
"Invalid pattern in parameter definition",
Some(node.identity.clone()),
);
(NodeKind::Error, StaticType::Error)
}
}
NodeKind::Assign {
info,
..
} => {
(
NodeKind::Assign {
target: Rc::new(Node {
identity: node.identity.clone(),
kind: NodeKind::Nop,
ty: specialized_ty.clone(),
comments: Rc::from([]),
}),
value: Rc::new(Node {
identity: node.identity.clone(),
kind: NodeKind::Nop,
ty: specialized_ty.clone(),
comments: Rc::from([]),
}),
info: AssignBinding {
addr: info.addr,
},
},
specialized_ty.clone(),
)
}
NodeKind::Identifier {
symbol,
binding: IdentifierBinding::Declaration { addr, kind: decl_kind },
} => {
ctx.set_type(*addr, specialized_ty.clone());
(
NodeKind::Identifier {
symbol: symbol.clone(),
binding: IdentifierBinding::Declaration {
addr: *addr,
kind: *decl_kind,
},
},
specialized_ty.clone(),
)
}
NodeKind::Tuple { elements } => {
return self.check_params_tuple(node, elements, specialized_ty, ctx, diag);
}
NodeKind::Nop => (NodeKind::Nop, StaticType::Void),
NodeKind::Error => (NodeKind::Error, StaticType::Error),
// All remaining variants are invalid in a parameter list.
// Identifier::Reference should not appear in parameter patterns.
NodeKind::Identifier { .. }
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::GetField { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Record { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {
diag.push_error(
"Invalid node in parameter list",
Some(node.identity.clone()),
);
(NodeKind::Error, StaticType::Error)
}
};
Node {
identity: node.identity.clone(),
kind,
ty,
comments: node.comments.clone(),
}
}
fn check_params_tuple<P: BoundLike>(
&self,
node: &Node<P>,
elements: &[Rc<Node<P>>],
specialized_ty: &StaticType,
ctx: &mut TypeContext,
diag: &mut Diagnostics,
) -> TypedNode {
match specialized_ty {
StaticType::Any
| StaticType::TypeVar(_) // TypeVar may resolve to any destructurable type
| StaticType::Tuple(_)
| StaticType::Vector(_, _)
| StaticType::Matrix(_, _)
| StaticType::List(_)
| StaticType::Record(_)
| StaticType::Error => {}
_ => {
diag.push_error(
format!(
"Cannot destructure type {} as a tuple/vector",
specialized_ty.display_compact()
),
Some(node.identity.clone()),
);
return Node {
identity: node.identity.clone(),
kind: NodeKind::Error,
ty: StaticType::Error,
comments: node.comments.clone(),
};
}
}
let mut typed_elements = Vec::new();
let mut elem_types = Vec::new();
for (i, el) in elements.iter().enumerate() {
let sub_ty = match specialized_ty {
StaticType::Tuple(t) => t.get(i).cloned().unwrap_or(StaticType::Any),
StaticType::Vector(inner, _) => (**inner).clone(),
StaticType::Matrix(inner, _) => (**inner).clone(),
StaticType::List(inner) => (**inner).clone(),
StaticType::Record(layout) => layout
.fields
.get(i)
.map(|(_, ty): &(Keyword, StaticType)| ty.clone())
.unwrap_or(StaticType::Any),
StaticType::Error => StaticType::Error,
_ => StaticType::Any,
};
let t = self.check_params(el.as_ref(), &sub_ty, ctx, diag);
elem_types.push(t.ty.clone());
typed_elements.push(Rc::new(t));
}
Node {
identity: node.identity.clone(),
kind: NodeKind::Tuple {
elements: typed_elements,
},
ty: StaticType::Tuple(elem_types),
comments: node.comments.clone(),
}
}
pub(super) fn check_node<P: BoundLike>(
&self,
node: &Node<P>,
ctx: &mut TypeContext,
diag: &mut Diagnostics,
) -> TypedNode {
let (kind, ty): (NodeKind<TypedPhase>, StaticType) = match &node.kind {
NodeKind::Nop => (NodeKind::Nop, StaticType::Void),
NodeKind::Constant(v) => {
let ty = v.static_type();
(NodeKind::Constant(v.clone()), ty)
}
NodeKind::Def {
pattern,
value,
info,
} => {
let val_typed = self.check_node(value, ctx, diag);
let ty = val_typed.ty.clone();
// Extract addr from pattern to register the type.
// Value restriction: only Function-typed values are generalized to Forall.
// Mutable state (Series, scalars) must remain monomorphic.
if let NodeKind::Identifier {
binding: IdentifierBinding::Declaration { addr, .. },
..
} = &pattern.kind
{
let stored_ty = if matches!(ty, StaticType::Function(..)) {
self.generalize(ty.clone(), ctx)
} else {
ty.clone()
};
ctx.set_type(*addr, stored_ty);
}
// For destructuring defs, check params on the pattern
if let NodeKind::Tuple { .. } = &pattern.kind {
let pat_typed = self.check_params(pattern.as_ref(), &val_typed.ty, ctx, diag);
let ty = val_typed.ty.clone();
return Node {
identity: node.identity.clone(),
kind: NodeKind::Def {
pattern: Rc::new(pat_typed),
value: Rc::new(val_typed),
info: DefBinding {
captured_by: info.captured_by.clone(),
},
},
ty,
comments: node.comments.clone(),
};
}
// Simple def — reconstruct the pattern node with the new type
let new_pattern: TypedNode = Node {
identity: pattern.identity.clone(),
kind: match &pattern.kind {
NodeKind::Identifier { symbol, binding } => NodeKind::Identifier {
symbol: symbol.clone(),
binding: match binding {
IdentifierBinding::Declaration { addr, kind: decl_kind } => {
IdentifierBinding::Declaration {
addr: *addr,
kind: *decl_kind,
}
}
IdentifierBinding::Reference(addr) => {
IdentifierBinding::Reference(*addr)
}
},
},
// Tuple patterns are handled above with an early return.
// All other variants are invalid as def patterns.
NodeKind::Nop
| NodeKind::Constant(_)
| NodeKind::FieldAccessor(_)
| NodeKind::Def { .. }
| NodeKind::Assign { .. }
| NodeKind::Lambda { .. }
| NodeKind::Call { .. }
| NodeKind::Again { .. }
| NodeKind::If { .. }
| NodeKind::Block { .. }
| NodeKind::Program { .. }
| NodeKind::Tuple { .. }
| NodeKind::Record { .. }
| NodeKind::GetField { .. }
| NodeKind::Expansion { .. }
| NodeKind::Extension(_)
| NodeKind::Error
| NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => NodeKind::Error,
},
ty: ty.clone(),
comments: pattern.comments.clone(),
};
(
NodeKind::Def {
pattern: Rc::new(new_pattern),
value: Rc::new(val_typed),
info: DefBinding {
captured_by: info.captured_by.clone(),
},
},
ty,
)
}
NodeKind::Identifier { symbol, binding } => {
if let IdentifierBinding::Reference(addr) = binding {
// Apply the current HM substitution so that TypeVars resolved in
// nested scopes (e.g. inside a `while` body) are visible here even
// when ctx.set_type could not propagate back through an upvalue address.
// Instantiate Forall types: each use site gets fresh TypeVars so that
// calls with different argument types remain independent.
let ty = Self::apply_subst(ctx.get_type(*addr), &self.subst.borrow());
let ty = self.instantiate(ty);
(
NodeKind::Identifier {
symbol: symbol.clone(),
binding: IdentifierBinding::Reference(*addr),
},
ty,
)
} else if let IdentifierBinding::Declaration { addr, kind: decl_kind } = binding {
let ty = Self::apply_subst(ctx.get_type(*addr), &self.subst.borrow());
(
NodeKind::Identifier {
symbol: symbol.clone(),
binding: IdentifierBinding::Declaration {
addr: *addr,
kind: *decl_kind,
},
},
ty,
)
} else {
(NodeKind::Error, StaticType::Error)
}
}
NodeKind::FieldAccessor(k) => {
(NodeKind::FieldAccessor(*k), StaticType::FieldAccessor(*k))
}
NodeKind::GetField { rec, field } => {
let rec_typed = self.check_node(rec, ctx, diag);
let field_ty = match &rec_typed.ty {
StaticType::Record(layout) => {
if let Some(idx) = layout.index_of(*field) {
layout.fields[idx].1.clone()
} else {
diag.push_error(
format!("Record does not have field :{}", field.name()),
Some(rec_typed.identity.clone()),
);
StaticType::Error
}
}
StaticType::Any => StaticType::Any,
StaticType::Error => StaticType::Error,
_ => {
diag.push_error(
format!(
"Cannot access field :{} on non-record type {}",
field.name(),
rec_typed.ty.display_compact()
),
Some(rec_typed.identity.clone()),
);
StaticType::Error
}
};
(
NodeKind::GetField {
rec: Rc::new(rec_typed),
field: *field,
},
field_ty,
)
}
NodeKind::Assign { target, value, info } => {
let val_typed = self.check_node(value, ctx, diag);
let ty = val_typed.ty.clone();
if let Some(addr) = info.addr {
ctx.set_type(addr, ty.clone());
}
// For destructuring assigns (addr = None), preserve the target pattern
// so the VM can unpack values. For simple assigns, the target is unused.
let target_typed = if info.addr.is_none() {
Rc::new(self.check_node(target, ctx, diag))
} else {
Rc::new(Node {
identity: node.identity.clone(),
kind: NodeKind::Nop,
ty: ty.clone(),
comments: Rc::from([]),
})
};
(
NodeKind::Assign {
target: target_typed,
value: Rc::new(val_typed),
info: AssignBinding {
addr: info.addr,
},
},
ty,
)
}
NodeKind::If {
cond,
then_br,
else_br,
} => {
let cond_typed = self.check_node(cond, ctx, diag);
let then_typed = self.check_node(then_br, ctx, diag);
let mut else_typed = None;
let mut final_ty = then_typed.ty.clone();
if let Some(e) = else_br {
let et = self.check_node(e, ctx, diag);
if et.ty != final_ty {
final_ty = Self::numeric_widen(&final_ty, &et.ty)
.or_else(|| Self::record_promote(&final_ty, &et.ty))
.unwrap_or(StaticType::Any);
}
else_typed = Some(Rc::new(et));
} else {
final_ty = StaticType::Optional(Box::new(then_typed.ty.clone()));
}
(
NodeKind::If {
cond: Rc::new(cond_typed),
then_br: Rc::new(then_typed),
else_br: else_typed,
},
final_ty,
)
}
NodeKind::Block { exprs } | NodeKind::Program { exprs } => {
let is_program = matches!(&node.kind, NodeKind::Program { .. });
let mut typed_exprs = Vec::new();
let mut last_ty = StaticType::Void;
for e in exprs {
let t = self.check_node(e, ctx, diag);
last_ty = t.ty.clone();
typed_exprs.push(Rc::new(t));
}
let kind = if is_program {
NodeKind::Program { exprs: typed_exprs }
} else {
NodeKind::Block { exprs: typed_exprs }
};
(kind, last_ty)
}
NodeKind::Lambda {
params,
body,
info,
} => {
let upvalues = &info.upvalues;
let positional_count = info.positional_count;
let mut upvalue_types = Vec::with_capacity(upvalues.len());
for &addr in upvalues {
upvalue_types.push(ctx.get_type(addr));
}
let mut lambda_ctx =
TypeContext::new(64, upvalue_types, ctx.root_types, Some(ctx));
let param_hint_ty = StaticType::Tuple(
(0..positional_count.unwrap_or(0)).map(|_| self.fresh_var()).collect(),
);
let params_typed = self.check_params(
params.as_ref(),
&param_hint_ty,
&mut lambda_ctx,
diag,
);
lambda_ctx.current_params_ty = Some(params_typed.ty.clone());
let body_typed = self.check_node(body, &mut lambda_ctx, diag);
let ret_ty = body_typed.ty.clone();
let fn_ty = StaticType::Function(Box::new(Signature {
params: params_typed.ty.clone(),
ret: ret_ty,
}));
(
NodeKind::Lambda {
params: Rc::new(params_typed),
body: Rc::new(body_typed),
info: LambdaBinding {
upvalues: upvalues.clone(),
positional_count,
},
},
fn_ty,
)
}
NodeKind::Call { callee, args } => {
let callee_typed = self.check_node(callee, ctx, diag);
let args_typed = if let NodeKind::Tuple { elements } = &args.kind {
let arg_count = elements.len();
let mut typed_elements: Vec<Option<Rc<TypedNode>>> = vec![None; arg_count];
let mut known_types: Vec<Option<StaticType>> = vec![None; arg_count];
let mut lambda_indices = Vec::new();
// Phase 1: Type non-lambda arguments first
for (i, e) in elements.iter().enumerate() {
if matches!(e.kind, NodeKind::Lambda { .. }) {
lambda_indices.push(i);
} else {
let t = self.check_node(e, ctx, diag);
known_types[i] = Some(t.ty.clone());
typed_elements[i] = Some(Rc::new(t));
}
}
// Phase 2: Type lambda arguments with parameter hints (if available)
for i in lambda_indices {
let hints = extract_lambda_param_hints(
&callee_typed.ty,
i,
&known_types,
);
let t = if let Some(param_types) = hints {
self.check_lambda_with_param_hints(
&elements[i],
&param_types,
ctx,
diag,
)
} else {
self.check_node(&elements[i], ctx, diag)
};
known_types[i] = Some(t.ty.clone());
typed_elements[i] = Some(Rc::new(t));
}
let final_elements: Vec<Rc<TypedNode>> = typed_elements
.into_iter()
.map(|e| e.expect("all args should be typed"))
.collect();
let elem_types: Vec<StaticType> =
final_elements.iter().map(|e| e.ty.clone()).collect();
Node {
identity: args.identity.clone(),
kind: NodeKind::Tuple {
elements: final_elements,
},
ty: StaticType::Tuple(elem_types),
comments: args.comments.clone(),
}
} else {
self.check_node(args, ctx, diag)
};
let mut ret_ty = match callee_typed.ty.resolve_call(&args_typed.ty) {
Some(ty) => ty,
None => {
let callee_name = match &callee_typed.kind {
NodeKind::Identifier { symbol, .. } => format!("'{}'", symbol.name),
_ => "function".to_string(),
};
diag.push_error(
format!(
"{}: no matching overload for ({})",
callee_name, args_typed.ty.display_compact()
),
Some(node.identity.clone()),
);
StaticType::Error
}
};
// HM: propagate TypeVar constraints through overloaded calls so that
// e.g. `(+ Float TypeVar(1))` resolves TypeVar(1) = Float.
self.unify_matched_overload(&callee_typed.ty, &args_typed.ty, diag);
// HM step 9: when a TypeVar is called with a single Int argument
// (series lookback indexing pattern), record a lazy index-call constraint
// instead of eagerly unifying `TypeVar = Series(elem)`.
//
// The constraint pair (callee_var → result_var) is stored in
// `index_constraints`. When `bind_var` later resolves callee_var to
// `Series(inner)`, it automatically binds result_var to `inner`.
//
// This avoids over-constraining the function to Series-only: passing
// any other callable (e.g. a Function) simply leaves result_var
// unresolved and the call returns `Any` — no spurious type error.
if let StaticType::TypeVar(n) = &callee_typed.ty {
let is_index_call = matches!(&args_typed.ty,
StaticType::Tuple(elems) if elems.len() == 1
&& matches!(&elems[0], StaticType::Int)
);
if is_index_call && matches!(ret_ty, StaticType::Any) {
let existing = self.index_constraints.borrow().get(n).copied();
if let Some(existing_ret_id) = existing {
// Same TypeVar indexed again — all index results on the same
// parameter must share one element TypeVar. Unify the fresh
// var with the existing one so they resolve together.
let elem_var = self.fresh_var();
self.unify(elem_var.clone(), StaticType::TypeVar(existing_ret_id), diag);
ret_ty = Self::apply_subst(elem_var, &self.subst.borrow());
} else {
let elem_var = self.fresh_var();
let StaticType::TypeVar(elem_id) = &elem_var else { unreachable!() };
self.index_constraints.borrow_mut().insert(*n, *elem_id);
ret_ty = elem_var;
}
}
}
// HM step 10: unify Function parameter types with actual argument types,
// but only when the signature still contains TypeVars to resolve.
// Skip for fully concrete signatures (e.g. fn([any any])) to avoid
// false conflicts between Tuple and Vector representations.
if let StaticType::Function(sig) = &callee_typed.ty
&& Self::has_typevar_component(&sig.params)
{
let params = sig.params.clone();
self.unify(params, args_typed.ty.clone(), diag);
ret_ty = Self::apply_subst(ret_ty, &self.subst.borrow());
}
// Dispatch compiler hooks registered by the RTL (keyed by global slot index).
// Hooks handle type-inference extensions such as:
// - series: inject a fresh TypeVar for the element type
// - push: unify the series element TypeVar with the pushed value type
if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(idx)),
..
} = &callee_typed.kind
&& let Some(hook) = self.compiler_hooks.get(&idx.0) {
let hook_ctx = CheckerInferenceAccess { checker: self, ctx };
ret_ty = hook.post_call(&args_typed, ret_ty, &hook_ctx, diag);
}
(
NodeKind::Call {
callee: Rc::new(callee_typed),
args: Rc::new(args_typed),
},
ret_ty,
)
}
NodeKind::Again { args } => {
let args_typed = if let NodeKind::Tuple { elements } = &args.kind {
let mut typed_elements = Vec::new();
let mut elem_types = Vec::new();
for e in elements {
let t = self.check_node(e, ctx, diag);
elem_types.push(t.ty.clone());
typed_elements.push(Rc::new(t));
}
Node {
identity: args.identity.clone(),
kind: NodeKind::Tuple {
elements: typed_elements,
},
ty: StaticType::Tuple(elem_types),
comments: args.comments.clone(),
}
} else {
self.check_node(args, ctx, diag)
};
if let Some(expected_ty) = &ctx.current_params_ty {
self.unify(expected_ty.clone(), args_typed.ty.clone(), diag);
}
(
NodeKind::Again {
args: Rc::new(args_typed),
},
StaticType::Any,
)
}
NodeKind::Tuple { elements } => {
let mut typed_elements = Vec::new();
for e in elements {
typed_elements.push(Rc::new(self.check_node(e, ctx, diag)));
}
let ty = if typed_elements.is_empty() {
StaticType::Vector(Box::new(StaticType::Any), 0)
} else {
let first_ty = &typed_elements[0].ty;
let all_same = typed_elements.iter().all(|e| e.ty == *first_ty);
if all_same {
match first_ty {
StaticType::Vector(inner, len) => {
StaticType::Matrix(inner.clone(), vec![typed_elements.len(), *len])
}
StaticType::Matrix(inner, shape) => {
let mut new_shape = vec![typed_elements.len()];
new_shape.extend(shape);
StaticType::Matrix(inner.clone(), new_shape)
}
_ => {
StaticType::Vector(Box::new(first_ty.clone()), typed_elements.len())
}
}
} else {
StaticType::Tuple(typed_elements.iter().map(|e| e.ty.clone()).collect())
}
};
(
NodeKind::Tuple {
elements: typed_elements,
},
ty,
)
}
NodeKind::Record { fields, layout } => {
let mut typed_fields = Vec::with_capacity(fields.len());
let mut fields_ty = Vec::with_capacity(fields.len());
for (i, (key_node, val_node)) in fields.iter().enumerate() {
let kt = self.check_node(key_node, ctx, diag);
let vt = self.check_node(val_node, ctx, diag);
fields_ty.push((layout.fields[i].0, vt.ty.clone()));
typed_fields.push((Rc::new(kt), Rc::new(vt)));
}
let new_layout = RecordLayout::get_or_create(fields_ty);
(
NodeKind::Record {
fields: typed_fields,
layout: new_layout.clone(),
},
StaticType::Record(new_layout),
)
}
NodeKind::Expansion {
original_call,
expanded,
} => {
let expanded_typed = self.check_node(expanded, ctx, diag);
let ty = expanded_typed.ty.clone();
(
NodeKind::Expansion {
original_call: original_call.clone(),
expanded: Rc::new(expanded_typed),
},
ty,
)
}
NodeKind::Extension(_ext) => {
diag.push_error(
format!(
"TypeChecking for extension '{}' not implemented",
_ext.display_name()
),
Some(node.identity.clone()),
);
(NodeKind::Error, StaticType::Error)
}
NodeKind::Error => (NodeKind::Error, StaticType::Error),
// Syntax-only variants should not appear in bound phases
NodeKind::MacroDecl { .. }
| NodeKind::Template(_)
| NodeKind::Placeholder(_)
| NodeKind::Splice(_) => {
diag.push_error(
"Unexpected syntax-only node in type checking",
Some(node.identity.clone()),
);
(NodeKind::Error, StaticType::Error)
}
};
Node {
identity: node.identity.clone(),
kind,
ty,
comments: node.comments.clone(),
}
}
}
+169
View File
@@ -0,0 +1,169 @@
use crate::ast::compiler::call_hooks::InferenceAccess;
use crate::ast::diagnostics::Diagnostics;
use crate::ast::nodes::{Address, VirtualId};
use crate::ast::types::{Signature, StaticType};
use std::collections::HashMap;
use super::TypeChecker;
/// Manages the types of locals and upvalues during a single type-checking pass.
pub(super) struct TypeContext<'a> {
pub(super) _parent: Option<&'a TypeContext<'a>>,
/// Maps slot index -> Inferred Type
pub(super) slots: HashMap<u32, StaticType>,
/// Types of captured variables (passed from outer scope)
pub(super) upvalue_types: Vec<StaticType>,
/// Access to root types for unified resolution
pub(super) root_types: &'a std::cell::RefCell<Vec<StaticType>>,
/// The expected parameters of the current function (for 'again' validation)
pub(super) current_params_ty: Option<StaticType>,
}
impl<'a> TypeContext<'a> {
pub(super) fn new(
_slot_count: u32,
upvalue_types: Vec<StaticType>,
root_types: &'a std::cell::RefCell<Vec<StaticType>>,
parent: Option<&'a TypeContext<'a>>,
) -> Self {
Self {
_parent: parent,
slots: HashMap::new(),
upvalue_types,
root_types,
current_params_ty: None,
}
}
pub(super) fn get_type(&self, addr: Address<VirtualId>) -> StaticType {
match addr {
Address::Local(slot) => self
.slots
.get(&slot.0)
.cloned()
.unwrap_or(StaticType::Any),
Address::Global(idx) => self
.root_types
.borrow()
.get(idx.0 as usize)
.cloned()
.unwrap_or(StaticType::Any),
Address::Upvalue(idx) => self
.upvalue_types
.get(idx.0 as usize)
.cloned()
.unwrap_or(StaticType::Any),
}
}
pub(super) fn set_type(&mut self, addr: Address<VirtualId>, ty: StaticType) {
match addr {
Address::Local(slot) => {
self.slots.insert(slot.0, ty);
}
Address::Global(idx) => {
let mut rt = self.root_types.borrow_mut();
let i = idx.0 as usize;
if i >= rt.len() {
rt.resize(i + 1, StaticType::Any);
}
rt[i] = ty;
}
_ => {}
}
}
}
/// Temporary wrapper that gives call-hooks unified access to both the
/// type-checker's inference state and the current scope's slot types.
/// Created on the stack at each hook dispatch site; zero allocation cost.
pub(super) struct CheckerInferenceAccess<'a, 'b> {
pub(super) checker: &'a TypeChecker,
pub(super) ctx: &'b TypeContext<'a>,
}
impl InferenceAccess for CheckerInferenceAccess<'_, '_> {
fn fresh_var(&self) -> StaticType {
self.checker.fresh_var()
}
fn unify(&self, a: StaticType, b: StaticType, diag: &mut Diagnostics) {
self.checker.unify(a, b, diag);
}
fn bind_typevar(&self, id: u32, ty: StaticType) {
self.checker.bind_var(id, ty);
}
fn apply_subst_ty(&self, ty: StaticType) -> StaticType {
TypeChecker::apply_subst(ty, &self.checker.subst.borrow())
}
fn try_numeric_widen(&self, a: &StaticType, b: &StaticType) -> Option<StaticType> {
TypeChecker::numeric_widen(a, b)
}
fn try_record_promote(&self, a: &StaticType, b: &StaticType) -> Option<StaticType> {
TypeChecker::record_promote(a, b)
}
fn get_slot_type(&self, addr: Address<VirtualId>) -> StaticType {
self.ctx.get_type(addr)
}
}
/// Extracts expected lambda parameter types from the callee type for a specific argument position.
/// Used for bidirectional type inference: when a Call's callee expects a function at `arg_index`,
/// this returns the expected parameter types for that function, derived from the callee's signature
/// and the already-known types of non-lambda arguments.
pub(super) fn extract_lambda_param_hints(
callee_ty: &StaticType,
arg_index: usize,
known_arg_types: &[Option<StaticType>],
) -> Option<Vec<StaticType>> {
/// Helper: given the expected type at a specific parameter position in a signature,
/// extract the lambda parameter types if it expects a function.
fn hints_from_param_type(param_ty: &StaticType) -> Option<Vec<StaticType>> {
if let StaticType::Function(sig) = param_ty {
if let StaticType::Tuple(params) = &sig.params {
return Some(params.clone());
}
return Some(vec![sig.params.clone()]);
}
None
}
/// Helper: extract the expected type at `arg_index` from a signature's params tuple.
fn param_at(sig: &Signature, arg_index: usize) -> Option<&StaticType> {
if let StaticType::Tuple(params) = &sig.params {
params.get(arg_index)
} else if arg_index == 0 {
Some(&sig.params)
} else {
None
}
}
match callee_ty {
StaticType::Function(sig) => {
let expected = param_at(sig, arg_index)?;
hints_from_param_type(expected)
}
StaticType::FunctionOverloads(sigs) => {
// Try each overload — return hints from the first one that has a function at this position
for sig in sigs {
if let Some(expected) = param_at(sig, arg_index)
&& let Some(hints) = hints_from_param_type(expected)
{
return Some(hints);
}
}
None
}
StaticType::PolymorphicFn {
resolve_arg_hints: Some(resolver),
..
} => resolver(arg_index, known_arg_types),
_ => None,
}
}
+141
View File
@@ -0,0 +1,141 @@
use crate::ast::nodes::{
Address, IdentifierBinding, Node, NodeKind, TypedNode, TypedPhase,
};
use crate::ast::types::{NodeIdentity, StaticType};
use std::collections::HashMap;
use std::rc::Rc;
use super::TypeChecker;
impl TypeChecker {
/// Applies the final HM substitution to all type annotations in the tree and
/// elaborates `(series n)` calls with their inferred schema arguments.
/// Must be called after `check()` or `check_node_as_bound()` completes.
pub fn finalize(&self, node: TypedNode) -> TypedNode {
let subst = self.subst.borrow().clone();
self.finalize_node(node, &subst)
}
/// Walks the typed AST, applies the HM substitution to every type annotation,
/// and dispatches finalization hooks (e.g. schema injection for `series`).
fn finalize_node(&self, node: TypedNode, subst: &HashMap<u32, StaticType>) -> TypedNode {
let new_ty = Self::apply_subst(node.ty, subst);
let new_kind = self.finalize_kind(node.kind, subst, &new_ty, &node.identity);
Node { kind: new_kind, ty: new_ty, identity: node.identity, comments: node.comments }
}
fn finalize_kind(
&self,
kind: NodeKind<TypedPhase>,
subst: &HashMap<u32, StaticType>,
node_ty: &StaticType,
_identity: &Rc<NodeIdentity>,
) -> NodeKind<TypedPhase> {
match kind {
// Leaf nodes — nothing to recurse into
NodeKind::Nop => NodeKind::Nop,
NodeKind::Constant(v) => NodeKind::Constant(v),
NodeKind::FieldAccessor(k) => NodeKind::FieldAccessor(k),
NodeKind::Error => NodeKind::Error,
NodeKind::Identifier { symbol, binding } => NodeKind::Identifier { symbol, binding },
NodeKind::Extension(ext) => NodeKind::Extension(ext),
// Call: dispatch finalize hook registered by RTL (keyed by global slot index).
// Hooks may rewrite the call node — e.g. series injects a schema argument.
NodeKind::Call { callee, args } => {
let callee_fin = Rc::new(self.finalize_node((*callee).clone(), subst));
let args_fin = Rc::new(self.finalize_node((*args).clone(), subst));
if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(Address::Global(idx)),
..
} = &callee_fin.kind
&& let Some(hook) = self.compiler_hooks.get(&idx.0)
&& let Some(new_kind) = hook.finalize(
Rc::clone(&callee_fin),
Rc::clone(&args_fin),
node_ty,
subst,
) {
return new_kind;
}
NodeKind::Call { callee: callee_fin, args: args_fin }
}
NodeKind::If { cond, then_br, else_br } => NodeKind::If {
cond: Rc::new(self.finalize_node((*cond).clone(), subst)),
then_br: Rc::new(self.finalize_node((*then_br).clone(), subst)),
else_br: else_br.map(|e| Rc::new(self.finalize_node((*e).clone(), subst))),
},
NodeKind::Def { pattern, value, info } => NodeKind::Def {
pattern: Rc::new(self.finalize_node((*pattern).clone(), subst)),
value: Rc::new(self.finalize_node((*value).clone(), subst)),
info,
},
NodeKind::Assign { target, value, info } => NodeKind::Assign {
target: Rc::new(self.finalize_node((*target).clone(), subst)),
value: Rc::new(self.finalize_node((*value).clone(), subst)),
info,
},
NodeKind::Lambda { params, body, info } => NodeKind::Lambda {
params: Rc::new(self.finalize_node((*params).clone(), subst)),
body: Rc::new(self.finalize_node((*body).clone(), subst)),
info,
},
NodeKind::Again { args } => NodeKind::Again {
args: Rc::new(self.finalize_node((*args).clone(), subst)),
},
NodeKind::Block { exprs } => NodeKind::Block {
exprs: exprs
.into_iter()
.map(|e| Rc::new(self.finalize_node((*e).clone(), subst)))
.collect(),
},
NodeKind::Program { exprs } => NodeKind::Program {
exprs: exprs
.into_iter()
.map(|e| Rc::new(self.finalize_node((*e).clone(), subst)))
.collect(),
},
NodeKind::Tuple { elements } => NodeKind::Tuple {
elements: elements
.into_iter()
.map(|e| Rc::new(self.finalize_node((*e).clone(), subst)))
.collect(),
},
NodeKind::Record { fields, layout } => NodeKind::Record {
fields: fields
.into_iter()
.map(|(k, v)| {
(
Rc::new(self.finalize_node((*k).clone(), subst)),
Rc::new(self.finalize_node((*v).clone(), subst)),
)
})
.collect(),
layout,
},
NodeKind::GetField { rec, field } => NodeKind::GetField {
rec: Rc::new(self.finalize_node((*rec).clone(), subst)),
field,
},
NodeKind::Expansion { original_call, expanded } => NodeKind::Expansion {
original_call,
expanded: Rc::new(self.finalize_node((*expanded).clone(), subst)),
},
NodeKind::MacroDecl { name, params, body } => NodeKind::MacroDecl {
name,
params: Rc::new(self.finalize_node((*params).clone(), subst)),
body: Rc::new(self.finalize_node((*body).clone(), subst)),
},
NodeKind::Template(inner) => {
NodeKind::Template(Rc::new(self.finalize_node((*inner).clone(), subst)))
}
NodeKind::Placeholder(inner) => {
NodeKind::Placeholder(Rc::new(self.finalize_node((*inner).clone(), subst)))
}
NodeKind::Splice(inner) => {
NodeKind::Splice(Rc::new(self.finalize_node((*inner).clone(), subst)))
}
}
}
}
+406
View File
@@ -0,0 +1,406 @@
use crate::ast::diagnostics::Diagnostics;
use crate::ast::types::{RecordLayout, Signature, StaticType};
use std::collections::HashMap;
use super::context::TypeContext;
use super::TypeChecker;
impl TypeChecker {
/// Creates a fresh, unique type variable.
pub(super) fn fresh_var(&self) -> StaticType {
let id = self.var_counter.get();
self.var_counter.set(id + 1);
StaticType::TypeVar(id)
}
/// Binds a TypeVar to a concrete type in the substitution, then propagates
/// any pending index-call constraints registered for the series lookback pattern.
///
/// When `index_constraints[id]` is set and `ty` is a `Series`, the result TypeVar
/// is bound to the element type — connecting the callee TypeVar to its element type
/// without the eager over-constraint of unification. Non-indexable types leave the
/// result TypeVar unresolved.
pub(super) fn bind_var(&self, id: u32, ty: StaticType) {
self.subst.borrow_mut().insert(id, ty.clone());
let ret_id = self.index_constraints.borrow().get(&id).copied();
if let (Some(ret_id), StaticType::Series(elem)) = (ret_id, &ty) {
self.subst.borrow_mut().insert(ret_id, *elem.clone());
}
}
/// Recursively applies the substitution map to a type, replacing all
/// resolved `TypeVar`s with their concrete types.
pub(super) fn apply_subst(ty: StaticType, subst: &HashMap<u32, StaticType>) -> StaticType {
match ty {
StaticType::TypeVar(n) => {
if let Some(resolved) = subst.get(&n) {
// Guard against self-referential substitutions (TypeVar(n) → TypeVar(n))
// which would cause infinite recursion.
if *resolved == StaticType::TypeVar(n) {
return StaticType::TypeVar(n);
}
// Follow the chain (handles transitive substitutions)
Self::apply_subst(resolved.clone(), subst)
} else {
StaticType::TypeVar(n)
}
}
StaticType::Series(inner) => {
StaticType::Series(Box::new(Self::apply_subst(*inner, subst)))
}
StaticType::Stream(inner) => {
StaticType::Stream(Box::new(Self::apply_subst(*inner, subst)))
}
StaticType::Optional(inner) => {
StaticType::Optional(Box::new(Self::apply_subst(*inner, subst)))
}
StaticType::Function(sig) => StaticType::Function(Box::new(Signature {
params: Self::apply_subst(sig.params, subst),
ret: Self::apply_subst(sig.ret, subst),
})),
StaticType::Tuple(elems) => {
StaticType::Tuple(elems.into_iter().map(|t| Self::apply_subst(t, subst)).collect())
}
// Substitute into the body but skip over the bound vars: they are local to
// this schema and must not be replaced by the global substitution.
StaticType::Forall(vars, body) => {
let filtered: HashMap<u32, StaticType> =
subst.iter().filter(|(k, _)| !vars.contains(k)).map(|(k, v)| (*k, v.clone())).collect();
StaticType::Forall(vars, Box::new(Self::apply_subst(*body, &filtered)))
}
other => other,
}
}
/// Returns true if `TypeVar(var_id)` appears anywhere in `ty` under the
/// current substitution. Used to prevent infinite types (occurs check).
fn occurs(var_id: u32, ty: &StaticType, subst: &HashMap<u32, StaticType>) -> bool {
match ty {
StaticType::TypeVar(n) => {
if *n == var_id {
return true;
}
// Follow chain in substitution
if let Some(resolved) = subst.get(n) {
Self::occurs(var_id, resolved, subst)
} else {
false
}
}
StaticType::Series(inner)
| StaticType::Stream(inner)
| StaticType::Optional(inner) => Self::occurs(var_id, inner, subst),
StaticType::Function(sig) => {
Self::occurs(var_id, &sig.params, subst)
|| Self::occurs(var_id, &sig.ret, subst)
}
StaticType::Tuple(elems) => elems.iter().any(|t| Self::occurs(var_id, t, subst)),
// A bound var inside Forall does not count as a free occurrence.
StaticType::Forall(vars, body) => {
!vars.contains(&var_id) && Self::occurs(var_id, body, subst)
}
_ => false,
}
}
/// Unifies two types under the current substitution.
/// On success, the substitution is extended so that `ty1` and `ty2` become equal.
/// On failure (type mismatch or occurs check), a diagnostic error is emitted.
pub(super) fn unify(&self, ty1: StaticType, ty2: StaticType, diag: &mut Diagnostics) {
let subst = self.subst.borrow_mut();
let ty1 = Self::apply_subst(ty1, &subst);
let ty2 = Self::apply_subst(ty2, &subst);
match (ty1, ty2) {
(a, b) if a == b => {}
(StaticType::TypeVar(n), ty) | (ty, StaticType::TypeVar(n)) => {
if Self::occurs(n, &ty, &subst) {
diag.push_error(format!("Infinite type: ?{} = {}", n, ty.display_compact()), None);
return;
}
// Release the borrow before routing through bind_var so that
// constraint propagation can re-borrow subst without a panic.
drop(subst);
self.bind_var(n, ty);
}
(StaticType::Series(a), StaticType::Series(b)) => {
drop(subst);
self.unify(*a, *b, diag);
}
(StaticType::Stream(a), StaticType::Stream(b)) => {
drop(subst);
self.unify(*a, *b, diag);
}
(StaticType::Optional(a), StaticType::Optional(b)) => {
drop(subst);
self.unify(*a, *b, diag);
}
// Unify element-wise so that overload resolution can propagate TypeVar
// constraints: e.g. `(+ Float TypeVar(1))` matched against `(Float, Float)`
// triggers `unify(Float, TypeVar(1))` → `subst[1] = Float`.
(StaticType::Tuple(a), StaticType::Tuple(b)) => {
drop(subst);
for (ta, tb) in a.into_iter().zip(b.into_iter()) {
self.unify(ta, tb, diag);
}
}
// A homogeneous Vector is assignable from a Tuple — unify each element
// with the vector's inner type. Mirrors the `is_assignable_from` coercion.
(StaticType::Tuple(elems), StaticType::Vector(inner, len))
| (StaticType::Vector(inner, len), StaticType::Tuple(elems)) => {
if elems.len() != len {
diag.push_error(
format!("Type mismatch: expected tuple of length {}, got {}", len, elems.len()),
None,
);
return;
}
drop(subst);
for e in elems {
self.unify(e, (*inner).clone(), diag);
}
}
// Variadic(T) unifies with a Tuple by unifying each element with T.
(StaticType::Variadic(inner), StaticType::Tuple(elems))
| (StaticType::Tuple(elems), StaticType::Variadic(inner)) => {
drop(subst);
for e in elems {
self.unify(e, (*inner).clone(), diag);
}
}
// Any and Error are already handled by is_assignable_from — silently succeed
(StaticType::Any, _) | (_, StaticType::Any) => {}
(StaticType::Error, _) | (_, StaticType::Error) => {}
// Forall should be instantiated before unification; delegate to the body.
(StaticType::Forall(_, body), other) | (other, StaticType::Forall(_, body)) => {
drop(subst);
self.unify(*body, other, diag);
}
(a, b) => {
diag.push_error(format!("Type mismatch: expected {}, got {}", a.display_compact(), b.display_compact()), None);
}
}
}
/// After a `FunctionOverloads` call resolves successfully, unify the matched
/// overload's concrete parameter types with the actual argument types.
///
/// This propagates TypeVar constraints through overloaded calls — for example,
/// `(+ Float TypeVar(1))` matched against `(Float, Float) → Float` binds
/// `TypeVar(1) = Float` so that nested closures can resolve series element types.
///
/// The "concrete anchor" guard ensures we only unify when at least one argument
/// is a known concrete type. Without it, `(+ TypeVar TypeVar)` would
/// spuriously pick the first overload (e.g. Int) and bind both TypeVars to Int.
pub(super) fn unify_matched_overload(
&self,
callee_ty: &StaticType,
args_ty: &StaticType,
diag: &mut Diagnostics,
) {
let StaticType::FunctionOverloads(sigs) = callee_ty else { return };
// Require both a concrete anchor (to pin the overload choice) and at
// least one TypeVar (something to actually bind). Pure concrete calls
// like `(- DateTime DateTime)` have nothing to unify and would
// incorrectly trigger errors from coercion-only compatible types.
if !Self::has_concrete_component(args_ty) || !Self::has_typevar_component(args_ty) {
return;
}
// Only unify when EXACTLY ONE non-variadic overload matches.
// If multiple overloads match (e.g. `(* TypeVar Int)` matches both
// `(Int,Int)→Int` and `(Float,Float)→Float`), the choice is ambiguous
// and binding the TypeVar to the first match would be incorrect.
let is_variadic = |sig: &Signature| matches!(sig.params, StaticType::Any | StaticType::Variadic(_));
let mut unique_match: Option<&Signature> = None;
for sig in sigs {
if !is_variadic(sig) && sig.params.is_assignable_from(args_ty) {
if unique_match.is_some() {
return; // Ambiguous — more than one specific overload matches
}
unique_match = Some(sig);
}
}
if let Some(sig) = unique_match {
self.unify(sig.params.clone(), args_ty.clone(), diag);
}
}
/// Returns true if `ty` (or any element of a Tuple) is a concrete type —
/// i.e. not `Any`, `TypeVar`, or `Error`.
pub(super) fn has_concrete_component(ty: &StaticType) -> bool {
match ty {
StaticType::Tuple(elems) => elems.iter().any(Self::is_concrete),
other => Self::is_concrete(other),
}
}
/// Returns true if `ty` contains a TypeVar anywhere in its structure.
pub(super) fn has_typevar_component(ty: &StaticType) -> bool {
match ty {
StaticType::TypeVar(_) => true,
StaticType::Tuple(elems) => elems.iter().any(Self::has_typevar_component),
StaticType::Series(inner) | StaticType::Stream(inner) | StaticType::Optional(inner) => {
Self::has_typevar_component(inner)
}
StaticType::Function(sig) => {
Self::has_typevar_component(&sig.params) || Self::has_typevar_component(&sig.ret)
}
_ => false,
}
}
fn is_concrete(ty: &StaticType) -> bool {
!matches!(ty, StaticType::Any | StaticType::TypeVar(_) | StaticType::Error)
}
/// Returns the widened numeric type when one side is `Int` and the other `Float`.
/// This is the only implicit numeric promotion in Myc: Int is a subtype of Float.
pub(super) fn numeric_widen(a: &StaticType, b: &StaticType) -> Option<StaticType> {
match (a, b) {
(StaticType::Int, StaticType::Float) | (StaticType::Float, StaticType::Int) => {
Some(StaticType::Float)
}
_ => None,
}
}
/// If both types are Records with the same field names and compatible types
/// (same type or Int→Float widening), returns the promoted Record type.
pub(super) fn record_promote(a: &StaticType, b: &StaticType) -> Option<StaticType> {
let (StaticType::Record(la), StaticType::Record(lb)) = (a, b) else {
return None;
};
if la == lb {
return None;
}
if la.fields.len() != lb.fields.len() {
return None;
}
let mut promoted_fields = Vec::with_capacity(la.fields.len());
for ((ka, ta), (kb, tb)) in la.fields.iter().zip(lb.fields.iter()) {
if ka != kb {
return None;
}
let t = if ta == tb {
ta.clone()
} else if let Some(w) = Self::numeric_widen(ta, tb) {
w
} else {
return None;
};
promoted_fields.push((*ka, t));
}
Some(StaticType::Record(RecordLayout::get_or_create(promoted_fields)))
}
/// Collects all free TypeVar IDs that appear in `ty`, following the substitution
/// chain and skipping variables bound by `Forall`. Deduplicates via `out`.
fn collect_free_tvars(ty: &StaticType, subst: &HashMap<u32, StaticType>, out: &mut Vec<u32>) {
match ty {
StaticType::TypeVar(n) => {
if let Some(resolved) = subst.get(n) {
Self::collect_free_tvars(resolved, subst, out);
} else if !out.contains(n) {
out.push(*n);
}
}
StaticType::Series(inner) | StaticType::Stream(inner) | StaticType::Optional(inner) => {
Self::collect_free_tvars(inner, subst, out);
}
StaticType::Function(sig) => {
Self::collect_free_tvars(&sig.params, subst, out);
Self::collect_free_tvars(&sig.ret, subst, out);
}
StaticType::Tuple(elems) => {
for e in elems {
Self::collect_free_tvars(e, subst, out);
}
}
// Recurse into the body but skip the locally bound vars.
StaticType::Forall(vars, body) => {
let mut body_free = Vec::new();
Self::collect_free_tvars(body, subst, &mut body_free);
for v in body_free {
if !vars.contains(&v) && !out.contains(&v) {
out.push(v);
}
}
}
_ => {}
}
}
/// Collects all free TypeVar IDs that are currently visible in `ctx`.
/// Used by `generalize` to avoid quantifying TypeVars that are still shared
/// with other live bindings (series, scalars, outer function params).
fn ctx_free_tvars(ctx: &TypeContext, subst: &HashMap<u32, StaticType>) -> Vec<u32> {
let mut out = Vec::new();
for ty in ctx.slots.values() {
Self::collect_free_tvars(ty, subst, &mut out);
}
for ty in &ctx.upvalue_types {
Self::collect_free_tvars(ty, subst, &mut out);
}
for ty in ctx.root_types.borrow().iter() {
Self::collect_free_tvars(ty, subst, &mut out);
}
out
}
/// Generalizes a type at a `def` boundary (Algorithm W `gen` step).
/// Wraps all TypeVars that are free in `ty` but NOT free in `ctx` into a
/// `Forall`. Value restriction: only call this for `Function`-typed values.
pub(super) fn generalize(&self, ty: StaticType, ctx: &TypeContext) -> StaticType {
let subst = self.subst.borrow();
let resolved = Self::apply_subst(ty, &subst);
let mut tvars_in_ty = Vec::new();
Self::collect_free_tvars(&resolved, &subst, &mut tvars_in_ty);
let ctx_tvars = Self::ctx_free_tvars(ctx, &subst);
let quantified: Vec<u32> = tvars_in_ty.into_iter()
.filter(|v| !ctx_tvars.contains(v))
.collect();
if quantified.is_empty() {
resolved
} else {
StaticType::Forall(quantified, Box::new(resolved))
}
}
/// Instantiates a `Forall` type by replacing each bound TypeVar with a
/// fresh one (Algorithm W `inst` step). No-op for non-`Forall` types.
///
/// Also remaps any index-call constraints: if `index_constraints[old] = ret`
/// and both `old` and `ret` are bound vars, the fresh copies inherit the
/// same pairing so that `bind_var` propagation keeps working at each call site.
pub(super) fn instantiate(&self, ty: StaticType) -> StaticType {
let StaticType::Forall(vars, body) = ty else { return ty };
let mut local_subst: HashMap<u32, StaticType> = HashMap::new();
let mut var_mapping: HashMap<u32, u32> = HashMap::new();
for v in &vars {
let fresh = self.fresh_var();
if let StaticType::TypeVar(fresh_id) = &fresh {
var_mapping.insert(*v, *fresh_id);
}
local_subst.insert(*v, fresh);
}
// Copy index-call constraints for the freshly created TypeVars.
// Both the callee-var and its result-var must be in vars for the
// mapping to apply; partial remaps are dropped (they can't occur in
// a well-formed Forall, but the guard is cheap insurance).
let new_constraints: Vec<(u32, u32)> = {
let constraints = self.index_constraints.borrow();
vars.iter()
.filter_map(|v| {
let new_v = *var_mapping.get(v)?;
let old_ret = *constraints.get(v)?;
let new_ret = *var_mapping.get(&old_ret)?;
Some((new_v, new_ret))
})
.collect()
};
for (new_v, new_ret) in new_constraints {
self.index_constraints.borrow_mut().insert(new_v, new_ret);
}
Self::apply_subst(*body, &local_subst)
}
}
+130
View File
@@ -0,0 +1,130 @@
mod check;
mod context;
mod finalize;
mod inference;
#[cfg(test)]
mod tests;
use crate::ast::compiler::call_hooks::RtlCompilerHook;
use crate::ast::diagnostics::Diagnostics;
use crate::ast::nodes::{BoundLike, BoundPhase, LambdaBinding, Node, NodeKind, TypedNode};
use crate::ast::types::{Signature, StaticType};
use std::cell::Cell;
use std::collections::HashMap;
use std::rc::Rc;
use context::TypeContext;
pub struct TypeChecker {
pub(crate) root_types: Rc<std::cell::RefCell<Vec<StaticType>>>,
/// Shared monotonic counter for generating unique type variable IDs.
/// Points to the `Environment`'s counter so that IDs never collide
/// with TypeVars stored in `root_types` from earlier compilation passes.
pub(crate) var_counter: Rc<Cell<u32>>,
/// Global substitution map: TypeVar ID → resolved StaticType.
/// Shared across all scopes within a single type-checking pass.
pub(crate) subst: std::cell::RefCell<HashMap<u32, StaticType>>,
/// Index-call constraints: callee_var → result_var.
///
/// When a `TypeVar` is used as a callable with a single `Int` argument —
/// the series lookback pattern `(s 0)` — we record the pairing here instead
/// of eagerly unifying `TypeVar = Series(elem)`. The constraint is evaluated
/// lazily in `bind_var`: when the callee TypeVar is bound to a concrete type,
/// the element type is extracted directly from the `Series` variant.
pub(crate) index_constraints: std::cell::RefCell<HashMap<u32, u32>>,
/// Compiler hooks keyed by global slot index.
/// Populated from the frozen RTL snapshot; empty for non-RTL call sites.
pub(crate) compiler_hooks: Rc<HashMap<u32, Rc<dyn RtlCompilerHook>>>,
}
impl TypeChecker {
pub fn new(
root_types: Rc<std::cell::RefCell<Vec<StaticType>>>,
var_counter: Rc<Cell<u32>>,
compiler_hooks: Rc<HashMap<u32, Rc<dyn RtlCompilerHook>>>,
) -> Self {
Self {
root_types,
var_counter,
subst: std::cell::RefCell::new(HashMap::new()),
index_constraints: std::cell::RefCell::new(HashMap::new()),
compiler_hooks,
}
}
pub fn check(
&self,
node: &Node<BoundPhase>,
arg_types: &[StaticType],
diag: &mut Diagnostics,
) -> TypedNode {
let typed = self.check_node_as_bound(node, arg_types, diag);
self.finalize(typed)
}
/// Allows re-checking a node from any phase as if it were a bound node.
/// This is useful for specialization where we re-type a TypedNode with more specific info.
pub fn check_node_as_bound<P: BoundLike>(
&self,
node: &Node<P>,
arg_types: &[StaticType],
diag: &mut Diagnostics,
) -> TypedNode {
match &node.kind {
NodeKind::Lambda { params, body, info } => {
let upvalues = &info.upvalues;
let positional_count = info.positional_count;
let mut upvalue_types = Vec::with_capacity(upvalues.len());
for &_addr in upvalues {
upvalue_types.push(StaticType::Any);
}
let root_ctx = TypeContext::new(0, vec![], &self.root_types, None);
let mut lambda_ctx =
TypeContext::new(64, upvalue_types, &self.root_types, Some(&root_ctx));
let arg_tuple_ty = if arg_types.is_empty() {
StaticType::Any
} else {
StaticType::Tuple(arg_types.to_vec())
};
let params_typed =
self.check_params(params.as_ref(), &arg_tuple_ty, &mut lambda_ctx, diag);
let body_typed = self.check_node(body, &mut lambda_ctx, diag);
let ret_ty = body_typed.ty.clone();
let final_params_ty = params_typed.ty.clone();
let fn_ty = StaticType::Function(Box::new(Signature {
params: final_params_ty,
ret: ret_ty,
}));
Node {
identity: node.identity.clone(),
kind: NodeKind::Lambda {
params: Rc::new(params_typed),
body: Rc::new(body_typed),
info: LambdaBinding {
upvalues: upvalues.clone(),
positional_count,
},
},
ty: fn_ty,
comments: node.comments.clone(),
}
}
NodeKind::Block { .. } | NodeKind::Program { .. } => {
let mut ctx = TypeContext::new(64, vec![], &self.root_types, None);
self.check_node(node, &mut ctx, diag)
}
_ => {
let mut root_ctx = TypeContext::new(0, vec![], &self.root_types, None);
self.check_node(node, &mut root_ctx, diag)
}
}
}
}
+193
View File
@@ -0,0 +1,193 @@
use super::*;
use crate::ast::environment::Environment;
use crate::ast::types::StaticType;
fn check_source(source: &str) -> TypedNode {
let env = Environment::new();
env.compile(source).into_result().unwrap()
}
fn get_ret_type(node: &TypedNode) -> StaticType {
if let StaticType::Function(sig) = &node.ty {
sig.ret.clone()
} else {
node.ty.clone()
}
}
#[test]
fn test_inference_constants() {
assert_eq!(get_ret_type(&check_source("10")), StaticType::Int);
assert_eq!(get_ret_type(&check_source("10.5")), StaticType::Float);
assert_eq!(get_ret_type(&check_source("true")), StaticType::Bool);
assert_eq!(get_ret_type(&check_source("\"hello\"")), StaticType::Text);
}
#[test]
fn test_inference_variable_propagation() {
// (do (def x 10) x) -> The last 'x' must be Int
let typed = check_source("(do (def x 10) x)");
// compile() wraps in a Program node; the (do ...) block is the single child.
if let NodeKind::Program { exprs: prog } = &typed.kind {
if let NodeKind::Block { exprs } = &prog[0].kind {
let last_expr = exprs.last().unwrap();
assert_eq!(
last_expr.ty,
StaticType::Int,
"Variable 'x' should be inferred as Int"
);
} else {
panic!("Expected Block inside Program");
}
} else {
panic!("Expected Program, got {:?}", typed.kind);
}
}
#[test]
fn test_inference_block_type() {
// Block type = last expression type
assert_eq!(get_ret_type(&check_source("(do 1 2.5)")), StaticType::Float);
assert_eq!(
get_ret_type(&check_source("(do 1.5 \"test\")")),
StaticType::Text
);
}
#[test]
fn test_inference_lambda_return() {
// (fn [a] 10) -> fn(any) -> Int
// Since it's already a Lambda, it's NOT wrapped further.
let typed = check_source("(fn [a] 10)");
if let StaticType::Function(sig) = &typed.ty {
assert_eq!(sig.ret, StaticType::Int);
} else {
panic!("Expected function type, got {:?}", typed.ty);
}
// Nested: (fn [] (do 1 2.5)) -> fn() -> Float
let typed_nested = check_source("(fn [] (do 1 2.5))");
if let StaticType::Function(sig) = &typed_nested.ty {
assert_eq!(sig.ret, StaticType::Float);
} else {
panic!("Expected function type");
}
}
#[test]
fn test_inference_assignment_updates_type() {
// (do (def x 10) (assign x 20.5) x) -> x becomes Float after assignment
let typed = check_source("(do (def x 10) (assign x 20.5) x)");
if let NodeKind::Program { exprs: prog } = &typed.kind {
if let NodeKind::Block { exprs } = &prog[0].kind {
let last_expr = exprs.last().unwrap();
assert_eq!(
last_expr.ty,
StaticType::Float,
"Variable 'x' should be specialized to Float after assignment"
);
} else {
panic!("Expected Block inside Program");
}
} else {
panic!("Expected Program, got {:?}", typed.kind);
}
}
#[test]
fn test_operator_overloading_inference() {
assert_eq!(get_ret_type(&check_source("(+ 1 2)")), StaticType::Int);
assert_eq!(
get_ret_type(&check_source("(+ 1.0 2.0)")),
StaticType::Float
);
assert_eq!(
get_ret_type(&check_source("(+ \"a\" \"b\")")),
StaticType::Text
);
assert_eq!(get_ret_type(&check_source("(/ 1 2)")), StaticType::Float);
}
#[test]
fn test_datetime_inference() {
// date("2023-01-01") -> DateTime
assert_eq!(
get_ret_type(&check_source("(date \"2023-01-01\")")),
StaticType::DateTime
);
// DateTime + Int -> DateTime
assert_eq!(
get_ret_type(&check_source("(+ (date \"2023-01-01\") 86400000)")),
StaticType::DateTime
);
// DateTime - DateTime -> Int (Duration)
assert_eq!(
get_ret_type(&check_source(
"(- (date \"2023-01-02\") (date \"2023-01-01\"))"
)),
StaticType::Int
);
// DateTime comparison -> Bool
assert_eq!(
get_ret_type(&check_source(
"(> (date \"2023-01-02\") (date \"2023-01-01\"))"
)),
StaticType::Bool
);
}
#[test]
fn test_inference_tuple_vector_matrix() {
// Heterogeneous -> Tuple
assert_eq!(
get_ret_type(&check_source("[1 3.14 \"text\"]")),
StaticType::Tuple(vec![StaticType::Int, StaticType::Float, StaticType::Text])
);
// Homogeneous -> Vector
assert_eq!(
get_ret_type(&check_source("[10 20 30]")),
StaticType::Vector(Box::new(StaticType::Int), 3)
);
// Nested Homogeneous -> Matrix
assert_eq!(
get_ret_type(&check_source("[[1 2] [3 4]]")),
StaticType::Matrix(Box::new(StaticType::Int), vec![2, 2])
);
// Deep Matrix
assert_eq!(
get_ret_type(&check_source("[[[1 2]] [[3 4]]]")),
StaticType::Matrix(Box::new(StaticType::Int), vec![2, 1, 2])
);
// Shape mismatch -> Tuple of Vectors
let mixed = get_ret_type(&check_source("[[1 2] [3 4 5]]"));
if let StaticType::Tuple(elements) = mixed {
assert_eq!(
elements[0],
StaticType::Vector(Box::new(StaticType::Int), 2)
);
assert_eq!(
elements[1],
StaticType::Vector(Box::new(StaticType::Int), 3)
);
} else {
panic!("Expected Tuple for shape mismatch, got {:?}", mixed);
}
}
#[test]
fn test_inference_record() {
use crate::ast::types::Keyword;
let typed = check_source("{:x 1 :y 0.3}");
let ty = get_ret_type(&typed);
if let StaticType::Record(layout) = ty {
assert_eq!(layout.fields.len(), 2);
assert_eq!(layout.fields[0], (Keyword::intern("x"), StaticType::Int));
assert_eq!(layout.fields[1], (Keyword::intern("y"), StaticType::Float));
} else {
panic!("Expected Record, got {:?}", ty);
}
}
+10
View File
@@ -51,4 +51,14 @@ impl Diagnostics {
identity: id,
});
}
/// Formats all error-level diagnostics as a newline-joined string.
pub fn format_errors(&self) -> String {
self.items
.iter()
.filter(|d| d.level == DiagnosticLevel::Error)
.map(|d| d.message.as_str())
.collect::<Vec<_>>()
.join("\n")
}
}
+559 -444
View File
File diff suppressed because it is too large Load Diff
+108 -6
View File
@@ -1,4 +1,4 @@
use crate::ast::types::SourceLocation;
use crate::ast::types::{CommentLine, SourceLocation};
use std::iter::Peekable;
use std::rc::Rc;
use std::str::Chars;
@@ -27,12 +27,15 @@ pub enum TokenKind {
pub struct Token {
pub kind: TokenKind,
pub location: SourceLocation,
pub comments: Rc<[CommentLine]>,
}
pub struct Lexer<'a> {
input: Peekable<Chars<'a>>,
line: u32,
col: u32,
at_line_start: bool,
comments_buffer: Vec<CommentLine>,
}
impl<'a> Lexer<'a> {
@@ -41,11 +44,31 @@ impl<'a> Lexer<'a> {
input: input.chars().peekable(),
line: 1,
col: 1,
at_line_start: true,
comments_buffer: Vec::new(),
}
}
pub fn next_token(&mut self) -> Result<Token, String> {
self.skip_whitespace();
self.skip_whitespace_and_comments()?;
let comments: Rc<[CommentLine]> = if self.comments_buffer.is_empty() {
Rc::from([])
} else {
let mut start = 0;
while start < self.comments_buffer.len()
&& self.comments_buffer[start] == CommentLine::Blank
{
start += 1;
}
let mut end = self.comments_buffer.len();
while end > start && self.comments_buffer[end - 1] == CommentLine::Blank {
end -= 1;
}
let res: Rc<[CommentLine]> = self.comments_buffer[start..end].iter().cloned().collect();
self.comments_buffer.clear();
res
};
let start_location = SourceLocation {
line: self.line,
@@ -62,6 +85,7 @@ impl<'a> Lexer<'a> {
return Ok(Token {
kind: TokenKind::EOF,
location: start_location,
comments,
});
}
};
@@ -119,6 +143,7 @@ impl<'a> Lexer<'a> {
Ok(Token {
kind,
location: start_location,
comments,
})
}
@@ -126,28 +151,82 @@ impl<'a> Lexer<'a> {
self.input.peek()
}
fn skip_whitespace(&mut self) {
fn skip_whitespace_and_comments(&mut self) -> Result<(), String> {
let mut consecutive_newlines = 0;
while let Some(&c) = self.peek() {
if c.is_whitespace() || is_invisible(c) || c == ',' {
if c == '\n' {
self.line += 1;
self.col = 1;
self.at_line_start = true;
consecutive_newlines += 1;
} else {
self.col += 1;
}
self.input.next();
} else if c == ';' || c == '#' {
for c in self.input.by_ref() {
if c == '\n' {
} else if c == ';' {
if !self.at_line_start {
return Err(format!("Inline comments are not allowed at {}:{}", self.line, self.col));
}
if !self.comments_buffer.is_empty() && consecutive_newlines > 1 {
for _ in 0..(consecutive_newlines - 1) {
self.comments_buffer.push(CommentLine::Blank);
}
}
consecutive_newlines = 0;
self.input.next(); // consume first ';'
self.col += 1;
// Check for a second ';' → Doc level
let is_doc = if let Some(&';') = self.peek() {
self.input.next(); // consume second ';'
self.col += 1;
true
} else {
false
};
// Consume optional single space after the semicolon(s)
if let Some(&' ') = self.peek() {
self.input.next();
self.col += 1;
}
let mut comment_text = String::new();
while let Some(&cc) = self.peek() {
if cc == '\n' {
break;
}
comment_text.push(self.input.next().unwrap());
self.col += 1;
}
let line = if is_doc {
CommentLine::Doc(Rc::from(comment_text))
} else {
CommentLine::Comment(Rc::from(comment_text))
};
self.comments_buffer.push(line);
} else if c == '#' {
if !self.at_line_start {
return Err(format!("Inline directives are not allowed at {}:{}", self.line, self.col));
}
for cc in self.input.by_ref() {
if cc == '\n' {
self.line += 1;
self.col = 1;
self.at_line_start = true;
consecutive_newlines += 1;
break;
}
}
} else {
self.at_line_start = false;
break;
}
}
Ok(())
}
fn read_while<F>(&mut self, mut predicate: F) -> String
@@ -156,6 +235,11 @@ impl<'a> Lexer<'a> {
{
let mut s = String::new();
while let Some(&c) = self.peek() {
if is_invisible(c) {
self.input.next();
self.col += 1;
continue;
}
if predicate(c) {
s.push(self.input.next().unwrap());
self.col += 1;
@@ -217,3 +301,21 @@ fn is_invisible(c: char) -> bool {
_ => false,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_lex_invisible_chars() {
// U+200B Zero Width Space should be ignored inside identifiers
let source = "a\u{200B}b";
let mut lexer = Lexer::new(source);
let token = lexer.next_token().unwrap();
if let TokenKind::Identifier(id) = token.kind {
assert_eq!(id.as_ref(), "ab");
} else {
panic!("Expected identifier, got {:?}", token.kind);
}
}
}
+249
View File
@@ -0,0 +1,249 @@
use std::cell::RefCell;
use rmcp::{
ServerHandler,
handler::server::{router::tool::ToolRouter, wrapper::Parameters},
model::{ServerCapabilities, ServerInfo},
schemars, tool, tool_handler, tool_router,
transport::stdio,
};
use crate::ast::{environment::Environment, rtl};
const LANGUAGE_SPEC: &str = include_str!("../../docs/BNF.md");
/// Input struct for tools that take a Myc source code string.
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct CodeInput {
/// The Myc source code to process.
pub code: String,
}
/// Input struct for tools that look up a single RTL symbol by name.
#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct SymbolInput {
/// The RTL symbol name to look up (e.g. "+", "push", "series").
pub name: String,
}
thread_local! {
/// Persistent environment for REPL-style evaluation across multiple calls.
/// Lives in a thread-local because `Environment` uses `Rc<RefCell<...>>`
/// internally and is not `Send`/`Sync`, while `rmcp` requires the server
/// struct to be `Send + Sync`. Safe because we run on a single-threaded
/// Tokio runtime.
static REPL_ENV: RefCell<Option<Environment>> = const { RefCell::new(None) };
}
/// MCP server exposing the Myc compiler as a set of tools for LLMs.
#[derive(Clone)]
pub struct MycMcpServer {
// Used by the generated `ServerHandler` impl from `#[tool_router]`.
#[allow(dead_code)]
tool_router: ToolRouter<Self>,
}
impl MycMcpServer {
pub fn new() -> Self {
Self {
tool_router: Self::tool_router(),
}
}
/// Creates a fresh, fully initialized Myc environment.
/// A new environment is created per tool call because `Environment` uses
/// `Rc<RefCell<...>>` internally and is intentionally single-threaded.
fn make_env() -> Environment {
let mut env = Environment::new();
env.optimization = true;
env
}
}
impl Default for MycMcpServer {
fn default() -> Self {
Self::new()
}
}
#[tool_router]
impl MycMcpServer {
/// Evaluates a Myc expression and returns the result as a string.
#[tool(description = "Evaluate a Myc expression or script and return the result. \
The Myc language is a Lisp-like DSL for financial analysis with first-class ASTs, \
closures, and series (time-series queues). \
Use get_language_spec to learn the syntax first.")]
fn eval_myc(&self, Parameters(CodeInput { code }): Parameters<CodeInput>) -> String {
let env = Self::make_env();
match env.run_script(&code) {
Ok(value) => format!("{}", value),
Err(e) => format!("Error: {}", e),
}
}
/// Dumps the compiled and optimized AST for a Myc expression (verbose debug format).
#[tool(description = "Compile a Myc expression and return a human-readable dump of \
the compiled AST. Useful for understanding how the compiler transforms code \
through its optimization and lowering passes. \
Returns the full debug format including binding addresses and stack offsets. \
Use dump_ast_compact for a cleaner view focused on types and purity.")]
fn dump_ast(&self, Parameters(CodeInput { code }): Parameters<CodeInput>) -> String {
let env = Self::make_env();
match env.dump_ast(&code) {
Ok(dump) => dump,
Err(e) => format!("Error: {}", e),
}
}
/// Dumps the compiled AST in compact, human-readable format.
#[tool(description = "Compile a Myc expression and return a compact AST dump showing \
inferred types (e.g. 'fn([int]) -> bool') and purity markers ('!' for impure, \
'~' for side-effect-free). '[tail]' marks tail-call positions. \
Omits internal details like stack offsets and binding addresses. \
Prefer this over dump_ast when you want to understand the type structure of an expression.")]
fn dump_ast_compact(&self, Parameters(CodeInput { code }): Parameters<CodeInput>) -> String {
let env = Self::make_env();
match env.dump_ast_compact(&code) {
Ok(dump) => dump,
Err(e) => format!("Error: {}", e),
}
}
/// Checks the syntax and types of a Myc expression without executing it.
#[tool(description = "Check the syntax and types of a Myc expression without running it. \
Returns 'OK' if the code is valid, or a list of compiler diagnostics on error. \
Note: accepts a single expression only. Wrap multiple expressions in (do ...): \
e.g. (do (def x 42) (+ x 1)).")]
fn check_syntax(&self, Parameters(CodeInput { code }): Parameters<CodeInput>) -> String {
let env = Self::make_env();
let result = env.compile(&code);
if result.diagnostics.has_errors() {
let errors: Vec<String> = result
.diagnostics
.items
.into_iter()
.map(|d| format!("{:?}: {}", d.level, d.message))
.collect();
errors.join("\n")
} else {
"OK".to_string()
}
}
/// Lists all built-in functions and constants available in the Myc RTL.
#[tool(description = "List all built-in functions and constants available in the Myc \
runtime library (RTL). Returns a sorted list of all registered identifiers \
such as +, -, push, series, print, etc.")]
fn list_builtins(&self) -> String {
let env = Self::make_env();
// Only RTL bindings are in the fixed scope (scope 0), which is what
// list_bindings() returns. This excludes user-defined symbols.
let names = env.list_bindings();
names.join("\n")
}
/// Returns the full Myc language specification (BNF grammar and semantics).
#[tool(description = "Return the complete Myc language specification, including BNF grammar, \
core semantics, data types, special forms, macro system, and standard library overview. \
Read this first to understand how to write valid Myc code.")]
fn get_language_spec(&self) -> String {
LANGUAGE_SPEC.to_string()
}
/// Returns documentation for all documented RTL symbols.
#[tool(description = "Return documentation for ALL built-in RTL functions and constants at once. \
Prefer get_symbol_doc for individual lookups to save tokens. \
Use this only when you need a full overview of the entire standard library.")]
fn get_rtl_docs(&self) -> String {
let env = Self::make_env();
let docs = env.list_rtl_docs();
if docs.is_empty() {
"No documented RTL symbols found.".to_string()
} else {
docs.join("\n\n")
}
}
/// Returns documentation for a single RTL symbol by name.
#[tool(description = "Return the type signature, description, and examples for a single \
built-in RTL symbol. Use list_builtins to see all available names, then call this \
for each symbol you need details on. More token-efficient than get_rtl_docs.")]
fn get_symbol_doc(&self, Parameters(SymbolInput { name }): Parameters<SymbolInput>) -> String {
let env = Self::make_env();
match env.get_rtl_doc(&name) {
Some(doc) => doc,
None => format!("No documentation found for symbol '{}'.", name),
}
}
/// Evaluates a Myc expression in a persistent REPL session.
/// Bindings from previous calls are preserved.
#[tool(description = "Evaluate a Myc expression in a persistent REPL session. \
Unlike eval_myc, bindings (variables, functions, macros) from previous calls \
are preserved across invocations. Use repl_reset to start a fresh session. \
Example workflow: call repl_eval with '(def x 42)', then '(+ x 1)' returns 43.")]
fn repl_eval(&self, Parameters(CodeInput { code }): Parameters<CodeInput>) -> String {
REPL_ENV.with(|cell| {
let mut env_opt = cell.borrow_mut();
let env = env_opt.get_or_insert_with(Self::make_env);
match env.run_script(&code) {
Ok(value) => format!("{}", value),
Err(e) => format!("Error: {}", e),
}
})
}
/// Resets the REPL session, discarding all accumulated bindings.
#[tool(description = "Reset the persistent REPL session, discarding all variables, \
functions, and macros defined in previous repl_eval calls. \
The next repl_eval call will start with a fresh environment.")]
fn repl_reset(&self) -> String {
REPL_ENV.with(|cell| {
*cell.borrow_mut() = None;
});
"REPL session reset.".to_string()
}
}
#[tool_handler]
impl ServerHandler for MycMcpServer {
fn get_info(&self) -> ServerInfo {
ServerInfo::new(
ServerCapabilities::builder()
.enable_tools()
.build(),
)
.with_instructions(
"This server exposes the Myc compiler — a Lisp-like DSL for financial analysis. \
Use `get_language_spec` to learn the language syntax and semantics, \
`list_builtins` to see all available built-in functions, \
`check_syntax` to validate code, \
`eval_myc` to run expressions, and \
`dump_ast` to inspect the compiled AST. \
For interactive sessions, use `repl_eval` to evaluate expressions with \
persistent bindings across calls, and `repl_reset` to start fresh.",
)
}
}
/// Starts the MCP server on stdio (JSON-RPC over stdin/stdout).
///
/// Uses a single-threaded Tokio runtime because `Environment` relies on
/// `Rc<RefCell<...>>` and is not `Send`. The server handles one request
/// at a time, which matches the single-threaded design of the Myc runtime.
pub fn run() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Register RTL once just to warm the prelude path — the actual environment
// used per tool call is created fresh in `make_env()`.
// This also validates that the embedded system library compiles correctly.
let _ = rtl::register as fn(&Environment);
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()?
.block_on(async {
let server = MycMcpServer::new();
let service = rmcp::serve_server(server, stdio()).await?;
service.waiting().await?;
Ok(())
})
}
+3
View File
@@ -1,7 +1,10 @@
pub mod closure;
pub mod compiler;
pub use ::data_server;
pub mod diagnostics;
pub mod environment;
pub mod lexer;
pub mod mcp_server;
pub mod nodes;
pub mod parser;
pub mod rtl;
+560 -53
View File
@@ -1,7 +1,9 @@
use crate::ast::types::{Identity, Object, Value};
use std::any::Any;
use crate::ast::types::{CommentLine, Identity, Keyword, Purity, RecordLayout, StaticType, Value};
use std::fmt::Debug;
use std::rc::Rc;
use std::sync::Arc;
// ── Symbol ─────────────────────────────────────────────────────────────────────
/// A name with an optional context for macro hygiene.
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
@@ -30,24 +32,353 @@ impl From<&str> for Symbol {
}
}
/// A generic AST Node wrapper to preserve identity and metadata
// ── Address types ──────────────────────────────────────────────────────────────
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct VirtualId(pub u32);
impl std::fmt::Display for VirtualId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "V{}", self.0)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct StackOffset(pub u32);
impl std::fmt::Display for StackOffset {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "S{}", self.0)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct UpvalueIdx(pub u32);
impl std::fmt::Display for UpvalueIdx {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "U{}", self.0)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct GlobalIdx(pub u32);
impl std::fmt::Display for GlobalIdx {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "G{}", self.0)
}
}
#[derive(Debug, PartialEq, Eq, Hash)]
pub enum Address<L> {
Local(L), // Local address (VirtualId or StackOffset)
Upvalue(UpvalueIdx), // Index in the closure's upvalue array
Global(GlobalIdx), // Index in the global environment vector
}
impl<L: Clone> Clone for Address<L> {
fn clone(&self) -> Self {
match self {
Address::Local(l) => Address::Local(l.clone()),
Address::Upvalue(u) => Address::Upvalue(*u),
Address::Global(g) => Address::Global(*g),
}
}
}
impl<L: Copy> Copy for Address<L> {}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum DeclarationKind {
Variable,
Parameter,
}
// ── CompactMetadata trait ──────────────────────────────────────────────────────
/// Provides a compact, human-readable annotation for node metadata.
/// Used by the GUI AST dump to show type and purity without verbose debug output.
pub trait CompactMetadata {
/// Returns a short annotation such as `"int"`, `"fn([]) -> bool !"`.
/// Returns `None` for phases that carry no type information.
fn compact_label(&self) -> Option<String>;
}
impl CompactMetadata for () {
fn compact_label(&self) -> Option<String> {
None
}
}
impl CompactMetadata for StaticType {
fn compact_label(&self) -> Option<String> {
Some(format!("{self}"))
}
}
impl CompactMetadata for NodeMetrics {
fn compact_label(&self) -> Option<String> {
Some(format!(
"{}{}",
self.original.ty,
purity_suffix(self.purity)
))
}
}
impl CompactMetadata for RuntimeMetadata {
fn compact_label(&self) -> Option<String> {
Some(format!(
"{}{}{}",
self.ty,
if self.is_tail { " [tail]" } else { "" },
purity_suffix(self.original.ty.purity)
))
}
}
fn purity_suffix(p: Purity) -> &'static str {
match p {
Purity::Pure => "",
Purity::SideEffectFree => " ~",
Purity::Impure => " !",
}
}
// ── CompilerPhase trait ────────────────────────────────────────────────────────
pub trait CompilerPhase: std::fmt::Debug + 'static {
type Metadata: std::fmt::Debug + Clone + PartialEq + CompactMetadata;
type LocalAddress: std::fmt::Debug + Clone + Copy + PartialEq + Eq + std::hash::Hash;
/// Semantic role and address of an identifier (reference vs. declaration).
type Binding: std::fmt::Debug + Clone + PartialEq;
/// Binding metadata for `def` nodes (captured_by info).
type DefInfo: std::fmt::Debug + Clone + PartialEq;
/// Target address for `assign` nodes.
type AssignInfo: std::fmt::Debug + Clone + PartialEq;
/// Closure metadata for `lambda` nodes (upvalues, positional count).
type LambdaInfo: std::fmt::Debug + Clone + PartialEq;
/// Record field layout for O(1) access.
type RecordLayout: std::fmt::Debug + Clone + PartialEq;
}
// ── Phase-specific binding info types ──────────────────────────────────────────
/// Semantic role of an identifier after binding.
#[derive(Debug, Clone, PartialEq)]
pub struct Node<K, T = ()> {
pub enum IdentifierBinding<L = VirtualId> {
/// A variable reference (read access).
Reference(Address<L>),
/// A variable or parameter declaration (write/bind target).
Declaration {
addr: Address<L>,
kind: DeclarationKind,
},
}
/// Binding metadata attached to `def` nodes.
#[derive(Debug, Clone, PartialEq)]
pub struct DefBinding {
/// Identities of lambdas that capture this definition's variable.
pub captured_by: Vec<Identity>,
}
/// Target address attached to `assign` nodes.
/// `Some(addr)` for simple assignment, `None` for destructuring assignment
/// (where addresses live on the individual `Identifier` nodes in the target pattern).
#[derive(Debug, Clone, PartialEq)]
pub struct AssignBinding<L = VirtualId> {
pub addr: Option<Address<L>>,
}
/// Closure metadata attached to `lambda` nodes.
#[derive(Debug, Clone, PartialEq)]
pub struct LambdaBinding<L = VirtualId> {
/// Addresses of captured variables from enclosing scopes.
pub upvalues: Vec<Address<L>>,
/// Number of positional parameters (None = variadic).
pub positional_count: Option<u32>,
}
// ── Compiler phases ────────────────────────────────────────────────────────────
/// The initial phase produced by the parser. All annotation slots are `()`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SyntaxPhase;
impl CompilerPhase for SyntaxPhase {
type Metadata = ();
type LocalAddress = ();
type Binding = ();
type DefInfo = ();
type AssignInfo = ();
type LambdaInfo = ();
type RecordLayout = ();
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct BoundPhase;
impl CompilerPhase for BoundPhase {
type Metadata = ();
type LocalAddress = VirtualId;
type Binding = IdentifierBinding<VirtualId>;
type DefInfo = DefBinding;
type AssignInfo = AssignBinding<VirtualId>;
type LambdaInfo = LambdaBinding<VirtualId>;
type RecordLayout = Arc<RecordLayout>;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TypedPhase;
impl CompilerPhase for TypedPhase {
type Metadata = StaticType;
type LocalAddress = VirtualId;
type Binding = IdentifierBinding<VirtualId>;
type DefInfo = DefBinding;
type AssignInfo = AssignBinding<VirtualId>;
type LambdaInfo = LambdaBinding<VirtualId>;
type RecordLayout = Arc<RecordLayout>;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AnalyzedPhase;
impl CompilerPhase for AnalyzedPhase {
type Metadata = NodeMetrics;
type LocalAddress = VirtualId;
type Binding = IdentifierBinding<VirtualId>;
type DefInfo = DefBinding;
type AssignInfo = AssignBinding<VirtualId>;
type LambdaInfo = LambdaBinding<VirtualId>;
type RecordLayout = Arc<RecordLayout>;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RuntimePhase;
impl CompilerPhase for RuntimePhase {
type Metadata = RuntimeMetadata;
type LocalAddress = StackOffset;
type Binding = IdentifierBinding<StackOffset>;
type DefInfo = DefBinding;
type AssignInfo = AssignBinding<StackOffset>;
type LambdaInfo = LambdaBinding<StackOffset>;
type RecordLayout = Arc<RecordLayout>;
}
/// Convenience alias for all post-binding phases that share the same
/// concrete binding, def, assign, lambda, and record-layout types.
/// Covers `BoundPhase`, `TypedPhase`, `AnalyzedPhase` (but not `SyntaxPhase` or `RuntimePhase`).
pub trait BoundLike:
CompilerPhase<
LocalAddress = VirtualId,
Binding = IdentifierBinding<VirtualId>,
DefInfo = DefBinding,
AssignInfo = AssignBinding<VirtualId>,
LambdaInfo = LambdaBinding<VirtualId>,
RecordLayout = Arc<RecordLayout>,
>
{
}
impl<P> BoundLike for P where
P: CompilerPhase<
LocalAddress = VirtualId,
Binding = IdentifierBinding<VirtualId>,
DefInfo = DefBinding,
AssignInfo = AssignBinding<VirtualId>,
LambdaInfo = LambdaBinding<VirtualId>,
RecordLayout = Arc<RecordLayout>,
>
{
}
// ── Node<P> ────────────────────────────────────────────────────────────────────
/// A unified AST node, decorated with phase-specific information P.
/// Replaces both `SyntaxNode` (parser output) and the old bound AST node.
#[derive(Debug, PartialEq)]
pub struct Node<P: CompilerPhase = BoundPhase> {
pub identity: Identity,
pub kind: K,
pub ty: T,
pub kind: NodeKind<P>,
pub ty: P::Metadata,
pub comments: Rc<[CommentLine]>,
}
impl Object for Node<UntypedKind> {
fn type_name(&self) -> &'static str {
"ast-node"
impl<P: CompilerPhase> Clone for Node<P> {
fn clone(&self) -> Self {
Self {
identity: self.identity.clone(),
kind: self.kind.clone(),
ty: self.ty.clone(),
comments: self.comments.clone(),
}
fn as_any(&self) -> &dyn Any {
self
}
}
/// The base for custom node types (extensions)
/// Type alias: the parser AST is now `Node<SyntaxPhase>`.
pub type SyntaxNode = Node<SyntaxPhase>;
/// Type alias: the parser AST kind is now `NodeKind<SyntaxPhase>`.
pub type SyntaxKind = NodeKind<SyntaxPhase>;
pub type TypedNode = Node<TypedPhase>;
#[derive(Debug, Clone, PartialEq)]
pub struct NodeMetrics {
pub original: Rc<TypedNode>,
pub purity: Purity,
pub is_recursive: bool,
}
pub type AnalyzedNode = Node<AnalyzedPhase>;
#[derive(Clone)]
pub struct RuntimeMetadata {
pub ty: StaticType,
pub is_tail: bool,
pub original: Rc<AnalyzedNode>,
pub stack_size: u32,
}
impl std::fmt::Debug for RuntimeMetadata {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Metadata")
.field("ty", &self.ty)
.field("is_tail", &self.is_tail)
.field("stack_size", &self.stack_size)
.finish()
}
}
impl PartialEq for RuntimeMetadata {
fn eq(&self, other: &Self) -> bool {
self.ty == other.ty
&& self.is_tail == other.is_tail
&& Rc::ptr_eq(&self.original, &other.original)
&& self.stack_size == other.stack_size
}
}
pub type ExecNode = Node<RuntimePhase>;
pub type GlobalFunctionRegistry = std::collections::HashMap<GlobalIdx, Rc<Node<BoundPhase>>>;
pub type GlobalAnalyzedRegistry = std::collections::HashMap<GlobalIdx, Rc<AnalyzedNode>>;
pub trait BoundExtension<P: CompilerPhase>: std::fmt::Debug {
fn clone_box(&self) -> Box<dyn BoundExtension<P>>;
fn display_name(&self) -> String;
}
impl<P: CompilerPhase> Clone for Box<dyn BoundExtension<P>> {
fn clone(&self) -> Self {
self.clone_box()
}
}
// ── CustomNode ─────────────────────────────────────────────────────────────────
/// The base for custom node types (extensions).
pub trait CustomNode: Debug {
fn display_name(&self) -> &'static str;
fn clone_box(&self) -> Box<dyn CustomNode>;
@@ -59,71 +390,247 @@ impl Clone for Box<dyn CustomNode> {
}
}
#[derive(Debug, Clone)]
pub enum UntypedKind {
impl PartialEq for Box<dyn CustomNode> {
fn eq(&self, _other: &Self) -> bool {
false
}
}
// ── NodeKind<P> ────────────────────────────────────────────────────────────────
/// A key-value pair in a record literal: `(key_node, value_node)`.
pub type RecordFieldPair<P> = (Rc<Node<P>>, Rc<Node<P>>);
/// Unified AST node kind, replacing both `SyntaxKind` and `BoundKind<P>`.
///
/// The structure is always consistent with the syntax the user wrote.
/// Phase-specific information is carried in `P::*` annotation slots,
/// which are `()` in `SyntaxPhase` and concrete types in later phases.
#[derive(Debug)]
pub enum NodeKind<P: CompilerPhase = SyntaxPhase> {
Nop,
Constant(Value),
/// A general identifier (used for both references and declarations in the untyped AST).
Identifier(Symbol),
/// A first-class field accessor (e.g. .name)
FieldAccessor(crate::ast::types::Keyword),
Identifier {
symbol: Symbol,
binding: P::Binding,
},
FieldAccessor(Keyword),
If {
cond: Box<Node<UntypedKind>>,
then_br: Box<Node<UntypedKind>>,
else_br: Option<Box<Node<UntypedKind>>>,
cond: Rc<Node<P>>,
then_br: Rc<Node<P>>,
else_br: Option<Rc<Node<P>>>,
},
Def {
target: Box<Node<UntypedKind>>,
value: Box<Node<UntypedKind>>,
pattern: Rc<Node<P>>,
value: Rc<Node<P>>,
info: P::DefInfo,
},
Assign {
target: Box<Node<UntypedKind>>,
value: Box<Node<UntypedKind>>,
target: Rc<Node<P>>,
value: Rc<Node<P>>,
info: P::AssignInfo,
},
Lambda {
params: Box<Node<UntypedKind>>,
body: Rc<Node<UntypedKind>>,
params: Rc<Node<P>>,
body: Rc<Node<P>>,
info: P::LambdaInfo,
},
Call {
callee: Box<Node<UntypedKind>>,
args: Box<Node<UntypedKind>>,
callee: Rc<Node<P>>,
args: Rc<Node<P>>,
},
Again {
args: Box<Node<UntypedKind>>,
},
Pipe {
inputs: Vec<Node<UntypedKind>>,
lambda: Box<Node<UntypedKind>>,
args: Rc<Node<P>>,
},
Block {
exprs: Vec<Node<UntypedKind>>,
exprs: Vec<Rc<Node<P>>>,
},
/// Top-level program node. Like Block, but does not introduce a scope —
/// definitions propagate to the enclosing scope.
Program {
exprs: Vec<Rc<Node<P>>>,
},
Tuple {
elements: Vec<Node<UntypedKind>>,
elements: Vec<Rc<Node<P>>>,
},
Record {
fields: Vec<(Node<UntypedKind>, Node<UntypedKind>)>,
fields: Vec<RecordFieldPair<P>>,
layout: P::RecordLayout,
},
/// A macro declaration that can be expanded at compile time.
/// Macro declaration (only valid in `SyntaxPhase`).
MacroDecl {
name: Symbol,
params: Box<Node<UntypedKind>>,
body: Box<Node<UntypedKind>>,
params: Rc<Node<P>>,
body: Rc<Node<P>>,
},
/// A template for AST nodes, allowing for substitutions. (Quasiquote)
Template(Box<Node<UntypedKind>>),
/// A placeholder inside a template to be replaced by a node or value. (Unquote)
Placeholder(Box<Node<UntypedKind>>),
/// A placeholder that flattens a sequence or merges a record into its surroundings. (Splice)
Splice(Box<Node<UntypedKind>>),
/// Represents an expanded macro call, preserving the original call for debugging.
/// Quasiquote template (only valid in `SyntaxPhase`).
Template(Rc<Node<P>>),
/// Unquote placeholder inside a template (only valid in `SyntaxPhase`).
Placeholder(Rc<Node<P>>),
/// Splice placeholder inside a template (only valid in `SyntaxPhase`).
Splice(Rc<Node<P>>),
/// Optimized field access (only created during lowering for `RuntimePhase`).
GetField {
rec: Rc<Node<P>>,
field: Keyword,
},
/// Expanded macro call, preserving the original call for debugging.
Expansion {
/// The original call from the source AST.
call: Box<Node<UntypedKind>>,
/// The resulting AST after macro expansion.
expanded: Box<Node<UntypedKind>>,
original_call: Rc<Node<SyntaxPhase>>,
expanded: Rc<Node<P>>,
},
/// A diagnostic poison node, allowing compilation to continue after an error.
Error,
Extension(Box<dyn CustomNode>),
Extension(Box<dyn BoundExtension<P>>),
}
impl<P: CompilerPhase> Clone for NodeKind<P> {
fn clone(&self) -> Self {
match self {
NodeKind::Nop => NodeKind::Nop,
NodeKind::Constant(v) => NodeKind::Constant(v.clone()),
NodeKind::Identifier { symbol, binding } => NodeKind::Identifier {
symbol: symbol.clone(),
binding: binding.clone(),
},
NodeKind::FieldAccessor(k) => NodeKind::FieldAccessor(*k),
NodeKind::If { cond, then_br, else_br } => NodeKind::If {
cond: cond.clone(),
then_br: then_br.clone(),
else_br: else_br.clone(),
},
NodeKind::Def { pattern, value, info } => NodeKind::Def {
pattern: pattern.clone(),
value: value.clone(),
info: info.clone(),
},
NodeKind::Assign { target, value, info } => NodeKind::Assign {
target: target.clone(),
value: value.clone(),
info: info.clone(),
},
NodeKind::Lambda { params, body, info } => NodeKind::Lambda {
params: params.clone(),
body: body.clone(),
info: info.clone(),
},
NodeKind::Call { callee, args } => NodeKind::Call {
callee: callee.clone(),
args: args.clone(),
},
NodeKind::Again { args } => NodeKind::Again { args: args.clone() },
NodeKind::Block { exprs } => NodeKind::Block { exprs: exprs.clone() },
NodeKind::Program { exprs } => NodeKind::Program { exprs: exprs.clone() },
NodeKind::Tuple { elements } => NodeKind::Tuple { elements: elements.clone() },
NodeKind::Record { fields, layout } => NodeKind::Record {
fields: fields.clone(),
layout: layout.clone(),
},
NodeKind::GetField { rec, field } => NodeKind::GetField {
rec: rec.clone(),
field: *field,
},
NodeKind::MacroDecl { name, params, body } => NodeKind::MacroDecl {
name: name.clone(),
params: params.clone(),
body: body.clone(),
},
NodeKind::Template(inner) => NodeKind::Template(inner.clone()),
NodeKind::Placeholder(inner) => NodeKind::Placeholder(inner.clone()),
NodeKind::Splice(inner) => NodeKind::Splice(inner.clone()),
NodeKind::Expansion { original_call, expanded } => NodeKind::Expansion {
original_call: original_call.clone(),
expanded: expanded.clone(),
},
NodeKind::Error => NodeKind::Error,
NodeKind::Extension(ext) => NodeKind::Extension(ext.clone()),
}
}
}
impl<P: CompilerPhase> PartialEq for NodeKind<P> {
fn eq(&self, other: &Self) -> bool {
match (self, other) {
(NodeKind::Nop, NodeKind::Nop) => true,
(NodeKind::Constant(a), NodeKind::Constant(b)) => a == b,
(NodeKind::Identifier { symbol: sa, binding: ba }, NodeKind::Identifier { symbol: sb, binding: bb }) => {
sa == sb && ba == bb
}
(NodeKind::FieldAccessor(a), NodeKind::FieldAccessor(b)) => a == b,
(NodeKind::If { cond: ca, then_br: ta, else_br: ea }, NodeKind::If { cond: cb, then_br: tb, else_br: eb }) => {
Rc::ptr_eq(ca, cb) && Rc::ptr_eq(ta, tb) && match (ea, eb) {
(Some(a), Some(b)) => Rc::ptr_eq(a, b),
(None, None) => true,
_ => false,
}
}
(NodeKind::Def { pattern: pa, value: va, info: ia }, NodeKind::Def { pattern: pb, value: vb, info: ib }) => {
Rc::ptr_eq(pa, pb) && Rc::ptr_eq(va, vb) && ia == ib
}
(NodeKind::Assign { target: ta, value: va, info: ia }, NodeKind::Assign { target: tb, value: vb, info: ib }) => {
Rc::ptr_eq(ta, tb) && Rc::ptr_eq(va, vb) && ia == ib
}
(NodeKind::Lambda { params: pa, body: ba, info: ia }, NodeKind::Lambda { params: pb, body: bb, info: ib }) => {
Rc::ptr_eq(pa, pb) && Rc::ptr_eq(ba, bb) && ia == ib
}
(NodeKind::Call { callee: ca, args: aa }, NodeKind::Call { callee: cb, args: ab }) => {
Rc::ptr_eq(ca, cb) && Rc::ptr_eq(aa, ab)
}
(NodeKind::Again { args: aa }, NodeKind::Again { args: ab }) => Rc::ptr_eq(aa, ab),
(NodeKind::Block { exprs: ea }, NodeKind::Block { exprs: eb })
| (NodeKind::Program { exprs: ea }, NodeKind::Program { exprs: eb }) => {
ea.len() == eb.len() && ea.iter().zip(eb.iter()).all(|(a, b)| Rc::ptr_eq(a, b))
}
(NodeKind::Tuple { elements: ea }, NodeKind::Tuple { elements: eb }) => {
ea.len() == eb.len() && ea.iter().zip(eb.iter()).all(|(a, b)| Rc::ptr_eq(a, b))
}
(NodeKind::Record { fields: fa, layout: la }, NodeKind::Record { fields: fb, layout: lb }) => {
la == lb
&& fa.len() == fb.len()
&& fa.iter().zip(fb.iter()).all(|((ka, va), (kb, vb))| Rc::ptr_eq(ka, kb) && Rc::ptr_eq(va, vb))
}
(NodeKind::GetField { rec: ra, field: fa }, NodeKind::GetField { rec: rb, field: fb }) => {
Rc::ptr_eq(ra, rb) && fa == fb
}
(NodeKind::MacroDecl { name: na, params: pa, body: ba }, NodeKind::MacroDecl { name: nb, params: pb, body: bb }) => {
na == nb && Rc::ptr_eq(pa, pb) && Rc::ptr_eq(ba, bb)
}
(NodeKind::Template(a), NodeKind::Template(b)) => Rc::ptr_eq(a, b),
(NodeKind::Placeholder(a), NodeKind::Placeholder(b)) => Rc::ptr_eq(a, b),
(NodeKind::Splice(a), NodeKind::Splice(b)) => Rc::ptr_eq(a, b),
(NodeKind::Expansion { original_call: ca, expanded: ea }, NodeKind::Expansion { original_call: cb, expanded: eb }) => {
Rc::ptr_eq(ca, cb) && Rc::ptr_eq(ea, eb)
}
(NodeKind::Error, NodeKind::Error) => true,
_ => false,
}
}
}
impl<P: CompilerPhase> NodeKind<P> {
pub fn display_name(&self) -> String {
match self {
NodeKind::Nop => "NOP".to_string(),
NodeKind::Constant(v) => format!("CONST({})", v),
NodeKind::Identifier { symbol, .. } => format!("ID({})", symbol.name),
NodeKind::FieldAccessor(k) => format!("FIELD_ACCESSOR(.{})", k.name()),
NodeKind::If { .. } => "IF".to_string(),
NodeKind::Def { .. } => "DEF".to_string(),
NodeKind::Assign { .. } => "ASSIGN".to_string(),
NodeKind::Lambda { .. } => "LAMBDA".to_string(),
NodeKind::Call { .. } => "CALL".to_string(),
NodeKind::Again { .. } => "AGAIN".to_string(),
NodeKind::Block { .. } => "BLOCK".to_string(),
NodeKind::Program { .. } => "PROGRAM".to_string(),
NodeKind::Tuple { elements } => format!("TUPLE({})", elements.len()),
NodeKind::Record { fields, .. } => format!("RECORD({})", fields.len()),
NodeKind::GetField { field, .. } => format!("GET_FIELD(.{})", field.name()),
NodeKind::MacroDecl { name, .. } => format!("MACRO({})", name.name),
NodeKind::Template(_) => "TEMPLATE".to_string(),
NodeKind::Placeholder(_) => "PLACEHOLDER".to_string(),
NodeKind::Splice(_) => "SPLICE".to_string(),
NodeKind::Expansion { .. } => "EXPANSION".to_string(),
NodeKind::Extension(ext) => ext.display_name(),
NodeKind::Error => "ERROR".to_string(),
}
}
}
+201 -150
View File
@@ -1,7 +1,7 @@
use crate::ast::diagnostics::Diagnostics;
use crate::ast::lexer::{Lexer, Token, TokenKind};
use crate::ast::nodes::{Node, Symbol, UntypedKind};
use crate::ast::types::{Identity, Keyword, NodeIdentity, Value};
use crate::ast::nodes::{Symbol, SyntaxKind, SyntaxNode};
use crate::ast::types::{Identity, Keyword, NodeIdentity, SourceLocation, Value};
use std::rc::Rc;
pub struct Parser<'a> {
@@ -20,7 +20,8 @@ impl<'a> Parser<'a> {
diagnostics.push_error(e, None);
Token {
kind: TokenKind::EOF,
location: crate::ast::types::SourceLocation { line: 1, col: 1 },
location: SourceLocation { line: 1, col: 1 },
comments: Rc::from([]),
}
}
};
@@ -40,6 +41,7 @@ impl<'a> Parser<'a> {
Token {
kind: TokenKind::EOF,
location: self.current_token.location,
comments: Rc::from([]),
}
}
};
@@ -50,7 +52,7 @@ impl<'a> Parser<'a> {
&self.current_token.kind
}
pub fn parse_expression(&mut self) -> Node<UntypedKind> {
pub fn parse_expression(&mut self) -> SyntaxNode {
let token_loc = self.current_token.location;
let identity = NodeIdentity::new(token_loc);
@@ -59,48 +61,53 @@ impl<'a> Parser<'a> {
TokenKind::LeftBracket => self.parse_vector_literal(),
TokenKind::LeftBrace => self.parse_record_literal(),
TokenKind::Quote => {
self.advance(); // consume '
let token = self.advance(); // consume '
let expr = self.parse_expression();
Node {
SyntaxNode {
identity: identity.clone(),
kind: UntypedKind::Call {
callee: Box::new(self.make_id_node("quote", identity.clone())),
args: Box::new(Node {
kind: SyntaxKind::Call {
callee: Rc::new(self.make_id_node("quote", identity.clone())),
args: Rc::new(SyntaxNode {
identity,
kind: UntypedKind::Tuple {
elements: vec![expr],
kind: SyntaxKind::Tuple {
elements: vec![Rc::new(expr)],
},
ty: (),
comments: Rc::from([]),
}),
},
ty: (),
comments: token.comments,
}
}
TokenKind::Backtick => {
self.advance(); // consume `
let token = self.advance(); // consume `
let expr = self.parse_expression();
Node {
SyntaxNode {
identity,
kind: UntypedKind::Template(Box::new(expr)),
kind: SyntaxKind::Template(Rc::new(expr)),
ty: (),
comments: token.comments,
}
}
TokenKind::Tilde => {
self.advance(); // consume ~
let token = self.advance(); // consume ~
if *self.peek() == TokenKind::At {
self.advance(); // consume @
let expr = self.parse_expression();
Node {
SyntaxNode {
identity,
kind: UntypedKind::Splice(Box::new(expr)),
kind: SyntaxKind::Splice(Rc::new(expr)),
ty: (),
comments: token.comments,
}
} else {
let expr = self.parse_expression();
Node {
SyntaxNode {
identity,
kind: UntypedKind::Placeholder(Box::new(expr)),
kind: SyntaxKind::Placeholder(Rc::new(expr)),
ty: (),
comments: token.comments,
}
}
}
@@ -112,6 +119,22 @@ impl<'a> Parser<'a> {
matches!(self.current_token.kind, TokenKind::EOF)
}
/// Parses a complete program as a sequence of top-level expressions
/// wrapped in a `Program` node.
pub fn parse_program(&mut self) -> SyntaxNode {
let identity = NodeIdentity::new(SourceLocation { line: 1, col: 1 });
let mut expressions = Vec::new();
while !self.at_eof() {
expressions.push(Rc::new(self.parse_expression()));
}
SyntaxNode {
identity,
kind: SyntaxKind::Program { exprs: expressions },
ty: (),
comments: Rc::from([]),
}
}
fn synchronize(&mut self) {
while !self.at_eof() {
match self.peek() {
@@ -126,42 +149,46 @@ impl<'a> Parser<'a> {
}
}
fn parse_atom(&mut self) -> Node<UntypedKind> {
fn parse_atom(&mut self) -> SyntaxNode {
let token = self.advance();
let identity = NodeIdentity::new(token.location);
let kind = match token.kind {
TokenKind::Integer(n) => UntypedKind::Constant(Value::Int(n)),
TokenKind::Float(n) => UntypedKind::Constant(Value::Float(n)),
TokenKind::String(s) => UntypedKind::Constant(Value::Text(s)),
TokenKind::Keyword(k) => UntypedKind::Constant(Value::Keyword(Keyword::intern(&k))),
TokenKind::Integer(n) => SyntaxKind::Constant(Value::Int(n)),
TokenKind::Float(n) => SyntaxKind::Constant(Value::Float(n)),
TokenKind::String(s) => SyntaxKind::Constant(Value::Text(s)),
TokenKind::Keyword(k) => SyntaxKind::Constant(Value::Keyword(Keyword::intern(&k))),
TokenKind::Identifier(id) => match id.as_ref() {
"..." => UntypedKind::Nop,
"..." => SyntaxKind::Nop,
s if s.starts_with('.') && s.len() > 1 => {
UntypedKind::FieldAccessor(Keyword::intern(&s[1..]))
SyntaxKind::FieldAccessor(Keyword::intern(&s[1..]))
}
_ => UntypedKind::Identifier(id.into()),
_ => SyntaxKind::Identifier {
symbol: id.into(),
binding: (),
},
TokenKind::EOF => UntypedKind::Error, // Error already logged by advance
},
TokenKind::EOF => SyntaxKind::Error, // Error already logged by advance
_ => {
self.diagnostics.push_error(
format!("Unexpected token in atom: {:?}", token.kind),
Some(identity.clone()),
);
UntypedKind::Error
SyntaxKind::Error
}
};
Node {
SyntaxNode {
identity,
kind,
ty: (),
comments: token.comments,
}
}
fn parse_list(&mut self) -> Node<UntypedKind> {
let start_loc = self.advance().location; // consume '('
let identity = NodeIdentity::new(start_loc);
fn parse_list(&mut self) -> SyntaxNode {
let token = self.advance(); // consume '('
let identity = NodeIdentity::new(token.location);
if *self.peek() == TokenKind::RightParen {
self.diagnostics.push_error(
@@ -169,20 +196,20 @@ impl<'a> Parser<'a> {
Some(identity.clone()),
);
self.advance(); // consume )
return Node {
return SyntaxNode {
identity,
kind: UntypedKind::Error,
kind: SyntaxKind::Error,
ty: (),
comments: token.comments,
};
}
let head = self.parse_expression();
let node = if let UntypedKind::Identifier(ref sym) = head.kind {
match sym.name.as_ref() {
let mut node = if let SyntaxKind::Identifier { ref symbol, .. } = head.kind {
match symbol.name.as_ref() {
"if" => self.parse_if(identity),
"fn" => self.parse_fn(identity),
"pipe" => self.parse_pipe(identity),
"again" => self.parse_again(identity),
"def" => self.parse_def(identity),
"assign" => self.parse_assign(identity),
@@ -195,119 +222,121 @@ impl<'a> Parser<'a> {
};
self.expect(TokenKind::RightParen);
node.comments = token.comments;
node
}
fn parse_again(&mut self, identity: Identity) -> Node<UntypedKind> {
fn parse_again(&mut self, identity: Identity) -> SyntaxNode {
let mut elements = Vec::new();
while *self.peek() != TokenKind::RightParen && *self.peek() != TokenKind::EOF {
elements.push(self.parse_expression());
elements.push(Rc::new(self.parse_expression()));
}
let args_node = Node {
let args_node = SyntaxNode {
identity: identity.clone(),
kind: UntypedKind::Tuple { elements },
kind: SyntaxKind::Tuple { elements },
ty: (),
comments: Rc::from([]),
};
Node {
SyntaxNode {
identity,
kind: UntypedKind::Again {
args: Box::new(args_node),
kind: SyntaxKind::Again {
args: Rc::new(args_node),
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_if(&mut self, identity: Identity) -> Node<UntypedKind> {
let cond = Box::new(self.parse_expression());
let then_br = Box::new(self.parse_expression());
fn parse_if(&mut self, identity: Identity) -> SyntaxNode {
let cond = Rc::new(self.parse_expression());
let then_br = Rc::new(self.parse_expression());
let mut else_br = None;
if *self.peek() != TokenKind::RightParen {
else_br = Some(Box::new(self.parse_expression()));
else_br = Some(Rc::new(self.parse_expression()));
}
Node {
SyntaxNode {
identity,
kind: UntypedKind::If {
kind: SyntaxKind::If {
cond,
then_br,
else_br,
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_def(&mut self, identity: Identity) -> Node<UntypedKind> {
let target = Box::new(self.parse_pattern());
let value = Box::new(self.parse_expression());
fn parse_def(&mut self, identity: Identity) -> SyntaxNode {
let pattern = Rc::new(self.parse_pattern());
let value = Rc::new(self.parse_expression());
Node {
SyntaxNode {
identity,
kind: UntypedKind::Def { target, value },
ty: (),
}
}
fn parse_assign(&mut self, identity: Identity) -> Node<UntypedKind> {
// (assign target value)
let target = Box::new(self.parse_expression());
let value = Box::new(self.parse_expression());
Node {
identity,
kind: UntypedKind::Assign { target, value },
ty: (),
}
}
fn parse_do(&mut self, identity: Identity) -> Node<UntypedKind> {
let mut exprs = Vec::new();
while *self.peek() != TokenKind::RightParen && *self.peek() != TokenKind::EOF {
exprs.push(self.parse_expression());
}
Node {
identity,
kind: UntypedKind::Block { exprs },
ty: (),
}
}
fn parse_pipe(&mut self, identity: Identity) -> Node<UntypedKind> {
let inputs_node = self.parse_expression();
let inputs = match inputs_node.kind {
UntypedKind::Tuple { elements } => elements,
_ => vec![inputs_node],
};
let lambda = Box::new(self.parse_expression());
Node {
identity,
kind: UntypedKind::Pipe { inputs, lambda },
ty: (),
}
}
fn parse_fn(&mut self, identity: Identity) -> Node<UntypedKind> {
let params = Box::new(self.parse_param_vector());
let body = self.parse_expression();
Node {
identity,
kind: UntypedKind::Lambda {
params,
body: Rc::new(body),
kind: SyntaxKind::Def {
pattern,
value,
info: (),
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_macro_decl(&mut self, identity: Identity) -> Node<UntypedKind> {
fn parse_assign(&mut self, identity: Identity) -> SyntaxNode {
// (assign target value)
let target = Rc::new(self.parse_expression());
let value = Rc::new(self.parse_expression());
SyntaxNode {
identity,
kind: SyntaxKind::Assign {
target,
value,
info: (),
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_do(&mut self, identity: Identity) -> SyntaxNode {
let mut exprs = Vec::new();
while *self.peek() != TokenKind::RightParen && *self.peek() != TokenKind::EOF {
exprs.push(Rc::new(self.parse_expression()));
}
SyntaxNode {
identity,
kind: SyntaxKind::Block { exprs },
ty: (),
comments: Rc::from([]),
}
}
fn parse_fn(&mut self, identity: Identity) -> SyntaxNode {
let params = Rc::new(self.parse_param_vector());
let body = self.parse_expression();
SyntaxNode {
identity,
kind: SyntaxKind::Lambda {
params,
body: Rc::new(body),
info: (),
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_macro_decl(&mut self, identity: Identity) -> SyntaxNode {
let name_node = self.parse_expression();
let name = match name_node.kind {
UntypedKind::Identifier(sym) => sym,
SyntaxKind::Identifier { symbol, .. } => symbol,
_ => {
self.diagnostics.push_error(
"Expected identifier for macro name",
@@ -316,20 +345,21 @@ impl<'a> Parser<'a> {
Symbol::from("error")
}
};
let params = Box::new(self.parse_param_vector());
let params = Rc::new(self.parse_param_vector());
let body = self.parse_expression();
Node {
SyntaxNode {
identity,
kind: UntypedKind::MacroDecl {
kind: SyntaxKind::MacroDecl {
name,
params,
body: Box::new(body),
body: Rc::new(body),
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_param_vector(&mut self) -> Node<UntypedKind> {
fn parse_param_vector(&mut self) -> SyntaxNode {
if *self.peek() != TokenKind::LeftBracket {
self.diagnostics.push_error(
format!(
@@ -338,16 +368,17 @@ impl<'a> Parser<'a> {
),
Some(NodeIdentity::new(self.current_token.location)),
);
return Node {
return SyntaxNode {
identity: NodeIdentity::new(self.current_token.location),
kind: UntypedKind::Error,
kind: SyntaxKind::Error,
ty: (),
comments: Rc::from([]),
};
}
self.parse_pattern()
}
fn parse_pattern(&mut self) -> Node<UntypedKind> {
fn parse_pattern(&mut self) -> SyntaxNode {
let next = self.peek();
match next {
TokenKind::Identifier(_) => {
@@ -356,10 +387,14 @@ impl<'a> Parser<'a> {
TokenKind::Identifier(s) => s.into(),
_ => unreachable!(),
};
Node {
SyntaxNode {
identity: NodeIdentity::new(token.location),
kind: UntypedKind::Identifier(sym),
kind: SyntaxKind::Identifier {
symbol: sym,
binding: (),
},
ty: (),
comments: token.comments,
}
}
TokenKind::LeftBracket => {
@@ -368,14 +403,15 @@ impl<'a> Parser<'a> {
let mut elements = Vec::new();
while *self.peek() != TokenKind::RightBracket && *self.peek() != TokenKind::EOF {
elements.push(self.parse_pattern());
elements.push(Rc::new(self.parse_pattern()));
}
self.expect(TokenKind::RightBracket);
Node {
SyntaxNode {
identity,
kind: UntypedKind::Tuple { elements },
kind: SyntaxKind::Tuple { elements },
ty: (),
comments: token.comments,
}
}
TokenKind::Tilde => {
@@ -383,17 +419,19 @@ impl<'a> Parser<'a> {
if *self.peek() == TokenKind::At {
self.advance(); // consume @
let expr = self.parse_expression();
Node {
SyntaxNode {
identity: NodeIdentity::new(token.location),
kind: UntypedKind::Splice(Box::new(expr)),
kind: SyntaxKind::Splice(Rc::new(expr)),
ty: (),
comments: token.comments,
}
} else {
let expr = self.parse_expression();
Node {
SyntaxNode {
identity: NodeIdentity::new(token.location),
kind: UntypedKind::Placeholder(Box::new(expr)),
kind: SyntaxKind::Placeholder(Rc::new(expr)),
ty: (),
comments: token.comments,
}
}
}
@@ -405,59 +443,63 @@ impl<'a> Parser<'a> {
),
Some(NodeIdentity::new(self.current_token.location)),
);
self.advance();
Node {
let token = self.advance();
SyntaxNode {
identity: NodeIdentity::new(self.current_token.location),
kind: UntypedKind::Error,
kind: SyntaxKind::Error,
ty: (),
comments: token.comments,
}
}
}
}
fn parse_call(&mut self, callee: Node<UntypedKind>, identity: Identity) -> Node<UntypedKind> {
fn parse_call(&mut self, callee: SyntaxNode, identity: Identity) -> SyntaxNode {
let mut elements = Vec::new();
while *self.peek() != TokenKind::RightParen && *self.peek() != TokenKind::EOF {
elements.push(self.parse_expression());
elements.push(Rc::new(self.parse_expression()));
}
// The arguments are wrapped in a Tuple node, reusing the call's identity/location.
let args_node = Node {
let args_node = SyntaxNode {
identity: identity.clone(),
kind: UntypedKind::Tuple { elements },
kind: SyntaxKind::Tuple { elements },
ty: (),
comments: Rc::from([]),
};
Node {
SyntaxNode {
identity,
kind: UntypedKind::Call {
callee: Box::new(callee),
args: Box::new(args_node),
kind: SyntaxKind::Call {
callee: Rc::new(callee),
args: Rc::new(args_node),
},
ty: (),
comments: Rc::from([]),
}
}
fn parse_vector_literal(&mut self) -> Node<UntypedKind> {
fn parse_vector_literal(&mut self) -> SyntaxNode {
let token = self.advance();
let mut elements = Vec::new();
while *self.peek() != TokenKind::RightBracket && *self.peek() != TokenKind::EOF {
let expr = self.parse_expression();
elements.push(expr);
elements.push(Rc::new(expr));
}
self.expect(TokenKind::RightBracket);
Node {
SyntaxNode {
identity: NodeIdentity::new(token.location),
kind: UntypedKind::Tuple { elements },
kind: SyntaxKind::Tuple { elements },
ty: (),
comments: token.comments,
}
}
fn parse_record_literal(&mut self) -> Node<UntypedKind> {
fn parse_record_literal(&mut self) -> SyntaxNode {
let token = self.advance();
let mut fields = Vec::new();
@@ -467,7 +509,7 @@ impl<'a> Parser<'a> {
// strictly we could allow any expression and check at runtime.
// Delphi enforces keywords. We can do minimal check here.
match &key_node.kind {
UntypedKind::Constant(Value::Keyword(_)) => {}
SyntaxKind::Constant(Value::Keyword(_)) => {}
_ => {
self.diagnostics.push_error(
"Record keys must be keywords (syntactically)",
@@ -485,14 +527,18 @@ impl<'a> Parser<'a> {
}
let val_node = self.parse_expression();
fields.push((key_node, val_node));
fields.push((Rc::new(key_node), Rc::new(val_node)));
}
self.expect(TokenKind::RightBrace);
Node {
SyntaxNode {
identity: NodeIdentity::new(token.location),
kind: UntypedKind::Record { fields },
kind: SyntaxKind::Record {
fields,
layout: (),
},
ty: (),
comments: token.comments,
}
}
@@ -509,15 +555,20 @@ impl<'a> Parser<'a> {
Token {
kind,
location: self.current_token.location,
comments: Rc::from([]),
}
}
}
fn make_id_node(&self, name: &str, identity: Identity) -> Node<UntypedKind> {
Node {
fn make_id_node(&self, name: &str, identity: Identity) -> SyntaxNode {
SyntaxNode {
identity,
kind: UntypedKind::Identifier(Symbol::from(name)),
kind: SyntaxKind::Identifier {
symbol: Symbol::from(name),
binding: (),
},
ty: (),
comments: Rc::from([]),
}
}
}
+214 -32
View File
@@ -1,5 +1,5 @@
use crate::ast::environment::Environment;
use crate::ast::types::{Purity, Signature, StaticType, Value};
use crate::ast::types::{Keyword, Purity, Signature, StaticType, Value};
use std::rc::Rc;
pub fn register(env: &Environment) {
@@ -8,6 +8,7 @@ pub fn register(env: &Environment) {
register_comparison(env);
register_logic(env);
register_io(env);
register_testing(env);
}
fn register_constants(env: &Environment) {
@@ -15,9 +16,12 @@ fn register_constants(env: &Environment) {
// In Delphi RTL: CFalse, CTrue, CNaN
// We register NaN as a value, not a function.
env.register_constant("NaN", StaticType::Float, Value::Float(f64::NAN));
env.register_constant("true", StaticType::Bool, Value::Bool(true));
env.register_constant("false", StaticType::Bool, Value::Bool(false));
env.register_constant("NaN", StaticType::Float, Value::Float(f64::NAN))
.doc("Not-a-Number float constant. Use to represent undefined numeric results.");
env.register_constant("true", StaticType::Bool, Value::Bool(true))
.doc("Boolean true constant.");
env.register_constant("false", StaticType::Bool, Value::Bool(false))
.doc("Boolean false constant.");
}
fn register_arithmetic(env: &Environment) {
@@ -31,6 +35,14 @@ fn register_arithmetic(env: &Environment) {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Int]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Text, StaticType::Text]),
ret: StaticType::Text,
@@ -39,10 +51,10 @@ fn register_arithmetic(env: &Environment) {
params: StaticType::Tuple(vec![StaticType::DateTime, StaticType::Int]),
ret: StaticType::DateTime,
},
// Variadic
// Variadic: any number of numeric args
Signature {
params: StaticType::Any,
ret: StaticType::Any,
params: StaticType::Variadic(Box::new(StaticType::Float)),
ret: StaticType::Float,
},
]);
env.register_native_fn("+", add_ty, Purity::Pure, |args| {
@@ -76,7 +88,9 @@ fn register_arithmetic(env: &Environment) {
Value::Float(acc)
}
}
});
}).doc("Adds two or more values.")
.description("Supports int, float, text concatenation, and datetime+ms offset. Also variadic: (+ 1 2 3) => 6.")
.examples(&["(+ 1 2)", "(+ 1.5 2.5)", "(+ \"Hello\" \" World\")", "(+ my-date 86400000)"]);
// --- Subtract (-) ---
let sub_ty = StaticType::FunctionOverloads(vec![
@@ -88,6 +102,14 @@ fn register_arithmetic(env: &Environment) {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Int]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Int]),
ret: StaticType::Int,
@@ -104,10 +126,10 @@ fn register_arithmetic(env: &Environment) {
params: StaticType::Tuple(vec![StaticType::DateTime, StaticType::Int]),
ret: StaticType::DateTime,
},
// Variadic
// Variadic: any number of numeric args
Signature {
params: StaticType::Any,
ret: StaticType::Any,
params: StaticType::Variadic(Box::new(StaticType::Float)),
ret: StaticType::Float,
},
]);
env.register_native_fn("-", sub_ty, Purity::Pure, |args| {
@@ -150,7 +172,8 @@ fn register_arithmetic(env: &Environment) {
} else {
Value::Float(acc)
}
});
}).doc("Subtracts values. Also supports unary negation: (- x).")
.examples(&["(- 10 3)", "(- 5)", "(- 2026-01-01 2025-01-01)"]);
// --- Multiply (*) ---
let mul_ty = StaticType::FunctionOverloads(vec![
@@ -162,10 +185,18 @@ fn register_arithmetic(env: &Environment) {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Float]),
ret: StaticType::Float,
},
// Variadic
Signature {
params: StaticType::Any,
ret: StaticType::Any,
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Int]),
ret: StaticType::Float,
},
// Variadic: any number of numeric args
Signature {
params: StaticType::Variadic(Box::new(StaticType::Float)),
ret: StaticType::Float,
},
]);
env.register_native_fn("*", mul_ty, Purity::Pure, |args| {
@@ -192,7 +223,8 @@ fn register_arithmetic(env: &Environment) {
Value::Float(acc)
}
}
});
}).doc("Multiplies values. Also variadic: (* 2 3 4) => 24.")
.examples(&["(* 3 4)", "(* 1.5 2.0)"]);
// --- Divide (/) ---
let div_ty = StaticType::FunctionOverloads(vec![
@@ -204,6 +236,14 @@ fn register_arithmetic(env: &Environment) {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Float]),
ret: StaticType::Float,
},
Signature {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Int]),
ret: StaticType::Float,
},
]);
env.register_native_fn("/", div_ty, Purity::Pure, |args| {
if args.len() != 2 {
@@ -224,7 +264,8 @@ fn register_arithmetic(env: &Environment) {
} else {
Value::Float(a / b)
}
});
}).doc("Divides two numbers, always returns float. Returns NaN on division by zero.")
.examples(&["(/ 10 3)", "(/ 7.0 2.0)"]);
// --- Integer Divide (//) ---
let int_div_ty = StaticType::Function(Box::new(Signature {
@@ -247,7 +288,8 @@ fn register_arithmetic(env: &Environment) {
// Usually div is for integers. Let's stick to Int.
_ => Value::Void,
}
});
}).doc("Integer division. Returns int, truncates towards zero.")
.examples(&["(// 10 3)"]);
// --- Modulus (%) ---
let mod_ty = StaticType::Function(Box::new(Signature {
@@ -268,7 +310,8 @@ fn register_arithmetic(env: &Environment) {
}
_ => Value::Void,
}
});
}).doc("Modulo: remainder of integer division.")
.examples(&["(% 10 3)"]);
}
fn register_comparison(env: &Environment) {
@@ -300,7 +343,7 @@ fn register_comparison(env: &Environment) {
(Value::DateTime(a), Value::DateTime(b)) => Value::Bool(a > b),
_ => Value::Bool(false),
}
});
}).doc("Greater-than comparison. Supports int, float, datetime.");
// --- Less Than (<) ---
env.register_native_fn("<", cmp_ty.clone(), Purity::Pure, |args| {
@@ -315,7 +358,7 @@ fn register_comparison(env: &Environment) {
(Value::DateTime(a), Value::DateTime(b)) => Value::Bool(a < b),
_ => Value::Bool(false),
}
});
}).doc("Less-than comparison. Supports int, float, datetime.");
// --- Greater Or Equal (>=) ---
env.register_native_fn(">=", cmp_ty.clone(), Purity::Pure, |args| {
@@ -330,7 +373,7 @@ fn register_comparison(env: &Environment) {
(Value::DateTime(a), Value::DateTime(b)) => Value::Bool(a >= b),
_ => Value::Bool(false),
}
});
}).doc("Greater-than-or-equal comparison. Supports int, float, datetime.");
// --- Less Or Equal (<=) ---
env.register_native_fn("<=", cmp_ty.clone(), Purity::Pure, |args| {
@@ -345,7 +388,7 @@ fn register_comparison(env: &Environment) {
(Value::DateTime(a), Value::DateTime(b)) => Value::Bool(a <= b),
_ => Value::Bool(false),
}
});
}).doc("Less-than-or-equal comparison. Supports int, float, datetime.");
// --- Equal (=) ---
let eq_ty = StaticType::Function(Box::new(Signature {
@@ -370,10 +413,16 @@ fn register_comparison(env: &Environment) {
(Value::Record(la, va), Value::Record(lb, vb)) => {
Value::Bool(std::sync::Arc::ptr_eq(la, lb) && va == vb)
}
(Value::Tuple(a), Value::Tuple(b)) => {
Value::Bool(a.len() == b.len()
&& a.iter().zip(b.iter()).all(|(x, y)| values_equal(x, y)))
}
(Value::FieldAccessor(a), Value::FieldAccessor(b)) => Value::Bool(a == b),
(Value::Void, Value::Void) => Value::Bool(true),
_ => Value::Bool(false),
}
});
}).doc("Structural equality. Records are equal if they have the same layout and field values.")
.examples(&["(= 1 1)", "(= {:a 1} {:a 1})", "(= :foo :foo)"]);
// --- Not Equal (<>) ---
env.register_native_fn("<>", eq_ty, Purity::Pure, |args| {
@@ -389,10 +438,15 @@ fn register_comparison(env: &Environment) {
(Value::Bool(a), Value::Bool(b)) => Value::Bool(a != b),
(Value::Keyword(a), Value::Keyword(b)) => Value::Bool(a != b),
(Value::DateTime(a), Value::DateTime(b)) => Value::Bool(a != b),
(Value::Tuple(a), Value::Tuple(b)) => {
Value::Bool(!(a.len() == b.len()
&& a.iter().zip(b.iter()).all(|(x, y)| values_equal(x, y))))
}
(Value::FieldAccessor(a), Value::FieldAccessor(b)) => Value::Bool(a != b),
(Value::Void, Value::Void) => Value::Bool(false),
_ => Value::Bool(true),
}
});
}).doc("Structural inequality. The inverse of =.");
}
fn register_logic(env: &Environment) {
@@ -416,7 +470,9 @@ fn register_logic(env: &Environment) {
Value::Int(i) => Value::Int(!i), // Bitwise NOT
_ => Value::Void,
}
});
}).doc("Logical NOT for bool, bitwise NOT for int.")
.examples(&["(not true)", "(not 0b1010)"]);
// --- And (and) ---
let logic_op_ty = StaticType::FunctionOverloads(vec![
@@ -438,7 +494,7 @@ fn register_logic(env: &Environment) {
(Value::Int(a), Value::Int(b)) => Value::Int(a & b),
_ => Value::Void,
}
});
}).doc("Logical AND for bool, bitwise AND for int.");
// --- Or (or) ---
env.register_native_fn("or", logic_op_ty.clone(), Purity::Pure, |args| {
@@ -450,7 +506,7 @@ fn register_logic(env: &Environment) {
(Value::Int(a), Value::Int(b)) => Value::Int(a | b),
_ => Value::Void,
}
});
}).doc("Logical OR for bool, bitwise OR for int.");
// --- Xor (xor) ---
env.register_native_fn("xor", logic_op_ty.clone(), Purity::Pure, |args| {
@@ -462,7 +518,7 @@ fn register_logic(env: &Environment) {
(Value::Int(a), Value::Int(b)) => Value::Int(a ^ b),
_ => Value::Void,
}
});
}).doc("Logical XOR for bool, bitwise XOR for int.");
// --- Shift Left (<<) ---
let shift_ty = StaticType::Function(Box::new(Signature {
@@ -477,7 +533,8 @@ fn register_logic(env: &Environment) {
(Value::Int(a), Value::Int(b)) => Value::Int(a << b),
_ => Value::Void,
}
});
}).doc("Bitwise left shift.")
.examples(&["(<< 1 4)"]);
// --- Shift Right (>>) ---
env.register_native_fn(">>", shift_ty, Purity::Pure, |args| {
@@ -488,7 +545,8 @@ fn register_logic(env: &Environment) {
(Value::Int(a), Value::Int(b)) => Value::Int(a >> b),
_ => Value::Void,
}
});
}).doc("Bitwise right shift.")
.examples(&["(>> 16 4)"]);
}
fn register_io(env: &Environment) {
@@ -505,5 +563,129 @@ fn register_io(env: &Environment) {
}
println!();
Value::Void
});
}).doc("Prints all arguments to stdout followed by a newline. Returns void.")
.examples(&["(print \"Hello World\")", "(print x y z)"]);
}
/// Structural equality that compares record values by field content,
/// not by Arc pointer identity. Required because the compiler may fold
/// record literals at compile-time, yielding a different Arc than one
/// created at runtime via make_record.
fn values_equal(a: &Value, b: &Value) -> bool {
match (a, b) {
(Value::Record(la, va), Value::Record(lb, vb)) => {
la.fields.len() == lb.fields.len()
&& la
.fields
.iter()
.zip(lb.fields.iter())
.all(|((ka, _), (kb, _))| ka == kb)
&& va
.iter()
.zip(vb.iter())
.all(|(x, y)| values_equal(x, y))
}
(Value::Tuple(a), Value::Tuple(b)) => {
a.len() == b.len() && a.iter().zip(b.iter()).all(|(x, y)| values_equal(x, y))
}
_ => a == b,
}
}
fn register_testing(env: &Environment) {
let any_ty = StaticType::Function(Box::new(Signature {
params: StaticType::Any,
ret: StaticType::Void,
}));
// (assert cond) / (assert cond "msg")
env.register_native_fn("assert", any_ty.clone(), Purity::Impure, |args| {
let cond = match args.first() {
Some(Value::Bool(b)) => *b,
Some(other) => panic!("assert expects a bool, got {}", other),
None => panic!("assert requires at least 1 argument"),
};
if !cond {
match args.get(1) {
Some(Value::Text(msg)) => panic!("assertion failed: {}", msg),
_ => panic!("assertion failed"),
}
}
Value::Void
})
.doc("Asserts that a condition is true. Panics with an optional message on failure.")
.examples(&["(assert (= 1 1))", "(assert (> x 0) \"x must be positive\")"]);
// (assert-eq expected actual) / (assert-eq expected actual "msg")
env.register_native_fn("assert-eq", any_ty.clone(), Purity::Impure, |args| {
if args.len() < 2 {
panic!("assert-eq requires at least 2 arguments");
}
let expected = &args[0];
let actual = &args[1];
if !values_equal(expected, actual) {
match args.get(2) {
Some(Value::Text(msg)) => panic!(
"assertion failed: {}\n expected: {}\n got: {}",
msg, expected, actual
),
_ => panic!(
"assertion failed: expected {}, got {}",
expected, actual
),
}
}
Value::Void
})
.doc("Asserts that expected == actual. Panics with a diff message on failure.")
.examples(&["(assert-eq 42 (factorial 5 1))", "(assert-eq :ok (status) \"wrong status\")"]);
// (type-of x) -> keyword
let type_of_ty = StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any]),
ret: StaticType::Keyword,
}));
env.register_native_fn("type-of", type_of_ty, Purity::Pure, |args| {
let name = match args.first() {
// Match on Value directly to avoid lossy static_type() conversion
// (e.g. Value::Function → StaticType::Any loses the "function" info).
Some(v) => match v {
Value::Void => "void",
Value::Bool(_) => "bool",
Value::Int(_) => "int",
Value::Float(_) => "float",
Value::DateTime(_) => "datetime",
Value::Text(_) => "text",
Value::Keyword(_) => "keyword",
Value::Tuple(_) => "tuple",
Value::Record(_, _) => "record",
Value::FieldAccessor(_) => "field-accessor",
Value::Function(_) | Value::Closure(_) => "function",
Value::Quote(_) => "quote",
Value::Series(_) => "series",
Value::Stream(_) => "stream",
Value::Cell(c) => {
// Recurse through Cell to get the inner type.
// We can't recurse here directly, so use static_type fallback.
match c.borrow().static_type() {
StaticType::Void => "void",
StaticType::Bool => "bool",
StaticType::Int => "int",
StaticType::Float => "float",
StaticType::DateTime => "datetime",
StaticType::Text => "text",
StaticType::Keyword => "keyword",
StaticType::Record(_) => "record",
StaticType::Series(_) => "series",
StaticType::Stream(_) => "stream",
_ => "unknown",
}
}
},
None => panic!("type-of requires 1 argument"),
};
Value::Keyword(Keyword::intern(name))
})
.doc("Returns the type of a value as a keyword (e.g. :int, :float, :stream).")
.examples(&["(type-of 42)", "(type-of my-stream)"]);
}
+219
View File
@@ -0,0 +1,219 @@
//! RTL registration for file-based market data streams.
//!
//! Bridges the `Arc`-based [`::data_server::DataServer`] cache with the `Rc`-based VM world
//! by registering generator closures that read pre-parsed chunks and convert
//! them into `Value::Record` items for `RootStream::tick()`.
use crate::ast::data_server::get_data_server;
use crate::ast::environment::Environment;
use crate::ast::rtl::streams::{RootStream, StreamNode};
use crate::ast::types::{Keyword, Purity, RecordLayout, Signature, StaticType, Value};
use std::rc::Rc;
/// Registers `create-m1-stream` and `create-tick-stream` as RTL functions.
pub fn register(env: &Environment) {
register_m1_stream(env);
register_tick_stream(env);
}
// -- M1 Stream ---------------------------------------------------------------
fn register_m1_stream(env: &Environment) {
let generators = env.pipeline_generators.clone();
let m1_layout = RecordLayout::get_or_create(vec![
(Keyword::intern("time"), StaticType::DateTime),
(Keyword::intern("open"), StaticType::Float),
(Keyword::intern("high"), StaticType::Float),
(Keyword::intern("low"), StaticType::Float),
(Keyword::intern("close"), StaticType::Float),
(Keyword::intern("spread"), StaticType::Float),
(Keyword::intern("volume"), StaticType::Int),
]);
let ret_type = StaticType::Stream(Box::new(StaticType::Record(m1_layout.clone())));
env.register_native_fn(
"create-m1-stream",
StaticType::Function(Box::new(Signature {
params: StaticType::Any,
ret: ret_type,
})),
Purity::Impure,
move |args: &[Value]| {
let symbol = match &args[0] {
Value::Text(s) => s.as_ref(),
_ => panic!("create-m1-stream: first argument must be a string"),
};
let from_ms = args.get(1).map(|v| match v {
Value::DateTime(ms) => *ms,
_ => panic!("create-m1-stream: 'from' must be a DateTime"),
});
let to_ms = args.get(2).map(|v| match v {
Value::DateTime(ms) => *ms,
_ => panic!("create-m1-stream: 'to' must be a DateTime"),
});
let server = get_data_server()
.expect("create-m1-stream: DataServer not initialized. Call init_data_server() first.");
// Unknown symbol → Void (no stream created)
if !server.has_symbol(symbol) {
return Value::Void;
}
let mut iter = server.stream_m1_windowed(symbol, from_ms, to_ms);
// Create the pipeline entry point
let root_stream = Rc::new(RootStream::new());
let layout = m1_layout.clone();
let stream_node = StreamNode {
inner: root_stream.clone(),
element_type: StaticType::Record(layout.clone()),
};
// Pre-fetch the first chunk (None if time window is empty)
let mut current_chunk = iter.as_mut().and_then(|it| it.next_chunk());
let mut pos = 0;
let generator = move || -> bool {
loop {
let chunk = match &current_chunk {
Some(c) => c,
None => return false,
};
if pos < chunk.len() {
let rec = &chunk[pos];
pos += 1;
let value = Value::Record(
layout.clone(),
Rc::new(vec![
Value::DateTime(rec.time_ms),
Value::Float(rec.open),
Value::Float(rec.high),
Value::Float(rec.low),
Value::Float(rec.close),
Value::Float(rec.spread),
Value::Int(rec.volume),
]),
);
root_stream.tick(value);
return true;
}
// Advance to next chunk (triggers prefetch internally)
current_chunk = iter.as_mut().and_then(|it| it.next_chunk());
pos = 0;
}
};
generators.borrow_mut().push(Box::new(generator));
Value::Stream(Rc::new(stream_node))
},
)
.doc("Creates a stream of M1 (minute) OHLCV bars from tick data files.")
.description("Reads pre-cached binary data from the DataServer. Optional from/to DateTime arguments restrict the time window (millisecond precision). Returns Void if the symbol is unknown. Returns an empty (immediately exhausted) stream if the symbol exists but the time window contains no data.")
.examples(&[
"(def ohlc (create-m1-stream \"EURUSD\"))",
"(def ohlc (create-m1-stream \"EURUSD\" (date \"2020-01-01\") (date \"2020-12-31\")))",
]);
}
// -- Tick Stream -------------------------------------------------------------
fn register_tick_stream(env: &Environment) {
let generators = env.pipeline_generators.clone();
let tick_layout = RecordLayout::get_or_create(vec![
(Keyword::intern("time"), StaticType::DateTime),
(Keyword::intern("ask"), StaticType::Float),
(Keyword::intern("bid"), StaticType::Float),
]);
let ret_type = StaticType::Stream(Box::new(StaticType::Record(tick_layout.clone())));
env.register_native_fn(
"create-tick-stream",
StaticType::Function(Box::new(Signature {
params: StaticType::Any,
ret: ret_type,
})),
Purity::Impure,
move |args: &[Value]| {
let symbol = match &args[0] {
Value::Text(s) => s.as_ref(),
_ => panic!("create-tick-stream: first argument must be a string"),
};
let from_ms = args.get(1).map(|v| match v {
Value::DateTime(ms) => *ms,
_ => panic!("create-tick-stream: 'from' must be a DateTime"),
});
let to_ms = args.get(2).map(|v| match v {
Value::DateTime(ms) => *ms,
_ => panic!("create-tick-stream: 'to' must be a DateTime"),
});
let server = get_data_server()
.expect("create-tick-stream: DataServer not initialized.");
// Unknown symbol → Void (no stream created)
if !server.has_symbol(symbol) {
return Value::Void;
}
let mut iter = server.stream_tick_windowed(symbol, from_ms, to_ms);
let root_stream = Rc::new(RootStream::new());
let layout = tick_layout.clone();
let stream_node = StreamNode {
inner: root_stream.clone(),
element_type: StaticType::Record(layout.clone()),
};
// Pre-fetch the first chunk (None if time window is empty)
let mut current_chunk = iter.as_mut().and_then(|it| it.next_chunk());
let mut pos = 0;
let generator = move || -> bool {
loop {
let chunk = match &current_chunk {
Some(c) => c,
None => return false,
};
if pos < chunk.len() {
let rec = &chunk[pos];
pos += 1;
let value = Value::Record(
layout.clone(),
Rc::new(vec![
Value::DateTime(rec.time_ms),
Value::Float(rec.ask),
Value::Float(rec.bid),
]),
);
root_stream.tick(value);
return true;
}
current_chunk = iter.as_mut().and_then(|it| it.next_chunk());
pos = 0;
}
};
generators.borrow_mut().push(Box::new(generator));
Value::Stream(Rc::new(stream_node))
},
)
.doc("Creates a stream of tick data (ask/bid) from tick data files.")
.description("Reads pre-cached binary data from the DataServer. Optional from/to DateTime arguments restrict the time window (millisecond precision). Returns Void if the symbol is unknown. Returns an empty (immediately exhausted) stream if the symbol exists but the time window contains no data.")
.examples(&[
"(def ticks (create-tick-stream \"EURUSD\"))",
"(def ticks (create-tick-stream \"EURUSD\" (date \"2020-01-01\") (date \"2020-12-31\")))",
]);
}
+4 -2
View File
@@ -25,7 +25,9 @@ pub fn register(env: &Environment) {
}
}
Value::Void
});
}).doc("Parses a date string and returns a datetime (milliseconds since epoch).")
.description("Supported formats: \"YYYY-MM-DD\" and \"YYYY-MM-DD HH:MM:SS\".")
.examples(&["(date \"2024-01-15\")", "(date \"2024-01-15 09:30:00)\""]);
let now_ty = StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![]),
@@ -35,5 +37,5 @@ pub fn register(env: &Environment) {
env.register_native_fn("now", now_ty, Purity::SideEffectFree, |_| {
let ts = chrono::Utc::now().timestamp_millis();
Value::DateTime(ts)
});
}).doc("Returns the current UTC timestamp as milliseconds since the Unix epoch.");
}
+94
View File
@@ -0,0 +1,94 @@
use std::cell::RefCell;
use std::collections::HashMap;
use std::rc::Rc;
use crate::ast::compiler::call_hooks::RtlCompilerHook;
/// A generator function for data pipelines. Returns `true` while data is available.
pub type PipelineGenerator = Box<dyn FnMut() -> bool>;
/// Documentation entry for a single RTL symbol (function or constant).
/// Populated via the builder returned by `register_native_fn` / `register_constant`.
pub struct RtlDocEntry {
pub name: String,
/// Short one-line description (required).
pub one_liner: &'static str,
/// Optional longer explanation.
pub description: Option<&'static str>,
/// Optional Myc code examples.
pub examples: Option<&'static [&'static str]>,
}
/// Builder returned by `register_native_fn` and `register_constant`.
/// Call `.doc(...)` to attach documentation; optional `.description(...)` and `.examples(...)`
/// can be chained. The entry is written to the registry when the builder is dropped.
/// Optionally, call `.with_compiler_hook(...)` to register an [`RtlCompilerHook`]
/// for this slot — keyed by its global slot index, not by name.
pub struct RtlRegistration {
name: String,
slot_idx: u32,
rtl_docs: Rc<RefCell<Vec<RtlDocEntry>>>,
hook_registry: Rc<RefCell<HashMap<u32, Rc<dyn RtlCompilerHook>>>>,
one_liner: Option<&'static str>,
description: Option<&'static str>,
examples: Option<&'static [&'static str]>,
}
impl RtlRegistration {
pub(crate) fn new(
name: String,
slot_idx: u32,
rtl_docs: Rc<RefCell<Vec<RtlDocEntry>>>,
hook_registry: Rc<RefCell<HashMap<u32, Rc<dyn RtlCompilerHook>>>>,
) -> Self {
Self {
name,
slot_idx,
rtl_docs,
hook_registry,
one_liner: None,
description: None,
examples: None,
}
}
/// Attach a required one-line description.
pub fn doc(mut self, one_liner: &'static str) -> Self {
self.one_liner = Some(one_liner);
self
}
/// Attach an optional longer description (must call `.doc()` first).
pub fn description(mut self, desc: &'static str) -> Self {
self.description = Some(desc);
self
}
/// Attach optional usage examples as Myc code strings.
pub fn examples(mut self, ex: &'static [&'static str]) -> Self {
self.examples = Some(ex);
self
}
/// Register a compiler hook for this slot.
/// Hooks are keyed by the slot's global index — no name matching in the compiler.
pub fn with_compiler_hook(self, hook: Rc<dyn RtlCompilerHook>) -> Self {
self.hook_registry
.borrow_mut()
.insert(self.slot_idx, hook);
self
}
}
impl Drop for RtlRegistration {
fn drop(&mut self) {
if let Some(one_liner) = self.one_liner {
self.rtl_docs.borrow_mut().push(RtlDocEntry {
name: self.name.clone(),
one_liner,
description: self.description,
examples: self.examples,
});
}
}
}
+45 -43
View File
@@ -1,11 +1,15 @@
use crate::ast::environment::Environment;
use crate::ast::types::{Purity, Signature, StaticType, Value};
use crate::ast::types::{NativeFunction, Purity, Signature, StaticType, Value};
use std::cell::RefCell;
use std::f64::consts;
use std::rc::Rc;
pub fn register(env: &Environment) {
// Constants
env.register_constant("PI", StaticType::Float, Value::Float(consts::PI));
env.register_constant("E", StaticType::Float, Value::Float(consts::E));
env.register_constant("PI", StaticType::Float, Value::Float(consts::PI))
.doc("Mathematical constant π ≈ 3.14159265.");
env.register_constant("E", StaticType::Float, Value::Float(consts::E))
.doc("Mathematical constant e ≈ 2.71828182 (Euler's number).");
// Helper to get f64 from Value (Int or Float)
fn to_f64(v: &Value) -> Option<f64> {
@@ -17,7 +21,7 @@ pub fn register(env: &Environment) {
}
// Unary functions (float -> float)
let register_unary = |name: &'static str, f: fn(f64) -> f64| {
let register_unary = |name: &'static str, f: fn(f64) -> f64, doc: &'static str| {
let ty = StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Float]),
ret: StaticType::Float,
@@ -28,11 +32,11 @@ pub fn register(env: &Environment) {
} else {
Value::Void
}
});
}).doc(doc);
};
// Unary functions (float -> int)
let register_to_int = |name: &'static str, f: fn(f64) -> f64| {
let register_to_int = |name: &'static str, f: fn(f64) -> f64, doc: &'static str| {
let ty = StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Float]),
ret: StaticType::Int,
@@ -43,29 +47,29 @@ pub fn register(env: &Environment) {
} else {
Value::Void
}
});
}).doc(doc);
};
register_unary("sqrt", f64::sqrt);
register_unary("sin", f64::sin);
register_unary("cos", f64::cos);
register_unary("tan", f64::tan);
register_unary("asin", f64::asin);
register_unary("acos", f64::acos);
register_unary("atan", f64::atan);
register_unary("exp", f64::exp);
register_unary("ln", f64::ln);
register_unary("log10", f64::log10);
register_unary("fract", f64::fract);
register_unary("sqrt", f64::sqrt, "Returns the square root of x.");
register_unary("sin", f64::sin, "Returns the sine of x (radians).");
register_unary("cos", f64::cos, "Returns the cosine of x (radians).");
register_unary("tan", f64::tan, "Returns the tangent of x (radians).");
register_unary("asin", f64::asin, "Returns the arcsine of x in radians.");
register_unary("acos", f64::acos, "Returns the arccosine of x in radians.");
register_unary("atan", f64::atan, "Returns the arctangent of x in radians.");
register_unary("exp", f64::exp, "Returns e raised to the power of x.");
register_unary("ln", f64::ln, "Returns the natural logarithm of x.");
register_unary("log10",f64::log10,"Returns the base-10 logarithm of x.");
register_unary("fract",f64::fract,"Returns the fractional part of x.");
// Rounding functions returning Int
register_to_int("round", f64::round);
register_to_int("floor", f64::floor);
register_to_int("ceil", f64::ceil);
register_to_int("trunc", f64::trunc);
register_to_int("round", f64::round, "Rounds x to the nearest integer.");
register_to_int("floor", f64::floor, "Rounds x down to the largest integer ≤ x.");
register_to_int("ceil", f64::ceil, "Rounds x up to the smallest integer ≥ x.");
register_to_int("trunc", f64::trunc, "Truncates x towards zero.");
// Binary functions (float, float -> float)
let register_binary = |name: &'static str, f: fn(f64, f64) -> f64| {
let register_binary = |name: &'static str, f: fn(f64, f64) -> f64, doc: &'static str| {
let ty = StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Float, StaticType::Float]),
ret: StaticType::Float,
@@ -76,12 +80,12 @@ pub fn register(env: &Environment) {
} else {
Value::Void
}
});
}).doc(doc);
};
register_binary("atan2", f64::atan2);
register_binary("pow", f64::powf);
register_binary("log", f64::log);
register_binary("atan2", f64::atan2, "Returns the arctangent of y/x, using signs to determine the quadrant.");
register_binary("pow", f64::powf, "Returns x raised to the power y.");
register_binary("log", f64::log, "Returns the logarithm of x in the given base: (log base x).");
// Specialized abs to handle Int -> Int, Float -> Float
let abs_ty = StaticType::Function(Box::new(Signature {
@@ -97,12 +101,13 @@ pub fn register(env: &Environment) {
Value::Float(f) => Value::Float(f.abs()),
_ => Value::Void,
}
});
}).doc("Returns the absolute value of a number. Works for both int and float.")
.examples(&["(abs -5)", "(abs -3.14)"]);
// Variadic min / max
// Variadic min / max — each argument must be numeric
let variadic_ty = StaticType::Function(Box::new(Signature {
params: StaticType::Any,
ret: StaticType::Any,
params: StaticType::Variadic(Box::new(StaticType::Float)),
ret: StaticType::Float,
}));
env.register_native_fn("min", variadic_ty.clone(), Purity::Pure, |args| {
@@ -118,7 +123,7 @@ pub fn register(env: &Environment) {
}
}
best
});
}).doc("Returns the minimum of its arguments. Variadic: (min 3 1 2) => 1.");
env.register_native_fn("max", variadic_ty, Purity::Pure, |args| {
if args.is_empty() {
@@ -133,7 +138,7 @@ pub fn register(env: &Environment) {
}
}
best
});
}).doc("Returns the maximum of its arguments. Variadic: (max 3 1 2) => 3.");
// Random generator factory (Closure-based)
let generator_sig = Signature {
@@ -152,11 +157,7 @@ pub fn register(env: &Environment) {
},
]);
env.register_native_fn(
"make-random",
make_random_ty,
Purity::SideEffectFree,
|args| {
env.register_native_fn("make-random", make_random_ty, Purity::SideEffectFree, |args| {
let rng = if args.is_empty() {
fastrand::Rng::new()
} else if let Value::Int(s) = args[0] {
@@ -165,12 +166,13 @@ pub fn register(env: &Environment) {
fastrand::Rng::new()
};
let prng = std::rc::Rc::new(std::cell::RefCell::new(rng));
let prng = Rc::new(RefCell::new(rng));
Value::Function(std::rc::Rc::new(crate::ast::types::NativeFunction {
func: std::rc::Rc::new(move |_args| Value::Float(prng.borrow_mut().f64())),
Value::Function(Rc::new(NativeFunction {
func: Rc::new(move |_args| Value::Float(prng.borrow_mut().f64())),
purity: Purity::Impure,
}))
},
);
}).doc("Creates a random number generator function returning floats in [0, 1).")
.description("Optionally seeded: (make-random seed) for reproducible sequences.")
.examples(&["(def rng (make-random 42))", "(rng)"]);
}
+98
View File
@@ -1,17 +1,115 @@
pub mod core;
pub mod data_streams;
pub mod datetime;
pub mod docs;
pub mod intrinsics;
pub mod math;
pub mod series;
pub mod streams;
pub mod type_registry;
pub use docs::{PipelineGenerator, RtlDocEntry, RtlRegistration};
use crate::ast::environment::Environment;
pub fn register(env: &Environment) {
core::register(env);
data_streams::register(env);
datetime::register(env);
math::register(env);
series::register(env);
streams::register(env);
}
/// Registers a native RTL function and optionally attaches documentation.
///
/// Required: `doc:` — a short one-line description.
/// Optional: `description:` — a longer explanation.
/// Optional: `examples:` — a `&'static [&'static str]` of Myc code snippets.
///
/// # Example
/// ```ignore
/// rtl_fn!(env, "+",
/// doc: "Adds two values.",
/// add_ty, Pure, |args| { ... }
/// );
///
/// rtl_fn!(env, "push",
/// doc: "Pushes a new record into a series.",
/// description: "The record must match the series layout.",
/// examples: &["(push my_ticks {:price 10.5})"],
/// push_ty, Impure, |args| { ... }
/// );
/// ```
#[macro_export]
macro_rules! rtl_fn {
// With description and examples
($env:expr, $name:expr,
doc: $doc:expr,
description: $desc:expr,
examples: $ex:expr,
$ty:expr, $purity:expr, $impl:expr $(,)?) => {
$env.register_native_fn($name, $ty, $purity, $impl)
.doc($doc)
.description($desc)
.examples($ex);
};
// With description only
($env:expr, $name:expr,
doc: $doc:expr,
description: $desc:expr,
$ty:expr, $purity:expr, $impl:expr $(,)?) => {
$env.register_native_fn($name, $ty, $purity, $impl)
.doc($doc)
.description($desc);
};
// With examples only
($env:expr, $name:expr,
doc: $doc:expr,
examples: $ex:expr,
$ty:expr, $purity:expr, $impl:expr $(,)?) => {
$env.register_native_fn($name, $ty, $purity, $impl)
.doc($doc)
.examples($ex);
};
// doc only
($env:expr, $name:expr,
doc: $doc:expr,
$ty:expr, $purity:expr, $impl:expr $(,)?) => {
$env.register_native_fn($name, $ty, $purity, $impl)
.doc($doc);
};
}
/// Registers a native RTL constant and optionally attaches documentation.
/// Same optional fields as `rtl_fn!`.
#[macro_export]
macro_rules! rtl_const {
// With description and examples
($env:expr, $name:expr,
doc: $doc:expr,
description: $desc:expr,
examples: $ex:expr,
$ty:expr, $val:expr $(,)?) => {
$env.register_constant($name, $ty, $val)
.doc($doc)
.description($desc)
.examples($ex);
};
// With description only
($env:expr, $name:expr,
doc: $doc:expr,
description: $desc:expr,
$ty:expr, $val:expr $(,)?) => {
$env.register_constant($name, $ty, $val)
.doc($doc)
.description($desc);
};
// doc only
($env:expr, $name:expr,
doc: $doc:expr,
$ty:expr, $val:expr $(,)?) => {
$env.register_constant($name, $ty, $val)
.doc($doc);
};
}
+29
View File
@@ -0,0 +1,29 @@
;; Myc System Library
;; Standard macros and core utilities.
(macro while [cond body]
`((fn [] (if ~cond
(do ~body (again))
)))
)
(macro repeat [var limit body]
`((fn [~var __limit]
(if (< ~var __limit)
(do
~body
(again (+ ~var 1) __limit)
)
))
0 ~limit)
)
;; Creates a stateful cache (Series) from a stateless stream.
;; The element type is inferred from the stream's item type.
(macro cache [lookback src]
`(do
(def data (series ~lookback))
(pipe [~src] (fn [s] (do (push data s))))
data
)
)
@@ -4,9 +4,7 @@ use std::fmt::{self, Debug};
use std::rc::Rc;
use std::sync::Arc;
use crate::ast::environment::Environment;
use crate::ast::types::{Keyword, Object, RecordLayout, Series, StaticType, Value};
use crate::ast::types::{Purity, Signature};
use crate::ast::types::{Keyword, PushableStorage, RecordLayout, SeriesStorage, StaticType, Value};
// ============================================================================
// 1. Core Storage Engine
@@ -77,22 +75,44 @@ impl<T> RingBuffer<T> {
/// (No pointers/heaps like String or Record).
pub trait ScalarValue: Copy + Debug + 'static {
fn to_value(self) -> Value;
/// Converts a dynamic `Value` back to this scalar type, returning `None` on mismatch.
fn from_value(v: Value) -> Option<Self>;
/// Returns the `StaticType` corresponding to this scalar type.
fn static_type() -> StaticType;
}
impl ScalarValue for f64 {
fn to_value(self) -> Value {
Value::Float(self)
}
fn from_value(v: Value) -> Option<Self> {
v.as_float()
}
fn static_type() -> StaticType {
StaticType::Float
}
}
impl ScalarValue for i64 {
fn to_value(self) -> Value {
Value::Int(self)
}
fn from_value(v: Value) -> Option<Self> {
v.as_int()
}
fn static_type() -> StaticType {
StaticType::Int
}
}
impl ScalarValue for bool {
fn to_value(self) -> Value {
Value::Bool(self)
}
fn from_value(v: Value) -> Option<Self> {
if let Value::Bool(b) = v { Some(b) } else { None }
}
fn static_type() -> StaticType {
StaticType::Bool
}
}
/// A highly optimized, type-safe series for primitive values (Float, Int, Bool).
@@ -122,20 +142,19 @@ impl<T: ScalarValue> Debug for ScalarSeries<T> {
}
}
// Makes the series usable as a dynamic Object for the VM.
impl<T: ScalarValue> Object for ScalarSeries<T> {
fn type_name(&self) -> &'static str {
impl<T: ScalarValue> SeriesStorage for ScalarSeries<T> {
fn series_type_name(&self) -> &'static str {
self.type_name
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
T::static_type()
}
}
impl<T: ScalarValue> Series for ScalarSeries<T> {
fn get_item(&self, index: usize) -> Option<Value> {
self.data.borrow().get(index).map(|&v| v.to_value())
}
@@ -145,6 +164,17 @@ impl<T: ScalarValue> Series for ScalarSeries<T> {
fn total_count(&self) -> u64 {
self.data.borrow().total_count()
}
fn as_pushable(&self) -> Option<&dyn PushableStorage> {
Some(self)
}
}
impl<T: ScalarValue> PushableStorage for ScalarSeries<T> {
fn push_value(&self, value: Value) {
if let Some(v) = T::from_value(value) {
self.data.borrow_mut().push(v);
}
}
}
// ============================================================================
@@ -171,19 +201,19 @@ impl Debug for ValueSeries {
}
}
impl Object for ValueSeries {
fn type_name(&self) -> &'static str {
impl SeriesStorage for ValueSeries {
fn series_type_name(&self) -> &'static str {
"ValueSeries"
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
StaticType::Any
}
}
impl Series for ValueSeries {
fn get_item(&self, index: usize) -> Option<Value> {
self.data.borrow().get(index).cloned()
}
@@ -193,48 +223,29 @@ impl Series for ValueSeries {
fn total_count(&self) -> u64 {
self.data.borrow().total_count()
}
fn as_pushable(&self) -> Option<&dyn PushableStorage> {
Some(self)
}
}
impl PushableStorage for ValueSeries {
fn push_value(&self, value: Value) {
self.data.borrow_mut().push(value);
}
}
// ============================================================================
// 3. RecordSeries (SoA - Struct of Arrays)
// ============================================================================
/// An interface allowing iteration over fields of a RecordSeries independently
/// of the exact type (Type Erasure for member arrays).
pub trait SeriesMember: Object + Series {
/// Pushes a dynamic Value into the series, performing the necessary downcast.
fn push_value(&self, value: Value);
}
/// Marker supertrait for RecordSeries field buffers.
/// Ensures each field buffer supports indexed access, dynamic dispatch, and value insertion.
pub trait SeriesMember: PushableStorage {}
impl SeriesMember for ScalarSeries<f64> {
fn push_value(&self, value: Value) {
if let Some(v) = value.as_float() {
self.data.borrow_mut().push(v);
}
}
}
impl SeriesMember for ScalarSeries<i64> {
fn push_value(&self, value: Value) {
if let Some(v) = value.as_int() {
self.data.borrow_mut().push(v);
}
}
}
impl SeriesMember for ScalarSeries<bool> {
fn push_value(&self, value: Value) {
if let Value::Bool(v) = value {
self.data.borrow_mut().push(v);
}
}
}
impl SeriesMember for ValueSeries {
fn push_value(&self, value: Value) {
self.data.borrow_mut().push(value);
}
}
impl SeriesMember for ScalarSeries<f64> {}
impl SeriesMember for ScalarSeries<i64> {}
impl SeriesMember for ScalarSeries<bool> {}
impl SeriesMember for ValueSeries {}
/// The "Struct of Arrays" implementation for Records (e.g., Ticks).
/// Instead of holding a large array of Records (AoS), this series
@@ -308,24 +319,24 @@ impl Debug for RecordSeries {
f,
"RecordSeries[fields: {}, len: {}]",
self.fields.len(),
Series::len(self)
self.len()
)
}
}
impl Object for RecordSeries {
fn type_name(&self) -> &'static str {
impl SeriesStorage for RecordSeries {
fn series_type_name(&self) -> &'static str {
"RecordSeries"
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
StaticType::Record(self.layout.clone())
}
}
impl Series for RecordSeries {
fn get_item(&self, index: usize) -> Option<Value> {
self.get_record(index)
}
@@ -338,6 +349,23 @@ impl Series for RecordSeries {
.map(|f| f.borrow().total_count())
.unwrap_or(0)
}
fn as_pushable(&self) -> Option<&dyn PushableStorage> {
Some(self)
}
}
impl PushableStorage for RecordSeries {
fn push_value(&self, value: Value) {
if let Value::Record(_, values) = value {
for (i, v) in values.iter().enumerate() {
if let Some(f) = self.fields.get(i) {
f.borrow().push_value(v.clone());
}
}
} else {
panic!("push to RecordSeries expects a record");
}
}
}
// ============================================================================
@@ -362,24 +390,24 @@ impl Debug for SeriesView {
f,
"SeriesView[field: {}, len: {}]",
self.field_name.name(),
Series::len(self)
self.len()
)
}
}
impl Object for SeriesView {
fn type_name(&self) -> &'static str {
impl SeriesStorage for SeriesView {
fn series_type_name(&self) -> &'static str {
"SeriesView"
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
self.inner.borrow().inner_static_type()
}
}
impl Series for SeriesView {
fn get_item(&self, index: usize) -> Option<Value> {
self.inner.borrow().get_item(index)
}
@@ -413,19 +441,19 @@ impl<T: ScalarValue> Debug for SharedSeries<T> {
}
}
impl<T: ScalarValue> Object for SharedSeries<T> {
fn type_name(&self) -> &'static str {
impl<T: ScalarValue> SeriesStorage for SharedSeries<T> {
fn series_type_name(&self) -> &'static str {
self.type_name
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
T::static_type()
}
}
impl<T: ScalarValue> Series for SharedSeries<T> {
fn get_item(&self, index: usize) -> Option<Value> {
self.buffer
.borrow()
@@ -451,19 +479,19 @@ impl Debug for SharedValueSeries {
}
}
impl Object for SharedValueSeries {
fn type_name(&self) -> &'static str {
impl SeriesStorage for SharedValueSeries {
fn series_type_name(&self) -> &'static str {
"SharedValueSeries"
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
StaticType::Any
}
}
impl Series for SharedValueSeries {
fn get_item(&self, index: usize) -> Option<Value> {
self.buffer.borrow().get(index).cloned()
}
@@ -487,24 +515,24 @@ impl Debug for SharedRecordSeries {
f,
"SharedRecordSeries[fields: {}, len: {}]",
self.field_buffers.len(),
Series::len(self)
self.len()
)
}
}
impl Object for SharedRecordSeries {
fn type_name(&self) -> &'static str {
impl SeriesStorage for SharedRecordSeries {
fn series_type_name(&self) -> &'static str {
"SharedRecordSeries"
}
fn as_any(&self) -> &dyn Any {
self
}
fn as_series(&self) -> Option<&dyn Series> {
Some(self)
fn into_rc_any(self: Rc<Self>) -> Rc<dyn Any> {
self
}
fn inner_static_type(&self) -> StaticType {
StaticType::Record(self.layout.clone())
}
}
impl Series for SharedRecordSeries {
fn get_item(&self, index: usize) -> Option<Value> {
if self.field_buffers.is_empty() {
return None;
@@ -533,116 +561,3 @@ impl Series for SharedRecordSeries {
.unwrap_or(0)
}
}
// ============================================================================
// 5. Script Integration (RTL Registration)
// ============================================================================
pub fn register(env: &Environment) {
// (series lookback_limit template_or_type_keyword) -> Series
env.register_native_fn(
"series",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Any]),
ret: StaticType::Any,
})),
Purity::Impure,
|args: &[Value]| {
let lookback_limit = args[0].as_int().unwrap_or(0) as usize;
let arg = &args[1];
match arg {
Value::Keyword(k) => {
match k.name().to_lowercase().as_str() {
"float" => Value::Object(Rc::new(ScalarSeries::<f64>::new("FloatSeries", lookback_limit))),
"int" | "datetime" => Value::Object(Rc::new(ScalarSeries::<i64>::new("IntSeries", lookback_limit))),
"bool" => Value::Object(Rc::new(ScalarSeries::<bool>::new("BoolSeries", lookback_limit))),
"text" => Value::Object(Rc::new(ValueSeries::new(lookback_limit))),
_ => panic!("Unknown or unsupported series type keyword: :{}", k.name()),
}
}
Value::Record(schema_layout, schema_values) => {
// Interpret the record as a schema: { :field :type_keyword }
let mut fields = Vec::with_capacity(schema_layout.fields.len());
for (i, (name, _)) in schema_layout.fields.iter().enumerate() {
let type_keyword = match &schema_values[i] {
Value::Keyword(tk) => tk.name().to_lowercase(),
_ => panic!("Field :{} in series schema must be a type keyword (e.g., :float, :int)", name.name()),
};
let ty = match type_keyword.as_str() {
"float" => StaticType::Float,
"int" => StaticType::Int,
"bool" => StaticType::Bool,
"datetime" => StaticType::DateTime,
"text" => StaticType::Text,
_ => panic!("Unsupported type keyword :{} for series field :{}", type_keyword, name.name()),
};
fields.push((*name, ty));
}
let final_layout = RecordLayout::get_or_create(fields);
Value::Object(Rc::new(RecordSeries::new(final_layout, lookback_limit)))
}
_ => panic!("series expects a lookback_limit and a type keyword (:float, :int...) or a schema record (e.g., {{:price :float}})"),
}
},
);
// (push series value) -> Void
env.register_native_fn(
"push",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any, StaticType::Any]),
ret: StaticType::Void,
})),
Purity::Impure,
|args: &[Value]| {
let (series_val, val) = (&args[0], &args[1]);
if let Value::Object(obj) = series_val {
let any = obj.as_any();
// Polymorphic push logic
if let Some(rs) = any.downcast_ref::<RecordSeries>() {
if let Value::Record(_, values) = val {
for (i, v) in values.iter().enumerate() {
if let Some(f) = rs.fields.get(i) {
f.borrow().push_value(v.clone());
}
}
} else {
panic!("push to RecordSeries expects a record");
}
} else if let Some(s) = any.downcast_ref::<ScalarSeries<f64>>() {
s.push_value(val.clone());
} else if let Some(s) = any.downcast_ref::<ScalarSeries<i64>>() {
s.push_value(val.clone());
} else if let Some(s) = any.downcast_ref::<ScalarSeries<bool>>() {
s.push_value(val.clone());
} else if let Some(s) = any.downcast_ref::<ValueSeries>() {
s.push_value(val.clone());
} else {
panic!("Object is not a mutable series");
}
return Value::Void;
}
panic!("push expects a series object as the first argument")
},
);
// (len series) -> Int
env.register_native_fn(
"len",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any]),
ret: StaticType::Int,
})),
Purity::Pure,
|args: &[Value]| {
if let Value::Object(obj) = &args[0]
&& let Some(s) = obj.as_series()
{
return Value::Int(s.len() as i64);
}
Value::Int(0)
},
);
}
+230
View File
@@ -0,0 +1,230 @@
pub mod data;
pub use data::*;
use std::collections::HashMap;
use std::rc::Rc;
use std::sync::Arc;
use crate::ast::compiler::call_hooks::{InferenceAccess, RtlCompilerHook};
use crate::ast::diagnostics::Diagnostics;
use crate::ast::environment::Environment;
use crate::ast::nodes::{IdentifierBinding, Node, NodeKind, TypedNode, TypedPhase};
use crate::ast::types::{NativeFunction, NodeIdentity, Purity, SeriesStorage, Signature, SourceLocation, StaticType, Value};
/// Creates a type-specialized series with the given lookback depth.
/// Dispatches on `StaticType` to allocate the optimal storage backend:
/// - `Float` → `ScalarSeries<f64>`
/// - `Int`/`DateTime` → `ScalarSeries<i64>`
/// - `Bool` → `ScalarSeries<bool>`
/// - `Record` → `RecordSeries` (SoA layout)
/// - Anything else → `ValueSeries` (boxed fallback)
pub fn create_typed_series(element_type: &StaticType, lookback: usize) -> Rc<dyn SeriesStorage> {
match element_type {
StaticType::Float => Rc::new(ScalarSeries::<f64>::new("FloatSeries", lookback)),
StaticType::Int | StaticType::DateTime => Rc::new(ScalarSeries::<i64>::new("IntSeries", lookback)),
StaticType::Bool => Rc::new(ScalarSeries::<bool>::new("BoolSeries", lookback)),
StaticType::Record(layout) => Rc::new(RecordSeries::new(Arc::clone(layout), lookback)),
_ => Rc::new(ValueSeries::new(lookback)),
}
}
// ============================================================================
// 4. Compiler Hooks (registered via RTL bootstrap, no name coupling)
// ============================================================================
/// Compiler hook for `(series n)`.
///
/// - **post_call:** Replaces the `Series(Any)` return type with a fresh TypeVar
/// so that subsequent `push` calls can unify the element type (HM inference).
/// - **finalize:** Replaces the callee with a pre-configured factory closure that
/// directly allocates the correct storage type (e.g. `ScalarSeries::<f64>` for
/// Float, `RecordSeries` with a captured `Arc<RecordLayout>` for record types).
pub struct SeriesHook;
impl RtlCompilerHook for SeriesHook {
fn post_call(
&self,
_args: &TypedNode,
ret_ty: StaticType,
ctx: &dyn InferenceAccess,
_diag: &mut Diagnostics,
) -> StaticType {
if let StaticType::Series(inner) = &ret_ty
&& **inner == StaticType::Any
{
return StaticType::Series(Box::new(ctx.fresh_var()));
}
ret_ty
}
fn finalize(
&self,
_callee: Rc<TypedNode>,
args: Rc<TypedNode>,
node_ty: &StaticType,
_subst: &HashMap<u32, StaticType>,
) -> Option<NodeKind<TypedPhase>> {
let StaticType::Series(inner) = node_ty else { return None };
let NodeKind::Tuple { elements } = &args.kind else { return None };
if elements.len() != 1 {
return None;
}
// Build a factory Value::Function whose closure already knows the exact
// storage type — no keyword encoding, no runtime dispatch.
let elem_ty = inner.as_ref().clone();
let factory_value: Value = Value::Function(Rc::new(NativeFunction {
func: Rc::new(move |args: &[Value]| {
let n = args[0].as_int().unwrap_or(0) as usize;
Value::Series(create_typed_series(&elem_ty, n))
}),
purity: Purity::Impure,
}));
let factory_node = Rc::new(Node {
kind: NodeKind::Constant(factory_value),
ty: StaticType::Any,
identity: NodeIdentity::new(SourceLocation { line: 0, col: 0 }),
comments: Rc::from([]),
});
Some(NodeKind::Call { callee: factory_node, args: Rc::clone(&args) })
}
}
/// Compiler hook for `(push series val)`.
///
/// Unifies the series element TypeVar with the pushed value type, applying
/// Int→Float numeric promotion and record field promotion when needed.
///
/// We intentionally do NOT bake the resolved type into ctx after normal
/// unification. Keeping `Series(TypeVar(n))` in ctx allows a later push
/// to recover the TypeVar ID and rebind it (e.g. Int → Float promotion).
/// All identifier lookups call `apply_subst`, so the resolved type is
/// always correct regardless of what ctx stores.
pub struct PushHook;
impl RtlCompilerHook for PushHook {
fn post_call(
&self,
args: &TypedNode,
ret_ty: StaticType,
ctx: &dyn InferenceAccess,
diag: &mut Diagnostics,
) -> StaticType {
let NodeKind::Tuple { elements } = &args.kind else { return ret_ty };
if elements.len() < 2 {
return ret_ty;
}
let series_arg = &elements[0];
let value_arg = &elements[1];
let StaticType::Series(inner) = &series_arg.ty else { return ret_ty };
let inner_ty = (**inner).clone();
let val_ty = value_arg.ty.clone();
if let NodeKind::Identifier {
binding: IdentifierBinding::Reference(addr),
..
} = &series_arg.kind
{
// Recover the raw inner type from the scope slot (before apply_subst)
// so we can find the TypeVar ID even after the first push resolved it.
let raw_inner = match ctx.get_slot_type(*addr) {
StaticType::Series(ri) => *ri,
_ => inner_ty.clone(),
};
if let Some(promoted) = ctx
.try_numeric_widen(&inner_ty, &val_ty)
.or_else(|| ctx.try_record_promote(&inner_ty, &val_ty))
{
// Rebind the TypeVar to the promoted type so that finalize
// injects the correct schema (e.g. :float instead of :int).
if let StaticType::TypeVar(n) = raw_inner {
ctx.bind_typevar(n, promoted.clone());
}
} else {
ctx.unify(inner_ty, val_ty, diag);
}
} else {
ctx.unify(inner_ty, val_ty, diag);
}
ret_ty
}
}
// ============================================================================
// 5. Script Integration (RTL Registration)
// ============================================================================
pub fn register(env: &Environment) {
// (series lookback_limit) -> Series
// The element type is inferred by the HM type checker from subsequent push calls.
// After type checking, the compiler elaborates (series n) into (series n schema)
// so the runtime always receives the explicit schema argument.
env.register_native_fn(
"series",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Int]),
ret: StaticType::Series(Box::new(StaticType::Any)),
})),
Purity::Impure,
|args: &[Value]| {
// The finalize hook replaces this call with a pre-configured factory
// for resolved element types. This fallback only runs when the element
// type could not be determined at compile time (e.g. push inside a
// nested closure the type checker could not reach).
let lookback_limit = args[0].as_int().unwrap_or(0) as usize;
Value::Series(Rc::new(ValueSeries::new(lookback_limit)))
},
).with_compiler_hook(Rc::new(SeriesHook))
.doc("Creates a new series (ring buffer) with the given lookback limit.")
.description("Takes a single lookback limit (int). The element type is inferred at compile time from subsequent push calls via HM type inference. The compiler injects a second schema argument automatically; the runtime also accepts an explicit schema for cases where inference is not possible.")
.examples(&[
"(series 100)",
"(do (def s (series 5)) (push s 1.5) (s 0))",
]);
// (push series value) -> Void
env.register_native_fn(
"push",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any, StaticType::Any]),
ret: StaticType::Void,
})),
Purity::Impure,
|args: &[Value]| {
let (series_val, val) = (&args[0], &args[1]);
if let Value::Series(s) = series_val {
match s.as_pushable() {
Some(p) => p.push_value(val.clone()),
None => panic!("push expects a mutable series, got read-only {}", s.series_type_name()),
}
return Value::Void;
}
panic!("push expects a series as the first argument")
},
).with_compiler_hook(Rc::new(PushHook))
.doc("Pushes a new value into a series. After push, index 0 returns this value.")
.examples(&[
"(push my-ticks {:price 10.5 :volume 100})",
"(push prices 42.0)",
]);
// (len series) -> Int
env.register_native_fn(
"len",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any]),
ret: StaticType::Int,
})),
Purity::Pure,
|args: &[Value]| {
if let Value::Series(s) = &args[0] {
return Value::Int(s.len() as i64);
}
Value::Int(0)
},
).doc("Returns the current number of items stored in a series.")
.examples(&["(len my-ticks)"]);
}
-536
View File
@@ -1,536 +0,0 @@
use crate::ast::types::{PipeFn, Value};
use std::cell::RefCell;
use std::rc::Rc;
/// A Signal is the "packet" flowing through the reactive pipeline.
/// It represents a value produced at a specific logical time (cycle_id).
#[derive(Debug, Clone)]
pub struct Signal {
pub cycle_id: u64,
pub value: Value,
}
/// A Stream is a stateless provider of signals.
/// It doesn't "own" the data, it just knows how to get the current one.
pub trait Stream {
fn current_signal(&self) -> Option<Signal>;
}
/// An Observer is a node in the pipeline that reacts to new signals.
/// (e.g., a Pipe or a SharedSeries buffer).
pub trait Observer {
/// Notifies the observer about a new signal in the current cycle.
/// `source_index` identifies which input stream provided the value.
fn notify(&mut self, source_index: usize, cycle_id: u64, value: Value);
}
/// A lightweight adapter to map an unknown source index to a specific target index.
/// This prevents index collisions when a Pipe listens to multiple independent RootStreams.
pub struct SourceAdapter {
pub target: Rc<RefCell<dyn Observer>>,
pub target_index: usize,
}
impl Observer for SourceAdapter {
fn notify(&mut self, _ignored_source: usize, cycle_id: u64, value: Value) {
self.target
.borrow_mut()
.notify(self.target_index, cycle_id, value);
}
}
/// Polymorphic Interface for any stream that can accept observers (like Delphi's IStream).
pub trait ObservableStream {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>);
}
impl<T: ObservableStream> ObservableStream for std::cell::RefCell<T> {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.borrow().add_observer(observer);
}
}
/// A generic wrapper to pass ANY ObservableStream (Root, Pipe, etc.) as an Object to the VM.
#[derive(Clone)]
pub struct StreamNode {
pub inner: Rc<dyn ObservableStream>,
}
impl std::fmt::Debug for StreamNode {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "StreamNode")
}
}
impl crate::ast::types::Object for StreamNode {
fn type_name(&self) -> &'static str {
"StreamNode"
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
}
/// The RootStream is the "Clock" and data source of the entire pipeline.
/// It generates the monotonic `cycle_id` and triggers the observers.
pub struct RootStream {
current_cycle: std::cell::Cell<u64>,
observers: RefCell<Vec<Rc<RefCell<dyn Observer>>>>,
}
impl ObservableStream for RootStream {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.observers.borrow_mut().push(observer);
}
}
impl Default for RootStream {
fn default() -> Self {
Self::new()
}
}
impl RootStream {
pub fn new() -> Self {
Self {
current_cycle: std::cell::Cell::new(0),
observers: RefCell::new(Vec::new()),
}
}
/// Advances the pipeline to the next cycle and propagates a value.
pub fn tick(&self, value: Value) {
let next_cycle = self.current_cycle.get() + 1;
self.current_cycle.set(next_cycle);
// Propagate to all observers.
// We use a local borrow of the observers list to keep the cell borrow short.
let obs_list = self.observers.borrow();
for obs in obs_list.iter() {
// Root observers are always at source_index 0.
obs.borrow_mut().notify(0, next_cycle, value.clone());
}
}
pub fn current_cycle(&self) -> u64 {
self.current_cycle.get()
}
pub fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.observers.borrow_mut().push(observer);
}
}
/// A PipeStream is a reactive node that transforms inputs via a lambda.
/// It implements "Barrier Synchronization": It only executes when all inputs
/// have reported a value for the same cycle_id.
pub struct PipeStream {
pub name: String,
/// The inputs this pipe is observing.
/// In a real system, these would be other Streams.
/// For the MVP, we assume the Pipe is notified by the Root or its parents.
pub input_count: usize,
/// Tracks the last cycle_id received from each input.
last_cycle_per_input: Vec<u64>,
/// Stores the current value for each input to construct the argument tuple.
current_values: Vec<Value>,
/// The current output signal of this pipe.
current_signal: RefCell<Option<Signal>>,
/// The executable closure representing the Lambda. Expects a slice of arguments.
pub executor: Option<Box<PipeFn>>,
/// Observers of THIS pipe.
observers: RefCell<Vec<Rc<RefCell<dyn Observer>>>>,
}
impl PipeStream {
pub fn new(
name: String,
input_count: usize,
executor: Option<Box<PipeFn>>,
) -> Self {
Self {
name,
input_count,
last_cycle_per_input: vec![0; input_count],
current_values: vec![Value::Void; input_count],
current_signal: RefCell::new(None),
executor,
observers: RefCell::new(Vec::new()),
}
}
}
impl ObservableStream for PipeStream {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.observers.borrow_mut().push(observer);
}
}
impl Stream for PipeStream {
fn current_signal(&self) -> Option<Signal> {
self.current_signal.borrow().clone()
}
}
impl Observer for PipeStream {
fn notify(&mut self, source_index: usize, cycle_id: u64, value: Value) {
let barrier_reached = {
if source_index < self.input_count {
self.last_cycle_per_input[source_index] = cycle_id;
self.current_values[source_index] = value;
}
// Check if all inputs reached the same cycle.
self.last_cycle_per_input.iter().all(|&c| c == cycle_id)
};
if barrier_reached {
// 1. Prepare Arguments for Lambda (Current values of all inputs) - NO CLONE NEEDED!
let args = &self.current_values;
// 2. Execute Lambda using the encapsulated VM executor
let result = if let Some(exec) = &mut self.executor {
exec(args)
} else {
self.current_values[0].clone() // Identity bypass (defaults to first input)
};
// 3. Handle Void case! (Filter pattern)
if matches!(result, Value::Void) {
return; // Act as a filter: do not emit, do not push.
}
// 4. Update Current Signal
let new_signal = Signal {
cycle_id,
value: result,
};
*self.current_signal.borrow_mut() = Some(new_signal.clone());
// 5. Notify Observers (Always at source_index 0 of the NEXT pipe)
let obs_list = self.observers.borrow();
for obs in obs_list.iter() {
obs.borrow_mut()
.notify(0, cycle_id, new_signal.value.clone());
}
}
}
}
/// A specialized observer that pushes incoming signals into a SharedSeries buffer.
pub struct SeriesPusher<T: crate::ast::rtl::series::ScalarValue> {
pub buffer: Rc<RefCell<crate::ast::rtl::series::RingBuffer<T>>>,
pub extractor: fn(Value) -> Option<T>,
}
impl<T: crate::ast::rtl::series::ScalarValue> Observer for SeriesPusher<T> {
fn notify(&mut self, _source_index: usize, _cycle_id: u64, value: Value) {
if let Some(v) = (self.extractor)(value) {
self.buffer.borrow_mut().push(v);
}
}
}
/// A specialized observer that pushes incoming signals into a generic SharedValueSeries buffer.
pub struct ValuePusher {
pub buffer: Rc<RefCell<crate::ast::rtl::series::RingBuffer<Value>>>,
}
impl Observer for ValuePusher {
fn notify(&mut self, _source_index: usize, _cycle_id: u64, value: Value) {
self.buffer.borrow_mut().push(value);
}
}
/// A specialized observer that splits a Record into its fields and pushes them into SoA buffers.
pub struct RecordPusher {
pub field_buffers: Vec<Rc<RefCell<dyn crate::ast::rtl::series::SeriesMember>>>,
}
impl Observer for RecordPusher {
fn notify(&mut self, _source_index: usize, _cycle_id: u64, value: Value) {
if let Value::Record(_, values) = value {
for (i, v) in values.iter().enumerate() {
if let Some(buf) = self.field_buffers.get(i) {
buf.borrow_mut().push_value(v.clone());
}
}
}
}
}
/// Factory function to build a specialized pipeline node based on the output type.
/// This keeps the VM "dumb" and moves the buffer selection logic to the RTL.
pub fn build_pipeline_node(
inputs: Vec<Rc<dyn ObservableStream>>,
executor: Box<PipeFn>,
_out_type: &StaticType,
) -> Rc<StreamNode> {
let pipe = Rc::new(RefCell::new(PipeStream::new(
"pipe".to_string(),
inputs.len(),
Some(executor),
)));
// Connect inputs to the pipe
for (i, input) in inputs.into_iter().enumerate() {
let adapter = Rc::new(RefCell::new(SourceAdapter {
target: pipe.clone(),
target_index: i,
}));
input.add_observer(adapter);
}
Rc::new(StreamNode { inner: pipe })
}
pub fn build_map_stream(input: Rc<dyn ObservableStream>, field: Keyword) -> StreamNode {
let executor: Box<PipeFn> = Box::new(move |args: &[Value]| -> Value {
let val = &args[0];
if let Value::Record(layout, values) = val
&& let Some(idx) = layout.index_of(field)
{
return values[idx].clone();
}
Value::Void // In streams, Void acts as a filter
});
let pipe = Rc::new(RefCell::new(PipeStream::new(
format!("map:{}", field.name()),
1,
Some(executor),
)));
let adapter = Rc::new(RefCell::new(SourceAdapter {
target: pipe.clone(),
target_index: 0,
}));
input.add_observer(adapter);
StreamNode {
inner: pipe,
}
}
// ============================================================================
// Script Integration (RTL Registration)
// ============================================================================
use crate::ast::environment::Environment;
use crate::ast::types::{Keyword, Purity, RecordLayout, Signature, StaticType};
pub fn register(env: &Environment) {
// (create-random-ohlc seed limit) -> StreamNode
let generators = env.pipeline_generators.clone();
// Define the OHLC layout for typing
let ohlc_layout = RecordLayout::get_or_create(vec![
(Keyword::intern("open"), StaticType::Float),
(Keyword::intern("high"), StaticType::Float),
(Keyword::intern("low"), StaticType::Float),
(Keyword::intern("close"), StaticType::Float),
]);
env.register_native_fn(
"create-random-ohlc",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Int]),
ret: StaticType::Stream(Box::new(StaticType::Record(ohlc_layout.clone()))),
})),
Purity::Impure, // Modifies global generator registry
move |args: &[Value]| {
if args.len() != 2 {
panic!("create-random-ohlc expects exactly 2 arguments (seed, limit)");
}
let seed = if let Value::Int(s) = args[0] {
s as u64
} else {
0
};
let limit = if let Value::Int(l) = args[1] {
l as usize
} else {
0
};
// 1. Create the RootStream
let root_stream = Rc::new(RootStream::new());
let stream_node = StreamNode {
inner: root_stream.clone(),
};
// 2. Setup the Layout for OHLC records
let layout = RecordLayout::get_or_create(vec![
(Keyword::intern("open"), StaticType::Float),
(Keyword::intern("high"), StaticType::Float),
(Keyword::intern("low"), StaticType::Float),
(Keyword::intern("close"), StaticType::Float),
]);
// 3. Create the stateful generator closure
let mut current_tick = 0;
let mut last_close = 100.0;
// We use a local PRNG instance for reproducibility based on the seed
let mut rng = fastrand::Rng::with_seed(seed);
let generator = move || -> bool {
if current_tick >= limit {
return false; // Exhausted
}
// Generate random OHLC (Random Walk)
let change = (rng.f64() - 0.5) * 2.0;
let open = last_close;
let high = open + (rng.f64() * 2.0).abs();
let low = open - (rng.f64() * 2.0).abs();
let close = open + change;
last_close = close;
let record = Value::Record(
layout.clone(),
Rc::new(vec![
Value::Float(open),
Value::Float(high),
Value::Float(low),
Value::Float(close),
]),
);
// Pump the signal into the RootStream
root_stream.tick(record);
current_tick += 1;
true // Still active
};
// 4. Register the generator in the Environment
generators.borrow_mut().push(Box::new(generator));
// 5. Return the stream reference to the script
Value::Object(Rc::new(stream_node))
},
);
// (create-ticker condition-closure) -> StreamNode
let ticker_generators = env.pipeline_generators.clone();
let globals = env.global_values.clone();
env.register_native_fn(
"create-ticker",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any]), // Expects a closure
ret: StaticType::Any, // Returns StreamNode
})),
Purity::Impure,
move |args: &[Value]| {
if args.len() != 1 {
panic!("create-ticker expects exactly 1 argument (the condition closure)");
}
let closure_obj = if let Value::Object(obj) = &args[0] {
obj.clone()
} else {
panic!("create-ticker expects a closure as its argument");
};
// 1. Create the RootStream
let root_stream = Rc::new(RootStream::new());
let stream_node = StreamNode {
inner: root_stream.clone(),
};
// 2. Setup isolated VM
let mut ticker_vm = crate::ast::vm::VM::new(globals.clone());
let my_closure = closure_obj.clone();
// 3. Create generator
let generator = move || -> bool {
match ticker_vm.run_with_args(my_closure.clone(), &[]) {
Ok(Value::Bool(b)) => {
if b {
// Ticker pulses with a simple `true` value or `Void`
// We use true here so the pipe receives something tangible.
root_stream.tick(Value::Bool(true));
true
} else {
false // Exhausted
}
}
Ok(_) => panic!("create-ticker closure must return a boolean"),
Err(e) => panic!("create-ticker closure execution failed: {}", e),
}
};
// 4. Register the generator
ticker_generators.borrow_mut().push(Box::new(generator));
// 5. Return stream
Value::Object(Rc::new(stream_node))
},
);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::ast::types::Value;
#[test]
fn test_root_to_pipe_flow() {
let root = RootStream::new();
let pipe = Rc::new(RefCell::new(PipeStream::new(
"test-pipe".to_string(),
1,
None,
)));
root.add_observer(pipe.clone());
// Cycle 1: Root ticks 10.0
root.tick(Value::Float(10.0));
let sig = pipe.borrow().current_signal().unwrap();
assert_eq!(sig.cycle_id, 1);
if let Value::Float(v) = sig.value {
assert_eq!(v, 10.0);
} else {
panic!("Value must be Float(10.0)");
}
// Cycle 2: Root ticks 20.0
root.tick(Value::Float(20.0));
let sig2 = pipe.borrow().current_signal().unwrap();
assert_eq!(sig2.cycle_id, 2);
if let Value::Float(v) = sig2.value {
assert_eq!(v, 20.0);
} else {
panic!("Value must be Float(20.0)");
}
}
#[test]
fn test_barrier_sync() {
// Pipe with 2 inputs
let pipe = Rc::new(RefCell::new(PipeStream::new(
"barrier-pipe".to_string(),
2,
None,
)));
// Manual notifications simulate different input streams
pipe.borrow_mut().notify(0, 1, Value::Float(10.0));
assert!(
pipe.borrow().current_signal().is_none(),
"Barrier should NOT be reached after 1st input"
);
pipe.borrow_mut().notify(1, 1, Value::Float(20.0));
assert!(
pipe.borrow().current_signal().is_some(),
"Barrier SHOULD be reached after 2nd input"
);
let sig = pipe.borrow().current_signal().unwrap();
assert_eq!(sig.cycle_id, 1);
}
}
+417
View File
@@ -0,0 +1,417 @@
use crate::ast::compiler::call_hooks::{InferenceAccess, RtlCompilerHook};
use crate::ast::diagnostics::Diagnostics;
use crate::ast::nodes::{Node, NodeKind, TypedNode, TypedPhase};
use crate::ast::rtl::series::create_typed_series;
use crate::ast::types::{
NativeFunction, NodeIdentity, PipeFn, Purity, SeriesStorage, SourceLocation, StaticType,
Value,
};
use crate::ast::vm::{GlobalStore, VM};
use std::collections::HashMap;
use std::rc::Rc;
use super::nodes::build_pipeline_node;
use super::{ObservableStream, StreamNode};
/// Extracts `ObservableStream` references from a runtime `Value`.
/// Accepts a single `StreamNode` or a `Tuple` of `StreamNode`s.
pub(super) fn extract_obs_streams(val: &Value) -> Vec<Rc<dyn ObservableStream>> {
match val {
Value::Tuple(elements) => elements
.iter()
.map(|v| {
if let Value::Stream(s) = v
&& let Some(sn) = s.as_any().downcast_ref::<StreamNode>()
{
sn.inner.clone()
} else {
panic!("pipe: each input must be a StreamNode");
}
})
.collect(),
Value::Stream(s) => {
if let Some(sn) = s.as_any().downcast_ref::<StreamNode>() {
vec![sn.inner.clone()]
} else {
panic!("pipe: input must be a StreamNode");
}
}
_ => panic!("pipe: first argument must be a stream or tuple of streams"),
}
}
/// Builds a `PipeFn` executor from a runtime `Value::Closure` or `Value::Function`.
pub(super) fn build_pipe_executor(val: &Value, globals: GlobalStore) -> Box<PipeFn> {
match val {
Value::Closure(rc) => {
let my_closure = rc.clone();
let mut pipe_vm = VM::new(globals);
Box::new(move |call_args: &[Value]| -> Value {
pipe_vm
.run_with_args(my_closure.clone(), call_args)
.unwrap_or_else(|e| panic!("Pipeline lambda execution failed: {}", e))
})
}
Value::Function(f) => {
let my_func = f.clone();
Box::new(move |call_args: &[Value]| -> Value { (my_func.func)(call_args) })
}
_ => panic!("pipe: second argument must be a function or closure"),
}
}
/// Compiler hook for `(pipe inputs lambda)`.
///
/// - **finalize:** Replaces the `pipe` identifier with a pre-configured factory closure
/// that captures both the resolved output element type and the global store. This mirrors
/// `SeriesHook::finalize` and makes the element type available to `build_pipeline_node`
/// at runtime without a runtime type lookup.
pub struct PipeHook {
pub(super) globals: GlobalStore,
}
impl RtlCompilerHook for PipeHook {
fn post_call(
&self,
args: &TypedNode,
ret_ty: StaticType,
_ctx: &dyn InferenceAccess,
diag: &mut Diagnostics,
) -> StaticType {
// Validate: input stream count must match lambda parameter count.
if let NodeKind::Tuple { elements } = &args.kind
&& elements.len() == 2
{
let expected = match &elements[0].ty {
StaticType::Stream(_) | StaticType::Series(_) => 1,
StaticType::Vector(_, count) => *count,
StaticType::Tuple(elems) => elems.len(),
_ => return ret_ty,
};
if let NodeKind::Lambda { params, .. } = &elements[1].kind {
let actual = match &params.kind {
NodeKind::Tuple { elements } => elements.len(),
_ => 1,
};
if actual != expected {
diag.push_error(
format!(
"pipe: lambda expects {} parameter(s) but {} input stream(s) provided",
actual, expected
),
Some(elements[1].identity.clone()),
);
}
}
}
ret_ty
}
fn finalize(
&self,
_callee: Rc<TypedNode>,
args: Rc<TypedNode>,
node_ty: &StaticType,
_subst: &HashMap<u32, StaticType>,
) -> Option<NodeKind<TypedPhase>> {
let StaticType::Stream(inner) = node_ty else { return None };
// Only finalize when the element type is concrete — not unresolved Any or TypeVar.
if matches!(inner.as_ref(), StaticType::Any | StaticType::TypeVar(_)) {
return None;
}
let element_type = *inner.clone();
let globals = self.globals.clone();
let factory: Value = Value::Function(Rc::new(NativeFunction {
func: Rc::new(move |call_args: &[Value]| {
let obs = extract_obs_streams(&call_args[0]);
let exec = build_pipe_executor(&call_args[1], globals.clone());
Value::Stream(build_pipeline_node(obs, exec, &element_type))
}),
purity: Purity::Impure,
}));
let factory_node = Rc::new(Node {
kind: NodeKind::Constant(factory),
ty: StaticType::Any,
identity: NodeIdentity::new(SourceLocation { line: 0, col: 0 }),
comments: Rc::from([]),
});
Some(NodeKind::Call { callee: factory_node, args })
}
}
/// Resolves the return type of a `pipe` call from its argument types.
/// Argument layout: `Tuple([inputs, lambda])` where inputs is a Tuple of streams
/// or a single StreamNode, and lambda is a `Function<args -> U>`.
/// Returns `Stream<U>`, unwrapping `Optional<U>` to `U` (filter pattern).
pub(super) fn pipe_type_resolver(args_ty: &StaticType) -> Option<StaticType> {
if let StaticType::Tuple(elements) = args_ty
&& elements.len() == 2
&& let StaticType::Function(sig) = &elements[1]
{
let inner = if let StaticType::Optional(inner) = &sig.ret {
*inner.clone()
} else {
sig.ret.clone()
};
return Some(StaticType::Stream(Box::new(inner)));
}
None
}
/// Provides expected lambda parameter types for bidirectional type inference.
/// For `pipe`, the lambda at position 1 receives the inner type of the input stream(s).
pub(super) fn pipe_arg_hint_resolver(
arg_index: usize,
known_args: &[Option<StaticType>],
) -> Option<Vec<StaticType>> {
if arg_index != 1 {
return None;
}
let input_ty = known_args.first()?.as_ref()?;
/// Extract the inner type from a single stream or series.
fn extract_inner(ty: &StaticType) -> StaticType {
match ty {
StaticType::Stream(inner) | StaticType::Series(inner) => *inner.clone(),
_ => StaticType::Any,
}
}
match input_ty {
StaticType::Stream(_) | StaticType::Series(_) => {
Some(vec![extract_inner(input_ty)])
}
// Vector: homogeneous fixed-size array, e.g. `[src]` → Vector(Stream<T>, 1)
StaticType::Vector(elem_ty, count) => {
Some(vec![extract_inner(elem_ty); *count])
}
// Tuple: heterogeneous, e.g. `[src1 src2]` with different stream types
StaticType::Tuple(elements) => {
Some(elements.iter().map(extract_inner).collect())
}
_ => None,
}
}
// ============================================================================
// pipe-series: Pipe with automatic value accumulation into Series
// ============================================================================
/// Compiler hook for `(pipe-series lookback inputs lambda)`.
///
/// - **post_call:** Validates 3 arguments (Int, Streams, Lambda) and checks that
/// the input stream count matches the lambda parameter count.
/// - **finalize:** Replaces the callee with a factory closure that builds a
/// wrapper-executor. The wrapper pushes values into internal Series, checks the
/// fill gate, and forwards Series objects to the user lambda.
pub(super) struct LookbackPipeHook {
pub(super) globals: GlobalStore,
}
impl RtlCompilerHook for LookbackPipeHook {
fn post_call(
&self,
args: &TypedNode,
ret_ty: StaticType,
_ctx: &dyn InferenceAccess,
diag: &mut Diagnostics,
) -> StaticType {
// Validate: (pipe-series Int Streams Lambda) — 3 elements
let NodeKind::Tuple { elements } = &args.kind else { return ret_ty };
if elements.len() != 3 || !matches!(elements[0].ty, StaticType::Int) {
return ret_ty;
}
let expected = match &elements[1].ty {
StaticType::Stream(_) | StaticType::Series(_) => 1,
StaticType::Vector(_, count) => *count,
StaticType::Tuple(elems) => elems.len(),
_ => return ret_ty,
};
if let NodeKind::Lambda { params, .. } = &elements[2].kind {
let actual = match &params.kind {
NodeKind::Tuple { elements } => elements.len(),
_ => 1,
};
if actual != expected {
diag.push_error(
format!(
"pipe-series: lambda expects {} parameter(s) but {} input stream(s) provided",
actual, expected
),
Some(elements[2].identity.clone()),
);
}
}
ret_ty
}
fn finalize(
&self,
_callee: Rc<TypedNode>,
args: Rc<TypedNode>,
node_ty: &StaticType,
_subst: &HashMap<u32, StaticType>,
) -> Option<NodeKind<TypedPhase>> {
let StaticType::Stream(inner) = node_ty else { return None };
if matches!(inner.as_ref(), StaticType::Any | StaticType::TypeVar(_)) {
return None;
}
let element_type = *inner.clone();
let globals = self.globals.clone();
let input_element_types = extract_input_element_types(&args);
let factory: Value = Value::Function(Rc::new(NativeFunction {
func: Rc::new(move |call_args: &[Value]| {
let lookback = call_args[0].as_int().unwrap() as usize;
let obs = extract_obs_streams(&call_args[1]);
let user_exec = build_pipe_executor(&call_args[2], globals.clone());
let wrapper = build_buffered_wrapper(lookback, &input_element_types, user_exec);
Value::Stream(build_pipeline_node(obs, wrapper, &element_type))
}),
purity: Purity::Impure,
}));
let factory_node = Rc::new(Node {
kind: NodeKind::Constant(factory),
ty: StaticType::Any,
identity: NodeIdentity::new(SourceLocation { line: 0, col: 0 }),
comments: Rc::from([]),
});
Some(NodeKind::Call { callee: factory_node, args })
}
}
/// Builds a wrapper executor that accumulates values into internal Series
/// and only calls the user lambda once the fill gate is satisfied.
pub(super) fn build_buffered_wrapper(
lookback: usize,
input_element_types: &[StaticType],
mut user_exec: Box<PipeFn>,
) -> Box<PipeFn> {
let series: Vec<Rc<dyn SeriesStorage>> = input_element_types
.iter()
.map(|ty| create_typed_series(ty, lookback))
.collect();
let mut fill_gate_open = false;
Box::new(move |args: &[Value]| {
// 1. Push incoming values into internal series
for (i, s) in series.iter().enumerate() {
s.as_pushable().unwrap().push_value(args[i].clone());
}
// 2. Fill gate: wait until all series have enough data
if !fill_gate_open {
if series.iter().all(|s| s.len() >= lookback) {
fill_gate_open = true;
} else {
return Value::Void; // Filtered by PipeStream::notify
}
}
// 3. Call user lambda with Series objects instead of raw values
let series_args: Vec<Value> = series
.iter()
.map(|s| Value::Series(Rc::clone(s)))
.collect();
user_exec(&series_args)
})
}
/// Extracts input element types from the typed AST for `pipe-series`.
/// Args layout: `Tuple([Int, inputs, Lambda])` — inputs at index 1.
fn extract_input_element_types(args: &TypedNode) -> Vec<StaticType> {
let NodeKind::Tuple { elements } = &args.kind else { return vec![] };
if elements.len() < 2 {
return vec![];
}
let input_ty = &elements[1].ty;
fn inner_type(ty: &StaticType) -> StaticType {
match ty {
StaticType::Stream(inner) | StaticType::Series(inner) => *inner.clone(),
_ => StaticType::Any,
}
}
match input_ty {
StaticType::Stream(_) | StaticType::Series(_) => vec![inner_type(input_ty)],
StaticType::Vector(elem_ty, count) => vec![inner_type(elem_ty); *count],
StaticType::Tuple(elems) => elems.iter().map(inner_type).collect(),
_ => vec![StaticType::Any],
}
}
/// Extracts element types from runtime stream values (for the native fn fallback).
pub(super) fn extract_runtime_input_types(val: &Value) -> Vec<StaticType> {
match val {
Value::Tuple(elements) => elements
.iter()
.map(|v| match v {
Value::Stream(s) => s.element_type(),
_ => StaticType::Any,
})
.collect(),
Value::Stream(s) => vec![s.element_type()],
_ => vec![StaticType::Any],
}
}
/// Resolves the return type of `pipe-series` from its argument types.
/// Argument layout: `Tuple([Int, inputs, Lambda])`.
/// Returns `Stream<U>`, unwrapping `Optional<U>` to `U` (filter pattern).
pub(super) fn buffered_pipe_type_resolver(args_ty: &StaticType) -> Option<StaticType> {
let StaticType::Tuple(elements) = args_ty else { return None };
if elements.len() != 3 || !matches!(elements[0], StaticType::Int) {
return None;
}
let StaticType::Function(sig) = &elements[2] else { return None };
let inner = if let StaticType::Optional(inner) = &sig.ret {
*inner.clone()
} else {
sig.ret.clone()
};
Some(StaticType::Stream(Box::new(inner)))
}
/// Provides expected lambda parameter types for `pipe-series`.
/// Lambda is at arg index 2. Parameters are wrapped as `Series<T>` instead of raw `T`.
pub(super) fn buffered_pipe_arg_hint_resolver(
arg_index: usize,
known_args: &[Option<StaticType>],
) -> Option<Vec<StaticType>> {
// Lambda is at position 2; only provide hints for that position
if arg_index != 2 {
return None;
}
// Inputs are at position 1
let input_ty = known_args.get(1)?.as_ref()?;
fn extract_inner(ty: &StaticType) -> StaticType {
match ty {
StaticType::Stream(inner) | StaticType::Series(inner) => *inner.clone(),
_ => StaticType::Any,
}
}
// Wrap each inner type in Series<T> — the lambda sees Series, not raw values
let wrap = |t: StaticType| StaticType::Series(Box::new(t));
match input_ty {
StaticType::Stream(_) | StaticType::Series(_) => {
Some(vec![wrap(extract_inner(input_ty))])
}
StaticType::Vector(elem_ty, count) => {
Some(vec![wrap(extract_inner(elem_ty)); *count])
}
StaticType::Tuple(elements) => {
Some(elements.iter().map(|e| wrap(extract_inner(e))).collect())
}
_ => None,
}
}
+95
View File
@@ -0,0 +1,95 @@
mod nodes;
mod hooks;
mod register;
#[cfg(test)]
mod tests;
pub use nodes::{
build_map_stream, build_pipeline_node, PipeStream, RecordPusher, RootStream, SeriesPusher,
ValuePusher,
};
pub use hooks::PipeHook;
pub use register::register;
use crate::ast::types::{StaticType, StreamStorage, Value};
use std::cell::RefCell;
use std::rc::Rc;
/// A Signal is the "packet" flowing through the reactive pipeline.
/// It represents a value produced at a specific logical time (cycle_id).
#[derive(Debug, Clone)]
pub struct Signal {
pub cycle_id: u64,
pub value: Value,
}
/// A Stream is a stateless provider of signals.
/// It doesn't "own" the data, it just knows how to get the current one.
pub trait Stream {
fn current_signal(&self) -> Option<Signal>;
}
/// An Observer is a node in the pipeline that reacts to new signals.
/// (e.g., a Pipe or a SharedSeries buffer).
pub trait Observer {
/// Notifies the observer about a new signal in the current cycle.
/// `source_index` identifies which input stream provided the value.
fn notify(&mut self, source_index: usize, cycle_id: u64, value: Value);
}
/// A lightweight adapter to map an unknown source index to a specific target index.
/// This prevents index collisions when a Pipe listens to multiple independent RootStreams.
pub struct SourceAdapter {
pub target: Rc<RefCell<dyn Observer>>,
pub target_index: usize,
}
impl Observer for SourceAdapter {
fn notify(&mut self, _ignored_source: usize, cycle_id: u64, value: Value) {
self.target
.borrow_mut()
.notify(self.target_index, cycle_id, value);
}
}
/// Polymorphic Interface for any stream that can accept observers (like Delphi's IStream).
pub trait ObservableStream {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>);
}
impl<T: ObservableStream> ObservableStream for std::cell::RefCell<T> {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.borrow().add_observer(observer);
}
}
/// A generic wrapper to pass ANY ObservableStream (Root, Pipe, etc.) as a Stream to the VM.
#[derive(Clone)]
pub struct StreamNode {
pub inner: Rc<dyn ObservableStream>,
/// The `StaticType` of the elements emitted by this stream.
/// Set at construction time and used by `Value::static_type()`.
pub element_type: StaticType,
}
impl std::fmt::Debug for StreamNode {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "StreamNode")
}
}
impl StreamStorage for StreamNode {
fn stream_type_name(&self) -> &'static str {
"stream"
}
fn as_any(&self) -> &dyn std::any::Any {
self
}
fn into_rc_any(self: std::rc::Rc<Self>) -> std::rc::Rc<dyn std::any::Any> {
self
}
fn element_type(&self) -> StaticType {
self.element_type.clone()
}
}
+259
View File
@@ -0,0 +1,259 @@
use crate::ast::rtl::series::{RingBuffer, ScalarValue, SeriesMember};
use crate::ast::types::{Keyword, PipeFn, StaticType, Value};
use std::cell::RefCell;
use std::rc::Rc;
use super::{ObservableStream, Observer, Signal, StreamNode};
/// The RootStream is the "Clock" and data source of the entire pipeline.
/// It generates the monotonic `cycle_id` and triggers the observers.
pub struct RootStream {
current_cycle: std::cell::Cell<u64>,
observers: RefCell<Vec<Rc<RefCell<dyn Observer>>>>,
}
impl ObservableStream for RootStream {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.observers.borrow_mut().push(observer);
}
}
impl Default for RootStream {
fn default() -> Self {
Self::new()
}
}
impl RootStream {
pub fn new() -> Self {
Self {
current_cycle: std::cell::Cell::new(0),
observers: RefCell::new(Vec::new()),
}
}
/// Advances the pipeline to the next cycle and propagates a value.
pub fn tick(&self, value: Value) {
let next_cycle = self.current_cycle.get() + 1;
self.current_cycle.set(next_cycle);
// Propagate to all observers.
// We use a local borrow of the observers list to keep the cell borrow short.
let obs_list = self.observers.borrow();
for obs in obs_list.iter() {
// Root observers are always at source_index 0.
obs.borrow_mut().notify(0, next_cycle, value.clone());
}
}
pub fn current_cycle(&self) -> u64 {
self.current_cycle.get()
}
pub fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.observers.borrow_mut().push(observer);
}
}
/// A PipeStream is a reactive node that transforms inputs via a lambda.
/// It implements "Barrier Synchronization": It only executes when all inputs
/// have reported a value for the same cycle_id.
pub struct PipeStream {
pub name: String,
/// The inputs this pipe is observing.
/// In a real system, these would be other Streams.
/// For the MVP, we assume the Pipe is notified by the Root or its parents.
pub input_count: usize,
/// Tracks the last cycle_id received from each input.
last_cycle_per_input: Vec<u64>,
/// Stores the current value for each input to construct the argument tuple.
current_values: Vec<Value>,
/// The current output signal of this pipe.
current_signal: RefCell<Option<Signal>>,
/// The executable closure representing the Lambda. Expects a slice of arguments.
pub executor: Option<Box<PipeFn>>,
/// Observers of THIS pipe.
observers: RefCell<Vec<Rc<RefCell<dyn Observer>>>>,
/// Output element type, injected by PipeHook::finalize. `Any` when unknown.
pub element_type: StaticType,
}
impl PipeStream {
/// Creates a PipeStream with unknown output element type (`StaticType::Any`).
pub fn new(name: String, input_count: usize, executor: Option<Box<PipeFn>>) -> Self {
Self::new_typed(name, input_count, executor, StaticType::Any)
}
/// Creates a PipeStream with a known output element type.
/// Called by `build_pipeline_node` when the type is available from `PipeHook::finalize`.
pub fn new_typed(
name: String,
input_count: usize,
executor: Option<Box<PipeFn>>,
element_type: StaticType,
) -> Self {
Self {
name,
input_count,
last_cycle_per_input: vec![0; input_count],
current_values: vec![Value::Void; input_count],
current_signal: RefCell::new(None),
executor,
observers: RefCell::new(Vec::new()),
element_type,
}
}
}
impl ObservableStream for PipeStream {
fn add_observer(&self, observer: Rc<RefCell<dyn Observer>>) {
self.observers.borrow_mut().push(observer);
}
}
impl super::Stream for PipeStream {
fn current_signal(&self) -> Option<Signal> {
self.current_signal.borrow().clone()
}
}
impl Observer for PipeStream {
fn notify(&mut self, source_index: usize, cycle_id: u64, value: Value) {
let barrier_reached = {
if source_index < self.input_count {
self.last_cycle_per_input[source_index] = cycle_id;
self.current_values[source_index] = value;
}
// Check if all inputs reached the same cycle.
self.last_cycle_per_input.iter().all(|&c| c == cycle_id)
};
if barrier_reached {
// 1. Prepare Arguments for Lambda (Current values of all inputs) - NO CLONE NEEDED!
let args = &self.current_values;
// 2. Execute Lambda using the encapsulated VM executor
let result = if let Some(exec) = &mut self.executor {
exec(args)
} else {
self.current_values[0].clone() // Identity bypass (defaults to first input)
};
// 3. Handle Void case! (Filter pattern)
if matches!(result, Value::Void) {
return; // Act as a filter: do not emit, do not push.
}
// 4. Update Current Signal
let new_signal = Signal {
cycle_id,
value: result,
};
*self.current_signal.borrow_mut() = Some(new_signal.clone());
// 5. Notify Observers (Always at source_index 0 of the NEXT pipe)
let obs_list = self.observers.borrow();
for obs in obs_list.iter() {
obs.borrow_mut()
.notify(0, cycle_id, new_signal.value.clone());
}
}
}
}
/// A specialized observer that pushes incoming signals into a SharedSeries buffer.
pub struct SeriesPusher<T: ScalarValue> {
pub buffer: Rc<RefCell<RingBuffer<T>>>,
pub extractor: fn(Value) -> Option<T>,
}
impl<T: ScalarValue> Observer for SeriesPusher<T> {
fn notify(&mut self, _source_index: usize, _cycle_id: u64, value: Value) {
if let Some(v) = (self.extractor)(value) {
self.buffer.borrow_mut().push(v);
}
}
}
/// A specialized observer that pushes incoming signals into a generic SharedValueSeries buffer.
pub struct ValuePusher {
pub buffer: Rc<RefCell<RingBuffer<Value>>>,
}
impl Observer for ValuePusher {
fn notify(&mut self, _source_index: usize, _cycle_id: u64, value: Value) {
self.buffer.borrow_mut().push(value);
}
}
/// A specialized observer that splits a Record into its fields and pushes them into SoA buffers.
pub struct RecordPusher {
pub field_buffers: Vec<Rc<RefCell<dyn SeriesMember>>>,
}
impl Observer for RecordPusher {
fn notify(&mut self, _source_index: usize, _cycle_id: u64, value: Value) {
if let Value::Record(_, values) = value {
for (i, v) in values.iter().enumerate() {
if let Some(buf) = self.field_buffers.get(i) {
buf.borrow_mut().push_value(v.clone());
}
}
}
}
}
/// Factory function to build a specialized pipeline node based on the output type.
/// This keeps the VM "dumb" and moves the buffer selection logic to the RTL.
pub fn build_pipeline_node(
inputs: Vec<Rc<dyn ObservableStream>>,
executor: Box<PipeFn>,
out_type: &StaticType,
) -> Rc<StreamNode> {
let pipe = Rc::new(RefCell::new(PipeStream::new_typed(
"pipe".to_string(),
inputs.len(),
Some(executor),
out_type.clone(),
)));
// Connect inputs to the pipe
for (i, input) in inputs.into_iter().enumerate() {
let adapter = Rc::new(RefCell::new(super::SourceAdapter {
target: pipe.clone(),
target_index: i,
}));
input.add_observer(adapter);
}
Rc::new(StreamNode { inner: pipe, element_type: out_type.clone() })
}
pub fn build_map_stream(input: Rc<dyn ObservableStream>, field: Keyword) -> StreamNode {
let executor: Box<PipeFn> = Box::new(move |args: &[Value]| -> Value {
let val = &args[0];
if let Value::Record(layout, values) = val
&& let Some(idx) = layout.index_of(field)
{
return values[idx].clone();
}
Value::Void // In streams, Void acts as a filter
});
let pipe = Rc::new(RefCell::new(PipeStream::new(
format!("map:{}", field.name()),
1,
Some(executor),
)));
let adapter = Rc::new(RefCell::new(super::SourceAdapter {
target: pipe.clone(),
target_index: 0,
}));
input.add_observer(adapter);
StreamNode {
inner: pipe,
element_type: StaticType::Any,
}
}
+232
View File
@@ -0,0 +1,232 @@
use crate::ast::environment::Environment;
use crate::ast::types::{Keyword, Purity, RecordLayout, Signature, StaticType, Value};
use std::rc::Rc;
use super::nodes::{build_pipeline_node, RootStream};
use super::hooks::{
build_buffered_wrapper, build_pipe_executor, buffered_pipe_arg_hint_resolver,
buffered_pipe_type_resolver, extract_obs_streams, extract_runtime_input_types,
pipe_arg_hint_resolver, pipe_type_resolver, LookbackPipeHook, PipeHook,
};
use super::StreamNode;
pub fn register(env: &Environment) {
// (create-random-ohlc seed limit) -> StreamNode
let generators = env.pipeline_generators.clone();
// Define the OHLC layout for typing
let ohlc_layout = RecordLayout::get_or_create(vec![
(Keyword::intern("open"), StaticType::Float),
(Keyword::intern("high"), StaticType::Float),
(Keyword::intern("low"), StaticType::Float),
(Keyword::intern("close"), StaticType::Float),
]);
env.register_native_fn(
"create-random-ohlc",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Int, StaticType::Int]),
ret: StaticType::Stream(Box::new(StaticType::Record(ohlc_layout.clone()))),
})),
Purity::Impure, // Modifies global generator registry
move |args: &[Value]| {
if args.len() != 2 {
panic!("create-random-ohlc expects exactly 2 arguments (seed, limit)");
}
let seed = if let Value::Int(s) = args[0] {
s as u64
} else {
0
};
let limit = if let Value::Int(l) = args[1] {
l as usize
} else {
0
};
// 1. Create the RootStream
let root_stream = Rc::new(RootStream::new());
// 2. Setup the Layout for OHLC records (before StreamNode so element_type is available)
let layout = RecordLayout::get_or_create(vec![
(Keyword::intern("open"), StaticType::Float),
(Keyword::intern("high"), StaticType::Float),
(Keyword::intern("low"), StaticType::Float),
(Keyword::intern("close"), StaticType::Float),
]);
let stream_node = StreamNode {
inner: root_stream.clone(),
element_type: StaticType::Record(layout.clone()),
};
// 3. Create the stateful generator closure
let mut current_tick = 0;
let mut last_close = 100.0;
// We use a local PRNG instance for reproducibility based on the seed
let mut rng = fastrand::Rng::with_seed(seed);
let generator = move || -> bool {
if current_tick >= limit {
return false; // Exhausted
}
// Generate random OHLC (Random Walk)
let change = (rng.f64() - 0.5) * 2.0;
let open = last_close;
let high = open + (rng.f64() * 2.0).abs();
let low = open - (rng.f64() * 2.0).abs();
let close = open + change;
last_close = close;
let record = Value::Record(
layout.clone(),
Rc::new(vec![
Value::Float(open),
Value::Float(high),
Value::Float(low),
Value::Float(close),
]),
);
// Pump the signal into the RootStream
root_stream.tick(record);
current_tick += 1;
true // Still active
};
// 4. Register the generator in the Environment
generators.borrow_mut().push(Box::new(generator));
// 5. Return the stream reference to the script
Value::Stream(Rc::new(stream_node))
},
).doc("Creates a stream of random OHLC (Open-High-Low-Close) candles.")
.description("Args: (seed: int, limit: int). Uses a random walk model. Connect via (pipe ...).")
.examples(&["(def ohlc (create-random-ohlc 42 1000))"]);
// (create-ticker condition-closure) -> StreamNode
let ticker_generators = env.pipeline_generators.clone();
// Captures global_store() during bootstrap (user_values = RTL scratch, rtl_len = 0).
// After Environment::new() resets user_values, this GlobalStore holds the OLD
// bootstrap Rc (RTL values only, frozen). Stream lambdas run in an RTL-only context.
let globals = env.global_store();
env.register_native_fn(
"create-ticker",
StaticType::Function(Box::new(Signature {
params: StaticType::Tuple(vec![StaticType::Any]), // Expects a closure
ret: StaticType::Any, // Returns StreamNode
})),
Purity::Impure,
move |args: &[Value]| {
if args.len() != 1 {
panic!("create-ticker expects exactly 1 argument (the condition closure)");
}
let closure_obj = if let Value::Closure(rc) = &args[0] {
rc.clone()
} else {
panic!("create-ticker expects a closure as its argument");
};
// 1. Create the RootStream
let root_stream = Rc::new(RootStream::new());
let stream_node = StreamNode {
inner: root_stream.clone(),
element_type: StaticType::Bool,
};
// 2. Setup isolated VM
let mut ticker_vm = crate::ast::vm::VM::new(globals.clone());
let my_closure = closure_obj.clone();
// 3. Create generator
let generator = move || -> bool {
match ticker_vm.run_with_args(my_closure.clone(), &[]) {
Ok(Value::Bool(b)) => {
if b {
// Ticker pulses with a simple `true` value or `Void`
// We use true here so the pipe receives something tangible.
root_stream.tick(Value::Bool(true));
true
} else {
false // Exhausted
}
}
Ok(_) => panic!("create-ticker closure must return a boolean"),
Err(e) => panic!("create-ticker closure execution failed: {}", e),
}
};
// 4. Register the generator
ticker_generators.borrow_mut().push(Box::new(generator));
// 5. Return stream
Value::Stream(Rc::new(stream_node))
},
).doc("Creates a stream driven by a boolean closure. Ticks as long as closure returns true.")
.examples(&["(def ticker (create-ticker (fn [] (< (now) end-time))))"]);
// (pipe inputs lambda) -> StreamNode
// inputs: a Tuple of StreamNodes or a single StreamNode
// lambda: a Closure or Function
// Same RTL-only bootstrap capture as create-ticker above.
// `fn_globals` is moved into the native fn fallback; `hook_globals` is captured by PipeHook.
let fn_globals = env.global_store();
let hook_globals = fn_globals.clone();
env.register_native_fn(
"pipe",
StaticType::PolymorphicFn {
resolve_return: pipe_type_resolver,
resolve_arg_hints: Some(pipe_arg_hint_resolver),
},
Purity::Impure,
move |args: &[Value]| {
// This fallback only runs when PipeHook::finalize did not replace the callee
// (e.g. the output element type could not be determined at compile time).
assert!(args.len() == 2, "pipe expects exactly 2 arguments (inputs, lambda)");
let obs = extract_obs_streams(&args[0]);
let exec = build_pipe_executor(&args[1], fn_globals.clone());
Value::Stream(build_pipeline_node(obs, exec, &StaticType::Any))
},
).with_compiler_hook(Rc::new(PipeHook { globals: hook_globals }))
.doc("Connects a lambda to one or more streams, producing a transformed output stream.")
.description("Returns void from lambda to act as a filter (value is dropped). Supports tuple inputs for barrier synchronization.")
.examples(&[
"(pipe ohlc (fn [bar] (.close bar)))",
"(pipe [stream-a stream-b] (fn [a b] (+ a b)))",
]);
// (pipe-series lookback inputs lambda) -> StreamNode
// Like pipe, but accumulates values into internal Series with the given lookback
// depth. The lambda receives Series objects and only fires once all series have
// at least `lookback` elements (fill gate).
let fn_globals = env.global_store();
let hook_globals = fn_globals.clone();
env.register_native_fn(
"pipe-series",
StaticType::PolymorphicFn {
resolve_return: buffered_pipe_type_resolver,
resolve_arg_hints: Some(buffered_pipe_arg_hint_resolver),
},
Purity::Impure,
move |args: &[Value]| {
assert!(args.len() == 3, "pipe-series expects 3 arguments (lookback, inputs, lambda)");
let lookback = args[0].as_int().unwrap() as usize;
let obs = extract_obs_streams(&args[1]);
let input_types: Vec<StaticType> = extract_runtime_input_types(&args[1]);
let user_exec = build_pipe_executor(&args[2], fn_globals.clone());
let wrapper = build_buffered_wrapper(lookback, &input_types, user_exec);
Value::Stream(build_pipeline_node(obs, wrapper, &StaticType::Any))
},
).with_compiler_hook(Rc::new(LookbackPipeHook { globals: hook_globals }))
.doc("Like pipe, but accumulates values into Series before firing the lambda.")
.description("The lambda only fires once all input series have at least N elements (fill gate). Lambda parameters are Series objects with lookback indexing (0 = newest).")
.examples(&[
"(pipe-series 20 ohlc (fn [bars] (- (.close (bars 0)) (.close (bars 19)))))",
"(pipe-series 2 [stream-a stream-b] (fn [a b] (+ (a 0) (b 0))))",
]);
}

Some files were not shown because too many files have changed in this diff Show More