This commit addresses the critical rule regarding exhaustive `match`
expressions on AST node kinds. By explicitly listing all variants
instead of using wildcards (`_ =>`), we ensure that new AST node types
are consciously handled throughout the compilation pipeline. This
practice prevents subtle bugs and makes the compiler more robust against
future changes.
The `CLAUDE.md` file has been updated to reflect this critical rule.
The deleted `BUG_program_node_typeinference.md` file indicates that a
previously identified bug related to type inference with `Program` nodes
has been resolved, likely as a consequence of enforcing these exhaustive
matches.
When expanding macros, the expander may encounter new macro
declarations.
These need to be stored back into the environment's macro registry so
that they are available for subsequent compilation passes or other
environments.
Persist macros after expansion
This commit modifies the `expand` method in `src/ast/environment.rs` to
persist macro declarations encountered during expansion. When a macro is
expanded, the `Expander` updates its internal registry. This change
ensures that the updated registry is then stored back into the
`Environment`'s `macro_registry`. This allows subsequent compilations or
expansions within the same environment to correctly recognize and
utilize these newly defined macros.
Additionally, a `NodeKind::Program` case was added to the
`CapturePass::transform` method in `src/ast/compiler/captures.rs`. This
ensures that expressions within a program node are also recursively
transformed, maintaining consistency in the AST processing pipeline.
Removes unnecessary lambda wrapping in the type checking pipeline.
Adds optimization logic for `NodeKind::Program` to handle sequences of
expressions.
Removes debug print statements from the type checker.
The type checker incorrectly inferred `Any` for lambda parameters when a
`Program` node was involved, preventing optimizations like constant
folding. This was because the `check_params_tuple` function was
resolving `TypeVar` to `StaticType::Any` instead of propagating the type
variable.
This commit addresses the issue by:
- Explicitly wrapping the bound AST in a parameterless lambda within
`compile_pipeline`. This ensures that the type checker always receives
a `Lambda` node, even if the original input was a `Program` node.
- Adding debug logging to the type checker to help diagnose similar
issues in the future.
Additionally, the commit fixes a bug where `parser.parse_program()` was
used instead of `parser.parse_expression()`, which would consume the
entire input and prevent checking for trailing expressions.
Introduced `scope_depth` to the `Binder` to track block nesting. This
allows for correctly determining whether a `def` binding should be
`Address::Global` (at scope depth 0) or `Address::Local`.
The `type_checker` was also updated to handle `Program` and `Block`
nodes by creating a new `TypeContext` for them.
Removed the `wrap_as_lambda` method from `Environment` as the compiler
now always produces a `Program` node, effectively handling the wrapping
at the AST level. Correspondingly, `instantiate` and
`run_script_compiled` were simplified to directly run the `Program`
node.
The `preload_dependencies` function has been refactored to directly run
parsed programs instead of collecting all forms into a single `Program`
node. This simplifies the compilation pipeline by allowing each
dependency to be processed individually.
Additionally, the `VM::run_with_observer` method has been updated to
manage its stack and frames more cleanly, avoiding the need for
`eval_in_frame`. The main loop now correctly extracts expressions from
the `Program` node for source code emission.
The `Program` node kind is introduced to represent top-level code
blocks, distinguishing them from `Block` nodes which introduce new
scopes. This commit updates various compiler passes to handle the
`Program` node, ensuring correct AST traversal and processing.
The `Program` node is similar to `Block`, but its definitions propagate
to the enclosing scope, unlike `Block` which creates a new, isolated
scope. This distinction is important for how variables and functions are
resolved within the compiled code.
The `fixed_scope_idx` field has been removed from `Binder` and
`Environment`. This field was used to enforce immutability on certain
scopes during binding.
The logic for handling the immutability of the root scope (scope 0) has
been moved and refactored. Now, during the `Environment::new` bootstrap
phase, the addresses within the RTL scope (scope 0) are directly
translated from `Address::Local` to `Address::Global`. This ensures that
the Binder inherently treats these as global and immutable without
needing an explicit `fixed_scope_idx` check.
The `Binder::bind_root` signature has been updated to reflect this
change by removing the `fixed_scope_idx` parameter. Consequently, tests
and other usages of `bind_root` have been adjusted.
This change simplifies the binding process by centralizing the
immutability handling to the environment setup phase, making the
Binder's logic cleaner.
The `Parser` can now parse a sequence of top-level expressions,
representing a complete program. This change modifies the `roundtrip`
function to iterate over these expressions and emit them, allowing for
multi-line program parsing.
Adds `egui_extras` to Cargo.toml and Cargo.lock to enable enhanced table
widgets.
Also introduces a new `StateTab` enum and integrates the
`egui_extras::TableBuilder` to display global variables in a structured
table within the application's state panel. This enhances the debugging
and introspection capabilities of the compiler UI by providing a clear
view of the environment's global state.
This commit reorganizes the stream-related logic by splitting the
`streams.rs` file into smaller, more manageable modules: `nodes.rs`,
`hooks.rs`, and `register.rs`.
The `nodes.rs` module now contains the core stream implementations like
`RootStream`, `PipeStream`, and the associated observer patterns.
The `hooks.rs` module encapsulates the compiler hook logic for `pipe`
and `pipe-series`, including type resolution and argument hinting. It
also includes helper functions for building pipe executors and
extracting stream information.
The `register.rs` module handles the registration of stream-related
built-in functions (`create-random-ohlc`, `create-ticker`, `pipe`,
`pipe-series`) within the environment.
This refactoring improves code organization, maintainability, and
testability by separating concerns into dedicated modules.
The TypeChecker implementation was becoming quite large, so it has been
refactored into several modules:
- `context`: Handles type checking contexts and inference access.
- `inference`: Contains the core Hindley-Milner inference logic
(unification, generalization, etc.).
- `finalize`: Manages the finalization step, applying substitutions and
dispatching hooks.
- `check`: Implements the main type checking logic for AST nodes.
This modularization improves code organization and maintainability.
The `analyzer.rs` and `lambda_collector.rs` files have been updated to
correctly traverse and process new node kinds in the Abstract Syntax
Tree (AST). This ensures that these new node types are properly included
in the analysis and lambda collection phases of the compiler.
The `pipe-buffered` function has been renamed to `pipe-series` to better
reflect its functionality. This change involves updating the function
name in the example file `examples/data_stream_pipe_buffered.myc` and
across the Rust source code in `src/ast/rtl/streams.rs`. Additionally,
test cases and documentation have been updated accordingly.
This change replaces a manual type check for the `again` function with a
call to the generic `unify` function. This improves consistency and
leverages existing type-checking logic.
A new test case `test_again_type_mismatch` has been added to
specifically verify that type mismatches in `again` calls are caught at
compile time, preventing potential infinite loops.
The `display_compact` method on `StaticType` has been enhanced to
provide more concise representations of types within error messages.
This change aims to improve the clarity of type-related diagnostics
generated by the compiler.
This commit introduces `StaticType::Variadic` to represent variadic
parameter types in function signatures. This allows for more precise
type checking of functions that accept a variable number of arguments,
such as arithmetic operators.
Previously, variadic functions were handled using `StaticType::Any`,
which was too permissive and could lead to runtime errors. The new
`Variadic` type enforces that all arguments must conform to a specified
inner type.
Changes include:
- Adding `StaticType::Variadic` to `src/ast/types.rs`.
- Updating the type checker to handle `Variadic` types correctly when
unifying with tuples or single arguments.
- Modifying built-in functions like arithmetic operators to use
`Variadic` for their parameter types.
- Improving error messages for function call mismatches to be more
specific.
- Adding a new test case to ensure arithmetic type mismatches are caught
at compile time.
- A new example `data_stream_pipe_buffered.myc` is added.
- The `HMA.myc` example now has a `Skip: output` directive.
Implements `pipe-buffered`, a new primitive that allows accumulating
values from multiple input streams into internal series buffers before
executing a user-provided lambda. This is useful for implementing
lookback logic and ensuring that the lambda only executes when
sufficient data is available in all input streams.
The function takes a `lookback` integer, input streams, and a lambda.
The lambda receives `Series` objects for each input, allowing indexed
access to past values. The `pipe-buffered` function manages the
buffering and fill-gate logic.
Introduces `create_typed_series` to centralize series creation logic.
This function dispatches on `StaticType` to select the appropriate
`SeriesStorage` implementation,
improving code reuse and maintainability within the series RTL.
The `SeriesHook` compiler hook is updated to use this new factory
function.
Additionally, new tests have been added to `rtl/streams.rs` and
`tests/pipeline.rs`
to specifically cover the behavior of `pipe-buffered`, focusing on its
fill gate
mechanism and multi-input handling.
Adds a check within the `PipeHook` to ensure the number of input streams
provided to a `pipe` operation matches the number of parameters expected
by the associated lambda function. This prevents runtime errors and
provides a clearer compile-time diagnostic.
Additionally, this commit corrects the order in which a `CallFrame` is
pushed onto the VM's frame stack. By pushing the frame earlier, it
ensures that slow-path `unpack` operations have access to a valid frame,
preventing "No call frame" errors.
This commit introduces time windowing functionality to the M1 and tick
data streams.
Key changes:
- **`DataServer::stream_m1_windowed` and
`DataServer::stream_tick_windowed`**: New methods added to
`DataServer` to allow streaming data within a specified `from_ms` and
`to_ms` range.
- **`SymbolChunkIter` modifications**: The `SymbolChunkIter` struct now
includes `from_ms`, `to_ms`, and `exhausted` fields to manage time
window filtering.
- **`filter_chunk` helper function**: A new utility function to
efficiently filter individual data chunks based on the time window.
This function also handles early termination when `to_ms` is exceeded.
- **`HasTimestamp` trait**: Introduced to provide a common interface for
accessing timestamps across different record types (`M1Parsed`,
`TickParsed`), enabling generic filtering.
- **`filter_files` method**: Added to `DataServer` to pre-filter file
keys based on the time window before creating iterators. This
optimizes file loading by skipping files that fall entirely outside
the desired range.
- **`create-m1-stream` and `create-tick-stream`**: The native functions
in `data_streams.rs` are updated to accept optional `DateTime`
arguments for `from` and `to` timestamps. They now also handle unknown
symbols by returning `Value::Void` and return an empty stream if the
time window results in no data.
- **Documentation**: Updated doc comments for the stream creation
functions to reflect the new time windowing capabilities and to
clarify behavior for unknown symbols and empty time windows.
These changes significantly enhance the flexibility of data streaming,
allowing users to query specific time ranges of M1 and tick data more
efficiently.
Introduces a reference counting mechanism for symbols within the
`FileCache`. When a `SymbolChunkIter` is created, it increments the
reference count for its symbol. When the iterator is dropped, the
reference count is decremented. If a symbol's reference count drops to
zero, all cached data associated with that symbol is evicted from the
cache, freeing up memory. This prevents stale data from being held
indefinitely when no active iterators are using it.
The `FileCache` now uses `std::sync::Condvar` to signal when files have
finished loading. This replaces the previous spin-wait mechanism,
significantly improving efficiency by allowing waiting threads to sleep
until explicitly woken up.
The `wait_for_load` method now blocks on the `Condvar`, and the
`insert_m1` and `insert_tick` methods call `notify_all` after updating
the cache and removing the loading guard. This ensures that all waiting
threads are woken up and can re-check the cache.
Additionally, `ensure_m1_loaded` and `ensure_tick_loaded` in
`DataServer` have been simplified to directly call `wait_for_load` when
another thread is already loading the file, removing the redundant
synchronous loading logic.
Refactor FileCache to use Condvar
This commit introduces a new data server module (`src/ast/data_server`)
designed for high-performance, thread-safe loading and caching of market
data.
Key components:
- `DataServer`: Manages symbol indexing and delegates loading/caching.
- `SymbolIndex`: Scans the data directory and maintains a sorted index
of data files per symbol.
- `FileCache`: Implements a thread-safe cache using `RwLock` and a
loading guard to prevent duplicate work.
- `loader`: Handles ZIP decompression and binary record parsing from
`.bin` files.
- `records`: Defines raw and parsed data structures for M1 and tick
data.
- `SymbolChunkIter`: Provides an iterator over pre-parsed data chunks,
prefetching subsequent files.
This architecture allows multiple VM threads to access market data
concurrently without redundant I/O, mirroring the strategy used in the
original Delphi `TDataServer`.
This commit refactors the way record types are computed within the
optimizer. Previously, the optimizer would often reuse the existing
`NodeMetrics` for records, which could lead to stale or polymorphic
types.
The changes introduce a new function, `compute_record_type`, which
explicitly recalculates the `StaticType` for a `Record` node based on
the concrete types of its child fields. This ensures that after
optimizations like inlining and folding, record types are as concrete as
possible, eliminating unresolved `TypeVars`.
Additionally, the `folder.rs` module is updated to recompute the
`RecordLayout` when creating constant record values, further improving
type accuracy. Two new tests are added to verify that inlined records,
including those with multiple fields, correctly result in concrete
types.
Refactor the `updated_metrics` function to `refreshed_metrics`. This
function now clones the `NodeMetrics` if the new type is identical to
the original, avoiding unnecessary allocations.
Also, adjust the `Block` and `Lambda` node kinds to correctly propagate
concrete types after inlining, especially when dealing with polymorphic
functions. This ensures that the return types of blocks and lambdas
reflect the actual computed types rather than stale type variables.
This change improves type inference accuracy and prepares for more
advanced optimizations by ensuring type information is correctly
propagated through the AST.
Introduce helper functions to recompute the `StaticType` of a tuple node
based on its children's types. This ensures that after inlining, tuples
within the AST have concrete types, rather than unresolved type
variables, even when derived from polymorphic functions.
Also adds tests to verify this behavior for both homogeneous and
heterogeneous tuples.
Introduce a new `dump_ast_compact` function for the environment and
Dumper. This function provides a simplified, human-readable
representation of the AST, focusing on inferred types and purity markers
instead of detailed debug information.
This change adds:
- A `compact` flag to the `Dumper` struct.
- The `dump_compact` method to the `Dumper`.
- The `dump_ast_compact` method to the `Environment`.
- A new `dump_ast_compact` command-line argument for the `ast` binary.
- A new MCP server endpoint `dump_ast_compact`.
- A `CompactMetadata` trait to abstract over different metadata types
for compact display.
- Updates to documentation and CLI help messages.
The `Object` trait is too generic and has been causing confusion. This
commit introduces `StreamStorage` as a dedicated trait for reactive
stream types.
This change involves:
- Renaming `Object` to `StreamStorage` in relevant places.
- Updating the `Value::Object` enum variant to `Value::Stream`.
- Modifying how streams are handled in the compiler, VM, and tests to
use the new `StreamStorage` trait.
- Adjusting example scripts and tests to reflect the change in type
representation.
- The `StreamNode` struct now explicitly holds its `element_type`.
This commit refactors the call hook mechanism in the compiler. Instead
of using function pointer types (`PostCallHook`, `FinalizeHook`) and a
struct to hold them (`CallHooks`), it introduces a trait
`RtlCompilerHook` with default implementations for `post_call` and
`finalize`.
This change aligns with Rust's idiomatic way of handling extension
points and allows for more flexible and extensible compiler behavior.
The `TypeChecker` now holds a map of `Rc<dyn RtlCompilerHook>`, enabling
different RTL functions to register their specific compiler hooks. The
`SeriesHook` and `PushHook` structs are introduced to demonstrate this
new pattern.
The `IndexElementFn` trait and its associated `series_element_type`
function have been removed. The `TypeChecker` now directly inspects the
`StaticType::Series` variant to extract the element type when
propagating index constraints. This simplifies the type-checking logic
by removing an unnecessary layer of indirection and adhering to the
project rule of keeping the AST simple and pushing complexity into the
system.
The `TypeInferenceAccess` and `TypeSlotAccess` traits have been
consolidated into a single `InferenceAccess` trait. This simplifies the
call hook interface by providing a unified way to access type inference
operations and scope slot types.
A new struct, `CheckerInferenceAccess`, is introduced to provide a
concrete implementation of `InferenceAccess` for the `TypeChecker`. This
struct wraps both the `TypeChecker` and the `TypeContext`, allowing call
hooks to interact with both without needing to know their specific
types. This avoids potential circular dependencies between modules.
The `get_slot_type` method has been moved to the new `InferenceAccess`
trait. This method is crucial for call hooks that need to recover the
original `TypeVar` ID after substitutions have occurred, particularly in
the `push_post_call` function for series.
Introduce `IndexElementFn` as a type alias for a function that maps a
`StaticType` to an optional `StaticType` representing its element type.
This allows the `TypeChecker` to determine the element type of indexable
types like `Series` without needing to be aware of their specific
implementations.
The `series_element_type` function is created and registered with the
`TypeChecker` to handle `Series` types. This design centralizes
type-specific logic for indexable types, making the `TypeChecker` more
generic and extensible for future indexable types such as `Stream`.
When a generic function indexes the same TypeVar parameter multiple
times,
all resulting TypeVars must share a single element type. This change
ensures
that subsequent indexing operations unify their fresh TypeVar with an
existing
one if the parameter has already been indexed, preventing incorrect type
inference and false negatives.
A new test, `test_let_poly_multi_index_all_results_typed`, is added to
verify this behavior.
Introduce `StaticType::Forall` to represent universally quantified
types. This enables Hindley-Milner's let-polymorphism, allowing
functions to be generalized and reused with different concrete types.
- **Generalization at `def`:** Function values are now generalized using
`self.generalize` when they are defined, wrapping their type in
`Forall`. This occurs at `def` boundaries.
- **Instantiation at use:** Function types are instantiated with fresh
type variables at each call site using `self.instantiate`. This
ensures that different uses of the same polymorphic function do not
interfere with each other's type inference.
- **Value restriction:** Only function-typed definitions are
generalized. Mutable state like series and scalars remain monomorphic
to prevent unexpected behavior.
- **Type context awareness:** The `generalize` function considers the
current type context to avoid quantifying type variables that are
still in use elsewhere.
Feat: Add Forall type for polymorphism
Introduces the `Forall` type to support polymorphic functions and enable
let-polymorphism.
- `Forall(vars, body)`: Represents a universally quantified type where
`vars` are the type variables bound by this quantification, and `body`
is the type itself.
- `TypeChecker::generalize`: Wraps a type in `Forall` when a `def`
boundary is encountered for function-typed values. This ensures that
each use site of a polymorphic function gets its own instantiation.
- `TypeChecker::instantiate`: Replaces the type variables within a
`Forall` type with fresh ones. This is performed at each call site of
a polymorphic function.
- Updates `TypeChecker::unify` and `TypeChecker::apply_subst` to
correctly handle `Forall` types, ensuring that bound variables within
a `Forall` are not affected by global substitutions and that `Forall`
types are instantiated before unification.
- Adds a new example script `examples/Propa.myc` to demonstrate basic
usage.
- Includes a regression test
`test_let_poly_non_series_callable_no_false_positive` to ensure that
passing non-series callables to generic functions does not lead to
type errors.
- Introduces `TypeChecker::bind_var` to handle lazy propagation of
index-call constraints, crucial for correctly typing series lookups
like `(s 0)`.
This commit introduces a new test suite for HM Let-Polymorphism.
These tests verify that user-defined generic functions work correctly
across multiple series element types without type conflicts. This is
crucial for ensuring that the type inference system can handle
polymorphic functions properly, especially when they are applied to
different types. The tests cover scenarios like generic accessors,
runtime correctness with different series types, record series, and
higher-order functions. They also include a test for the value
restriction to ensure that series bindings are not generalized, which
would lead to incorrect type errors.
Introduces `CallHooks` to enable extensions to the type inference and
AST finalization process. This allows for more dynamic and configurable
behavior without hardcoding symbol names in the compiler core.
Specifically, this commit adds:
- **`series_post_call`**: A hook for the `(series n)` function. It
ensures that the return type is a fresh `TypeVar` when the element
type is `Any`, allowing for proper type inference from subsequent
`push` calls.
- **`series_finalize`**: A hook for `(series n)` that rewrites the AST
node. It replaces the generic `series` call with a pre-configured
factory closure that allocates the correct series storage type (e.g.,
`ScalarSeries<f64>` for floats, `RecordSeries` for records). This
avoids runtime dispatch and relies on type information captured during
compilation.
- **`push_post_call`**: A hook for the `(push series val)` function. It
unifies the series element `TypeVar` with the pushed value's type. It
also handles numeric and record field promotion and updates the
`TypeVar` binding if a promotion occurs.
This commit introduces implicit numeric widening from Int to Float and a
corresponding record promotion mechanism.
When pushing values into a series, if the series' element type is a
TypeVar,
and the pushed value is of a different numeric type (e.g., Int and
Float),
the TypeVar will be unified with the wider type (Float). This ensures
that
operations on series elements handle type differences gracefully.
The record promotion handles cases where two records are involved, and
their
fields have compatible types, either identical or through numeric
widening.
This allows for more flexible type inference in complex structures.
When an identifier is used as an upvalue within a nested scope (e.g., a
`while` loop), the type checker needs to apply the global substitution
map to resolve `TypeVar`s that might have been determined in outer
scopes. This ensures that the correct type is used even if
`ctx.set_type` couldn't propagate the change directly through the
upvalue address.
This fix also includes a regression test to specifically cover this
scenario with series and closures.
The `series` function no longer requires an explicit schema. The element
type is now inferred at compile time from subsequent `push` calls using
Hindley-Milner type inference. This simplifies series creation and
aligns with the project's goal of making the untyped AST simple while
the system handles complexity.
Introduce `unify_matched_overload` to handle type variable propagation
for overloaded functions. This ensures that when an overload is chosen,
its concrete parameter types are unified with the actual argument types.
This is crucial for resolving type variables nested within closures
called through overloaded functions.
Additionally, this commit:
- Allows `series` to infer schema type from `push` calls, simplifying
its signature.
- Adds a fallback to `ValueSeries` in `rtl::series` if schema injection
fails.
- Updates `StaticType::TypeVar` to return `Any` when accessed,
simplifying field access logic.
- Adjusts tests to reflect the simplified `series` signature.
The `series` constructor now only accepts the `lookback` limit. The
element type is inferred by the type checker and implicitly added as a
schema argument during compilation. This simplifies the API and
centralizes type inference.
This change also includes:
- Updates to example scripts (`HMA.myc`, `sma.myc`, `soa_series.myc`) to
reflect the new `series` signature.
- Modifications to the `TypeChecker` to handle the inferred schema
injection.
- An addition of a regression test for type inference through nested
closures with `series`.
- Removal of `#[allow(dead_code)]` from several `TypeChecker` methods as
they are now used.
- Update to `rtl/prelude.myc` macro `cache`.
- Update to `rtl/series/mod.rs` to reflect new signature.
- Update to `ast/types.rs` to allow `series` to be indexed with an
integer to retrieve its element type.
Introduces a `TypeChecker` struct with logic for Hindley-Milner type
inference.
This includes:
- `var_counter` for generating unique type variable IDs.
- `subst` for the global substitution map.
- `fresh_var` to create new type variables.
- `apply_subst` to resolve type variables recursively.
- `occurs` to check for cycles during unification.
- `unify` to merge types and extend the substitution.
The `StaticType` enum is extended with a `TypeVar(u32)` variant to
represent unresolved type variables.
The `Display` and `is_assignable_from` implementations for `StaticType`
are updated to handle `TypeVar`.
The `GlobalStore` was refactored to clearly distinguish between
immutable RTL values and mutable user-defined global slots.
The `Environment` struct now holds:
- `rtl_values`: An immutable `Rc<[Value]>` for pre-defined RTL values.
- `user_values`: An `Rc<RefCell<Vec<Value>>>` for user-defined mutable
globals.
The `GlobalStore` struct now takes both `rtl` and `user` as parameters
and uses `rtl_len` to determine which store to access. This change
separates concerns and better reflects the immutability of RTL values
after the bootstrap phase, aligning with the project's concurrency
rules.
Documentation in `docs/Analysis_Environment.md` was updated to reflect
these structural changes.
Brummel