Michael Schimmel
a59367ba61
Introduces the `#use` preprocessor directive, allowing Myc scripts to explicitly declare dependencies on other Myc libraries. This change moves dependency management from a command-line argument to an in-script declaration, ensuring scripts are self-contained and can correctly resolve macros and other symbols. Key features include: - Declarations at the beginning of a file, evaluated before parsing. - Support for relative paths and a new `->` separator. - Automatic resolution of dependencies from specified search paths. - Idempotent loading to prevent duplicate parsing and evaluation. - No AST pollution; dependency management is a compile-time concern. The compiler's lexer has been updated to recognize `#` as a comment character, and the environment now manages search paths and loaded modules. This lays the groundwork for more complex library structures and improved code organization.
65 lines
4.2 KiB
Markdown
65 lines
4.2 KiB
Markdown
# Myc Dependency Management: The `#use` Directive
|
|
|
|
## Motivation
|
|
|
|
As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (`ast.exe`) only supported loading libraries via the `--lib` command-line argument.
|
|
|
|
We need a way to explicitly define dependencies *within* a Myc script, ensuring that a script can autonomously declare what it needs to run.
|
|
|
|
## Constraints & Design Decisions
|
|
|
|
1. **No AST Pollution:** The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern.
|
|
2. **Macro Availability:** Libraries often define macros. If script `A` depends on library `B`, and `B` defines a macro `foo`, script `A` must have access to `foo` during its own macro-expansion phase.
|
|
3. **Idempotency:** If `A` depends on `B` and `C`, and `B` also depends on `C`, library `C` must only be parsed and evaluated once.
|
|
4. **Location:** Dependencies must be declared at the very beginning of the file, before any actual Myc code.
|
|
5. **Path Format Restrictions:**
|
|
- Whitespace is **not allowed** in library paths and file names.
|
|
- The file extension is **always omitted** (the system implies `.myc`).
|
|
- The folder separator is replaced by `->`.
|
|
6. **Search Paths:** The `--lib` command-line argument will no longer proactively load all `.myc` files. Instead, it will append directories to a list of global **search paths**. When `#use` is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided `--lib` search paths.
|
|
|
|
## The Solution: `#use` Preprocessor Directives
|
|
|
|
We introduce a preprocessor directive `#use` that is evaluated *before* the main parsing phase begins.
|
|
|
|
### Syntax
|
|
|
|
```myc
|
|
#use math->indicators
|
|
#use utils
|
|
|
|
(do
|
|
;; Actual Myc code begins here
|
|
(def my_var (indicators.calculate ...))
|
|
)
|
|
```
|
|
*(Here `math->indicators` resolves to `math/indicators.myc` and `utils` resolves to `utils.myc`)*
|
|
|
|
### Implementation Plan
|
|
|
|
1. **Environment State:**
|
|
The `Environment` will be extended with a `HashSet<PathBuf>` (e.g., `loaded_modules`) to keep track of canonical paths that have already been loaded, ensuring idempotency.
|
|
|
|
2. **Pre-Parse Extraction:**
|
|
Before the AST `Parser` processes a script, a lightweight scraper will read the top lines of the file. It will extract all `#use` paths and stop as soon as it encounters a line that is not empty, not a comment (`;`), and not a `#use` directive.
|
|
|
|
3. **Recursive Resolution:**
|
|
- The scraper extracts the module path, replaces `->` with the OS-specific directory separator, and appends `.myc`.
|
|
- Paths are resolved relative to the file that declares the dependency.
|
|
- If a path is not in `loaded_modules`, it is added to the set.
|
|
- The file is read, and the extraction process runs recursively on its content.
|
|
- This creates a flat list of all dependencies in a valid topological order.
|
|
|
|
4. **Two-Pass Compilation:**
|
|
We reuse the existing robust two-pass mechanism used by the `--lib` flag:
|
|
- **Pass 1 (Discovery):** Parse all discovered dependency files and traverse their untyped ASTs to register `Def` (Globals) and `MacroDecl` (Macros) in the Environment. This makes all exported symbols and macros available.
|
|
- **Pass 2 (Compilation):** Type-check and compile all dependency files, then execute them to initialize their global state.
|
|
- Finally, compile and execute the original main script.
|
|
|
|
5. **Lexer Adaptation:**
|
|
Since `#use` is handled before the main parser, we will modify the `Lexer` to treat `#` as the start of a single-line comment (similar to `;`). This prevents the parser from crashing on the directives while keeping the line numbers accurate.
|
|
|
|
## Summary
|
|
|
|
By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules.
|