Michael Schimmel
84ef3f9aed
The `#use` directive now supports referencing entire directories, automatically including all `.myc` files within them. Additionally, environments are now initialized with the current working directory as a default search path, simplifying module resolution.
68 lines
4.6 KiB
Markdown
68 lines
4.6 KiB
Markdown
# Myc Dependency Management: The `#use` Directive
|
|
|
|
## Motivation
|
|
|
|
As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (`ast.exe`) only supported loading libraries via the `--lib` command-line argument.
|
|
|
|
We need a way to explicitly define dependencies *within* a Myc script, ensuring that a script can autonomously declare what it needs to run.
|
|
|
|
## Constraints & Design Decisions
|
|
|
|
1. **No AST Pollution:** The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern.
|
|
2. **Macro Availability:** Libraries often define macros. If script `A` depends on library `B`, and `B` defines a macro `foo`, script `A` must have access to `foo` during its own macro-expansion phase.
|
|
3. **Idempotency:** If `A` depends on `B` and `C`, and `B` also depends on `C`, library `C` must only be parsed and evaluated once.
|
|
4. **Location:** Dependencies must be declared at the very beginning of the file, before any actual Myc code.
|
|
5. **Path Format Restrictions:**
|
|
- Whitespace is **not allowed** in library paths and file names.
|
|
- The file extension is **always omitted** (the system implies `.myc`).
|
|
- The folder separator is replaced by `->`.
|
|
6. **Search Paths:** The `--lib` command-line argument will no longer proactively load all `.myc` files. Instead, it will append directories to a list of global **search paths**. When `#use` is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided `--lib` search paths.
|
|
|
|
## The Solution: `#use` Preprocessor Directives
|
|
|
|
We introduce a preprocessor directive `#use` that is evaluated *before* the main parsing phase begins. It can target either single files or entire directories.
|
|
|
|
### Syntax
|
|
|
|
```myc
|
|
#use math->indicators
|
|
#use utils
|
|
#use mylib
|
|
|
|
(do
|
|
;; Actual Myc code begins here
|
|
(def my_var (indicators.calculate ...))
|
|
)
|
|
```
|
|
*(Here `math->indicators` resolves to the file `math/indicators.myc`. `utils` resolves to `utils.myc`. If `mylib` is a directory instead of a file, it resolves to all `.myc` files within that directory.)*
|
|
|
|
### Implementation Plan
|
|
|
|
1. **Environment State:**
|
|
The `Environment` will be extended with a `HashSet<PathBuf>` (e.g., `loaded_modules`) to keep track of canonical paths that have already been loaded, ensuring idempotency.
|
|
|
|
2. **Pre-Parse Extraction:**
|
|
Before the AST `Parser` processes a script, a lightweight scraper will read the top lines of the file. It will extract all `#use` paths and stop as soon as it encounters a line that is not empty, not a comment (`;` or `#`), and not a `#use` directive.
|
|
|
|
3. **Recursive Resolution:**
|
|
- The scraper extracts the module path and replaces `->` with the OS-specific directory separator.
|
|
- It first checks if `<path>.myc` exists as a file. If it does, it resolves to that single file.
|
|
- If not, it checks if `<path>` is a directory. If so, it resolves to all `.myc` files inside that directory (sorted alphabetically).
|
|
- Paths are resolved relative to the file that declares the dependency, with a fallback to the global `--lib` search paths.
|
|
- If a path is not in `loaded_modules`, it is added to the set.
|
|
- The files are read, and the extraction process runs recursively on their content.
|
|
- This creates a flat list of all dependencies in a valid topological order.
|
|
|
|
4. **Two-Pass Compilation:**
|
|
We reuse the existing robust two-pass mechanism used by the `--lib` flag:
|
|
- **Pass 1 (Discovery):** Parse all discovered dependency files and traverse their untyped ASTs to register `Def` (Globals) and `MacroDecl` (Macros) in the Environment. This makes all exported symbols and macros available.
|
|
- **Pass 2 (Compilation):** Type-check and compile all dependency files, then execute them to initialize their global state.
|
|
- Finally, compile and execute the original main script.
|
|
|
|
5. **Lexer Adaptation:**
|
|
Since `#use` is handled before the main parser, we will modify the `Lexer` to treat `#` as the start of a single-line comment (similar to `;`). This prevents the parser from crashing on the directives while keeping the line numbers accurate.
|
|
|
|
## Summary
|
|
|
|
By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules.
|