Implement #use directive for dependency management
Introduces the `#use` preprocessor directive, allowing Myc scripts to explicitly declare dependencies on other Myc libraries. This change moves dependency management from a command-line argument to an in-script declaration, ensuring scripts are self-contained and can correctly resolve macros and other symbols. Key features include: - Declarations at the beginning of a file, evaluated before parsing. - Support for relative paths and a new `->` separator. - Automatic resolution of dependencies from specified search paths. - Idempotent loading to prevent duplicate parsing and evaluation. - No AST pollution; dependency management is a compile-time concern. The compiler's lexer has been updated to recognize `#` as a comment character, and the environment now manages search paths and loaded modules. This lays the groundwork for more complex library structures and improved code organization.
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
# Myc Dependency Management: The `#use` Directive
|
||||
|
||||
## Motivation
|
||||
|
||||
As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (`ast.exe`) only supported loading libraries via the `--lib` command-line argument.
|
||||
|
||||
We need a way to explicitly define dependencies *within* a Myc script, ensuring that a script can autonomously declare what it needs to run.
|
||||
|
||||
## Constraints & Design Decisions
|
||||
|
||||
1. **No AST Pollution:** The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern.
|
||||
2. **Macro Availability:** Libraries often define macros. If script `A` depends on library `B`, and `B` defines a macro `foo`, script `A` must have access to `foo` during its own macro-expansion phase.
|
||||
3. **Idempotency:** If `A` depends on `B` and `C`, and `B` also depends on `C`, library `C` must only be parsed and evaluated once.
|
||||
4. **Location:** Dependencies must be declared at the very beginning of the file, before any actual Myc code.
|
||||
5. **Path Format Restrictions:**
|
||||
- Whitespace is **not allowed** in library paths and file names.
|
||||
- The file extension is **always omitted** (the system implies `.myc`).
|
||||
- The folder separator is replaced by `->`.
|
||||
6. **Search Paths:** The `--lib` command-line argument will no longer proactively load all `.myc` files. Instead, it will append directories to a list of global **search paths**. When `#use` is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided `--lib` search paths.
|
||||
|
||||
## The Solution: `#use` Preprocessor Directives
|
||||
|
||||
We introduce a preprocessor directive `#use` that is evaluated *before* the main parsing phase begins.
|
||||
|
||||
### Syntax
|
||||
|
||||
```myc
|
||||
#use math->indicators
|
||||
#use utils
|
||||
|
||||
(do
|
||||
;; Actual Myc code begins here
|
||||
(def my_var (indicators.calculate ...))
|
||||
)
|
||||
```
|
||||
*(Here `math->indicators` resolves to `math/indicators.myc` and `utils` resolves to `utils.myc`)*
|
||||
|
||||
### Implementation Plan
|
||||
|
||||
1. **Environment State:**
|
||||
The `Environment` will be extended with a `HashSet<PathBuf>` (e.g., `loaded_modules`) to keep track of canonical paths that have already been loaded, ensuring idempotency.
|
||||
|
||||
2. **Pre-Parse Extraction:**
|
||||
Before the AST `Parser` processes a script, a lightweight scraper will read the top lines of the file. It will extract all `#use` paths and stop as soon as it encounters a line that is not empty, not a comment (`;`), and not a `#use` directive.
|
||||
|
||||
3. **Recursive Resolution:**
|
||||
- The scraper extracts the module path, replaces `->` with the OS-specific directory separator, and appends `.myc`.
|
||||
- Paths are resolved relative to the file that declares the dependency.
|
||||
- If a path is not in `loaded_modules`, it is added to the set.
|
||||
- The file is read, and the extraction process runs recursively on its content.
|
||||
- This creates a flat list of all dependencies in a valid topological order.
|
||||
|
||||
4. **Two-Pass Compilation:**
|
||||
We reuse the existing robust two-pass mechanism used by the `--lib` flag:
|
||||
- **Pass 1 (Discovery):** Parse all discovered dependency files and traverse their untyped ASTs to register `Def` (Globals) and `MacroDecl` (Macros) in the Environment. This makes all exported symbols and macros available.
|
||||
- **Pass 2 (Compilation):** Type-check and compile all dependency files, then execute them to initialize their global state.
|
||||
- Finally, compile and execute the original main script.
|
||||
|
||||
5. **Lexer Adaptation:**
|
||||
Since `#use` is handled before the main parser, we will modify the `Lexer` to treat `#` as the start of a single-line comment (similar to `;`). This prevents the parser from crashing on the directives while keeping the line numbers accurate.
|
||||
|
||||
## Summary
|
||||
|
||||
By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules.
|
||||
Reference in New Issue
Block a user