RustAst/docs/dependency_management.md

Myc Dependency Management: The #use Directive

Motivation

As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (ast.exe) only supported loading libraries via the --lib command-line argument.

We need a way to explicitly define dependencies within a Myc script, ensuring that a script can autonomously declare what it needs to run.

Constraints & Design Decisions

1. No AST Pollution: The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern.
2. Macro Availability: Libraries often define macros. If script A depends on library B, and B defines a macro foo, script A must have access to foo during its own macro-expansion phase.
3. Idempotency: If A depends on B and C, and B also depends on C, library C must only be parsed and evaluated once.
4. Location: Dependencies must be declared at the very beginning of the file, before any actual Myc code.
5. Path Format Restrictions:
   • Whitespace is not allowed in library paths and file names.
   • The file extension is always omitted (the system implies .myc).
   • The folder separator is replaced by ->.
6. Search Paths: The --lib command-line argument will no longer proactively load all .myc files. Instead, it will append directories to a list of global search paths. When #use is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided --lib search paths.

The Solution: #use Preprocessor Directives

We introduce a preprocessor directive #use that is evaluated before the main parsing phase begins.

Syntax

#use math->indicators
#use utils

(do
  ;; Actual Myc code begins here
  (def my_var (indicators.calculate ...))
)

(Here math->indicators resolves to math/indicators.myc and utils resolves to utils.myc)

Implementation Plan

1. Environment State: The Environment will be extended with a HashSet<PathBuf> (e.g., loaded_modules) to keep track of canonical paths that have already been loaded, ensuring idempotency.

2. Pre-Parse Extraction: Before the AST Parser processes a script, a lightweight scraper will read the top lines of the file. It will extract all #use paths and stop as soon as it encounters a line that is not empty, not a comment (;), and not a #use directive.

3. Recursive Resolution:

   • The scraper extracts the module path, replaces -> with the OS-specific directory separator, and appends .myc.
   • Paths are resolved relative to the file that declares the dependency.
   • If a path is not in loaded_modules, it is added to the set.
   • The file is read, and the extraction process runs recursively on its content.
   • This creates a flat list of all dependencies in a valid topological order.

4. Two-Pass Compilation: We reuse the existing robust two-pass mechanism used by the --lib flag:

   • Pass 1 (Discovery): Parse all discovered dependency files and traverse their untyped ASTs to register Def (Globals) and MacroDecl (Macros) in the Environment. This makes all exported symbols and macros available.
   • Pass 2 (Compilation): Type-check and compile all dependency files, then execute them to initialize their global state.
   • Finally, compile and execute the original main script.

5. Lexer Adaptation: Since #use is handled before the main parser, we will modify the Lexer to treat # as the start of a single-line comment (similar to ;). This prevents the parser from crashing on the directives while keeping the line numbers accurate.

Summary

By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules.