RustAst/docs/dependency_management.md

Myc Dependency Management: The #use Directive

Motivation

As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (ast.exe) only supported loading libraries via the --lib command-line argument.

We need a way to explicitly define dependencies within a Myc script, ensuring that a script can autonomously declare what it needs to run.

Constraints & Design Decisions

1. No AST Pollution: The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern.
2. Macro Availability: Libraries often define macros. If script A depends on library B, and B defines a macro foo, script A must have access to foo during its own macro-expansion phase.
3. Idempotency: If A depends on B and C, and B also depends on C, library C must only be parsed and evaluated once.
4. Location: Dependencies must be declared at the very beginning of the file, before any actual Myc code.
5. Path Format Restrictions:
   • Whitespace is not allowed in library paths and file names.
   • The file extension is always omitted (the system implies .myc).
   • The folder separator is replaced by ->.
6. Search Paths: The --lib command-line argument will no longer proactively load all .myc files. Instead, it will append directories to a list of global search paths. When #use is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided --lib search paths.

The Solution: #use Preprocessor Directives

We introduce a preprocessor directive #use that is evaluated before the main parsing phase begins. It can target either single files or entire directories.

Syntax

#use math->indicators
#use utils
#use mylib

(do
  ;; Actual Myc code begins here
  (def my_var (indicators.calculate ...))
)

(Here math->indicators resolves to the file math/indicators.myc. utils resolves to utils.myc. If mylib is a directory instead of a file, it resolves to all .myc files within that directory.)

Implementation Plan

1. Environment State: The Environment will be extended with a HashSet<PathBuf> (e.g., loaded_modules) to keep track of canonical paths that have already been loaded, ensuring idempotency.

2. Pre-Parse Extraction: Before the AST Parser processes a script, a lightweight scraper will read the top lines of the file. It will extract all #use paths and stop as soon as it encounters a line that is not empty, not a comment (; or #), and not a #use directive.

3. Recursive Resolution:

   • The scraper extracts the module path and replaces -> with the OS-specific directory separator.
   • It first checks if <path>.myc exists as a file. If it does, it resolves to that single file.
   • If not, it checks if <path> is a directory. If so, it resolves to all .myc files inside that directory (sorted alphabetically).
   • Paths are resolved relative to the file that declares the dependency, with a fallback to the global --lib search paths.
   • If a path is not in loaded_modules, it is added to the set.
   • The files are read, and the extraction process runs recursively on their content.
   • This creates a flat list of all dependencies in a valid topological order.

4. Two-Pass Compilation: We reuse the existing robust two-pass mechanism used by the --lib flag:

   • Pass 1 (Discovery): Parse all discovered dependency files and traverse their untyped ASTs to register Def (Globals) and MacroDecl (Macros) in the Environment. This makes all exported symbols and macros available.
   • Pass 2 (Compilation): Type-check and compile all dependency files, then execute them to initialize their global state.
   • Finally, compile and execute the original main script.

5. Lexer Adaptation: Since #use is handled before the main parser, we will modify the Lexer to treat # as the start of a single-line comment (similar to ;). This prevents the parser from crashing on the directives while keeping the line numbers accurate.

Summary

By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules.