Implement #use directive for dependency management

Introduces the `#use` preprocessor directive, allowing Myc scripts to
explicitly declare dependencies on other Myc libraries. This change
moves dependency management from a command-line argument to an in-script
declaration, ensuring scripts are self-contained and can correctly
resolve macros and other symbols.

Key features include:
- Declarations at the beginning of a file, evaluated before parsing.
- Support for relative paths and a new `->` separator.
- Automatic resolution of dependencies from specified search paths.
- Idempotent loading to prevent duplicate parsing and evaluation.
- No AST pollution; dependency management is a compile-time concern.

The compiler's lexer has been updated to recognize `#` as a comment
character, and the environment now manages search paths and loaded
modules. This lays the groundwork for more complex library structures
and improved code organization.

docs/dependency_management.md

@@ -0,0 +1,64 @@
+# Myc Dependency Management: The `#use` Directive
+
+## Motivation
+
+As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (`ast.exe`) only supported loading libraries via the `--lib` command-line argument. 
+
+We need a way to explicitly define dependencies *within* a Myc script, ensuring that a script can autonomously declare what it needs to run.
+
+## Constraints & Design Decisions
+
+1. **No AST Pollution:** The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern.
+2. **Macro Availability:** Libraries often define macros. If script `A` depends on library `B`, and `B` defines a macro `foo`, script `A` must have access to `foo` during its own macro-expansion phase.
+3. **Idempotency:** If `A` depends on `B` and `C`, and `B` also depends on `C`, library `C` must only be parsed and evaluated once.
+4. **Location:** Dependencies must be declared at the very beginning of the file, before any actual Myc code.
+5. **Path Format Restrictions:**
+   - Whitespace is **not allowed** in library paths and file names.
+   - The file extension is **always omitted** (the system implies `.myc`).
+   - The folder separator is replaced by `->`. 
+6. **Search Paths:** The `--lib` command-line argument will no longer proactively load all `.myc` files. Instead, it will append directories to a list of global **search paths**. When `#use` is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided `--lib` search paths. 
+
+## The Solution: `#use` Preprocessor Directives
+
+We introduce a preprocessor directive `#use` that is evaluated *before* the main parsing phase begins.
+
+### Syntax
+
+```myc
+#use math->indicators
+#use utils
+
+(do
+  ;; Actual Myc code begins here
+  (def my_var (indicators.calculate ...))
+)
+```
+*(Here `math->indicators` resolves to `math/indicators.myc` and `utils` resolves to `utils.myc`)*
+
+### Implementation Plan
+
+1. **Environment State:** 
+   The `Environment` will be extended with a `HashSet<PathBuf>` (e.g., `loaded_modules`) to keep track of canonical paths that have already been loaded, ensuring idempotency.
+
+2. **Pre-Parse Extraction:**
+   Before the AST `Parser` processes a script, a lightweight scraper will read the top lines of the file. It will extract all `#use` paths and stop as soon as it encounters a line that is not empty, not a comment (`;`), and not a `#use` directive.
+
+3. **Recursive Resolution:**
+   - The scraper extracts the module path, replaces `->` with the OS-specific directory separator, and appends `.myc`.
+   - Paths are resolved relative to the file that declares the dependency.
+   - If a path is not in `loaded_modules`, it is added to the set.
+   - The file is read, and the extraction process runs recursively on its content.
+   - This creates a flat list of all dependencies in a valid topological order.
+
+4. **Two-Pass Compilation:**
+   We reuse the existing robust two-pass mechanism used by the `--lib` flag:
+   - **Pass 1 (Discovery):** Parse all discovered dependency files and traverse their untyped ASTs to register `Def` (Globals) and `MacroDecl` (Macros) in the Environment. This makes all exported symbols and macros available.
+   - **Pass 2 (Compilation):** Type-check and compile all dependency files, then execute them to initialize their global state.
+   - Finally, compile and execute the original main script.
+
+5. **Lexer Adaptation:**
+   Since `#use` is handled before the main parser, we will modify the `Lexer` to treat `#` as the start of a single-line comment (similar to `;`). This prevents the parser from crashing on the directives while keeping the line numbers accurate.
+
+## Summary
+
+By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules.

examples/sma.myc

@@ -1,4 +1,5 @@
-;; Output: 5.5
+#use lib->sma
+;; Output: void
 
 (do
   (def sma20 (SMA 20))

src/ast/environment.rs

@@ -5,7 +5,8 @@ use crate::ast::nodes::{Node, Symbol, UntypedKind};
 use crate::ast::parser::Parser;
 use crate::ast::vm::{TracingObserver, VM};
 use std::cell::RefCell;
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
+use std::path::{Path, PathBuf};
 use std::rc::Rc;
 
 use crate::ast::compiler::bound_nodes::{
@@ -79,6 +80,8 @@ pub struct Environment {
     pub optimization: bool,
     pub macro_registry: Rc<RefCell<MacroRegistry>>,
     pub pipeline_generators: Rc<RefCell<Vec<PipelineGenerator>>>,
+    pub search_paths: Rc<RefCell<Vec<PathBuf>>>,
+    pub loaded_modules: Rc<RefCell<HashSet<PathBuf>>>,
 }
 
 struct EnvFunctionRegistry {
@@ -163,11 +166,17 @@ impl Environment {
             optimization: true,
             macro_registry: Rc::new(RefCell::new(MacroRegistry::new())),
             pipeline_generators: Rc::new(RefCell::new(Vec::new())),
+            search_paths: Rc::new(RefCell::new(Vec::new())),
+            loaded_modules: Rc::new(RefCell::new(HashSet::new())),
         };
         env.register_stdlib();
         env
     }
 
+    pub fn add_search_path(&self, path: impl AsRef<Path>) {
+        self.search_paths.borrow_mut().push(path.as_ref().to_path_buf());
+    }
+
     pub fn set_debug_mode(&mut self, enabled: bool) {
         self.debug_mode = enabled;
     }
@@ -205,41 +214,105 @@ impl Environment {
         *self.macro_registry.borrow_mut() = expander.into_registry();
     }
 
-    /// Loads all .myc files from a directory as a library.
-    /// This uses a two-pass approach:
-    /// 1. Discovery: Identify all top-level globals and macros across all files.
-    /// 2. Compilation: Expand, bind, and type-check each file, then run its initialization.
-    pub fn load_library(&self, path: impl AsRef<std::path::Path>) -> Result<(), String> {
-        let dir = path.as_ref();
-        if !dir.is_dir() {
-            return Err(format!("Path {} is not a directory", dir.display()));
+    /// Resolves a #use module path relative to a base path, then falls back to search paths.
+    fn resolve_module_path(&self, module_path: &str, base_path: &Path) -> Result<PathBuf, String> {
+        let relative_path = module_path.replace("->", std::path::MAIN_SEPARATOR_STR);
+        let file_name = format!("{}.myc", relative_path);
+
+        // 1. Try relative to base_path
+        let mut path = base_path.to_path_buf();
+        path.push(&file_name);
+        if path.is_file() {
+            return path.canonicalize().map_err(|e| format!("IO Error: {}", e));
         }
 
-        let mut files = Vec::new();
-        let mut entries: Vec<_> = std::fs::read_dir(dir)
-            .map_err(|e| e.to_string())?
-            .filter_map(|e| e.ok())
-            .collect();
-        // Sort entries for deterministic loading order
-        entries.sort_by_key(|e| e.path());
-
-        for entry in entries {
-            let path = entry.path();
-            if path.is_file() && path.extension().is_some_and(|ext| ext == "myc") {
-                let source = std::fs::read_to_string(&path).map_err(|e| e.to_string())?;
-                let mut parser = Parser::new(&source);
-                let untyped_ast = parser.parse_expression();
-                if parser.diagnostics.has_errors() {
-                    return Err(format!(
-                        "Parser errors in {}:\n{}",
-                        path.display(),
-                        parser.diagnostics.items[0].message
-                    ));
-                }
-                files.push((path, untyped_ast));
+        // 2. Try search paths
+        for sp in self.search_paths.borrow().iter() {
+            let mut path = sp.clone();
+            path.push(&file_name);
+            if path.is_file() {
+                return path.canonicalize().map_err(|e| format!("IO Error: {}", e));
             }
         }
 
+        Err(format!("Could not find module '{}' (looked for {})", module_path, file_name))
+    }
+
+    /// Recursively collects all dependencies starting from a given source string.
+    fn collect_dependencies(
+        &self,
+        source: &str,
+        base_path: &Path,
+        all_files: &mut Vec<(PathBuf, Node<UntypedKind>)>,
+    ) -> Result<(), String> {
+        let directives = self.extract_use_directives(source);
+
+        for module_path in directives {
+            let abs_path = self.resolve_module_path(&module_path, base_path)?;
+
+            if self.loaded_modules.borrow().contains(&abs_path) {
+                continue;
+            }
+
+            self.loaded_modules.borrow_mut().insert(abs_path.clone());
+
+            let lib_source = std::fs::read_to_string(&abs_path)
+                .map_err(|e| format!("Failed to read module {:?}: {}", abs_path, e))?;
+
+            // Pre-process dependencies of the library BEFORE adding it, ensuring topological order
+            let lib_base = abs_path.parent().unwrap_or(Path::new("."));
+            self.collect_dependencies(&lib_source, lib_base, all_files)?;
+
+            let mut parser = Parser::new(&lib_source);
+            let untyped_ast = parser.parse_expression();
+
+            if parser.diagnostics.has_errors() {
+                return Err(format!(
+                    "Parser error in module {:?}:\n{}",
+                    abs_path,
+                    parser.diagnostics.items[0].message
+                ));
+            }
+
+            all_files.push((abs_path, untyped_ast));
+        }
+        Ok(())
+    }
+
+    fn extract_use_directives(&self, source: &str) -> Vec<String> {
+        let mut paths = Vec::new();
+        for line in source.lines() {
+            let trimmed = line.trim();
+            if trimmed.is_empty() || trimmed.starts_with(';') {
+                continue;
+            }
+            if let Some(stripped) = trimmed.strip_prefix("#use ") {
+                let path = stripped.trim();
+                // Strip optional quotes if they somehow got in, though user spec says no spaces/quotes.
+                let clean_path = if (path.starts_with('"') && path.ends_with('"'))
+                    || (path.starts_with('\'') && path.ends_with('\''))
+                {
+                    &path[1..path.len() - 1]
+                } else {
+                    path
+                };
+                paths.push(clean_path.to_string());
+            } else {
+                // Stop at first non-directive/non-comment line
+                break;
+            }
+        }
+        paths
+    }
+
+    /// Used to pre-load all dependencies of a script before compiling it.
+    /// It returns the base path which can be used to resolve further things if necessary.
+    pub fn preload_dependencies(&self, source: &str, file_path: Option<&Path>) -> Result<(), String> {
+        let base_path = file_path.and_then(|p| p.parent()).unwrap_or_else(|| Path::new("."));
+        let mut files = Vec::new();
+        
+        self.collect_dependencies(source, base_path, &mut files)?;
+
         // Pass 1: Discovery (Globals and Macros)
         for (_, untyped_ast) in &files {
             self.discover_globals(untyped_ast);

src/ast/lexer.rs

@@ -136,7 +136,7 @@ impl<'a> Lexer<'a> {
                     self.col += 1;
                 }
                 self.input.next();
-            } else if c == ';' {
+            } else if c == ';' || c == '#' {
                 for c in self.input.by_ref() {
                     if c == '\n' {
                         self.line += 1;

src/bin/ast.rs

@@ -45,12 +45,9 @@ fn main() {
     let mut env = Environment::new();
     env.optimization = !cli.no_opt;
     
-    // Load libraries
+    // Load libraries (Now just search paths)
     for lib_path in &cli.lib {
-        if let Err(e) = env.load_library(lib_path) {
-            eprintln!("❌ Error loading library {:?}: {}", lib_path, e);
-            std::process::exit(1);
-        }
+        env.add_search_path(lib_path);
     }
 
     if cli.bench || cli.update_bench {
@@ -77,6 +74,11 @@ fn main() {
     }
 
     if let Some(script_str) = cli.eval {
+        if let Err(e) = env.preload_dependencies(&script_str, None) {
+            eprintln!("❌ Error resolving dependencies: {}", e);
+            std::process::exit(1);
+        }
+
         if cli.dump {
             match env.dump_ast(&script_str) {
                 Ok(dump) => println!("{}", dump),
@@ -90,6 +92,11 @@ fn main() {
     } else if let Some(file_path) = cli.file {
         match fs::read_to_string(&file_path) {
             Ok(content) => {
+                if let Err(e) = env.preload_dependencies(&content, Some(&file_path)) {
+                    eprintln!("❌ Error resolving dependencies for {:?}:\n{}", file_path, e);
+                    std::process::exit(1);
+                }
+
                 if cli.dump {
                     match env.dump_ast(&content) {
                         Ok(dump) => println!("{}", dump),

src/main.rs

@@ -98,6 +98,10 @@ impl CompilerApp {
         self.env.optimization = true;
 
         let res = (|| -> Result<(myc::ast::types::Value, std::time::Duration), String> {
+            if let Err(e) = self.env.preload_dependencies(&self.source_code, self.current_example_path.as_deref()) {
+                return Err(format!("Dependency Error: {}", e));
+            }
+
             let start_run = std::time::Instant::now();
             let result = if self.debug_enabled {
                 let (res, logs) = self.env.run_debug(&self.source_code)?;
@@ -158,6 +162,12 @@ impl CompilerApp {
     fn dump_ast(&mut self) {
         self.env = Environment::new();
         self.env.optimization = true; // Enable optimization for AST dump
+        
+        if let Err(e) = self.env.preload_dependencies(&self.source_code, self.current_example_path.as_deref()) {
+            self.output_log = format!("Dependency Error: {}", e);
+            return;
+        }
+
         match self.env.dump_ast(&self.source_code) {
             Ok(dump) => {
                 self.output_log = format!("--- BOUND AST DUMP ---\n\n{}", dump);

src/utils/tester.rs

@@ -28,6 +28,7 @@ pub fn run_functional_tests_with_optimization(enabled: bool) -> Vec<TestResult>
 
     for entry in entries.filter_map(|e| e.ok()) {
         let mut env = Environment::new(); // Fresh environment per test file
+        env.add_search_path(".");
         env.optimization = enabled;
         let path = entry.path();
         if path.extension().is_some_and(|ext| ext == "myc") {
@@ -39,6 +40,15 @@ pub fn run_functional_tests_with_optimization(enabled: bool) -> Vec<TestResult>
                 .map(|m| m.get(1).unwrap().as_str().trim().to_string());
 
             if let Some(expected) = expected_output {
+                if let Err(e) = env.preload_dependencies(&content, Some(&path)) {
+                    results.push(TestResult {
+                        name,
+                        success: false,
+                        message: format!("Dependency Error: {}", e),
+                    });
+                    continue;
+                }
+
                 match env.run_script(&content) {
                     Ok(val) => {
                         let val_str = format!("{}", val);
@@ -112,6 +122,19 @@ pub fn run_benchmarks(update: bool, filter: Option<&str>) -> Vec<BenchmarkResult
 
         // Compile once for this file (symbols/types are compatible with all fresh environments)
         let initial_env = Environment::new();
+        initial_env.add_search_path(".");
+        
+        if let Err(e) = initial_env.preload_dependencies(&content, Some(&path)) {
+            results.push(BenchmarkResult {
+                name,
+                median: Duration::ZERO,
+                baseline: None,
+                diff_pct: None,
+                status: format!("DEPENDENCY ERROR: {}", e),
+            });
+            continue;
+        }
+
         let compiled_once = match initial_env.compile(&content).into_result() {
             Ok(c) => c,
             Err(e) => {
@@ -130,6 +153,11 @@ pub fn run_benchmarks(update: bool, filter: Option<&str>) -> Vec<BenchmarkResult
         let measure_sum =
             |n: u32, node: &crate::ast::compiler::TypedNode| -> Result<Duration, String> {
                 let env = Environment::new();
+                env.add_search_path(".");
+                if let Err(e) = env.preload_dependencies(&content, Some(&path)) {
+                    return Err(format!("Dependency Error in measurement: {}", e));
+                }
+                
                 // Link once per sample
                 let linked = env.link(node.clone());
                 let func = env.instantiate(linked);