diff --git a/docs/dependency_management.md b/docs/dependency_management.md new file mode 100644 index 0000000..96ed165 --- /dev/null +++ b/docs/dependency_management.md @@ -0,0 +1,64 @@ +# Myc Dependency Management: The `#use` Directive + +## Motivation + +As the Myc language and its ecosystem grow, scripts become larger and need to be split into multiple files (libraries). Until now, the AST compiler tool (`ast.exe`) only supported loading libraries via the `--lib` command-line argument. + +We need a way to explicitly define dependencies *within* a Myc script, ensuring that a script can autonomously declare what it needs to run. + +## Constraints & Design Decisions + +1. **No AST Pollution:** The dependency declaration must not become part of the compiled AST. The AST is designed to be a pure representation of the runtime logic, which makes it suitable for visual programming and graphical representation. Dependency management is a compile-time/environment concern, not a runtime AST concern. +2. **Macro Availability:** Libraries often define macros. If script `A` depends on library `B`, and `B` defines a macro `foo`, script `A` must have access to `foo` during its own macro-expansion phase. +3. **Idempotency:** If `A` depends on `B` and `C`, and `B` also depends on `C`, library `C` must only be parsed and evaluated once. +4. **Location:** Dependencies must be declared at the very beginning of the file, before any actual Myc code. +5. **Path Format Restrictions:** + - Whitespace is **not allowed** in library paths and file names. + - The file extension is **always omitted** (the system implies `.myc`). + - The folder separator is replaced by `->`. +6. **Search Paths:** The `--lib` command-line argument will no longer proactively load all `.myc` files. Instead, it will append directories to a list of global **search paths**. When `#use` is resolved, the compiler first checks relative to the current file, and then falls back to searching within the provided `--lib` search paths. + +## The Solution: `#use` Preprocessor Directives + +We introduce a preprocessor directive `#use` that is evaluated *before* the main parsing phase begins. + +### Syntax + +```myc +#use math->indicators +#use utils + +(do + ;; Actual Myc code begins here + (def my_var (indicators.calculate ...)) +) +``` +*(Here `math->indicators` resolves to `math/indicators.myc` and `utils` resolves to `utils.myc`)* + +### Implementation Plan + +1. **Environment State:** + The `Environment` will be extended with a `HashSet` (e.g., `loaded_modules`) to keep track of canonical paths that have already been loaded, ensuring idempotency. + +2. **Pre-Parse Extraction:** + Before the AST `Parser` processes a script, a lightweight scraper will read the top lines of the file. It will extract all `#use` paths and stop as soon as it encounters a line that is not empty, not a comment (`;`), and not a `#use` directive. + +3. **Recursive Resolution:** + - The scraper extracts the module path, replaces `->` with the OS-specific directory separator, and appends `.myc`. + - Paths are resolved relative to the file that declares the dependency. + - If a path is not in `loaded_modules`, it is added to the set. + - The file is read, and the extraction process runs recursively on its content. + - This creates a flat list of all dependencies in a valid topological order. + +4. **Two-Pass Compilation:** + We reuse the existing robust two-pass mechanism used by the `--lib` flag: + - **Pass 1 (Discovery):** Parse all discovered dependency files and traverse their untyped ASTs to register `Def` (Globals) and `MacroDecl` (Macros) in the Environment. This makes all exported symbols and macros available. + - **Pass 2 (Compilation):** Type-check and compile all dependency files, then execute them to initialize their global state. + - Finally, compile and execute the original main script. + +5. **Lexer Adaptation:** + Since `#use` is handled before the main parser, we will modify the `Lexer` to treat `#` as the start of a single-line comment (similar to `;`). This prevents the parser from crashing on the directives while keeping the line numbers accurate. + +## Summary + +By using a preprocessor directive, we keep the core Lisp-like syntax and AST completely clean of module-loading mechanics, while providing a powerful, recursive, and idempotent dependency resolution system that plays perfectly with our macro expansion rules. diff --git a/examples/sma.myc b/examples/sma.myc index a8133f9..6f17ac9 100644 --- a/examples/sma.myc +++ b/examples/sma.myc @@ -1,4 +1,5 @@ -;; Output: 5.5 +#use lib->sma +;; Output: void (do (def sma20 (SMA 20)) diff --git a/src/ast/environment.rs b/src/ast/environment.rs index 7d124d3..30deac5 100644 --- a/src/ast/environment.rs +++ b/src/ast/environment.rs @@ -5,7 +5,8 @@ use crate::ast::nodes::{Node, Symbol, UntypedKind}; use crate::ast::parser::Parser; use crate::ast::vm::{TracingObserver, VM}; use std::cell::RefCell; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; +use std::path::{Path, PathBuf}; use std::rc::Rc; use crate::ast::compiler::bound_nodes::{ @@ -79,6 +80,8 @@ pub struct Environment { pub optimization: bool, pub macro_registry: Rc>, pub pipeline_generators: Rc>>, + pub search_paths: Rc>>, + pub loaded_modules: Rc>>, } struct EnvFunctionRegistry { @@ -163,11 +166,17 @@ impl Environment { optimization: true, macro_registry: Rc::new(RefCell::new(MacroRegistry::new())), pipeline_generators: Rc::new(RefCell::new(Vec::new())), + search_paths: Rc::new(RefCell::new(Vec::new())), + loaded_modules: Rc::new(RefCell::new(HashSet::new())), }; env.register_stdlib(); env } + pub fn add_search_path(&self, path: impl AsRef) { + self.search_paths.borrow_mut().push(path.as_ref().to_path_buf()); + } + pub fn set_debug_mode(&mut self, enabled: bool) { self.debug_mode = enabled; } @@ -205,41 +214,105 @@ impl Environment { *self.macro_registry.borrow_mut() = expander.into_registry(); } - /// Loads all .myc files from a directory as a library. - /// This uses a two-pass approach: - /// 1. Discovery: Identify all top-level globals and macros across all files. - /// 2. Compilation: Expand, bind, and type-check each file, then run its initialization. - pub fn load_library(&self, path: impl AsRef) -> Result<(), String> { - let dir = path.as_ref(); - if !dir.is_dir() { - return Err(format!("Path {} is not a directory", dir.display())); + /// Resolves a #use module path relative to a base path, then falls back to search paths. + fn resolve_module_path(&self, module_path: &str, base_path: &Path) -> Result { + let relative_path = module_path.replace("->", std::path::MAIN_SEPARATOR_STR); + let file_name = format!("{}.myc", relative_path); + + // 1. Try relative to base_path + let mut path = base_path.to_path_buf(); + path.push(&file_name); + if path.is_file() { + return path.canonicalize().map_err(|e| format!("IO Error: {}", e)); } - let mut files = Vec::new(); - let mut entries: Vec<_> = std::fs::read_dir(dir) - .map_err(|e| e.to_string())? - .filter_map(|e| e.ok()) - .collect(); - // Sort entries for deterministic loading order - entries.sort_by_key(|e| e.path()); - - for entry in entries { - let path = entry.path(); - if path.is_file() && path.extension().is_some_and(|ext| ext == "myc") { - let source = std::fs::read_to_string(&path).map_err(|e| e.to_string())?; - let mut parser = Parser::new(&source); - let untyped_ast = parser.parse_expression(); - if parser.diagnostics.has_errors() { - return Err(format!( - "Parser errors in {}:\n{}", - path.display(), - parser.diagnostics.items[0].message - )); - } - files.push((path, untyped_ast)); + // 2. Try search paths + for sp in self.search_paths.borrow().iter() { + let mut path = sp.clone(); + path.push(&file_name); + if path.is_file() { + return path.canonicalize().map_err(|e| format!("IO Error: {}", e)); } } + Err(format!("Could not find module '{}' (looked for {})", module_path, file_name)) + } + + /// Recursively collects all dependencies starting from a given source string. + fn collect_dependencies( + &self, + source: &str, + base_path: &Path, + all_files: &mut Vec<(PathBuf, Node)>, + ) -> Result<(), String> { + let directives = self.extract_use_directives(source); + + for module_path in directives { + let abs_path = self.resolve_module_path(&module_path, base_path)?; + + if self.loaded_modules.borrow().contains(&abs_path) { + continue; + } + + self.loaded_modules.borrow_mut().insert(abs_path.clone()); + + let lib_source = std::fs::read_to_string(&abs_path) + .map_err(|e| format!("Failed to read module {:?}: {}", abs_path, e))?; + + // Pre-process dependencies of the library BEFORE adding it, ensuring topological order + let lib_base = abs_path.parent().unwrap_or(Path::new(".")); + self.collect_dependencies(&lib_source, lib_base, all_files)?; + + let mut parser = Parser::new(&lib_source); + let untyped_ast = parser.parse_expression(); + + if parser.diagnostics.has_errors() { + return Err(format!( + "Parser error in module {:?}:\n{}", + abs_path, + parser.diagnostics.items[0].message + )); + } + + all_files.push((abs_path, untyped_ast)); + } + Ok(()) + } + + fn extract_use_directives(&self, source: &str) -> Vec { + let mut paths = Vec::new(); + for line in source.lines() { + let trimmed = line.trim(); + if trimmed.is_empty() || trimmed.starts_with(';') { + continue; + } + if let Some(stripped) = trimmed.strip_prefix("#use ") { + let path = stripped.trim(); + // Strip optional quotes if they somehow got in, though user spec says no spaces/quotes. + let clean_path = if (path.starts_with('"') && path.ends_with('"')) + || (path.starts_with('\'') && path.ends_with('\'')) + { + &path[1..path.len() - 1] + } else { + path + }; + paths.push(clean_path.to_string()); + } else { + // Stop at first non-directive/non-comment line + break; + } + } + paths + } + + /// Used to pre-load all dependencies of a script before compiling it. + /// It returns the base path which can be used to resolve further things if necessary. + pub fn preload_dependencies(&self, source: &str, file_path: Option<&Path>) -> Result<(), String> { + let base_path = file_path.and_then(|p| p.parent()).unwrap_or_else(|| Path::new(".")); + let mut files = Vec::new(); + + self.collect_dependencies(source, base_path, &mut files)?; + // Pass 1: Discovery (Globals and Macros) for (_, untyped_ast) in &files { self.discover_globals(untyped_ast); diff --git a/src/ast/lexer.rs b/src/ast/lexer.rs index 6a9c673..129aa5d 100644 --- a/src/ast/lexer.rs +++ b/src/ast/lexer.rs @@ -136,7 +136,7 @@ impl<'a> Lexer<'a> { self.col += 1; } self.input.next(); - } else if c == ';' { + } else if c == ';' || c == '#' { for c in self.input.by_ref() { if c == '\n' { self.line += 1; diff --git a/src/bin/ast.rs b/src/bin/ast.rs index 657ad8b..6c6d0a5 100644 --- a/src/bin/ast.rs +++ b/src/bin/ast.rs @@ -45,12 +45,9 @@ fn main() { let mut env = Environment::new(); env.optimization = !cli.no_opt; - // Load libraries + // Load libraries (Now just search paths) for lib_path in &cli.lib { - if let Err(e) = env.load_library(lib_path) { - eprintln!("❌ Error loading library {:?}: {}", lib_path, e); - std::process::exit(1); - } + env.add_search_path(lib_path); } if cli.bench || cli.update_bench { @@ -77,6 +74,11 @@ fn main() { } if let Some(script_str) = cli.eval { + if let Err(e) = env.preload_dependencies(&script_str, None) { + eprintln!("❌ Error resolving dependencies: {}", e); + std::process::exit(1); + } + if cli.dump { match env.dump_ast(&script_str) { Ok(dump) => println!("{}", dump), @@ -90,6 +92,11 @@ fn main() { } else if let Some(file_path) = cli.file { match fs::read_to_string(&file_path) { Ok(content) => { + if let Err(e) = env.preload_dependencies(&content, Some(&file_path)) { + eprintln!("❌ Error resolving dependencies for {:?}:\n{}", file_path, e); + std::process::exit(1); + } + if cli.dump { match env.dump_ast(&content) { Ok(dump) => println!("{}", dump), diff --git a/src/main.rs b/src/main.rs index a787a44..5baa1dc 100644 --- a/src/main.rs +++ b/src/main.rs @@ -98,6 +98,10 @@ impl CompilerApp { self.env.optimization = true; let res = (|| -> Result<(myc::ast::types::Value, std::time::Duration), String> { + if let Err(e) = self.env.preload_dependencies(&self.source_code, self.current_example_path.as_deref()) { + return Err(format!("Dependency Error: {}", e)); + } + let start_run = std::time::Instant::now(); let result = if self.debug_enabled { let (res, logs) = self.env.run_debug(&self.source_code)?; @@ -158,6 +162,12 @@ impl CompilerApp { fn dump_ast(&mut self) { self.env = Environment::new(); self.env.optimization = true; // Enable optimization for AST dump + + if let Err(e) = self.env.preload_dependencies(&self.source_code, self.current_example_path.as_deref()) { + self.output_log = format!("Dependency Error: {}", e); + return; + } + match self.env.dump_ast(&self.source_code) { Ok(dump) => { self.output_log = format!("--- BOUND AST DUMP ---\n\n{}", dump); diff --git a/src/utils/tester.rs b/src/utils/tester.rs index 24f2968..33c7e31 100644 --- a/src/utils/tester.rs +++ b/src/utils/tester.rs @@ -28,6 +28,7 @@ pub fn run_functional_tests_with_optimization(enabled: bool) -> Vec for entry in entries.filter_map(|e| e.ok()) { let mut env = Environment::new(); // Fresh environment per test file + env.add_search_path("."); env.optimization = enabled; let path = entry.path(); if path.extension().is_some_and(|ext| ext == "myc") { @@ -39,6 +40,15 @@ pub fn run_functional_tests_with_optimization(enabled: bool) -> Vec .map(|m| m.get(1).unwrap().as_str().trim().to_string()); if let Some(expected) = expected_output { + if let Err(e) = env.preload_dependencies(&content, Some(&path)) { + results.push(TestResult { + name, + success: false, + message: format!("Dependency Error: {}", e), + }); + continue; + } + match env.run_script(&content) { Ok(val) => { let val_str = format!("{}", val); @@ -112,6 +122,19 @@ pub fn run_benchmarks(update: bool, filter: Option<&str>) -> Vec c, Err(e) => { @@ -130,6 +153,11 @@ pub fn run_benchmarks(update: bool, filter: Option<&str>) -> Vec Result { let env = Environment::new(); + env.add_search_path("."); + if let Err(e) = env.preload_dependencies(&content, Some(&path)) { + return Err(format!("Dependency Error in measurement: {}", e)); + } + // Link once per sample let linked = env.link(node.clone()); let func = env.instantiate(linked);