From 40e79810e9afb8e9d8e0581a27e16609fdd220d7 Mon Sep 17 00:00:00 2001 From: Brummel Date: Wed, 17 Jun 2026 10:12:19 +0200 Subject: [PATCH] Add 'Parameter Search Methods' (parameter-search cluster) --- Parameter-Search-Methods.md | 90 +++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 Parameter-Search-Methods.md diff --git a/Parameter-Search-Methods.md b/Parameter-Search-Methods.md new file mode 100644 index 0000000..564729a --- /dev/null +++ b/Parameter-Search-Methods.md @@ -0,0 +1,90 @@ +# Parameter Search Methods + +*Parameter search* (in machine learning, *hyperparameter optimization*) is the +problem of choosing values for a system's tunable parameters so as to optimize an +objective that can only be measured by *running* the system — training a model, +executing a simulation, or backtesting a trading strategy — and reading off a +score. The objective is typically a black box: it is observed only through its +output, with no gradient or analytic form available. [L] + +This wiki summarizes the main families of parameter-search methods and the +property that most sharply distinguishes them: **how each trial depends on the +results of earlier trials**. + +## The organizing axis: evaluation dependency + +Every search method proposes a sequence of *trial points* (parameter +assignments), evaluates each, and keeps the best. Methods divide cleanly by +whether a trial point may be chosen *using the outcomes of earlier trials*: + +- **Non-adaptive (open-loop).** The complete set of trial points is fixed + *before any evaluation runs*. Evaluations are mutually independent. [L] This + family is [Grid search](Grid-Search) and [Random search](Random-Search) + (including quasi-random sampling). +- **Adaptive (closed-loop).** Each new trial is chosen *from the results so far*. + Evaluations form a dependency chain. [L] This family is + [Adaptive search](Adaptive-Search): Bayesian optimization, evolutionary / + genetic algorithms, and CMA-ES. + +This single axis governs three practical properties at once — sample efficiency, +parallelizability, and overfitting risk — so it has its own page: +[Evaluation dependency and parallelism](Evaluation-Dependency-and-Parallelism). + +```mermaid +flowchart TD + PS["Parameter search"] --> NA["Non-adaptive (open-loop)"] + PS --> AD["Adaptive (closed-loop)"] + NA --> GRID["Grid search"] + NA --> RAND["Random search"] + NA --> QR["Quasi-random / low-discrepancy"] + AD --> BO["Bayesian optimization"] + AD --> EV["Evolutionary / genetic"] + AD --> CMA["CMA-ES"] +``` + +## Choosing a method + +There is no universally best optimizer: averaged over all possible problems, all +search methods perform equally — the *No Free Lunch* theorem. [L] A method wins +only by matching the structure of the problem at hand. The following are +practitioner rules of thumb, not laws. [C] + +| Situation | Typical choice | +|---|---| +| Few parameters (about three or fewer), small known value sets | Grid search | +| Many parameters, continuous ranges, fixed evaluation budget | Random / quasi-random search | +| Few parameters actually matter, but which ones is unknown | Random search | +| Each evaluation is very expensive; a larger budget is available | Bayesian optimization | +| Continuous, rugged landscape, no usable gradient | CMA-ES / evolutionary | +| A robust baseline before investing in an adaptive method | Random search | + +A recurring caution: more aggressive optimization is not always better. Adaptive +methods that squeeze the best in-sample score are also the most prone to +**overfitting** the evaluation data — see the discussion under +[Adaptive search](Adaptive-Search). [C] + +## Pages + +- [Grid search](Grid-Search) — exhaustive enumeration of a value lattice. +- [Random search](Random-Search) — independent sampling over declared ranges. +- [Adaptive search](Adaptive-Search) — Bayesian, evolutionary, and CMA-ES. +- [Evaluation dependency and parallelism](Evaluation-Dependency-and-Parallelism) + — why the dependency structure decides parallelizability. + +## Claim legend + +- **[L]** law / exact — a definition, identity, or theorem that does not vary. +- **[C]** convention — a practitioner rule of thumb that varies by context. +- **[CORR]** corrected — a value fixed here against a common but wrong source. + +## References + +- Bergstra, J. & Bengio, Y. (2012). *Random Search for Hyper-Parameter + Optimization.* Journal of Machine Learning Research, 13, 281–305. + +- *Hyperparameter optimization.* Wikipedia. + +- *No free lunch in search and optimization.* Wikipedia. + +- *Walk forward optimization.* Wikipedia. +