Add 'Parameter Search Methods' (parameter-search cluster)

2026-06-17 10:12:19 +02:00
parent 8d3a4eb6c9
commit 40e79810e9
1 changed files with 90 additions and 0 deletions
@@ -0,0 +1,90 @@
+# Parameter Search Methods
+
+*Parameter search* (in machine learning, *hyperparameter optimization*) is the
+problem of choosing values for a system's tunable parameters so as to optimize an
+objective that can only be measured by *running* the system — training a model,
+executing a simulation, or backtesting a trading strategy — and reading off a
+score. The objective is typically a black box: it is observed only through its
+output, with no gradient or analytic form available. [L]
+
+This wiki summarizes the main families of parameter-search methods and the
+property that most sharply distinguishes them: **how each trial depends on the
+results of earlier trials**.
+
+## The organizing axis: evaluation dependency
+
+Every search method proposes a sequence of *trial points* (parameter
+assignments), evaluates each, and keeps the best. Methods divide cleanly by
+whether a trial point may be chosen *using the outcomes of earlier trials*:
+
+- **Non-adaptive (open-loop).** The complete set of trial points is fixed
+  *before any evaluation runs*. Evaluations are mutually independent. [L] This
+  family is [Grid search](Grid-Search) and [Random search](Random-Search)
+  (including quasi-random sampling).
+- **Adaptive (closed-loop).** Each new trial is chosen *from the results so far*.
+  Evaluations form a dependency chain. [L] This family is
+  [Adaptive search](Adaptive-Search): Bayesian optimization, evolutionary /
+  genetic algorithms, and CMA-ES.
+
+This single axis governs three practical properties at once — sample efficiency,
+parallelizability, and overfitting risk — so it has its own page:
+[Evaluation dependency and parallelism](Evaluation-Dependency-and-Parallelism).
+
+```mermaid
+flowchart TD
+  PS["Parameter search"] --> NA["Non-adaptive (open-loop)"]
+  PS --> AD["Adaptive (closed-loop)"]
+  NA --> GRID["Grid search"]
+  NA --> RAND["Random search"]
+  NA --> QR["Quasi-random / low-discrepancy"]
+  AD --> BO["Bayesian optimization"]
+  AD --> EV["Evolutionary / genetic"]
+  AD --> CMA["CMA-ES"]
+```
+
+## Choosing a method
+
+There is no universally best optimizer: averaged over all possible problems, all
+search methods perform equally — the *No Free Lunch* theorem. [L] A method wins
+only by matching the structure of the problem at hand. The following are
+practitioner rules of thumb, not laws. [C]
+
+| Situation | Typical choice |
+|---|---|
+| Few parameters (about three or fewer), small known value sets | Grid search |
+| Many parameters, continuous ranges, fixed evaluation budget | Random / quasi-random search |
+| Few parameters actually matter, but which ones is unknown | Random search |
+| Each evaluation is very expensive; a larger budget is available | Bayesian optimization |
+| Continuous, rugged landscape, no usable gradient | CMA-ES / evolutionary |
+| A robust baseline before investing in an adaptive method | Random search |
+
+A recurring caution: more aggressive optimization is not always better. Adaptive
+methods that squeeze the best in-sample score are also the most prone to
+**overfitting** the evaluation data — see the discussion under
+[Adaptive search](Adaptive-Search). [C]
+
+## Pages
+
+- [Grid search](Grid-Search) — exhaustive enumeration of a value lattice.
+- [Random search](Random-Search) — independent sampling over declared ranges.
+- [Adaptive search](Adaptive-Search) — Bayesian, evolutionary, and CMA-ES.
+- [Evaluation dependency and parallelism](Evaluation-Dependency-and-Parallelism)
+  — why the dependency structure decides parallelizability.
+
+## Claim legend
+
+- **[L]** law / exact — a definition, identity, or theorem that does not vary.
+- **[C]** convention — a practitioner rule of thumb that varies by context.
+- **[CORR]** corrected — a value fixed here against a common but wrong source.
+
+## References
+
+- Bergstra, J. & Bengio, Y. (2012). *Random Search for Hyper-Parameter
+  Optimization.* Journal of Machine Learning Research, 13, 281–305.
+  <https://jmlr.org/papers/v13/bergstra12a.html>
+- *Hyperparameter optimization.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Hyperparameter_optimization>
+- *No free lunch in search and optimization.* Wikipedia.
+  <https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization>
+- *Walk forward optimization.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Walk_forward_optimization>