Add 'Adaptive Search' (parameter-search cluster)

Adaptive-Search.md

@@ -0,0 +1,83 @@
+# Adaptive Search
+
+**Adaptive** (closed-loop) search methods choose each new trial *using the
+results of earlier trials*. Unlike [grid](Grid-Search) or
+[random](Random-Search) search — where every trial point is fixed before any
+evaluation runs — an adaptive method runs a feedback loop: evaluate, learn,
+propose the next point(s), repeat. [L] This buys **sample efficiency** at the
+cost of a **sequential dependency chain** (see
+[evaluation dependency and parallelism](Evaluation-Dependency-and-Parallelism)).
+
+## Bayesian optimization
+
+Bayesian optimization fits a cheap *surrogate model* of the objective — most
+commonly a Gaussian process (kriging) — and uses an *acquisition function* to
+decide where to sample next, trading off exploration of unknown regions against
+exploitation of promising ones. [L] Common acquisition functions are *probability
+of improvement*, *expected improvement*, and *Thompson sampling*. For a current
+best value $f^{*}$ (minimization), expected improvement scores a candidate $x$ as
+
+$$\alpha_{\text{EI}}(x) = \mathbb{E}\!\left[\max\!\left(0,\; f^{*} - f(x)\right)\right],$$
+
+the expected amount by which it would beat the incumbent. [L] Because each
+evaluation updates the surrogate's posterior, every query depends on all prior
+results — the method is *sequential by construction*. It is the method of choice
+when **each evaluation is expensive**, since it aims to minimize the number of
+queries. [C]
+
+## Evolutionary and genetic algorithms
+
+Evolutionary algorithms maintain a *population* of candidate solutions and
+improve it across **generations**. In each generation the *fitness* of every
+individual is evaluated; fitter individuals are stochastically *selected*, then
+combined by *crossover* and perturbed by *mutation* to form the next generation.
+[L] Each generation is therefore a function of the previous generation's fitness
+values — the dependency is *generational*. Evolutionary methods make few
+assumptions about the landscape and handle rugged, discontinuous objectives. [C]
+
+## CMA-ES
+
+The *Covariance Matrix Adaptation Evolution Strategy* is a population-based,
+derivative-free optimizer for **continuous** spaces. It samples candidates from a
+multivariate normal distribution and, each iteration, adapts that distribution's
+mean, step size, and *covariance matrix* from the ranking of the candidates — so
+the search distribution itself learns the local shape of the landscape. [L] It
+relies only on the *ranking* of candidates, not on objective values or gradients,
+which makes it robust on ill-conditioned, non-separable problems. [C]
+
+## The trade-off: efficiency vs dependency, and overfitting
+
+Adaptive methods typically reach a good optimum in **far fewer evaluations** than
+random search. [C] Two costs come with the feedback loop:
+
+1. **Limited parallelism.** A dependency chain cannot be fanned out the way
+   independent evaluations can; parallelism is confined to within a batch or
+   generation, separated by synchronization barriers
+   ([details](Evaluation-Dependency-and-Parallelism)). [L]
+2. **Overfitting risk.** An optimizer that aggressively maximizes the *measured*
+   score also fits the *noise* of the particular evaluation data — the harder it
+   optimizes, the more it can curve-fit a sample. [C] The standard mitigation is
+   to score on held-out data; in trading-strategy validation this is
+   [walk forward optimization](https://en.wikipedia.org/wiki/Walk_forward_optimization)
+   — optimize on an in-sample window, validate on rolling out-of-sample windows.
+   [C]
+
+A practical corollary: an adaptive method is worth its complexity mainly when
+evaluations are expensive and a *non-adaptive* baseline has been measured first.
+[C]
+
+## Claim legend
+
+- **[L]** law / exact — a definition, identity, or theorem that does not vary.
+- **[C]** convention — a practitioner rule of thumb that varies by context.
+
+## References
+
+- *Bayesian optimization.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Bayesian_optimization>
+- *Genetic algorithm.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Genetic_algorithm>
+- *CMA-ES.* Wikipedia.
+  <https://en.wikipedia.org/wiki/CMA-ES>
+- *Walk forward optimization.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Walk_forward_optimization>