Add 'Grid Search' (parameter-search cluster)

Grid-Search.md

@@ -0,0 +1,56 @@
+# Grid Search
+
+**Grid search** is the exhaustive evaluation of every combination drawn from a
+*manually specified, discrete value set per parameter* — the Cartesian product of
+those sets. [L] It is the most direct search: enumerate the lattice, evaluate
+every node, keep the best.
+
+## Cost: the product of the axes
+
+For $d$ parameters, where parameter $i$ contributes $k_i$ distinct values, the
+grid has
+
+$$N_{\text{grid}} = \prod_{i=1}^{d} k_i$$
+
+points. [L] Because the count is a *product*, every added parameter multiplies the
+work. In the simplest case of $d$ binary parameters this is already $2^{d}$,
+exponential in the number of parameters — an instance of the **curse of
+dimensionality**. [L] Ten parameters at ten values each is ten billion
+evaluations; the grid becomes unaffordable long before it is fine-grained enough.
+
+## Properties
+
+- **Complete within the lattice, but only the lattice.** Grid search finds the
+  best *node it enumerates*, but the resolution is chosen in advance; an optimum
+  that falls *between* grid lines is unreachable. [L]
+- **Deterministic and reproducible.** The same grid yields the same point set in
+  the same order, with no randomness to seed. [L]
+- **Independent evaluations.** No grid point depends on another's result, so the
+  whole grid is
+  [embarrassingly parallel](Evaluation-Dependency-and-Parallelism). [L]
+- **Wasteful in high dimensions.** When only a few parameters affect the
+  objective, grid search still spends its entire budget refining a coarse lattice
+  along the parameters that do not matter. [C]
+
+## When grid search fits
+
+Grid search is a reasonable default for **few parameters (about three or fewer)**
+over **small, known discrete value sets**, or when completeness over a specific
+lattice is itself the goal (e.g. a reproducible reference sweep). [C] Beyond a
+handful of dimensions, [random search](Random-Search) or an
+[adaptive method](Adaptive-Search) covers the space far more economically.
+
+## Claim legend
+
+- **[L]** law / exact — a definition, identity, or theorem that does not vary.
+- **[C]** convention — a practitioner rule of thumb that varies by context.
+
+## References
+
+- *Hyperparameter optimization* (Grid search section). Wikipedia.
+  <https://en.wikipedia.org/wiki/Hyperparameter_optimization>
+- *Curse of dimensionality.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Curse_of_dimensionality>
+- Bergstra, J. & Bengio, Y. (2012). *Random Search for Hyper-Parameter
+  Optimization.* JMLR, 13, 281–305.
+  <https://jmlr.org/papers/v13/bergstra12a.html>