Add 'Random Search' (parameter-search cluster)

Random-Search.md

@@ -0,0 +1,84 @@
+# Random Search
+
+**Random search** replaces exhaustive enumeration with *independent random
+draws*: each trial point is sampled from a declared distribution per parameter —
+typically uniform over a continuous range $[a_i, b_i]$ for each parameter $i$. [L]
+The number of trials, $n$, is chosen directly as a **budget**, independent of the
+number of parameters. [L]
+
+## Why a fixed budget beats a grid
+
+A [grid](Grid-Search)'s cost is the *product* of its axes, so it explodes with
+dimension. Random search fixes $n$ up front: adding a parameter does not multiply
+the cost — it only spreads the same $n$ draws across one more axis. Two
+consequences make it the stronger default in higher dimensions:
+
+1. **Gridless coverage.** Samples come from the *continuous* range, so the search
+   is not confined to a pre-chosen lattice; with more draws the space simply
+   becomes denser everywhere. [L]
+2. **Coverage per important axis.** Along any single parameter, $n$ random draws
+   give up to $n$ *distinct* values, whereas a grid of the same total size gives
+   only a coarse per-axis resolution. When just a few parameters truly matter —
+   but it is unknown *which* — this is decisive: Bergstra and Bengio showed that
+   random search matches or beats grid search for hyperparameter optimization
+   precisely because real problems tend to have **low effective dimensionality**.
+   [C]
+
+### How many samples?
+
+Coverage has an exact, dimension-independent form. If a target region occupies a
+fraction $v$ of the search volume, the probability that at least one of $n$
+independent uniform draws lands in it is
+
+$$P_{\text{hit}} = 1 - (1 - v)^{n}.$$
+
+To hit the best fraction $p$ of the space with confidence $1 - \varepsilon$ one
+needs
+
+$$n \ge \frac{\ln \varepsilon}{\ln(1 - p)}.$$
+
+[L] For example, to be 95% confident of landing in the top 5% region
+($p = 0.05$, $\varepsilon = 0.05$) requires $n \ge 59$ draws — **the same 59
+regardless of how many parameters there are.** [L] This is the quantitative core
+of the random-over-grid argument.
+
+## Quasi-random (low-discrepancy) sampling
+
+Pure pseudo-random draws clump: by chance they leave gaps and clusters.
+**Low-discrepancy sequences** (e.g. Sobol sequences, or Latin-hypercube designs)
+fill a space more evenly than pseudo-random points, so the proportion of points
+in any region tracks that region's volume more closely. [L] They are a drop-in
+improvement on uniform random with the same independence and budget properties,
+and reduce the sample count needed for a given coverage in quasi-Monte-Carlo
+settings. [C]
+
+## Properties
+
+- **Budget-driven and dimension-agnostic.** You set $n$; the cost is $n$. [L]
+- **Independent evaluations.** Draws do not depend on one another, so random
+  search is [embarrassingly parallel](Evaluation-Dependency-and-Parallelism). [L]
+- **Reproducible when seeded.** A fixed pseudo-random seed reproduces the exact
+  draw sequence; without one, two runs differ. [L]
+- **A strong baseline.** Cheap, simple, and parallel, random search is the
+  standard yardstick an [adaptive method](Adaptive-Search) must beat to justify
+  its added machinery. [C]
+
+## When random search fits
+
+Many parameters, continuous ranges, a fixed evaluation budget, or simply a
+trustworthy baseline before investing in adaptivity. [C]
+
+## Claim legend
+
+- **[L]** law / exact — a definition, identity, or theorem that does not vary.
+- **[C]** convention — a practitioner rule of thumb that varies by context.
+
+## References
+
+- Bergstra, J. & Bengio, Y. (2012). *Random Search for Hyper-Parameter
+  Optimization.* JMLR, 13, 281–305.
+  <https://jmlr.org/papers/v13/bergstra12a.html>
+- *Hyperparameter optimization* (Random search section). Wikipedia.
+  <https://en.wikipedia.org/wiki/Hyperparameter_optimization>
+- *Low-discrepancy sequence.* Wikipedia.
+  <https://en.wikipedia.org/wiki/Low-discrepancy_sequence>