Table of Contents

Parameter Search Methods

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Parameter Search Methods

Parameter search (in machine learning, hyperparameter optimization) is the problem of choosing values for a system's tunable parameters so as to optimize an objective that can only be measured by running the system — training a model, executing a simulation, or backtesting a trading strategy — and reading off a score. The objective is typically a black box: it is observed only through its output, with no gradient or analytic form available. [L]

This wiki summarizes the main families of parameter-search methods and the property that most sharply distinguishes them: how each trial depends on the results of earlier trials.

The organizing axis: evaluation dependency

Every search method proposes a sequence of trial points (parameter assignments), evaluates each, and keeps the best. Methods divide cleanly by whether a trial point may be chosen using the outcomes of earlier trials:

Non-adaptive (open-loop). The complete set of trial points is fixed before any evaluation runs. Evaluations are mutually independent. [L] This family is Grid search and Random search (including quasi-random sampling).
Adaptive (closed-loop). Each new trial is chosen from the results so far. Evaluations form a dependency chain. [L] This family is Adaptive search: Bayesian optimization, evolutionary / genetic algorithms, and CMA-ES.

This single axis governs three practical properties at once — sample efficiency, parallelizability, and overfitting risk — so it has its own page: Evaluation dependency and parallelism.

flowchart TD
  PS["Parameter search"] --> NA["Non-adaptive (open-loop)"]
  PS --> AD["Adaptive (closed-loop)"]
  NA --> GRID["Grid search"]
  NA --> RAND["Random search"]
  NA --> QR["Quasi-random / low-discrepancy"]
  AD --> BO["Bayesian optimization"]
  AD --> EV["Evolutionary / genetic"]
  AD --> CMA["CMA-ES"]

Choosing a method

There is no universally best optimizer: averaged over all possible problems, all search methods perform equally — the No Free Lunch theorem. [L] A method wins only by matching the structure of the problem at hand. The following are practitioner rules of thumb, not laws. [C]

Situation	Typical choice
Few parameters (about three or fewer), small known value sets	Grid search
Many parameters, continuous ranges, fixed evaluation budget	Random / quasi-random search
Few parameters actually matter, but which ones is unknown	Random search
Each evaluation is very expensive; a larger budget is available	Bayesian optimization
Continuous, rugged landscape, no usable gradient	CMA-ES / evolutionary
A robust baseline before investing in an adaptive method	Random search

A recurring caution: more aggressive optimization is not always better. Adaptive methods that squeeze the best in-sample score are also the most prone to overfitting the evaluation data — see the discussion under Adaptive search. [C]

Pages

Grid search — exhaustive enumeration of a value lattice.
Random search — independent sampling over declared ranges.
Adaptive search — Bayesian, evolutionary, and CMA-ES.
Evaluation dependency and parallelism — why the dependency structure decides parallelizability.

Claim legend

[L] law / exact — a definition, identity, or theorem that does not vary.
[C] convention — a practitioner rule of thumb that varies by context.
[CORR] corrected — a value fixed here against a common but wrong source.

References

Bergstra, J. & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13, 281–305. https://jmlr.org/papers/v13/bergstra12a.html
Hyperparameter optimization. Wikipedia. https://en.wikipedia.org/wiki/Hyperparameter_optimization
No free lunch in search and optimization. Wikipedia. https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization
Walk forward optimization. Wikipedia. https://en.wikipedia.org/wiki/Walk_forward_optimization

Home

Backtesting Engine Architecture

Strategy Analysis & Validation