SPX Market Regimes with a Hidden Markov Model

I built this project to understand whether a Hidden Markov Model could identify changing SPX market conditions without using information from the future.

The notebook combines daily returns, 20-day realised volatility, rolling drawdown and the VIX. Models and scalers are repeatedly refitted on past observations only, while test-period probabilities are produced with a causal forward filter rather than backward smoothing.

The inferred regimes are translated into a simple risk-control experiment and compared with SPX buy-and-hold, a constant-exposure benchmark and a causal volatility-targeting rule. The aim is to separate genuine timing value from the mechanical effect of holding less equity risk.

What I wanted to understand

Test whether latent market states can be identified causally and used to manage risk, then determine whether the HMM adds anything beyond much simpler exposure controls.

What the project covers

1. Engineer four market features from aligned SPX and VIX data: daily log return, 20-day realised volatility, rolling one-year drawdown and log VIX.
2. Select the number of states before the test period and refit a diagonal-Gaussian HMM through time using several random initialisations.
3. Order each refit's states by their estimated risk characteristics and compute causal filtered probabilities without future observations.
4. Convert regime probabilities into next-day SPX exposure and deduct turnover-based transaction costs.
5. Compare the HMM with buy-and-hold, an ex-post constant-exposure control and a causal 10% volatility-targeting benchmark.
6. Estimate market alpha and beta with Newey–West errors and use a paired moving-block bootstrap to quantify uncertainty in incremental returns.

What came out of the comparison

• The four states have a coherent economic ordering: annualised volatility rises from approximately 9.2% in Calm to 34.5% in Stress, while average VIX rises from approximately 14.9 to 29.5.
• Calm accounts for approximately 39% of the test sample and Stress for approximately 7.5%; estimated state durations range from roughly 20 to 36 trading days.
• The HMM allocation reduces the observed maximum drawdown materially relative to SPX buy-and-hold, but it also carries substantially less average equity exposure.
• Final HMM wealth is close to the simpler exposure-controlled benchmarks and below SPX buy-and-hold over this particular sample.
• Every paired moving-block bootstrap interval for incremental annual return crosses zero, so the experiment does not provide convincing evidence of alpha.

The useful result is not a profitable-strategy claim. The HMM identifies economically interpretable regimes and provides a structured way to reduce risk, but its timing does not convincingly outperform simpler controls once exposure and statistical uncertainty are made explicit.

predicted = previous_probability @ transition_matrix
filtered = predicted * emission_likelihood
filtered /= filtered.sum()

position = signal_exposure.shift(1).clip(0.0, 1.0)
net_return = gross_return - turnover * cost_bps * 1e-4

What the result does not claim

The analysis uses one SPX/VIX sample, a diagonal-Gaussian emission model and a predefined mapping from risk-ranked states to exposure. Cash returns and execution are simplified, regime probabilities can become overconfident, and the constant-exposure control is an ex-post attribution diagnostic rather than a deployable rule.