The NWP horseshoe

Numerical weather prediction’s (NWP) base paradigm is simple: weather is math, and math is sufficiently solvable. Just model phenomena on a grid using quantitative methods such as fluid mechanics and thermodynamics, then step forward in time from observed state. This observe-model-predict chain is the operative paradigm that powers everything else.

The use of the term “paradigm” is deliberate here. It comes from the systems thinking of Donella Meadows, which, funnily enough, built off the back of the seminal The Limits to Growth text, which itself asserted that a finite system cannot grow infinitely. A paradigm is the point of a system with the most leverage; the joint which everything else about the system turns on.

Atop the weather = math paradigm, a few different approaches have emerged, mostly combining traditional physics with modern computational methods. The UK’s Met Office recently dropped a useful taxonomy that shows five ways these combine to create NWP systems.

• Independent-Physics-Based. Solve the governing equations directly on a grid, with no learned components anywhere.
• Hybrid-Integrated-Machine-Learning. A trained model replaces or enhances one physics component from inside the dynamical model itself.
• Hybrid-Composite-ML. Physics-based and ML-based models run separately and exchange state, typically a higher resolution regional model taking boundary conditions from a global one.
• Augmented-ML. The underlying physics forecast runs untouched, and a separate learned model corrects or sharpens the output afterward.
• Independent-ML-Based. A trained model maps the current atmospheric state directly to a future state, with no physics equations run at inference at all.

The issue is that each of the Met Office’s penta-taxonomy kick in after forecasting went mechanistic. We can bolt two more paradigms onto the bottom.

• Individual-Embodied. A single trained observer synthesises experienced meteorological phenomena with hard-won expertise to make decisions, with minimal equations or datasets involved.
• Collective-Semi-Embodied. Community-level weather wisdom, refined across generations into memorable, transmissible rules is passed down via professions and communities of practice.

The interesting thing is that the Individual-Embodied brain and the Independent-ML-Based machine are doing the same thing: storing deep tacit knowledge somewhere uninspectable. Physics, on the other hand, sits out in the open, with legible, inspectable causality. Which means that the paradigms don’t form a line. They form a horseshoe.

The horseshoe theory of political science is typically credited to Jean-Pierre Faye. He noticed that Weimar-era communists and Nazis, supposed ideological opposites, were both working to torch the centrist factions. Faye’s claim is that when one bends the spectrum, the extremes end up closer to each other than to the middle. It’s a disputed theory but it does have a ring of truth in certain scenarios. Like NWP.

Both ends of the NWP horseshoe store knowledge as patterns. Patterns only cover what’s already been seen. The result is that physics tends to outperform ML when weather conditions are complex and/or historically unprecedented. Physics, sitting in the middle, doesn’t need precedent. It computes forward from observed state regardless of whether anything like this has happened before.

Physics matters most within complex, unprecedented, extreme situations. By definition, these events cannot be pattern-matched by humans or machines. But, unfortunately, we’re going to see more and more complex, unprecedented, extreme situations than ever. So, based on the horseshoe, it is not necessarily human expertise or ML that we need to figure out how to aggressively scale, but physics models.

The issue is that physics is horrifically costly at increasing scales. Halving grid spacing doubles resolution in each of three spatial dimensions, and the CFL stability condition forces a smaller time-step, leading to about 16x more compute for one doubling of resolution.

Push from today’s ~10km global models toward 1km, storm-resolving forecasts, and the naive multiplier runs to something like ten thousand times more compute. And it gets way worse as the resolution increases.

I am wondering: can we change that?