Galileo Measurement Science · Report 0.2

From Model Fit to Marketing Decisions

A controlled comparison of Galileo and open-source marketing mix modeling methods on whether estimated response curves lead to better budget decisions.

SmartInfer Research Lab · August 2026 · 90 synthetic datasets with known economic ground truth · Not peer reviewed

Executive summary

Marketing mix models are commonly judged by how well they reproduce historical outcomes. But a model can fit history and still recommend the wrong budget. This study asks a stricter question: when each model is given the same observations and the same allocation problem, how much economic value does its recommendation preserve relative to the known best allocation?

Main result. Galileo 0.2 had the lowest median normalized decision regret of the five evaluated methods. It beat the original Galileo 0 baseline on 84 of 90 matched datasets, Google Meridian on 55 of 90 (with two ties), Meta Robyn on 85 of 90, and PyMC-Marketing on 67 of 90.

The result is not universal dominance. PyMC-Marketing led the clean regime, while Google Meridian led when the baseline was strongly seasonal. Performance changed by data regime, which supports the report's central conclusion: MMM methods should be evaluated by the decisions they support and the conditions under which those decisions remain reliable.

What was compared

The public identifiers in this report describe research-series methods, not commercial software releases. Galileo 0 is the original production baseline used for this experiment. Galileo 0.2 is the subsequent research estimator. Internal development names are intentionally omitted from the public presentation but retained in the source manifests.

Public method	Role in the study
Galileo 0	Original deterministic production baseline: nonlinear Hill/adstock fitting with classical uncertainty.
Galileo 0.2	Bayesian Galileo research estimator with corrected steady-state response semantics.
Google Meridian	Google's Bayesian MMM, evaluated as an aggregate national model.
Meta Robyn	Meta's ridge/evolutionary MMM; executed as a separately frozen post-run extension.
PyMC-Marketing	Open-source Bayesian MMM included in the frozen four-model benchmark.

Experimental design

Nine controlled data-generating regimes were evaluated with ten deterministic seeds each. Every dataset contains known channel-response truth, so the economic value of both the model's allocation and the oracle allocation can be computed directly. The frozen benchmark required identical observations, channel definitions, train/holdout slices, budgets, constraints, and a common marginal allocator.

The primary metric is normalized decision regret. Zero is oracle-equivalent. One means the recommendation did no better than retaining the current allocation. Values above one mean the recommendation was worse than leaving the budget unchanged. Because regret is heavy-tailed, medians and full regime results are more informative than a single grand mean.

Overall decision results

Galileo 0.2

0.411

Galileo 0

23.349

Google Meridian

0.911

Meta Robyn

29.272

PyMC-Marketing

1.930

Median normalized regret across 90 datasets. Lower is better; the visual scale is capped at 30.

Method	N	Median	Bootstrap 95% CI	Mean	Best cells
Galileo 0.2	90	0.411	0.273–0.838	1.764	42/90
Galileo 0	90	23.349	12.307–37.098	28.734	3/90
Google Meridian	90	0.911	0.468–2.014	3.454	22/90
Meta Robyn	90	29.272	14.689–43.830	31.213	4/90
PyMC-Marketing	90	1.930	1.083–3.021	5.057	20/90

Bootstrap intervals use 10,000 deterministic resamples. “Best cells” counts exact lowest regret across the five methods; ties may count more than one method.

Results depend on the data regime

Regime	Galileo 0.2	Galileo 0	Google Meridian	Meta Robyn	PyMC-Marketing
D0 Clean, identifiable	9.137	75.376	9.134	79.857	5.948
D1 Strong collinearity	1.807	36.482	1.885	50.752	1.820
D2 Low outcome variation	0.536	43.339	2.306	42.414	3.503
D3 Organic-trend confounding	1.765	4.938	12.274	10.286	15.719
D4 Saturation choice	0.216	3.125	0.427	2.547	0.418
D5 Seasonal baseline	1.393	37.643	0.552	48.911	6.454
D6 Changing baseline	0.400	47.743	1.550	32.267	9.559
D7 Structural break	0.606	8.964	2.924	13.022	1.952
D8 Limited spend support	0.017	0.997	0.032	0.859	0.139

Mean normalized decision regret within each ten-seed regime. Lower is better. Means are shown here to preserve comparability with the frozen benchmark report; distributional results remain available in the paired data.

Galileo 0.2 was strongest in seven regimes. PyMC-Marketing led the clean regime, and Meridian led the seasonal-baseline regime. The original Galileo baseline and Robyn were especially vulnerable in the clean, collinear, low-variation, and seasonal settings under this allocation test.

Paired comparisons

Comparator	Galileo 0.2 wins	Median regret advantage	Mean advantage	Holm-adjusted p
Galileo 0	84/90	21.922	26.970	3.3e-15
Google Meridian	55/90	0.123	1.690	0.00035
Meta Robyn	85/90	18.938	29.448	2.4e-15
PyMC-Marketing	67/90	0.880	3.293	1.1e-06

One-sided paired Wilcoxon signed-rank tests evaluate whether comparator regret minus Galileo 0.2 regret is positive. P-values are Holm-adjusted across the four comparisons. Statistical significance on synthetic datasets is not evidence of real-customer lift.

What this demonstrates

Decision quality can distinguish models that historical-fit metrics alone may not separate.
The Galileo 0.2 response semantics materially improved allocation decisions relative to the original baseline.
Galileo 0.2 was competitive with established Bayesian MMM methods across the controlled regimes.
No estimator dominated every regime; seasonal structure remains an important source of model risk.

What this does not demonstrate

The experiment does not prove superior performance on real customer data.
It does not test Meridian's hierarchical multi-geo strengths.
Robyn was added in a separate post-run extension and was not part of the original four-model execution.
The transformations available through different frameworks are closely matched, not mathematically identical.
Bayesian posterior intervals and classical normal-approximation intervals do not have identical semantics.
The study evaluates aggregate allocation under a controlled oracle; it does not establish causal incrementality in an observational customer dataset.

Practical implication

A marketing model should not earn trust because it produces a plausible decomposition. It should earn trust by preserving decision value under known tests, identifying the regimes where its recommendations weaken, and refusing to turn unsupported precision into budget guidance.

Reproducibility

The review package includes the 90-row paired comparison file, aggregate statistics, regime results, and pairwise tests. The source benchmark retains dataset hashes, seeds, framework versions, failure rules, environment provenance, and cell-level model outputs. The Meta Robyn extension retains its own configuration and provenance because it was executed separately.

Source benchmark commit: 93f5125. Robyn extension commit: f39dd31.