A causal model ran clean, fit without error, and produced a plausible-looking number. It just wasn't controlling for the thing it claimed to control for. A field log of finding out why, written as it happened.
Phase 3's causal model is done: a two-way fixed-effects panel regression across four pathogen-antibiotic combinations, with antibiotic-consumption data added as a confounder, since heat and prescribing norms are plausibly correlated with each other. One combination, K. pneumoniae resistance to carbapenems, survives the mandatory placebo test. Standard next step before trusting a Phase 3 result: an independent pass tries to reproduce it from scratch.
Rebuilding the consumption-adjusted model independently, the confounder's own coefficient doesn't look right. The antibiotic-consumption term isn't behaving like a real regressor with real explanatory pull. Worth a closer look at exactly which column is going into the fit.
The confounder column was being selected like this:
[c for c in esac.columns if c not in ("country_code","year")][0]
The source file's columns are country_code, country_name, year, indicator, value. Filtering out country_code and year, then taking the first remaining column, doesn't land on value (the actual consumption figure). It lands on country_name, a text label.
The model was being fit on a country's own name, not its antibiotic use. And because country_name is identical for every year within a given country, the fixed-effects model's own machinery (drop_absorbed=True) silently folded it entirely into the country fixed effect it was already estimating. No crash. No warning. No NaN. The regression still ran, still produced a coefficient table, and every result built on it was captioned "consumption-adjusted" while controlling for nothing beyond what the country fixed effect was already doing.
Ran the buggy selection deliberately, to confirm this was really the mechanism, not a different bug that happened to produce a similarly odd-looking coefficient. It reproduced exactly: same buggy numbers as the original run, same silent absorption. Confirmed cause, not just a plausible theory of one.
Select the confounder column by name (value), not position. Reran all four pathogen-antibiotic combinations end to end, not just the one combination that had looked significant, since the same bug had silently touched every "consumption-adjusted" number in the draft.
Three of four combinations barely moved, as expected, since swapping a no-op regressor for a real but weak one shouldn't move much. The fourth, K. pneumoniae/carbapenems, moved from a pre-fix coefficient of −2.0061 (permutation p = 0.0131) to a corrected −1.7308 (permutation p = 0.0000). Different number. Same conclusion: still the one real, placebo-surviving result in the panel. The fix mattered enough to change the coefficient, not enough to overturn the finding.
Fixing the column meant antibiotic consumption was, for the first time, in the model. So: is it doing real confounding work? Checked its own coefficient directly: +0.9120, p = 0.1051. Not significant. The consumption confounder isn't meaningfully explaining anything in this fit, correctly wired in or not. The real reason this narrow 2010-2022 subsample shows a result the full 2000-2023 panel doesn't is most likely the narrower sample window itself, not the confounder. Reframed the write-up to say exactly that, instead of letting a fixed bug get credited with a discovery it didn't make.
The bug shipped in the first place because the placebo test was a manual step: someone runs the script, reads the console, trusts it. Built validate_pipeline.py to turn that into a gate: it refits every model, checks the permutation shuffle itself isn't degenerate, and flags drift against a stored baseline. Wired to GitHub Actions so it runs on every push, not only when someone remembers to run it by hand.
If you're joining external data into a fixed-effects or any absorbing-effects model, this failure mode is worth checking for directly. It doesn't announce itself.
Fixed-effects models can make a wrong column vanish cleanly: no crash, no NaN, a plausible-looking fit. Select confounder columns by name, never position, especially anywhere near drop_absorbed=True or an equivalent.
Confirming the buggy run reproduces exactly, on demand, is what separates "I found the cause" from "I found a plausible theory." Only fix after that confirmation.
After the fix, the confounder was finally in the model, and still not significant. The real explanation for the result was the sample window, not the confounder. Check what the newly-correct model is actually doing, not just whether the number moved.
The permutation test existed and was rigorous. It just wasn't automated yet. Turning it into a CI gate is what stops the same class of bug from shipping silently a second time.