Rishav Bhattarai
Debug log · PCVI Atlas

The Winchester Artifact

A small Virginia city topped a national health-risk ranking by a factor of eight. It wasn't the sickest place in America. It was a denominator in the wrong place. A field log of finding out why, written as it happened.

2,553 counties 8.79M claim rows 306 hospital referral regions 1 bug, caught before it shipped
Context

National scale-up run just finished: 8.79M Medicare Part D claims, matched against a 276-drug heat-sensitizing crosswalk, aggregated to county level, normalized per 1,000 residents 65+. Sanity-checking the top of the per-capita ranking before trusting it for anything.

First look at the ranking

Top of the list is dominated by tiny counties (Yukon-Koyukuk Census Area, AK (pop. 65+ = 913), Greeley County, KS (pop. 65+ = 232), sitting at >35,000 claims per 1,000 residents 65+, several multiples above the p99 (24,055). Small-N per-capita noise, the kind you expect and can filter with a population floor.

Applied a population floor

POP_65PLUS_FLOOR = 1000. Flagged rather than dropped (198/2,553 counties, 7.8%, fall below it, kept auditable in the output). Correlation moves from r = 0.1311 to r = 0.1405: it modestly strengthens, consistent with the excluded counties being noise, not signal. Good sign the floor is doing real work.

⚑ Checked the ranking again, post-floor

Still wrong at the top. Winchester city, VA leads at 56,438 claims per 1,000 residents 65+, with 4,874 residents 65+, more than 4x the floor threshold. This isn't a small-N artifact. The floor caught the extreme small counties; it can't distinguish a legitimately high-burden mid-sized county from something else entirely.

Stopped to look, instead of raising the floor again

Winchester City is a small independent city. Virginia's independent-city structure means it isn't part of any surrounding county, and it has a regional medical center serving a much larger catchment area than its own population. That's the "something else": patients from outside Winchester are being treated and prescribed there, and every one of those claims is attributed to Winchester's own tiny population, because the pipeline joins on prescriber ZIP → prescriber's own county, with no notion of where the patient lives.

⚑ Named the bug

Prescriber location ≠ patient location. Any regional hospital or pharmacy hub inflates its own home county's per-capita rate by construction: the numerator (claims) reflects a wide catchment area, the denominator (population) reflects only the hub's own small county. This is exactly the risk the project's original scoping document had flagged, and recommended a Hospital Referral Region sensitivity check for. It just hadn't bitten yet, so it hadn't been forced to the front of the queue until this ranking made it undeniable.

The fix: re-aggregate to Hospital Referral Region

HRRs (Dartmouth Atlas, 306 nationally) are built around actual hospital catchment and referral patterns, the standard health-services-research geography for exactly this mismatch. Two things needed reattributing:

Claims: prescriber ZIP → HRR directly (Dartmouth's ZipHsaHrr19 crosswalk, free, no login), bypassing county entirely.
Population: no ZCTA-level population file on hand, so each county is assigned to its plurality HRR (the HRR most of its ZCTAs map to) and county populations are summed per HRR, a minor approximation, since most counties don't actually split across HRR boundaries.

Result, checked directly

Winchester's rate dropped from 56,438 claims/1k-65+ (its own 4,874-person county-level denominator) to 6,535.5 (HRR-level, 105,269 people across its real 14-county catchment), right at the national median. The artifact is gone specifically because the denominator now matches the population the claims came from, not because of any smoothing or exclusion.

Didn't stop there: checked whether the fix broke something new

At HRR level, unfiltered, the correlation initially looked weaker than the county-level number (r ≈ 0.08 vs 0.13-0.14). That's informative on its own: part of the county-level "signal" may have been the same misattribution artifact in disguise: hot Sunbelt regional-hub counties being both genuinely hot and spuriously claims-inflated, correlating with heat for the wrong reason. But the unfiltered HRR number turned out to have its own small-N problem: some HRRs are still single-county with modest populations. New Haven, CT (pop. 65+ = 20,019) topped the unfiltered HRR ranking at ~117,800/1k-65+.

Before excluding New Haven, checked it wasn't the same bug again

Verified the individual prescribers behind the number were real, distinct physicians, not a mail-order pharmacy or PBM artifact routing claims through one address. It wasn't the attribution bug; it was a genuinely small population getting swung by a handful of high-volume prescribers. A different problem (small-N variance) needing a different fix: a population floor, this time at HRR level.

Final numbers, both fixes together

An HRR-level floor of ≥40,000 (≈p10 of the 293-HRR distribution) excluded 25/261 HRRs, flagged not dropped.

Pearson r, burden vs. summer heat exposure
Geographymean-HI rdays ≥105°F rnp
County-level, unfiltered0.13110.12402,553n/a
County-level, pop floor0.14050.13652,355n/a
HRR-level, unfiltered0.08410.07422610.176 (n.s.)
HRR-level, pop floor0.32470.3001236<0.0001

Once both the geography and the small-N problem are fixed, a moderate, statistically significant correlation shows up (r ≈ 0.32) that neither the raw county numbers nor the unfiltered HRR numbers showed cleanly. It also looks qualitatively sane in a way the earlier rankings didn't: hot Sunbelt HRRs (Hudson FL, Mesa AZ, Monroe LA, Metairie LA, Rome GA) at the top, cooler coastal and temperate HRRs (San Jose, Santa Rosa, Everett WA, Chicago) at the bottom. It's the first ranking in this project that looks like a real geographic pattern instead of a data-quality artifact.

The lesson, generalized

If you're working with prescriber-level, hospital-level, or any facility-anchored geographic health data, and aggregating up to a "where does this population live" geography, this failure mode will eventually find you too.

A population floor catches small-N noise. It does not catch catchment-area mismatch.

Those are different failure modes and need different fixes. Don't assume one floor fixes both just because both show up as extreme values at the top of a ranking.

When a fix changes the correlation you're chasing, check why before trusting the new number.

The HRR fix weakened the correlation before it strengthened it (unfiltered HRR: 0.08). The instinct to read that as "the fix isn't working" would have been wrong. It was revealing that part of the original signal was itself an artifact.

A population floor is a screen for hub effects, not a fix.

It only catches the extreme small end. Winchester (pop. 65+ = 4,874, more than 4x a floor of 1,000) sailed straight through a population floor while still being a pure catchment-area artifact. The actual fix had to match the geography to how the data is really generated, not just filter on population size.

When something looks like the same bug reappearing, verify that before applying the same fix.

New Haven, after the HRR fix, looked like Winchester all over again. It wasn't. Checking the underlying prescribers first avoided over-filtering a genuinely small but real signal.