Coverage-bias laboratory

Treating missing imagery as a missing-data problem

The project’s actual scientific contribution isn’t the vision model — it’s applying a rigorous missing-data framework to a question usually handled by reporting whatever imagery happens to exist.

Rubin’s missingness taxonomy

Rubin (1976) gives the canonical three-way split for why data might be missing. Which bucket coverage falls into determines whether a simple average of what’s photographed is a trustworthy city-level estimate.

MCAR

Missing Completely at Random

Whether a road segment has usable imagery is unrelated to anything, observed or not.

Very unlikely here. Contributors don't drive random roads with a camera mounted.

MAR

Missing at Random (conditional on observables)

Coverage depends on things we can measure and put in a model: road class, population density, nightlights, distance to city center.

The working assumption for the propensity model below — testable, not assumed to be true.

Systematic / MNAR

Missing Not at Random

Coverage depends on something unobserved that also affects the infrastructure attribute itself — e.g. informal settlements are both under-photographed and structurally different.

The scenario a reweighting correction cannot fully fix, and the masking experiment below is designed to probe for.

Propensity modeling

For every sampled road point, a propensity model estimates the probability that usable street-level imagery exists there, as a function of observable covariates: OSM road class, WorldPop population density, GHSL built-up/degree-of-urbanisation, VIIRS nightlight radiance, and distance from the city center. Fitted propensities feed two corrections downstream (see /infrastructure):

Inverse-probability weighting (IPW) — weight each covered point by 1 / propensity so under-photographed strata count more.
Post-stratification — bucket road points by covariate strata and reweight stratum means to match the full road-network universe, following Little (1993).

Both are standard survey-methodology tools (see Seaman & White, 2013 for the IPW misspecification caveat) applied here to a domain — crowdsourced street imagery coverage — where, per the literature review, no prior published work seems to have applied them.

The masking experiment

To test whether the correction actually works rather than just looking plausible, the design artificially masks additional imagery in a city with genuinely good coverage — using a known, controlled missingness mechanism — and checks whether the reweighted estimate recovers the true (fully-covered) value more closely than the naive one. A correction that only helps under the mechanism it was designed for, and fails under a systematic (MNAR-like) masking pattern, is itself a finding worth reporting, not a failure to hide.

Masking-experiment results

This run is real and complete — run it yourself with make bias-study. What’s real: the road network (9 cities, OpenStreetMap) and the covariates at every point (WorldPop population density, VIIRS nightlights, GHSL built-up surface). What’s simulated and labeled as such in the output file: the outcome (“infrastructure quality”) attached to each point, since no real infrastructure label exists yet, and the “fully covered” starting assumption, since real Mapillary coverage hasn’t been measured. The question this answers is narrower but real: if coverage is missing the way the MAR-like mask assumes, do the correction methods actually recover the true mean?

MCAR mask (coverage unrelated to covariates)

coverage 35.0% · 361,891 of 1,034,146 points · true mean 51.08

Method	Estimate	% error vs. true
Naive (covered points only)	51.06	0.05%
IPW — logistic propensity	51.12	0.06%
IPW — gradient-boosted propensity	51.27	0.36%
Post-stratification	51.06	0.05%

MAR-like mask (coverage tied to density + road class)

coverage 35.0% · 362,247 of 1,034,146 points · true mean 51.08

Method	Estimate	% error vs. true
Naive (covered points only)	61.75	20.89%
IPW — logistic propensity	55.29	8.24%
IPW — gradient-boosted propensity	52.25	2.29%
Post-stratification	56.59	10.78%

Source: data/results/masking_experiment_results.json, generated by src/bias/masking_experiment.py. Full design notes in docs/masking_experiment_design.md.

Propensity model on real coverage

Not yet measured

src/bias/propensity_model.py is written and tested, but fitting it to actual imagery coverage (rather than the masking experiment’s simulated missingness) requires the Mapillary coverage measurement in /coverage, which does not exist yet.

see docs/data_sources.md