Cross-city evaluation
Random split flatters models. Spatial structure doesn't lie.
Ploton et al. (2020, Nature Communications) showed that non-spatial validation on remote-sensing data can report >50% variance explained while spatially-blocked validation reveals near-zero true predictive power. This project evaluates every model three ways specifically to surface that gap rather than report only the flattering number.
Random split
Images shuffled and split without regard to location. The easiest split to report and the least trustworthy one here, because nearby street-view frames along the same road are highly correlated.
Inflates performance via spatial autocorrelation leakage — adjacent frames of the same block leak into both train and test.
Spatial split
Train and test sets are separated by a geographic buffer within the same cities, so no test frame sits near a training frame.
Removes fine-grained spatial leakage but still lets the model learn city-specific visual style (a particular city's building stock, road markings, sky).
City holdout
Entire cities are held out of training and evaluated only at test time — the strictest test of whether the model generalizes to a city with different architecture, road quality, and imagery style.
The most honest measure of real-world transfer, and the one most likely to show a meaningful accuracy drop relative to random split.
Why this matters for coverage bias
A model that only works via spatial leakage is also a model whose apparent accuracy will vary with how imagery happens to be distributed — precisely the confound this project is built to separate out. City holdout performance is the number that will actually inform whether reweighted infrastructure estimates (see /infrastructure) are trustworthy for a city with little or no labeled training data of its own.
Results
Not yet measured
No model has been trained yet, so there are no random-split, spatial-split, or city-holdout accuracy numbers to compare. This page describes the evaluation design, not results.
see docs/literature_review.md