Sensitivity Analysis for Site Ranking Weights

This page solves one task: measuring how much a site ranking changes when the criteria weights change, so you can tell a real-estate committee whether the shortlist is a robust conclusion or an artifact of one analyst’s weighting guess.

Every composite suitability score depends on weights — how much population matters versus income, competition, or rent. Those weights are judgment calls, and a ranking that reshuffles violently when a weight moves a few points is indefensible. Sensitivity analysis quantifies that fragility. It is the validation step that should run before any output from ranking and shortlisting candidate sites is presented as final, and it directly tests the weighting scheme built in building weighted site suitability scores.

What we measure #

The score for candidate $i$ is a weighted sum of normalized criteria $x_{ic}$ over criteria $c$ with weights $w_c$ (where $\sum_c w_c = 1$):

$$S_i(\mathbf{w}) = \sum_{c} w_c \, x_{ic}$$

The ranking is the ordering of the $S_i$. Sensitivity analysis perturbs $\mathbf{w}$ and asks how much the ordering moves. The natural metric for “how much did an ordering move” is Spearman’s rank correlation between the baseline ranking and the perturbed one:

$$\rho_s = 1 - \frac{6 \sum_i d_i^2}{n(n^2 - 1)}$$

where $d_i$ is the difference between a candidate’s baseline rank and its perturbed rank, and $n$ is the number of candidates. A $\rho_s$ near 1.0 means the ranking barely moved; a low or negative $\rho_s$ means the weights are driving the conclusion, not the data. We run two perturbation schemes: one-at-a-time (OAT), which nudges a single weight to attribute fragility to a specific criterion, and Monte Carlo over the weight simplex, which samples the full space of plausible weightings to get a distribution of $\rho_s$ and to flag individual sites whose rank is unstable.

The two schemes answer different questions and are complementary rather than redundant. OAT is diagnostic: by moving one weight while holding the rest fixed, it attributes any instability to a specific criterion, which tells the analyst which judgment call the conclusion actually hinges on. Its weakness is that it explores only the axes of the weight space, never the interior, so it can miss instability that appears only when several weights move together. Monte Carlo fills that gap by sampling the whole simplex — every point where the weights are non-negative and sum to one — producing an honest distribution of outcomes rather than a handful of one-dimensional slices. The constraint $\sum_c w_c = 1$ is what makes the weight space a simplex rather than a box, and sampling it correctly is why the implementation draws from a Dirichlet distribution rather than perturbing weights independently and hoping they still sum to one.

A single global Spearman statistic can hide a locally fragile shortlist: the overall ordering may be stable while two specific sites near the top-N boundary swap places under almost any reweighting. That is why the analysis tracks per-candidate rank volatility in addition to the aggregate correlation — a rank reversal inside the shortlist changes which sites get funded, whereas a reversal among the tail changes nothing anyone acts on.

Prerequisites #

• Python packages: numpy, pandas, and scipy (for scipy.stats.spearmanr and the Dirichlet sampler). Install with pip install numpy pandas scipy.
• A criteria matrix X of shape (n_candidates, n_criteria) with each column already normalized to a common scale — the job of normalizing mixed-scale site attributes for scoring. Sensitivity analysis on un-normalized criteria measures scale artifacts, not weight sensitivity.
• A baseline weight vector w0 summing to 1.
• A perturbation budget: the OAT step size (e.g. ±0.10) and the Monte Carlo sample count (a few thousand draws).

Configuration and execution parameters #

The Dirichlet concentration encodes how much weight uncertainty you actually believe in: a large value samples tightly around the baseline (you trust the weights), a small value explores the whole simplex (you do not). Sampling the full simplex uniformly is the conservative, stress-test choice.

Annotated implementation #

The routine computes the baseline ranking, runs OAT perturbations per criterion, then a Monte Carlo sweep, returning the Spearman distribution and per-site rank stability.

import numpy as np
import pandas as pd
from scipy.stats import spearmanr


def rank_from_weights(X: np.ndarray, w: np.ndarray) -> np.ndarray:
    """Return ordinal ranks (1 = best) for scores S = X @ w."""
    scores = X @ w
    order = np.argsort(-scores, kind="mergesort")   # stable, best first
    ranks = np.empty(len(scores), dtype=int)
    ranks[order] = np.arange(1, len(scores) + 1)
    return ranks


def oat_sensitivity(X: np.ndarray, w0: np.ndarray,
                    delta: float = 0.10) -> pd.DataFrame:
    """Nudge each weight by +/-delta, renormalize, measure Spearman vs baseline."""
    base = rank_from_weights(X, w0)
    rows = []
    for c in range(len(w0)):
        for sign in (+1, -1):
            w = w0.copy()
            w[c] = max(w[c] + sign * delta, 0.0)
            w = w / w.sum()                          # keep the simplex constraint
            rho, _ = spearmanr(base, rank_from_weights(X, w))
            rows.append({"criterion": c, "sign": sign, "spearman": rho})
    return pd.DataFrame(rows)


def monte_carlo_sensitivity(X: np.ndarray, w0: np.ndarray,
                            n_samples: int = 5000,
                            concentration: float = 20.0,
                            seed: int = 42) -> dict:
    """Sample weights ~ Dirichlet(concentration * w0); track rank stability."""
    rng = np.random.default_rng(seed)
    base = rank_from_weights(X, w0)
    n_cand = X.shape[0]
    alpha = concentration * w0                        # center mass on baseline
    rank_matrix = np.empty((n_samples, n_cand), dtype=int)
    spearmans = np.empty(n_samples)

    for s in range(n_samples):
        w = rng.dirichlet(alpha)                      # sums to 1 by construction
        r = rank_from_weights(X, w)
        rank_matrix[s] = r
        spearmans[s], _ = spearmanr(base, r)

    return {
        "baseline_rank": base,
        "spearman": spearmans,                        # distribution over draws
        "rank_std": rank_matrix.std(axis=0),          # per-candidate volatility
        "rank_mean": rank_matrix.mean(axis=0),
    }

Rank-reversal detection then flags the sites that actually matter — pairs that swap order within the shortlisted top-N, where a reversal changes a real decision rather than shuffling also-rans:

def top_n_reversals(base: np.ndarray, mc: dict, top_n: int = 15,
                    unstable_std: float = 2.0) -> pd.DataFrame:
    """Flag shortlisted sites whose rank is volatile across the weight sweep."""
    in_shortlist = base <= top_n
    flags = pd.DataFrame({
        "candidate": np.arange(len(base)),
        "baseline_rank": base,
        "rank_mean": mc["rank_mean"].round(1),
        "rank_std": mc["rank_std"].round(2),
        "in_shortlist": in_shortlist,
    })
    flags["unstable"] = in_shortlist & (mc["rank_std"] > unstable_std)
    return flags.sort_values("baseline_rank")

Failure modes and debugging #

The most common misread is treating a high Spearman as unambiguous good news. If two criteria are nearly collinear, reallocating weight between them changes little — the ranking is robust for the wrong reason. Inspect the criteria correlation structure before concluding the ranking is sound.

Verification #

1. Baseline recovery: with zero perturbation, rank_from_weights(X, w0) reproduces the ranking the scoring stage emitted.
2. Spearman distribution: summarize the Monte Carlo spearman array — a median above ~0.9 with a tight spread indicates a defensible ranking.
3. Instability flags: confirm unstable sites are genuinely volatile by inspecting their rank_std.

mc = monte_carlo_sensitivity(X, w0, n_samples=5000)
print(f"Spearman median: {np.median(mc['spearman']):.3f}")
print(f"Spearman 5th pct: {np.percentile(mc['spearman'], 5):.3f}")
flags = top_n_reversals(mc["baseline_rank"], mc, top_n=15)
print(flags[flags["unstable"]])

Report the 5th percentile of the Spearman distribution, not just the median: the committee needs the worst plausible case, not the average one. A ranking whose median $\rho_s$ is 0.95 but whose 5th percentile is 0.6 has a tail of weightings that reshuffle the shortlist, and any site flagged unstable should carry that caveat into the final report rather than being presented as a confident recommendation.

Related #

• Ranking & Shortlisting Candidate Sites — the ranking this analysis stress-tests before it is finalized.
• Building Weighted Site Suitability Scores — defines the weights whose robustness is measured here.
• Normalizing Mixed-Scale Site Attributes for Scoring — prerequisite normalization so sensitivity reflects weights, not scales.

← Back to Ranking & Shortlisting Candidate Sites