Evaluation

hqnn_forge.evaluation

hqnn_forge.evaluation

Post-hoc evaluation helpers for binary classifiers on imbalanced data.

Exported symbols

find_optimal_threshold Decision threshold that maximises a metric on held-out probabilities. ThresholdSearchResult (threshold, score) pair returned by find_optimal_threshold. matthews_corrcoef MCC from labels, pure torch/NumPy. f1_score F1 for the positive class. balanced_accuracy Mean of recall over the two classes. pr_auc Average precision (PR-AUC), threshold-free; not in METRICS. parameter_efficiency Score per thousand trainable parameters. wilcoxon_signed_rank Paired signed-rank test with its attainable p-value floor. WilcoxonResult Result of wilcoxon_signed_rank. rank_biserial_correlation Effect size for the paired comparison. multiclass_matthews_corrcoef, macro_f1_score, multiclass_balanced_accuracy The multiclass metrics, on hard labels; MULTICLASS_METRICS by name. bootstrap_ci Class-stratified bootstrap interval (BCa or percentile) for a metric. paired_bootstrap_ci The same for the difference between two models on the same samples. BootstrapResult Result of both. friedman_test Friedman / Iman–Davenport test of k models over N datasets. friedman_from_ranks The same from published average ranks. FriedmanResult Result of friedman_test. average_ranks Mean rank of each model over the datasets (1 = best). nemenyi_critical_difference Nemenyi CD for all-pairs comparison of average ranks. compare_to_control Holm-corrected z-tests of every model against a control. ControlComparison One row of compare_to_control. holm_correction Holm step-down adjusted p-values for any family. brier_score, expected_calibration_error, reliability_curve Calibration of the probabilities (#319). TemperatureScaler, PlattScaler Post-hoc calibration fitted on a validation split. multiclass_brier_score, top_label_ece, classwise_ece, MulticlassTemperatureScaler The same for (n, K) probabilities and logits (#360). plots Submodule: confusion matrix, fold boxplot, efficiency frontier (needs matplotlib; import it explicitly: from hqnn_forge.evaluation import plots).

MULTICLASS_METRICS module-attribute

MULTICLASS_METRICS: Mapping[str, Callable[[Tensor, Tensor], float]] = {'mcc': multiclass_matthews_corrcoef, 'f1': macro_f1_score, 'balanced_accuracy': multiclass_balanced_accuracy}

The multiclass counterparts of hqnn_forge.evaluation.METRICS, under the same names, so monitor="mcc" means the same thing for both.

METRICS module-attribute

METRICS: Mapping[str, Metric] = {'mcc': matthews_corrcoef, 'f1': f1_score, 'balanced_accuracy': balanced_accuracy}

Metrics selectable by name in find_optimal_threshold.

BootstrapResult

Bases: NamedTuple

Attributes:

Name	Type	Description
estimate	float	The metric on the original samples (for the paired variant: model a minus model b).
low, high		The interval bounds.
confidence	float	Nominal coverage, e.g. 0.95.
method	str	"bca" or "percentile".
distribution	NDArray[float64]	The n_resamples bootstrap values.

estimate instance-attribute

estimate: float

low instance-attribute

low: float

high instance-attribute

high: float

confidence instance-attribute

confidence: float

method instance-attribute

method: str

distribution instance-attribute

distribution: NDArray[float64]

MulticlassTemperatureScaler dataclass

softmax(z / temperature) for (n, K) logits; see the module docstring.

temperature instance-attribute

temperature: float

fit classmethod

fit(logits: object, y_true: object) -> MulticlassTemperatureScaler

The temperature minimising the validation cross-entropy (L-BFGS on log T).

The cross-entropy is convex in s = 1/T with slope Σ_i (E_i[z] − z_iy), where z_iy is the true-class logit of sample i and E_i the mean of its logits under softmax(s·z_i). At s → 0 that mean is the plain row mean and at s → ∞ the row maximum, so a finite T exists only if Σ_i (z_iy − mean_k z_ik) > 0 (the logits rank the true class better than chance) and some z_iy < max_k z_ik (not every true class has the top logit). Otherwise the optimum is T → ∞ or T → 0 and fitting raises ValueError.

PlattScaler dataclass

σ(a·z + b); see the module docstring.

a instance-attribute

a: float

b instance-attribute

b: float

fit classmethod

fit(logits: object, y_true: object) -> PlattScaler

(a, b) minimising the validation NLL: logistic regression on the logit.

Its maximum-likelihood estimate is finite exactly when no threshold on z separates the classes, ties allowed (quasi-complete separation); otherwise |a| → ∞ and fitting raises ValueError.

TemperatureScaler dataclass

σ(z / temperature); see the module docstring.

temperature instance-attribute

temperature: float

fit classmethod

fit(logits: object, y_true: object) -> TemperatureScaler

The temperature minimising the validation NLL (L-BFGS on log T, so T > 0).

The NLL is convex in s = 1/T with slope −Σ m_i σ(−s·m_i), where m_i = (2y_i − 1)·z_i is the signed margin. At s → 0 the slope is −Σ m_i / 2 and at s → ∞ it is −Σ_{m_i<0} m_i, so a finite T exists only if Σ m_i > 0 (the logits rank better than chance) and some m_i < 0 (the classes are not separated at 0). Otherwise the optimum is T → ∞ or T → 0 and fitting raises ValueError.

ControlComparison

Bases: NamedTuple

One model against the control in compare_to_control.

Attributes:

Name	Type	Description
model	int	Column index of the model.
rank_difference	float	R_model - R_control; negative when the model ranks better.
z	float	rank_difference / sqrt(k(k+1) / (6N)).
p_value	float	Two-sided normal p-value of z.
p_adjusted	float	p_value after Holm's correction over the k - 1 comparisons.

model instance-attribute

model: int

rank_difference instance-attribute

rank_difference: float

z instance-attribute

z: float

p_value instance-attribute

p_value: float

p_adjusted instance-attribute

p_adjusted: float

FriedmanResult

Bases: NamedTuple

Result of friedman_test.

Attributes:

Name	Type	Description
statistic	float	Friedman's χ²_F on k - 1 degrees of freedom.
p_value	float	Upper tail of the chi-square distribution at statistic.
iman_davenport	float	Iman and Davenport's F_F = (N-1) χ²_F / (N(k-1) - χ²_F) on (k-1, (k-1)(N-1)) degrees of freedom; less conservative than χ²_F, and the variant Demšar recommends.
iman_davenport_p_value	float	Upper tail of the F distribution at iman_davenport.
average_ranks	NDArray[float64]	Mean rank of each model over the datasets, shape (k,); rank 1 is the best.
n_datasets, n_models		N and k.

statistic instance-attribute

statistic: float

p_value instance-attribute

p_value: float

iman_davenport instance-attribute

iman_davenport: float

iman_davenport_p_value instance-attribute

iman_davenport_p_value: float

average_ranks instance-attribute

average_ranks: NDArray[float64]

n_datasets instance-attribute

n_datasets: int

n_models instance-attribute

n_models: int

WilcoxonResult

Bases: NamedTuple

Result of wilcoxon_signed_rank.

Attributes:

Name	Type	Description
statistic	float	W+, the sum of the ranks of the positive differences a - b.
p_value	float	p-value for alternative.
n	int	Number of non-zero differences the test used.
min_p_value	float	Smallest p-value attainable for this n, rank pattern and alternative. If it exceeds your significance level, the test cannot reject whatever the data.
method	str	"exact" or "normal".

statistic instance-attribute

statistic: float

p_value instance-attribute

p_value: float

n instance-attribute

n: int

min_p_value instance-attribute

min_p_value: float

method instance-attribute

method: str

ThresholdSearchResult

Bases: NamedTuple

Best threshold found and the metric value it achieves.

threshold instance-attribute

threshold: float

score instance-attribute

score: float

bootstrap_ci

bootstrap_ci(y_true: ArrayLike, y_score: ArrayLike, metric: MetricArg = 'mcc', *, n_resamples: int = 2000, confidence: float = 0.95, method: Method = 'bca', rng: Generator | int | None = None) -> BootstrapResult

Class-stratified bootstrap interval for metric(y_true, y_score).

Parameters:

Name	Type	Description	Default
y_true	ArrayLike	0/1 labels; both classes must be present.	required
y_score	ArrayLike	Hard 0/1 predictions for the named metrics; whatever a callable metric takes otherwise (e.g. probabilities for PR-AUC).	required
metric	MetricArg	"mcc" (default), "f1", "balanced_accuracy", or a callable metric(y_true, y_score) -> float.	'mcc'
n_resamples	int	Bootstrap resamples. Default 2000.	2000
confidence	float	Nominal coverage. Default 0.95.	0.95
method	Method	"bca" (default) or "percentile"; see the module docstring.	'bca'
rng	Generator | int | None	A numpy.random.Generator or a seed. Only this generator is used, never a global one, so equal seeds give equal intervals.	None

Returns:

Type	Description
BootstrapResult

paired_bootstrap_ci

paired_bootstrap_ci(y_true: ArrayLike, score_a: ArrayLike, score_b: ArrayLike, metric: MetricArg = 'mcc', *, n_resamples: int = 2000, confidence: float = 0.95, method: Method = 'bca', rng: Generator | int | None = None) -> BootstrapResult

Interval for metric(a) - metric(b), two models scored on the same samples.

Both models are evaluated on the same resample each time, so what the samples have in common -- a hard case is hard for both -- cancels out of the difference. Two separate intervals that overlap do not show that the models are alike; an interval of the difference that excludes 0 does show that they differ. Parameters as for bootstrap_ci.

brier_score

brier_score(y_true: object, prob: object) -> float

mean((prob − y_true)²).

classwise_ece

classwise_ece(y_true: object, prob: object, n_bins: int = 10, strategy: BinStrategy = 'uniform') -> float

Mean over classes of the one-vs-rest ECE of each class's probability.

expected_calibration_error

expected_calibration_error(y_true: object, prob: object, n_bins: int = 10, strategy: BinStrategy = 'uniform') -> float

Σ_b (n_b / n) · |frequency_b − mean probability_b| over the bins.

multiclass_brier_score

multiclass_brier_score(y_true: object, prob: object) -> float

mean_i Σ_k (p_ik − onehot(y_i)_k)²; see the module docstring.

reliability_curve

reliability_curve(y_true: object, prob: object, n_bins: int = 10, strategy: BinStrategy = 'uniform') -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]

(mean probability, observed frequency, count) per non-empty bin.

With the default uniform bins this is sklearn.calibration.calibration_curve (which returns the frequency first and the mean probability second).

top_label_ece

top_label_ece(y_true: object, prob: object, n_bins: int = 10, strategy: BinStrategy = 'uniform') -> float

ECE of the top-class confidence against top-class correctness.

macro_f1_score

macro_f1_score(y_true: Tensor, y_pred: Tensor) -> float

Unweighted mean of per-class F1 over the classes seen in either argument.

multiclass_balanced_accuracy

multiclass_balanced_accuracy(y_true: Tensor, y_pred: Tensor) -> float

Mean recall over the classes present in y_true.

multiclass_matthews_corrcoef

multiclass_matthews_corrcoef(y_true: Tensor, y_pred: Tensor) -> float

Gorodkin's R_K; 0 when either marginal is constant (undefined).

average_ranks

average_ranks(scores: ArrayLike, *, higher_is_better: bool = True) -> npt.NDArray[np.float64]

Mean rank of each model (column) over the datasets (rows) of scores.

Within a dataset the best model gets rank 1; tied scores share their average rank, as in Demšar (2006).

compare_to_control

compare_to_control(scores: ArrayLike, control: int, *, higher_is_better: bool = True) -> list[ControlComparison]

Every model against control by the z-test on average ranks, with Holm's correction over the k - 1 comparisons (Demšar 2006, §3.2.2).

Parameters:

Name	Type	Description	Default
scores	ArrayLike	(n_datasets, n_models) matrix, as for friedman_test.	required
control	int	Column index of the control model, e.g. the classical baseline.	required

Returns:

Type	Description
list of ControlComparison	One per non-control model, in column order.

friedman_from_ranks

friedman_from_ranks(average_ranks: ArrayLike, n_datasets: int) -> FriedmanResult

friedman_test from the models' average ranks alone, as published tables usually report them.

Parameters:

Name	Type	Description	Default
average_ranks	ArrayLike	Mean rank of each of the k models; they must sum to k(k+1)/2, up to rounding to two decimals (0.005 k).	required
n_datasets	int	Number of datasets N the ranks were averaged over.	required

friedman_test

friedman_test(scores: ArrayLike, *, higher_is_better: bool = True) -> FriedmanResult

Friedman test that k models perform alike over N datasets.

Each dataset ranks the models (rank 1 best, ties averaged); under the null hypothesis every model's average rank is (k+1)/2. No tie correction is applied to χ²_F, as in Demšar (2006); with ties that makes the test slightly conservative.

Parameters:

Name	Type	Description	Default
scores	ArrayLike	(n_datasets, n_models) matrix, one score per model and dataset, e.g. the mean test MCC of each model on each dataset.	required
higher_is_better	bool	False for losses and error rates.	True

Returns:

Type	Description
FriedmanResult	Report iman_davenport_p_value; if it rejects, follow with nemenyi_critical_difference or compare_to_control.

Raises:

Type	Description
ValueError	If scores is not a finite matrix with at least 2 rows and columns.

holm_correction

holm_correction(p_values: ArrayLike) -> npt.NDArray[np.float64]

Holm's step-down adjusted p-values, in the order given.

The i-th smallest of m p-values is multiplied by m - i + 1, made non-decreasing along that order and capped at 1. Rejecting every hypothesis whose adjusted p-value is at most α controls the family-wise error rate at α -- usable on any family, e.g. the per-dataset Wilcoxon p-values of one comparison.

nemenyi_critical_difference

nemenyi_critical_difference(n_models: int, n_datasets: int, alpha: float = 0.05) -> float

Nemenyi critical difference CD = q_α sqrt(k(k+1) / (6N)).

Two models whose average ranks differ by at least CD differ significantly at family-wise level alpha over all k(k-1)/2 pairs. q_α is the upper-α point of the studentised range of k normals at infinite degrees of freedom, divided by √2, computed here by numerical integration (Demšar 2006, Table 5a, tabulates it for k ≤ 10).

rank_biserial_correlation

rank_biserial_correlation(scores_a: ArrayLike, scores_b: ArrayLike) -> float

Matched-pairs rank-biserial correlation, the effect size for the signed-rank test.

r = (W+ − W−) / (W+ + W−) over the non-zero differences (Kerby 2014). It lies in [-1, 1]: +1 when a beats b on every fold, -1 when it loses on every fold, 0 when wins and losses balance by rank. If every difference is zero the models are indistinguishable and 0.0 is returned.

Parameters:

Name	Type	Description	Default
scores_a	ArrayLike	Paired metric values, one per fold, same length.	required
scores_b	ArrayLike	Paired metric values, one per fold, same length.	required

wilcoxon_signed_rank

wilcoxon_signed_rank(scores_a: ArrayLike, scores_b: ArrayLike, alternative: Alternative = 'two-sided') -> WilcoxonResult

Wilcoxon signed-rank test on paired per-fold scores.

Zero differences are dropped before ranking (Wilcoxon's original treatment). Tied absolute differences get average ranks.

Parameters:

Name	Type	Description	Default
scores_a	ArrayLike	Paired metric values, one per fold, same length.	required
scores_b	ArrayLike	Paired metric values, one per fold, same length.	required
alternative	Alternative	"two-sided" (default); "greater" tests whether a tends to exceed b; "less" the reverse.	'two-sided'

Returns:

Type	Description
WilcoxonResult

Raises:

Type	Description
ValueError	If the inputs differ in length, are empty or non-finite, or if every difference is zero (the test is undefined; the rank-biserial correlation is 0 in that case).

Examples:

>>> res = wilcoxon_signed_rank([0.58, 0.61, 0.55, 0.60, 0.57],
...                            [0.56, 0.57, 0.54, 0.55, 0.56])
>>> res.statistic, res.p_value, res.min_p_value
(15.0, 0.0625, 0.0625)

balanced_accuracy

balanced_accuracy(y_true: object, y_pred: object) -> float

Mean of the recall on each class; 0.5 is chance.

Returns 0.0 when a class is absent from y_true and the mean is therefore taken over one class only. scikit-learn returns that one class's recall instead (1.0 for a perfect labelling of a single-class split); 0.0 is used here so that find_optimal_threshold cannot prefer such a labelling.

f1_score

f1_score(y_true: object, y_pred: object) -> float

F1 of the positive class; 0.0 when there are no positives in either vector.

find_optimal_threshold

find_optimal_threshold(y_true: object, y_prob: object, metric: str | Metric = 'mcc') -> ThresholdSearchResult

Threshold on y_prob that maximises metric against y_true.

Every distinct labelling the probabilities can produce is scored: with a sample positive when prob >= threshold, all thresholds between two adjacent unique probabilities agree, so the candidates are those intervals plus the all-negative one above the largest probability. That is exhaustive, so no grid resolution has to be chosen.

The threshold returned is the midpoint of the winning interval rather than the probability at its upper end, so it sits as far as possible from the nearest validation sample: perfectly separable data with a gap between 0.08 and 0.92 returns 0.5, not 0.92. Ties between intervals are broken towards the candidate whose midpoint is closest to 0.5.

Named metrics are scored from cumulative confusion counts, so the whole search costs one sort. A callable metric is invoked once per candidate instead, since it is only defined on hard labels.

Parameters:

Name	Type	Description	Default
y_true	object	Binary labels, shape (n,).	required
y_prob	object	Positive-class probabilities, shape (n,), e.g. from model.predict_proba(X_val).	required
metric	str | Metric	"mcc" (default), "f1", "balanced_accuracy", or any callable (y_true, y_pred) -> float. A callable is always passed hard 0/1 predictions, (y_prob >= threshold).long(), never the probabilities. Threshold-free metrics such as PR-AUC or ROC-AUC therefore do not belong here: given hard labels they score one degenerate operating point, not the ranking. pr_auc is refused outright; other such scorers are not detected.	'mcc'

Returns:

Type	Description
ThresholdSearchResult	(threshold, score). The threshold may exceed 1.0 when labelling every sample negative is optimal and some probability is exactly 1.0.

Raises:

Type	Description
ValueError	If metric is an unknown name or is pr_auc (by name or as the function), if the inputs are empty or differ in length, if y_true is not binary, or if y_prob contains NaN or lies outside [0, 1].

Examples:

>>> import torch
>>> y = torch.tensor([0, 0, 0, 1, 1])
>>> p = torch.tensor([0.1, 0.2, 0.4, 0.45, 0.9])
>>> result = find_optimal_threshold(y, p)
>>> round(result.threshold, 3), result.score
(0.425, 1.0)

matthews_corrcoef

matthews_corrcoef(y_true: object, y_pred: object) -> float

Matthews correlation coefficient in [-1, 1].

(tp·tn − fp·fn) / sqrt((tp+fp)(tp+fn)(tn+fp)(tn+fn)). Returns 0.0 when any factor in the denominator is zero (only one class present in y_true or in y_pred), as scikit-learn does.

parameter_efficiency

parameter_efficiency(model: Module | int, score: float) -> float

score per thousand trainable parameters, e.g. MCC/kParam.

A model's count includes every trainable parameter, also those that can never move the output (circuit_summary(...).n_inert_params; 20 of the published SHNN's 122). That is the convention of the published results. To state efficiency over the live parameters instead, pass that count as an integer, and say which one was used.

Parameters:

Name	Type	Description	Default
model	Module | int	A module exposing count_parameters() (the hybrid classifiers), any nn.Module (trainable parameters are counted directly), or an integer parameter count.	required
score	float	The metric value achieved by that model.	required

Raises:

Type	Description
ValueError	If the parameter count is not positive.

pr_auc

pr_auc(y_true: object, y_prob: object) -> float

Area under the precision-recall curve, as average precision.

With the operating points taken in order of decreasing threshold,

AP = Σ_k (R_k − R_{k−1}) · P_k

the step interpolation of sklearn.metrics.average_precision_score: each gain in recall is credited at the precision of the operating point that achieves it. It is not the trapezoidal auc(recall, precision), which interpolates linearly between operating points and so credits precision no threshold actually attains.

Samples that share a probability form one operating point: a threshold cannot separate them, so the result does not depend on the order in which tied samples arrive.

PR-AUC is threshold-free, the usual companion of MCC on imbalanced data; it is not in METRICS, and find_optimal_threshold refuses it, because no threshold maximises it. It depends only on the ranking, so any score monotone in the positive-class probability (a logit, say) gives the same value.

Parameters:

Name	Type	Description	Default
y_true	object	Binary labels, shape (n,).	required
y_prob	object	Positive-class scores, shape (n,), e.g. from model.predict_proba(X_val).	required

Returns:

Type	Description
float	In [0, 1]. 0.0 when y_true has no positives, where recall is undefined; scikit-learn returns the same value, with a warning.

Raises:

Type	Description
ValueError	If the inputs are empty or differ in length, if y_true is not binary, or if y_prob contains NaN or ±inf.

Examples:

>>> round(pr_auc([0, 1, 1, 0, 1], [0.1, 0.4, 0.35, 0.8, 0.9]), 4)  # 1/3 + 2/9 + 1/4
0.8056