Benchmark

hqnn_forge.benchmark

hqnn_forge.benchmark

Train a hybrid model and its matched classical control on identical folds of several datasets, and report one row per dataset and model.

This answers the library's question -- does a small quantum layer earn its parameters? -- the same way every time, instead of by a comparison assembled by hand.

What both models share

For each dataset, the outer folds come from stratified_kfold, and in every fold the hybrid model and its control (classical_baseline, matched to the hybrid's live parameter count) get exactly the same:

• Scaling. Features are standardised with the mean and standard deviation of the fold's training part only, then applied to every row of the fold.
• Validation split. The training part is split once more, stratified: one validation_folds-th is held out for early stopping and threshold tuning. It holds only real rows.
• Oversampling. With oversample=True, SMOTE runs on the remaining training rows only, after scaling, since its neighbour search measures distances. Test and validation rows never contribute a synthetic sample.
• Training. The same loss, optimiser, learning rate, batch size, epoch budget and early stopping, the same batch order (one generator seed per fold), and the same initialisation seed. Each model is built and trained inside torch.random.fork_rng, so the run neither depends on nor disturbs the caller's global RNG.
• Threshold. The threshold that maximises MCC on the validation split at the best epoch (TrainingHistory.best_threshold), applied unchanged to the fold's test rows. The test rows are used for nothing else.

Reported per dataset and model

mcc_mean/mcc_std over the test folds (with n_seeds > 1, of each fold's mean over its initialisation seeds, and mcc_seed_std the mean across-seed standard deviation), n_parameters (the total trainable count, count_parameters(), including circuit weights that can never move the output) and MCC per 1,000 of them (parameter_efficiency of the mean). The control is matched to the hybrid's live count, so the two rows' n_parameters differ by the hybrid's inert weights as well as by the width rounding: for the published SHNN, 122 (102 live) against 101 (#234). Then the wall-clock training time summed over folds (simulating the circuit is part of an honest efficiency comparison), and the paired Wilcoxon signed-rank test of hybrid against control over the per-fold MCCs, with its effect size. The test columns are the same on both rows of a dataset.

wilcoxon_min_p is the smallest p-value the test could have produced with this many folds: with 5 folds, two-sided, it is 0.0625, so no outcome reaches 0.05. A difference then needs more folds (n_splits) to be declared, not a lower threshold.

wilcoxon_p_holm adjusts the per-dataset p-values as one family (one hypothesis per dataset, despite the duplicate model rows). Undefined tests remain NaN and keep an unrejectable slot in the family.

Across datasets

With more than one dataset, BenchmarkResult.cross_dataset reports a two-sided Wilcoxon test over the per-dataset mean MCCs, rank-biserial effect (positive favours the hybrid), and smallest attainable p-value. Each dataset contributes one pair, regardless of its size or the number of folds/seeds. Five non-zero pairs cannot reach 0.05, even when every dataset favours the same model. The comparison is included in saved experiment records.

The runner compares exactly two models. For three or more models, use friedman_test (including Iman--Davenport), followed by nemenyi_critical_difference for all pairs or compare_to_control with Holm-adjusted p-values against one control (Demšar 2006).

References

• Demšar (2006), "Statistical Comparisons of Classifiers over Multiple Data Sets", Journal of Machine Learning Research 7, 1--30.

HybridBuilder module-attribute

HybridBuilder = Callable[[int], nn.Module]

LossBuilder module-attribute

LossBuilder = Callable[[], nn.Module]

MODELS module-attribute

MODELS: tuple[str, str] = ('hybrid', 'control')

The two models of every dataset, in report order.

COLUMNS module-attribute

COLUMNS: tuple[str, ...] = ('dataset', 'model', 'architecture', 'n_samples', 'n_positives', 'n_folds', 'n_parameters', 'mcc_mean', 'mcc_std', 'n_seeds', 'mcc_seed_std', 'mcc_per_kparam', 'brier_mean', 'ece_mean', 'fold_mcc', 'train_seconds', 'wilcoxon_p', 'wilcoxon_p_holm', 'wilcoxon_min_p', 'rank_biserial')

Columns of BenchmarkResult.records and of the CSV, in order.

ECE_BINS module-attribute

ECE_BINS = 10

Bins of the per-fold expected calibration error. Equal-count ("quantile") bins: on imbalanced data equal-width ones put nearly every sample in the bin nearest 0 and leave the rest near-empty.

TUNABLE module-attribute

TUNABLE: frozenset[str] = frozenset({'lr', 'batch_size', 'max_epochs', 'patience'})

Training settings a search space may vary. The architecture is not tunable: the control's size is matched to the hybrid's, and tuning one model's architecture would break that match.

FoldResult dataclass

One model on one outer fold. The index arrays point into the dataset's rows, so the two models of a fold can be checked to have seen the same data.

dataset instance-attribute

dataset: str

model instance-attribute

model: str

fold instance-attribute

fold: int

train_idx instance-attribute

train_idx: NDArray[intp]

Rows the model was trained on (before SMOTE added synthetic ones).

val_idx instance-attribute

val_idx: NDArray[intp]

Rows used for early stopping and threshold tuning.

test_idx instance-attribute

test_idx: NDArray[intp]

Rows the reported score comes from.

n_synthetic instance-attribute

n_synthetic: int

split_seed instance-attribute

split_seed: int

Seed of the dataset's outer stratified split (the same for every fold).

inner_seed instance-attribute

inner_seed: int

Seed of the split of the fold's training rows into train and validation.

smote_seed instance-attribute

smote_seed: int

Seed of the fold's SMOTE draw (unused with oversample=False).

init_seed instance-attribute

init_seed: int

batch_seed instance-attribute

batch_seed: int

threshold instance-attribute

threshold: float

mcc instance-attribute

mcc: float

train_seconds instance-attribute

train_seconds: float

epochs instance-attribute

epochs: int

device instance-attribute

device: str | None

PennyLane device the circuit actually ran on (after any fallback); None for the control.

seed_index class-attribute instance-attribute

seed_index: int = 0

Which of the n_seeds initialisations of this fold (init_seed is its seed).

hyperparameters class-attribute instance-attribute

hyperparameters: dict[str, Any] = field(default_factory=dict)

The training settings tuning chose for this fold and model; empty without tuning.

noise_mcc class-attribute instance-attribute

noise_mcc: dict[float, float] = field(default_factory=dict)

Hybrid only: test MCC under depolarising noise of each swept probability.

brier class-attribute instance-attribute

brier: float = math.nan

Brier score of the test probabilities (#319).

ece class-attribute instance-attribute

ece: float = math.nan

Expected calibration error of the test probabilities, ECE_BINS quantile bins.

BenchmarkResult dataclass

Attributes:

Name	Type	Description
records	list[dict[str, Any]]	One dict per dataset and model with the keys in COLUMNS; fold_mcc is a tuple of the per-fold scores.
folds	list[FoldResult]	Every FoldResult, in dataset, fold, model order.
settings	dict[str, Any]	The run_benchmark arguments other than the datasets and the builder, with loss as its import path.
models	dict[str, dict[str, dict[str, Any]]]	Per dataset: {"hybrid": {"class": ..., "config": ...}, "control": {...}}, each model's class name and get_config().
datasets	dict[str, dict[str, Any]]	Per dataset: n_samples, n_features, n_positives and the SHA-256 of the float64 features and int64 labels, so a re-run can check it was given the same data.

records instance-attribute

records: list[dict[str, Any]]

folds instance-attribute

folds: list[FoldResult]

settings instance-attribute

settings: dict[str, Any]

models instance-attribute

models: dict[str, dict[str, dict[str, Any]]]

datasets instance-attribute

datasets: dict[str, dict[str, Any]]

noise class-attribute instance-attribute

noise: list[dict[str, Any]] = field(default_factory=list)

With noise_levels: one row per dataset and noise level -- the hybrid's and the noise-free control's mean MCC over folds, and the one-sided paired Wilcoxon test that the hybrid is better (p_hybrid_better, min_p).

noise_summary class-attribute instance-attribute

noise_summary: dict[str, dict[str, Any]] = field(default_factory=dict)

Per dataset: whether the hybrid is significantly better than the control without noise (better_noiseless), and the first swept noise level at which it no longer is (lost_at; None if it never was, or never stopped being within the sweep).

cross_dataset class-attribute instance-attribute

cross_dataset: dict[str, Any] | None = None

With more than one dataset: two-sided Wilcoxon comparison of hybrid and control's per-dataset mcc_mean, each dataset weighted equally. datasets gives their order, models the subtraction order (hybrid minus control), n_datasets the total, and n_pairs the count of non-zero differences (a difference of at most 1e-12 is a round-off tie). wilcoxon_statistic is W+; wilcoxon_p, wilcoxon_min_p and rank_biserial describe this cross-dataset test, separate from the within-dataset record columns. All mean MCCs tied: statistic and p-values are NaN, effect is 0. One dataset: None.

FitScore

Bases: NamedTuple

One trained model scored on its test rows.

mcc instance-attribute

mcc: float

threshold instance-attribute

threshold: float

seconds instance-attribute

seconds: float

epochs instance-attribute

epochs: int

brier instance-attribute

brier: float

ece instance-attribute

ece: float

Tuning dataclass

The same random-search budget for the hybrid model and its control.

Per outer fold, each model gets n_trials configurations drawn from its own search space (all of them if the space has fewer), and each is scored by the mean MCC over the same inner_folds stratified folds of the outer training part -- never the outer test rows. The best configuration is then trained as usual. The budget is counted in trials, not seconds, so the slower simulated model gets as many configurations as the control.

Attributes:

Name	Type	Description
n_trials	int	Configurations per model and outer fold.
search_spaces	Mapping[str, Mapping[str, Sequence[Any]]]	{"hybrid": {...}, "control": {...}}, each mapping a setting in TUNABLE to the values to draw from.
inner_folds	int	Stratified folds of the outer training part a configuration is scored on. Default 3.

n_trials instance-attribute

n_trials: int

search_spaces instance-attribute

search_spaces: Mapping[str, Mapping[str, Sequence[Any]]]

inner_folds class-attribute instance-attribute

inner_folds: int = 3

as_dict

as_dict() -> dict[str, Any]

default_hybrid

default_hybrid(n_input_features: int) -> nn.Module

HybridBinaryClassifier(n_input_features, n_qubits=8, n_layers=2).

fingerprint

fingerprint(X: NDArray[float64], y: NDArray[int64]) -> str

SHA-256 over the shape and bytes of X (float64) and y (int64).

run_benchmark

run_benchmark(datasets: Mapping[str, tuple[ArrayLike, ArrayLike]], hybrid: HybridBuilder = default_hybrid, *, n_splits: int = 5, validation_folds: int = 5, oversample: bool = True, loss: LossBuilder = FocalLoss, lr: float = 0.01, max_epochs: int = 100, batch_size: int = 256, patience: int | None = 10, random_state: int = 0, smote_kwargs: Mapping[str, Any] | None = None, record_path: str | PathLike[str] | None = None, n_seeds: int = 1, tuning: Tuning | None = None, noise_levels: Sequence[float] | None = None, noise_position: Position = 'all', alpha: float = 0.05) -> BenchmarkResult

Compare hybrid with its matched classical control on every dataset.

See the module docstring for what the two models share in each fold.

Parameters:

Name	Type	Description	Default
datasets	Mapping[str, tuple[ArrayLike, ArrayLike]]	Name → (X, y), with y binary 0/1 and 1 the rare class, e.g. {"credit-card": load_credit_card_fraud()[:2]}.	required
hybrid	HybridBuilder	hybrid(n_input_features) returns a fresh, untrained HybridBinaryClassifier or ParallelHybridClassifier; the control is classical_baseline of it. Called once per fold.	default_hybrid
n_splits	int	Outer folds per dataset. At least 2; see the module docstring for the smallest p-value a given number of folds allows.	5
validation_folds	int	One validation_folds-th of each training part is held out for validation. Default 5, i.e. 20 %.	5
oversample	bool	SMOTE the training rows (smote_kwargs are passed on).	True
loss	LossBuilder	Called once per model and fold for a fresh loss, default FocalLoss.	FocalLoss
lr	float	Adam learning rate and the train_model settings, the same for both models.	0.01
max_epochs	float	Adam learning rate and the train_model settings, the same for both models.	0.01
batch_size	float	Adam learning rate and the train_model settings, the same for both models.	0.01
patience	float	Adam learning rate and the train_model settings, the same for both models.	0.01
random_state	int	Root seed. Every fold split, SMOTE draw, initialisation and batch order derives from it, so a run is repeatable; the seeds used are in BenchmarkResult.folds.	0
n_seeds	int	Train each model this many times per fold, with distinct recorded initialisation seeds (split, SMOTE and batch order stay the fold's). The fold's score is the mean over its seeds, so the paired test still pairs folds, and mcc_seed_std reports how much a model's score moves with the initialisation alone: per fold the sample standard deviation (ddof=1) of its seeds' MCCs, averaged over folds. Default 1: one seed, as before. With more than one, hybrid must build its model with init_seed=None (a model's own seed would override the runner's and repeat the same weights), and a sampling one (shots) with seed=None too (a device seed would repeat the same shot noise); a ValueError is raised otherwise.	1
noise_levels	Sequence[float] | None	Also score each trained hybrid model on its test rows under depolarising noise of each probability (inserted as hqnn_forge.noise.apply_depolarizing_noise does, at noise_position), at the fold's threshold, and compare it with the noise-free control per level (BenchmarkResult.noise). This is inference-time noise on a model trained without it: it asks whether an advantage measured in noiseless simulation survives on a noisy device, not what training under noise would give.	None
noise_position	Sequence[float] | None	Also score each trained hybrid model on its test rows under depolarising noise of each probability (inserted as hqnn_forge.noise.apply_depolarizing_noise does, at noise_position), at the fold's threshold, and compare it with the noise-free control per level (BenchmarkResult.noise). This is inference-time noise on a model trained without it: it asks whether an advantage measured in noiseless simulation survives on a noisy device, not what training under noise would give.	None
alpha	float	Significance level for BenchmarkResult.noise_summary.	0.05
tuning	Tuning | None	Tune both models' training settings with the same budget in every outer fold before training them; see Tuning. Default: no tuning, the settings above for both.	None
record_path	str | PathLike[str] | None	Also write an experiment record (config, seeds, fold indices, dependency versions, devices, metrics) there as JSON; see hqnn_forge.experiment.save_record.	None

Returns:

Type	Description
BenchmarkResult	Per-dataset records (including Holm-adjusted fold-test p-values) and, for multiple datasets, cross_dataset statistics over mean MCCs.

write_csv

write_csv(records: Sequence[Mapping[str, Any]], path: str | PathLike[str]) -> None

Write records to path with the columns in COLUMNS; fold_mcc is written as its scores joined by ;.