Experiment records

hqnn_forge.experiment

hqnn_forge.experiment

Experiment records: what is needed to check or repeat a benchmark result, written as JSON beside it.

A checkpoint (hqnn_forge.utils.checkpoint) is enough to reload a model, not to reproduce a result. A record written by save_record (or by run_benchmark(..., record_path=...)) holds:

• config -- every run_benchmark setting, and per dataset the class and get_config() of the hybrid model and of its control;
• seeds -- the root random_state and, per fold, every seed derived from it: the outer split, the train/validation split, SMOTE, initialisation and batch order. A builder that fixes its own init_seed overrides the per-fold initialisation seed; that init_seed is then in the model config;
• folds -- the train, validation and test row indices of every fold, and the threshold, MCC, epochs and seconds per model;
• environment -- Python, platform, hqnn_forge and the versions of PennyLane, pennylane-lightning, torch, NumPy, scikit-learn and SciPy, and per fold the PennyLane device the circuit really ran on, since the device factory falls back silently apart from a warning;
• datasets -- size, positives and a SHA-256 of the data, not the data;
• metrics -- the per-dataset, per-model rows of the result, including the Holm-adjusted wilcoxon_p_holm;
• cross_dataset -- the hybrid/control comparison over per-dataset mean MCCs (cross_dataset), null for a single dataset.

load_record reads one back and reports which recorded versions differ from the running environment. rerun_benchmark repeats the run from a record on the same data: on CPU it reproduces the per-fold scores exactly.

The file is strict JSON: NaN (an undefined Wilcoxon p-value) is written as null.

RECORD_FORMAT_VERSION module-attribute

RECORD_FORMAT_VERSION = 1

Bumped whenever the layout below changes incompatibly.

RECORDED_PACKAGES module-attribute

RECORDED_PACKAGES: tuple[str, ...] = ('pennylane', 'pennylane-lightning', 'torch', 'numpy', 'scikit-learn', 'scipy')

Distributions whose installed version is recorded. None means absent.

COMPARED_VERSIONS module-attribute

COMPARED_VERSIONS: tuple[str, ...] = ('python', 'implementation', 'hqnn_forge')

The entries of environment besides packages that compare_environment compares; the platform string is not one.

environment

environment() -> dict[str, Any]

Python, platform, hqnn_forge and RECORDED_PACKAGES versions.

to_record

to_record(result: BenchmarkResult) -> dict[str, Any]

The JSON-ready experiment record of result in the current environment.

save_record

save_record(result: BenchmarkResult, path: str | PathLike[str]) -> None

Write result's experiment record to path as JSON.

Raises:

Type	Description
TypeError	If a model config holds something that is not plain data, e.g. a custom encoder module; such a run cannot be recorded.

compare_environment

compare_environment(recorded: Mapping[str, Any]) -> dict[str, tuple[Any, Any]]

{name: (recorded, current)} for every recorded version that differs from the running environment: python, hqnn_forge and each package. The platform string is informational and not compared.

load_record

load_record(path: str | PathLike[str]) -> tuple[dict[str, Any], dict[str, tuple[Any, Any]]]

Read a record written by save_record.

Returns:

Type	Description
(record, differences)	The record as a dict, and compare_environment of its environment: empty when every recorded version matches.

Raises:

Type	Description
ValueError	If the file is not an experiment record of a known format version.

rerun_benchmark

rerun_benchmark(record: Mapping[str, Any], datasets: Mapping[str, tuple[ArrayLike, ArrayLike]], *, loss: Any = None) -> BenchmarkResult

Repeat the run described by record on the same datasets.

The hybrid model is rebuilt from the recorded class and config (the builder function itself is not recorded), with every recorded setting and the root seed, so the folds, SMOTE draws, initialisations and batch orders are the ones recorded. On CPU the per-fold scores then match exactly.

Parameters:

Name	Type	Description	Default
record	Mapping[str, Any]	As returned by load_record.	required
datasets	Mapping[str, tuple[ArrayLike, ArrayLike]]	The same datasets under the same names and in the same order; each is checked against its recorded SHA-256.	required
loss	Any	The loss builder, if the recorded import path cannot be imported (a loss defined inside a function or a script).	None

Raises:

Type	Description
ValueError	If a dataset is missing, extra, reordered or differs from the recorded one, or if the rebuilt models' configs differ from the recorded ones.