Benchmarking

The sktime.benchmarking module contains functionality to perform benchmarking.

Benchmarking Framework

BaseBenchmark([id_format, backend, ...])

Base class for benchmarks.

ForecastingBenchmark([id_format, backend, ...])

Forecasting benchmark.

ClassificationBenchmark([id_format, ...])

Classification benchmark.

RegressionBenchmark([id_format, backend, ...])

Regression benchmark.

Storage Backends

JSONStorageHandler(path)	Storage handler for JSON files, with ending .json.
ParquetStorageHandler(path)	Storage handler for Parquet files, with ending .parquet.
CSVStorageHandler(path)	Storage handler for CSV files, with ending .csv.
NullStorageHandler(path)	Storage handler for no file access.

Benchmark analyzers

Benchmark analyzers consume the results of BaseBenchmark.run and compute ranking, omnibus / pairwise significance tests, and critical-difference diagrams.

BaseBenchmarkAnalyzer([metric, lower_is_better])	Base class for benchmark analyzers.
AverageRank([metric, lower_is_better])	Average ranks of estimators across datasets.
FriedmanTest([metric, lower_is_better])	Friedman test for differences between estimators across datasets.
NemenyiTest([metric, lower_is_better])	Nemenyi post-hoc test for pairwise differences between estimators.
WilcoxonSignedRankTest([metric, lower_is_better])	Wilcoxon signed-rank test on each unique pair of estimators.
SignTest([metric, lower_is_better])	Sign test for consistent differences between estimator pairs.
RankSumTest([metric, lower_is_better])	Wilcoxon rank-sum test on each ordered pair of estimators.
TwoSampleTTest([metric, lower_is_better, ...])	Independent two-sample t-test on each ordered pair of estimators.
CriticalDifferenceDiagram([metric, ...])	Critical-difference (CD) diagram evaluator.