Benchmarking

The sktime.benchmarking module contains functionality to perform benchmarking.

Benchmarking Framework v2

BaseBenchmark([id_format, backend, ...])

Base class for benchmarks.

ForecastingBenchmark([id_format, backend, ...])

Forecasting benchmark.

ClassificationBenchmark([id_format, ...])

Classification benchmark.

Storage Backends

JSONStorageHandler(path)	Storage handler for JSON files, with ending .json.
ParquetStorageHandler(path)	Storage handler for Parquet files, with ending .parquet.
CSVStorageHandler(path)	Storage handler for CSV files, with ending .csv.
NullStorageHandler(path)	Storage handler for no file access.

Benchmarking Framework v1

BaseMetric(name, **kwargs)
BaseResults()	Base results class.
BaseDataset(name)	Base dataset class.
HDDBaseResults(path)	HDD results.
HDDBaseDataset(path, name)	HDD dataset.

UEADataset(path, name[, suffix_train, ...])	Represent a dataset in UEA/UCR format on the hard-drive.
RAMDataset(dataset, name)	Represent a dataset in RAM.

Evaluator(results)

Analyze results of machine learning experiments.

run_clustering_experiment(trainX, clusterer, ...)	Run a clustering experiment and save the results to file.
load_and_run_clustering_experiment(...[, ...])	Run a clustering experiment.
run_classification_experiment(X_train, ...)	Run a classification experiment and save the results to file.
load_and_run_classification_experiment(...)	Load a dataset and run a classification experiment.

Orchestrator(tasks, datasets, strategies, ...)

Fit and predict one or more estimators on one or more datasets.

RAMResults()	In-memory results.
HDDResults(path)	HDD results.

BaseStrategy(estimator[, name])	Abstract base strategy class.
BaseSupervisedLearningStrategy(estimator[, name])	Abstract strategy class for time series supervised learning.
TSCStrategy(estimator[, name])	Strategy for time series classification.
TSRStrategy(estimator[, name])	Strategy for time series regression.

BaseTask(target[, features, metadata])	Abstract base task class.
TSCTask(target[, features, metadata])	Time series classification task.
TSRTask(target[, features, metadata])	Time series regression task.

PairwiseMetric(func[, name])	Compute metric pairwise.
AggregateMetric(func[, method, name])	Compute metric pairwise.

Post-hoc tests and utilities