evaluate#

evaluate(forecaster, cv, y, X=None, strategy: str = 'refit', scoring: Callable | list[Callable] | None = None, return_data: bool = False, error_score: str | int | float = nan, backend: str | None = None, cv_X=None, backend_params: dict | None = None, return_model: bool = False, cv_global=None, cv_global_temporal=None)[source]#

Evaluate forecaster using timeseries cross-validation.

All-in-one statistical performance benchmarking utility for forecasters which runs a simple backtest experiment and returns a summary pd.DataFrame.

The experiment run is the following:

In case of non-global evaluation (cv_global=None):

Denote by \(y_{train, 1}, y_{test, 1}, \dots, y_{train, K}, y_{test, K}\) the train/test folds produced by the generator cv.split_series(y). Denote by \(X_{train, 1}, X_{test, 1}, \dots, X_{train, K}, X_{test, K}\) the train/test folds produced by the generator cv_X.split_series(X) (if X is None, consider these to be None as well).

  1. Initialize the counter to i = 1

  2. Fit the forecaster to \(y_{train, 1}\), \(X_{train, 1}\), with fh set to the absolute indices of \(y_{test, 1}\).

  3. Use the forecaster to make a prediction y_pred with the exogenous

    data \(X_{test, i}\). Predictions are made using either predict, predict_proba or predict_quantiles, depending on scoring.

  4. Compute the scoring function on y_pred versus \(y_{test, i}\)

  5. If i == K, terminate, otherwise

  6. Set i = i + 1

  7. Ingest more data \(y_{train, i}\), \(X_{train, i}\), how depends on strategy:

  • if strategy == "refit", reset and fit forecaster via fit, on \(y_{train, i}\), \(X_{train, i}\) to forecast \(y_{test, i}\)

  • if strategy == "update", update forecaster via update, on \(y_{train, i}\), \(X_{train, i}\) to forecast \(y_{test, i}\)

  • if strategy == "no-update_params", forward forecaster via update, with argument update_params=False, to the cutoff of \(y_{train, i}\)

  1. Go to 3

In case of global evaluation (cv_global is not None):

There are two running indices: j for the instance splitter cv_global, and i for the temporal splitter cv.

\(y_{pretrain, j}, y_{global_test, j}\) are produced by cv_global.split_series(y) and are different time series. \(y_{global_test, j}\) is further split into \(y_{train, i, j}, y_{test, i, j}\) by cv.split_series(y_test). Exogenous folds \(X_{pretrain, j}\), \(X_{train, i, j}\), \(X_{test, i, j}\) are produced analogue.

For each instance fold j and temporal fold i:

  1. If i == 0 or strategy == "refit", clone the forecaster, pretrain on \(y_{pretrain, j}\), \(X_{pretrain, j}\), then fit on \(y_{train, i, j}\), \(X_{train, i, j}\), with fh set to the absolute indices of \(y_{test, i, j}\).

  2. Otherwise ingest more data \(y_{train, i, j}\), \(X_{train, i, j}\) depending on strategy:

  • if strategy == "update", update via update, with update_params=True

  • if strategy == "no-update_params", update via update, with update_params=False

  1. Predict y_pred with exogenous data \(X_{test, i, j}\).

  2. Compute the scoring function on y_pred versus \(y_{test, i, j}\).

Results returned in this function’s return are:

  • results of scoring calculations, from 4, in the i-th loop

  • runtimes for fitting and/or predicting, from 2, 3, 7, in the i-th loop

  • cutoff state of forecaster, at 3, in the i-th loop

  • \(y_{train, i}\), \(y_{test, i}\) (and y_pretrain in global mode), y_pred (optional)

  • fitted forecaster for each fold (optional)

A distributed and-or parallel back-end can be chosen via the backend parameter.

Parameters:
forecastersktime BaseForecaster descendant (concrete forecaster)

sktime forecaster to benchmark

cvsktime BaseSplitter descendant

determines split of y and possibly X into test and train folds y is always split according to cv, see above

  • if cv_X is not passed, X splits are subset to loc equal to y

  • if cv_X is passed, X is split according to cv_X

ysktime time series container

Target (endogeneous) time series used in the evaluation experiment

Xsktime time series container, of same mtype as y

Exogenous time series used in the evaluation experiment

strategy{“refit”, “update”, “no-update_params”}, optional, default=”refit”

defines the ingestion mode when the forecaster sees new data when window expands

  • “refit” = forecaster is refitted to each training window

  • “update” = forecaster is updated with training window data, in sequence provided

  • “no-update_params” = forecaster is updated via update, with update_params=False, to the cutoff of each new training window

scoringsubclass of sktime.performance_metrics.BaseMetric or list of same,

default=None. Used to get a score function that takes y_pred and y_test arguments and accept y_train as keyword argument. If None, then uses scoring = MeanAbsolutePercentageError(symmetric=True).

return_databool, default=False

Returns three additional columns in the DataFrame, by default False. The cells of the columns contain each a pd.Series for y_train, y_pred, y_test.

return_modelbool, default=False

If True, returns an additional column ‘fitted_forecaster’ containing the fitted forecaster for each fold.

error_score“raise” or numeric, default=np.nan

Value to assign to the score if an exception occurs in estimator fitting. If set to “raise”, the exception is raised. If a numeric value is given, FitFailedWarning is raised.

backendstring, by default “None”.

Parallelization backend to use for runs. Runs parallel evaluate if specified and strategy="refit".

  • “None”: executes loop sequentially, simple list comprehension

  • “loky”, “multiprocessing” and “threading”: uses joblib.Parallel loops

  • “joblib”: custom and 3rd party joblib backends, e.g., spark

  • “dask”: uses dask, requires dask package in environment

  • “dask_lazy”: same as “dask”, but changes the return to (lazy) dask.dataframe.DataFrame.

  • “ray”: uses ray, requires ray package in environment

Recommendation: Use “dask” or “loky” for parallel evaluate. “threading” is unlikely to see speed ups due to the GIL and the serialization backend (cloudpickle) for “dask” and “loky” is generally more robust than the standard pickle library used in “multiprocessing”.

cv_Xsktime BaseSplitter descendant, optional

determines split of X into test and train folds default is X being split to identical loc indices as y if passed, must have same number of splits as cv

backend_paramsdict, optional

additional parameters passed to the backend as config. Directly passed to utils.parallel.parallelize. Valid keys depend on the value of backend:

  • “None”: no additional parameters, backend_params is ignored

  • “loky”, “multiprocessing” and “threading”: default joblib backends any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, with the exception of backend which is directly controlled by backend. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults.

  • “joblib”: custom and 3rd party joblib backends, e.g., spark. any valid keys for joblib.Parallel can be passed here, e.g., n_jobs, backend must be passed as a key of backend_params in this case. If n_jobs is not passed, it will default to -1, other parameters will default to joblib defaults.

  • “dask”: any valid keys for dask.compute can be passed, e.g., scheduler

  • “ray”: The following keys can be passed:

    • “ray_remote_args”: dictionary of valid keys for ray.init

    • “shutdown_ray”: bool, default=True; False prevents ray from shutting

      down after parallelization.

    • “logger_name”: str, default=”ray”; name of the logger to use.

    • “mute_warnings”: bool, default=False; if True, suppresses warnings

cv_global: sklearn splitter, or sktime instance splitter, default=None

If cv_global is passed, then global benchmarking is applied, as follows:

  1. The cv_global splitter is used to split data at instance level, into a global pretrain set y_pretrain, and a global test set y_test_global. This is index j.

  2. cv then splits the global test set y_test_global temporally, to obtain temporal splits y_train, y_test. This is index i.

  3. If i == 0 or strategy == "refit", the estimator is cloned, pretrained on y_pretrain, and fitted on y_train. Otherwise it is updated on y_train according to strategy.

  4. The estimator produces predictions``y_pred``, of y_test.

Overall, with y_pretrain, y_train, y_test as above, the following evaluation will be applied at the start of each instance fold (i == 0) and on every fold if strategy == "refit":

forecaster.pretrain(y=y_pretrain, fh=cv.fh)
forecaster.fit(y=y_train, fh=cv.fh)
y_pred = forecaster.predict()
metric(y_test, y_pred)
cv_global_temporal: SingleWindowSplitter, default=None

ignored if cv_global is None. If passed, it splits the Panel temporally before the instance split from cv_global is applied. This avoids temporal leakage in the global evaluation across time series. Has to be a SingleWindowSplitter. cv is applied on the test set of the combined application of cv_global and cv_global_temporal.

Returns:
resultspd.DataFrame or dask.dataframe.DataFrame

DataFrame that contains several columns with information regarding each refit/update and prediction of the forecaster. Row index is splitter index of train/test fold in cv. Entries in the i-th row are for the i-th train/test split in cv. Columns are as follows:

  • test_{scoring.name}: (float) Model performance score. If scoring is a

list, then there is a column withname test_{scoring.name} for each scorer.

  • fit_time: (float) Time in sec for fit or update on train fold.

  • pred_time: (float) Time in sec to predict from fitted estimator.

  • len_train_window: (int) Length of train window.

  • cutoff: (int, pd.Timestamp, pd.Period) cutoff = last time index in train fold.

  • y_train: (pd.Series) only present if return_data=True,

train fold of the i-th split in cv, used to fit/update the forecaster.

  • y_pretrain: (pd.Series) present if return_data=True and

cv_global is passed, global pretrain fold used in pretrain.

  • y_pred: (pd.Series) present if return_data=True,

forecasts from fitted forecaster for the i-th test fold indices of cv.

  • y_test: (pd.Series) present if return_data=True,

testing fold of the i-th split in cv, used to compute the metric.

  • fitted_forecaster: (BaseForecaster) present if return_model=True,

fitted forecaster for the i-th split in cv.

Examples

The type of evaluation that is done by evaluate depends on metrics in param scoring. Default is MeanAbsolutePercentageError.

>>> from sktime.datasets import load_airline
>>> from sktime.forecasting.model_evaluation import evaluate
>>> from sktime.split import ExpandingWindowSplitter
>>> from sktime.forecasting.naive import NaiveForecaster
>>> y = load_airline()[:24]
>>> forecaster = NaiveForecaster(strategy="mean", sp=3)
>>> cv = ExpandingWindowSplitter(initial_window=12, step_length=6, fh=[1, 2, 3])
>>> results = evaluate(forecaster=forecaster, y=y, cv=cv)

To do global evaluation, provide cv_global and use forecasters supporting pretraining.

>>> from sklearn.model_selection import KFold
>>> from sktime.datasets import ForecastingData
>>> from sktime.forecasting.model_evaluation import evaluate
>>> from sktime.forecasting.ttm import TinyTimeMixerForecaster
>>> from sktime.split import InstanceSplitter, SingleWindowSplitter

>>> data = ForecastingData(
...     "australian_electricity_demand_dataset"
... ).load("y")

>>> cv = SingleWindowSplitter(fh=range(1, 48))

>>> results = evaluate(
...     TinyTimeMixerForecaster(),
...     y=data,
...     cv=cv,
...     cv_global=InstanceSplitter(KFold(5)),
...     cv_global_temporal=SingleWindowSplitter(fh=range(48 * 24)),
...     strategy="update",
... )

Optionally, users may select other metrics that can be supplied by scoring argument. These can be forecast metrics of any kind as stated here i.e., point forecast metrics, interval metrics, quantile forecast metrics. To evaluate estimators using a specific metric, provide them to the scoring arg.

>>> from sktime.performance_metrics.forecasting import MeanAbsoluteError
>>> loss = MeanAbsoluteError()
>>> results = evaluate(forecaster=forecaster, y=y, cv=cv, scoring=loss)

Optionally, users can provide a list of metrics to scoring argument.

>>> from sktime.performance_metrics.forecasting import MeanSquaredError
>>> results = evaluate(
...     forecaster=forecaster,
...     y=y,
...     cv=cv,
...     scoring=[MeanSquaredError(square_root=True), MeanAbsoluteError()],
... )

An example of an interval metric is the PinballLoss. It can be used with all probabilistic forecasters.

>>> from sktime.forecasting.naive import NaiveVariance
>>> from sktime.performance_metrics.forecasting.probabilistic import PinballLoss
>>> loss = PinballLoss()
>>> forecaster = NaiveForecaster(strategy="drift")
>>> results = evaluate(forecaster=NaiveVariance(forecaster),
... y=y, cv=cv, scoring=loss)

To return fitted models for each fold, set return_model=True:

>>> results = evaluate(
...     forecaster=forecaster,
...     y=y,
...     cv=cv,
...     scoring=loss,
...     return_model=True
... )
>>> fitted_forecaster = results.iloc[0]["fitted_forecaster"]
On this page