next up previous
Next: Simple nonlinear prediction Up: Nonlinear prediction Previous: Nonlinear prediction

Model validation

Before entering the methods, we have to discuss how to assess the results. The most obvious quantity for the quantification of predictability is the average forecast error, i.e. the root of the mean squared (rms) deviation of the individual prediction from the actual future value. If it is computed on those values which were also used to construct the model (or to perform the predictions), it is called the in-sample error. It is always advisable to save some data for an out-of-sample test. If the out-of-sample error is considerably larger than the in-sample error, data are either non-stationary or one has overfitted the data, i.e. the fit extracted structure from random fluctuations. A model with less parameters will then serve better. In cases where the data base is poor, on can apply complete cross-validation or take-one-out statistics, i.e. one constructs as many models as one performs forecasts, and in each case ignores the point one wants to predict. By construction, this method is realized in the local approaches, but not in the global ones.

The most significant, but least quantitative way of model validation is to iterate the model and to compare this synthetic time series to the experimental data. If they are compatible (e.g. in a delay plot), then the model is likely to be reasonable. Quantitatively, it is not easy to define the compatibility. One starts form an observed delay vector as intial condition, performs the first forecast, combines the forecast with all but the last components of the initial vector to a new delay vector, performs the next forecast, and so on. The resulting time series should then be compared to the measured data, most easily the attractor in a delay representation.


next up previous
Next: Simple nonlinear prediction Up: Nonlinear prediction Previous: Nonlinear prediction

Thomas Schreiber
Wed Jan 6 15:38:27 CET 1999