Model Comparison Statistics

The error statistic is selected at the beginning of each statistical report generation process. The statistics for each parameter and station, as well as parameter composite statistics, are reported. The user may also select to automatically generate reports as shown in Figure 1. The reports will be automatically saved in the project directory in the “#calib_stats” folder.  

Figure 1  Calibration plot generation options.

The user should select the Combine Error Statistics checkbox to plot all the statistics on one form rather than each on a separate form. The user may select HTML Format so that the statistics are formatted for easy copying into a report or displaying online.  To set EE so that if provides automatic summaries of statistics at the end of the model run the user should select Automatically Generate Reports checkbox as well as selecting Generate Plots and Statistic at End of Model Run in the Run Model form before running the model.

Extracting the data from the EFDC output can be time consuming if it is a large model run for a long period of time.  To avoid having to extract the results each time the statistics are generated, the user is advised to check the Use Existing Model Extractions check box, which will use the extracted data still in computer memory   

There are twelve pre-defined statistics available for the model-data statistical reports which are described below.  The difference between the average observation and model values (Mean Error - ME) is taken to show the deviation in terms of the unit of measurement.

RMSE: The root mean squared error (or RMSE) reflects sample standard deviation of differences between simulated and observed values. Similarly, the relative RMSE (RRMSE) is the RMSE normalized by the maximum range of observed values.

The Index of Agreement (IOA) is a standardized measure of the degree of model prediction error, with values ranging from 0 (no agreement) to 1 (perfect match).

The R² value is a statistical measure of how close a model’s predictions are to a fitted regression line. It is sometimes referred to the coefficient of determination.

Three widely used error measures are used here. “O” denotes observations and “P” denotes model predictions at the corresponding locations and times, the mean of the observed and predicted variables for “N” observations at a single or multiple observation stations is given in equations below.

Mean Observed:

Mean Predicted:

Standard Deviation Observed:

Standard Deviation Predicted:

Mean Error (ME):

Relative Mean Error (RME):

Absolute Mean Error (MAE):

Root Mean Square Error (RMSE):

Relative Root Mean Square Error (RRMSE):

Coefficient of Determination (R², Square of Correlation Coefficient):

Nash-Sutcliffe Index of Efficiency (NSI):

Coefficient of Efficiency (COE):

Index of Agreement (IOA):

An example of a report generated by EE is shown in Figure 2.  The bottom of the report contains a listing of the composite statistics for the entire model run. The parameters may be copied to the clipboard for pasting into Excel or some text editor.  A good practice is to save these statistics in each run’s directory for quick future reference. 

Figure 2  Example time series calibration statistics report.

EE used the same approach for calibration statistics that is does for the time series statistics, such that it generates a set of model/data pairs focusing on the measured data points.  EE uses a linear interpolation approach to generate the model value corresponding to the measured data point in time.  So, if there were 20,000 data points (from 15 minute data for example) which are corresponding to 5,000 model model snapshots, EE will generate 20,000 model/data pairs.