Error Analysis

You can use the Error Analysis method to determine the algorithm for error calculation. The Error Analysis method is calculated for all selected Forecast Methods. The winner is the forecast method with the lowest error.

  1. From the menu, select Data > Statistical Forecast Methods > Competitive Forecast Methods.
  2. Select a parameter and then click New or Edit.
  3. In Competitive Forecast Properties, select Error Analysis.
  4. In Method, select the applicable forecast method.
    Measure Formula
    Accuracy Signal AS = ([M1] - [M2]) / ([M1] + [M2])

    Accuracy Signal indicates how close the forecast is to a perfect forecast and whether it is overshooting or undershooting.

    • AS = 0: Perfect
    • AS < 0: Undershooting AS = -0.333, means M2 = 2 * M1, and AS= -0.048 means M2 is 10% more than M2
    • AS > 0: Overshooting AS = 0.333, means M1 = 2*M2
    Note: AS has the same scale for positive and negative values.
    Mean Absolute Error (MAE)

    This calculates the mean absolute error for demand or calculated forecast.

    dmp_MAE
    Mean Absolute Percentage Error (MAPE) This is a common and useful indicator that shows the relative difference between two measures. Examples include Demand versus Forecast and Forecast versus Sales.

    MAPE = Avg(3,Abs(([M1] - [M2]) / [M1]))
    Mean Error (ME)

    This calculates mean error for demand or calculated forecast.

    dmp_ME
    Mean Percentage Error (MPE)

    This calculates mean percentage error for demand or calculated forecast.

    dmp_MPE
    Mean Square Error (MSE) MSE = Avg(3,([M1] - [M2])^2)
    Symmetric MAPE (SMAPE) Symmetric MAPE (SMAPE) measures the percentage difference between the forecast and actual demand, where both values contribute equally to the denominator:

    SMAPE = |Actual − Forecast| / ((|Actual| + |Forecast|) / 2)

    A value of 0 means a perfect forecast. A value of 0.05 means the forecast deviates approximately 5% from actual demand.

    Unlike traditional MAPE, which divides only by the actual value, SMAPE treats over-forecasting and under-forecasting equally. This avoids bias where forecasting 100 when the actual value is 50 (100% error) is penalized differently than forecasting 50 when the actual value is 100 (50% error). With SMAPE, both cases produce the same error.

    This makes SMAPE particularly suitable for competitive forecast selection, where methods are compared on a fair and balanced basis regardless of whether they tend to forecast above or below actual demand.
    Theil's U-statistic Theil's U = Sqr(Sum(3,(([M1][P + 1] - [M2][P + 1]) / [M2][P])^2)/Sum(3,(([M2][P + 1] - [M2][P]) / [M2][P])^2))

    This provides a relative comparison and also squares the errors involved so that large errors are given more weight than small errors. Theil's U describes how good a fit two measures are.

    • U = 1: The naïve method is as good as the technique used, for example, taking a copy of the last year's sales or repeating the last demand is as good as a guess.
    • U < 1: The method used is better than a simple naive guess. The smaller Theil's U is, the better (U = 0 is a perfect prediction).
    • U > 1: The method used is worse than simple using a naïve method. The copy of last year's sales gives a better forecast.
  5. In Limits, specify values in the Lower and Upper limits. If the calculation of the error analysis method is outside these limits, the forecast method is not selected as the winner. You can perform competitive forecasting without finding a winner that meets the criteria. You can define a scoreboard to identify key records without a winner for manual inspection. You can determine a different forecast method or competitive method that is better suited.
  6. In Period Range, specify this information:
    Use Forecast defined period range
    Select this option to limit or control the periods to which the forecast applies.
    Hold-out periods (before forecast point)
    Specify a value to define the periods used for the Error Analysis. Hold-out periods are three monthly periods and a maximum of one-third of the total available periods.
    See Hold-out periods.
    Use Smart Combined Forecast
    Select this option to combine multiple forecasting models to generate a more accurate prediction.
    See Smart Combined Forecast.