Choose MAE or RMSE from the decision cost

MAE averages absolute errors. RMSE squares errors, averages them and takes a square root, giving larger errors greater influence. Choose the measure according to the decision consequences and the prediction target, rather than whichever produces the smaller-looking number.

An original five-case example shows why rankings can differ. Every actual outcome is ten. Candidate A predicts ten four times and twenty once. Candidate B predicts thirteen in every case.

Calculate both rankings

A's errors are (0,0,0,0,10), giving MAE=2 and RMSE=sqrt(20), approximately 4.4721. B's errors are (3,3,3,3,3), giving both MAE and RMSE equal to three.

MAE prefers A's four exact predictions and one large miss. RMSE prefers B's consistent smaller misses. The disagreement follows directly from how the metrics weight errors; neither result is a computational contradiction.

The MAE and RMSE references document the implementations used to verify this calculation.

Add direction to the decision cost

Candidate C predicts ten four times and zero once. Its absolute and squared errors match A's, so MAE and RMSE cannot distinguish them. Yet a decision may treat one unit of underprediction as much more costly than one unit of overprediction.

For an explicitly hypothetical cost rule of one cost unit per overpredicted unit and five per underpredicted unit, A costs ten, B costs fifteen and C costs fifty in total. These are authored teaching costs, not measured financial losses.

import numpy as np
from metric_cases import regression_case

r = regression_case()
A = r['A_one_overprediction']
B = r['B_constant_overprediction']
C = r['C_one_underprediction']
assert A['mae']==2 and np.isclose(A['rmse'],np.sqrt(20))
assert B['mae']==B['rmse']==3
assert A['mae']<B['mae'] and A['rmse']>B['rmse']
assert A['mae']==C['mae'] and A['rmse']==C['rmse']
assert [A['total_asymmetric_cost'],B['total_asymmetric_cost'],C['total_asymmetric_cost']]==[10,15,50]
print(r)

Run in the evaluation lab. The example is separate from the classification experiment and does not imply that a regression model was trained on these five cases.

Match the target statistic to the loss

Under standard population-risk conditions, squared error favors a conditional mean and absolute error favors a conditional median. If the outcome distribution is skewed, these can be different predictions. A model trained for one statistic should not be judged as though it were necessarily estimating the other.

For a fixed evaluation set, minimizing RMSE gives the same ranking as minimizing MSE because the square root is increasing. RMSE returns the result to the target's units; MSE uses squared units. Neither is a percentage unless the target itself has that interpretation.

Inspect errors behind the aggregate

Report large-error cases, directional bias and relevant slices. An aggregate can hide systematic underprediction for an important group even when overall MAE is acceptable. Also check whether extreme observations are valid cases, data errors or a changed process before deciding how to handle them.

If operational costs have capacity limits, thresholds or asymmetric consequences, write that decision loss explicitly and justify its parameters. A convenient statistical metric can remain a diagnostic while the decision rule uses a more appropriate cost measure.

Exercise: change the underprediction penalty from five to two, then introduce a maximum capacity that clips predictions above fifteen. Recompute the realized decision costs and explain whether the original metric ranking still supports the same choice.

NeuraPath's Data Science course connects evaluation metrics with business decisions. A defensible metric choice explains which mistakes it emphasizes and which consequences require additional analysis.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Compare models with uncertainty instead of one lucky score.
• Continue with Classification metrics when accuracy hides failure.