When you launch a machine learning model, you want to see great results right away, right? But what is considered a "good" result? Everything seems to work, but how well is a big question.
It often happens that a model produces great numbers for one metric, but does not bring any benefit to the real task. Why? Because the wrong evaluation metric was chosen. Yes, a metric is not just a technical detail, but a key benchmark by which you judge the quality of the model.
How to Choose the Right Evaluation Metric for Your ML Model
In this article, we will figure out how to choose the right metric for different types of tasks. We will discuss both technical and business aspects, including how to take into account the roi ads from advertising.
ROI Ads are useful because they show how much profit each advertising dollar brings, helping businesses understand the real value of their marketing efforts. By using ROI as a metric, you can align your ML model’s performance with actual financial outcomes, not just clicks or conversions.
Why is the choice of metric so important?
Imagine: you built a model, tested it, got 90% accuracy, and are happy with yourself. But there is a nuance - if your data contains 90% of objects of the same class, then a "dumb" model that always predicts the same class will give the same accuracy. That is, the figure is beautiful, but there is no benefit.
That is why it is important to understand the context of the task. For classification, there are some metrics; for regression, there are others. In some cases, it is important not to miss rare cases, and in others, not to make false positives.
A real-life example: metric ≠ success
Let's say you are working with an advertising model that must predict who will click on an ad. Everything looks perfect: the accuracy is high, and the model works stably. But the advertising budget is wasted. Why? Because clicks do not bring money, those who do not buy click.
This is where an important point comes up: a business metric, such as ROI (return on investment), can be much more important than standard technical metrics. Sometimes it is better to target fewer users, but with a higher probability of purchase.
If you have a classification
For tasks where you need to distribute objects into classes (for example, "will buy" or "will not buy"), there are several popular metrics:
Accuracy (precision) is the percentage of correct predictions. Easy to understand, but not always useful.
Precision (prediction accuracy) - of all the "yes" that the model gave, how many were correct?
Recall (completeness) - how many of the real "yes" the model was able to find.
F1-score is a compromise between precision and recall.
If it is more important not to miss cases (for example, diagnosing diseases), look at recall. If it is more important not to scare people with false alarms, we prioritize precision.
If you have a regression
When the task is to predict a number (for example, price, demand, or conversion), other metrics are used:
MSE (mean square error) - the greater the deviation, the worse.
RMSE - the same error, but with a root, closer to "reality" in scale.
MAE (mean absolute error) - shows the average error without quadratic gains.
R² (determination coefficient) - how well the model explains the data (from 0 to 1).
It is important to understand here: MSE/MAE show accuracy, and R² - the explanatory power of the model.
Tasks where regular metrics are not enough
There are complex cases, for example:
Class imbalance (99% vs. 1%)
Need to consider the confidence of the model, not just a binary answer.s
Result dependence on the selected threshold
In such cases, it is better to use:
AUC-ROC - measures how well the model distinguishes classes at different thresholds.
LogLoss - penalizes for confident but incorrect answers (important for probabilistic models).
But what if ROI is important?
A separate case is business models, where the metric directly affects money. If the model is good in precision/recall, but does not bring profit, it is useless.
In marketing, the following are often used:
ROI - how much profit does each invested ruble bring in
CPA (cost of customer acquisition)
LTV (lifetime value of a customer)
If you are choosing between two models, be sure to look at which one brings more value in money, and not just in percentages.
How to choose a metric: a short algorithm
First, ask yourself: What is success for me? Then use these steps.
Determine the type of task: classification, regression, or something non-standard.
Look at the data - are there any imbalances, outliers, or noise?
Think: what is more important - not to miss or not to mix up?
Connect the business goal, especially if you work with advertising, sales, or in the financial sector.
Test several metrics in parallel - and compare not only by numbers, but also by common sense.
In a nutshell
A metric is a compass. It shows whether you are moving in the right direction. The choice of metric determines what you consider a “good result,” and this is much more important than just high accuracy.
Don’t forget about the business. Even a model that is great in numbers can fail if it does not bring value. So, combine a technical approach with real tasks, and let the metrics work for you.