How we test AI models

A test set is a portion of data that the AI model has never seen during training. It is used after training is complete to check how well the model performs on new, real-world examples. Think of it like a final exam — the model studied on the training data, and the test set is the exam it has not seen before.

A training set is the collection of data examples that an AI model learns from. The model finds patterns in this data so it can make predictions later on new inputs. Just like you learn math from worked examples in a textbook, the AI learns from examples in the training set.

If we test on training data, the model might simply have memorized the answers rather than truly learning the underlying patterns. This produces a falsely high score that does not reflect how the model will perform on genuinely new examples. It is like grading yourself on homework you already have the answer key to — it does not prove you really learned the material.

Accuracy is the percentage of predictions the model gets right out of all predictions it makes. For example, if a model correctly classifies 90 out of 100 images, its accuracy is 90%. It is the simplest and most widely understood way to summarise how well a model is performing.

correct = 90
total   = 100
accuracy = correct / total * 100
print(f"Accuracy: {accuracy}%")  # 90.0%

Overfitting happens when a model learns the training data too well — including its random noise and quirks — so it performs poorly on new data it has never seen. Think of it as memorising practice exam questions word-for-word but failing when the real exam asks slightly different questions. A telltale sign is very high training accuracy paired with much lower test accuracy.

Underfitting happens when a model is too simple to capture the patterns in the training data, so it performs badly on both training and test sets. It is like submitting a barely-started homework assignment — the model has not learned enough from the data it was given. Unlike overfitting, the model does poorly even on the examples it was supposed to learn from.

A confusion matrix is a table that shows exactly how many predictions the model got right and wrong for each category. For a cat-vs-dog classifier it shows how many cats were correctly labelled 'cat', how many were wrongly called 'dog', and vice versa. This detailed breakdown helps developers understand precisely where the model is making mistakes, not just how many errors it made in total.

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_true, y_pred)
print(cm)
# Output layout: [[TN, FP],
#                  [FN, TP]]

A validation set is a portion of data used during the training process to tune the model and catch overfitting before the final evaluation. The test set is held back until the very end to give a completely unbiased measure of the finished model's performance. Think of the validation set as practice tests and the test set as the one official final exam.

In k-fold cross-validation the dataset is divided into k equal parts called folds. The model is trained k times, each time using a different fold as the validation set and the remaining folds as training data. The results from all k rounds are averaged, giving a much more stable and reliable performance estimate than any single split — especially valuable when the dataset is small.

from sklearn.model_selection import KFold
kf = KFold(n_splits=5)
for train_idx, val_idx in kf.split(X):
    X_train, X_val = X[train_idx], X[val_idx]
    # train on X_train, evaluate on X_val

Precision measures how many of the model's positive predictions were actually correct. For example, if a spam filter labels 20 emails as spam but only 16 truly are spam, its precision is 80%. High precision means the model rarely raises false alarms, which matters in situations where false positives are costly or annoying.

Recall measures how many of the actual positive cases the model successfully found. For example, if there are 50 real disease cases in a dataset and the model correctly identifies 40 of them, recall is 80%. High recall matters when missing a real positive is dangerous — such as failing to detect a disease.

The F1 score is the harmonic mean of precision and recall, combining both into a single number that reflects how well a model balances finding true positives and avoiding false alarms. It is especially useful when a dataset is imbalanced — for example, far more 'not fraud' cases than 'fraud' cases — because accuracy alone can be very misleading in those situations. A high F1 score means the model is both precise and thorough.

Data leakage happens when information from the test set accidentally influences the training process, making the model appear far better than it really is. For example, accidentally training on data that includes future information — information unavailable at real prediction time — produces great-looking results that completely fall apart in the real world. It is one of the most common and costly mistakes in AI experiments.

A benchmark dataset is a well-known, standardised collection of data that researchers use to compare different AI models on equal footing. Famous examples include ImageNet for image recognition and MNIST for handwritten digit recognition. Using the same benchmark means everyone can see clearly which model truly performs best without the results being affected by different data choices.

A baseline model is the simplest possible approach you compare your AI against — for instance, always predicting the most common class in the dataset. It sets a minimum performance bar that any useful model must beat. If your sophisticated AI only barely outperforms the baseline, it may not be worth the extra cost and complexity.

Generalization is an AI model's ability to perform well on data it has never seen before, not just on the training examples it learned from. A model that generalises well has discovered the true underlying patterns rather than memorising specific examples. Achieving good generalization is the ultimate goal of machine learning.

Mean Absolute Error measures how far off a model's predictions are on average by computing the absolute difference between each prediction and the true value, then averaging all those differences. For example, if a model predicts house prices as [10, 20] (in thousands) but actual prices are [8, 24], the MAE is (|10-8| + |20-24|) / 2 = (2 + 4) / 2 = 3. Smaller MAE means the model's guesses are closer to reality.

predictions = [10, 20]
actuals     = [8,  24]
mae = sum(abs(p - a) for p, a in zip(predictions, actuals)) / len(predictions)
print(mae)  # 3.0

When one class is far more common than another, a model can achieve very high accuracy simply by always predicting the majority class, while being completely useless. For instance, a dataset with 950 non-fraud and 50 fraud cases gives 95% accuracy to a model that always says 'not fraud' — yet it catches zero actual fraud cases. Metrics like precision, recall, and the F1 score give a much fairer picture in these situations.

An ROC (Receiver Operating Characteristic) curve plots a model's true positive rate against its false positive rate at many different decision thresholds, showing the trade-off between catching real positives and triggering false alarms. The Area Under the Curve (AUC) summarises overall performance in a single number — a perfect model scores 1.0, while a model that is no better than random guessing scores 0.5. ROC curves are particularly useful when you need to choose the right balance for your specific application.

Hyperparameters are settings you choose before training — such as learning rate or tree depth — that control how the model learns. We try many combinations, evaluate each on the validation set, and keep the best one. Using the test set for this process would be cheating: the model would be tailored to the test set, making the final score look much better than it really is in the real world.

An ablation study tests a model by systematically removing one component at a time to measure how much each part contributes to overall performance. For example, if removing a data-cleaning step causes accuracy to drop by 10%, that step is clearly important. It is a scientific way of understanding what actually makes a model work well, rather than guessing.

Out-of-distribution data looks very different from what the model was trained on — for example, a model trained on daytime photos encountering night-time photos. Models often fail badly on OOD data even when they score well on standard tests, which can cause serious problems once deployed in the real world. Testing on OOD examples reveals how robust and trustworthy a model truly is.

Some AI outputs — such as creative writing, chatbot replies, or generated images — are difficult to measure with simple numbers, so human raters read or view the outputs and score them for quality, helpfulness, or accuracy. Human evaluation gives a realistic picture of whether the AI is genuinely useful, even though it is slower and more expensive than automatic metrics. It is commonly used to test large language models and AI image generators.

A/B testing compares two model versions by showing each to a different group of real users at the same time and measuring which one performs better in practice. For example, a streaming service might test two recommendation models to see which one leads to more watch time. It is a real-world evaluation that can reveal things offline accuracy metrics completely miss.

Error analysis means carefully examining the examples a model got wrong to discover patterns in its mistakes. For instance, an image classifier might fail mostly on blurry photos or photos with unusual lighting conditions. Understanding the types of errors helps developers decide what to fix next — whether that means collecting more specific data, engineering better features, or choosing a different model architecture.

Fairness testing checks whether an AI model performs equally well across different groups of people — for example, whether a face-recognition system is equally accurate for people of all skin tones. If a model is unfair, it can cause real harm to under-represented groups in high-stakes decisions like hiring or medical diagnosis. Fairness testing is an essential part of responsible and ethical AI development.

Adversarial examples are inputs that have been deliberately modified with tiny, often invisible changes specifically designed to fool an AI model into making the wrong prediction. For example, adding carefully crafted pixel-level noise to a photo can make an image classifier confidently label a cat as a dog, even though the image looks unchanged to a human. Testing with adversarial examples exposes weaknesses in model robustness and security that normal testing would completely miss.

RMSE measures prediction error by squaring each error, averaging them, then taking the square root — which means large mistakes are penalised far more heavily than small ones. MAE, by contrast, treats all errors equally regardless of size. If your application cares much more about avoiding very large mistakes than small ones — like predicting medication doses — RMSE is the better metric to use.

If data is stored in a sorted order — for example, all cat images first followed by all dog images — splitting without shuffling means the training set might see only one class and the test set the other, making the evaluation completely meaningless. Shuffling ensures both sets contain a representative mix of all categories. This gives a fair and realistic measure of how the model will actually perform.

The loss score measures how wrong the model's predictions are — lower is better. During healthy training, both training loss and validation loss should decrease together. If the training loss keeps falling while the validation loss starts rising, the model is overfitting: it is getting better at the training data but worse at generalising to new examples.

The holdout method means setting aside (holding out) a fixed portion of your data — typically 20% — as a test set that the model never sees during training. After training is complete, the model is evaluated on this held-out portion to get an honest, unbiased measure of real-world performance. It is the simplest and most commonly used evaluation strategy.

A ground truth label is the verified correct answer for a data example — for instance, a doctor confirming that a medical scan shows a tumour. AI models are tested by comparing their predictions against ground truth labels to measure performance. If the ground truth labels are wrong, every metric calculated from them will be misleading, which is why label quality is just as important as model quality.

When we evaluate a model separately on data from different groups — different ages, genders, skin tones, or languages — we can spot performance gaps that a single overall score would hide. For example, an AI hiring tool that ranks equally qualified men 15% higher than women has learned a gender bias from its training data. Discovering such bias during testing gives developers the opportunity to fix it before the model causes real-world harm.

Offline evaluation tests a model on a fixed historical dataset before it is deployed — like measuring accuracy on a held-out test set in a lab. Online evaluation measures how the model actually performs once it is live, using real users and real-time data in the wild. Both are important: offline testing is safe and controlled, while online testing reveals real-world behaviour that offline metrics can easily miss.