Training vs testing data

Training data is the set of examples a model learns from. The computer studies these examples to find patterns, similar to how you learn math by practicing solved problems. The quality and amount of training data strongly affect how well the model performs.

Testing data is a separate set of examples used to check how well a model has learned. The model has never seen this data during training, so it shows whether the model can handle new, unseen situations. It acts like a final exam after studying.

If the model is tested on the same data it learned from, it may simply memorize answers and look smarter than it really is. Keeping them separate gives an honest measure of how the model performs on new data. This is like testing a student with questions they have not seen before.

It means 80 out of every 100 examples are used to teach the model, and the remaining 20 are kept aside to test it. This gives the model plenty of examples to learn from while still saving some for an unbiased check. The exact ratio can change, but 80/20 is a popular choice.

It means the model remembers the exact training examples and their answers rather than understanding the general pattern. Such a model does great on training data but fails on new data. This problem is called overfitting.

This large gap suggests the model has overfit the training data. It memorized the training examples instead of learning general patterns, so it struggles with new, unseen testing data. A good model performs similarly on both sets.

Overfitting is when a model learns the training data too well, including its noise and random details, so it cannot handle new data. It is like a student who memorizes exact textbook answers but cannot solve a slightly different question. Overfitting shows up as high training accuracy but low testing accuracy.

Underfitting is when a model is too simple and fails to learn the patterns even in the training data. It performs poorly on both training and testing data. It is like a student who did not study enough to understand the basics.

During training the model 'studies' the examples, just like a student practices problems. The testing data contains questions the model has never seen, like a real exam, so it reveals true understanding rather than memorization. Both check whether learning will work in new situations.

Data leakage happens when information from the testing set accidentally gets into training, for example by including the same examples in both sets. This makes the model look better than it really is because it has secretly already seen the test answers. Leakage gives misleading, overly optimistic results.

Shuffling mixes up the order so both sets get a fair, varied sample of examples. If data is ordered (for example all cats first, then all dogs), an unshuffled split could put all dogs in the test set. Shuffling helps both sets represent the whole dataset.

A validation set is a third portion of data used while building the model to tune settings and compare versions. The testing set is saved for the very end to give a final, unbiased score. Using a separate validation set keeps the final test honest.

Good training data would include many images of handwritten digits along with their correct labels (0 through 9). It should cover different handwriting styles, sizes, and pen colors so the model learns to generalize. Variety in the examples helps the model handle new handwriting.

More good examples let the model see a wider range of situations and learn patterns more reliably. With limited data, the model may miss important cases or memorize the few examples it has. However, the data must be accurate and relevant, not just large in quantity.

Generalization is a model's ability to perform well on new, unseen data, not just the examples it trained on. It shows the model learned the real pattern instead of memorizing. We measure generalization using the testing data.

The train_test_split function from sklearn.model_selection is commonly used. It randomly divides the features and labels into training and testing portions based on a chosen ratio. This makes splitting data quick and consistent.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

It tells the function to put 20% of the data into the testing set and keep the remaining 80% for training. The value is a fraction between 0 and 1. Changing it adjusts how much data is reserved for testing.

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2)

Setting random_state to a fixed number makes the random split the same every time you run the code. This reproducibility lets you and others get identical results and compare experiments fairly. Without it, the split would change on each run.

train_test_split(X, y, test_size=0.2, random_state=42)

Cross-validation splits the data into several parts and repeatedly trains and tests using different parts each time. Every example gets a turn at being in the test set, giving a more reliable estimate of performance. It is especially useful when you have limited data.

The test should reflect the situations the model will actually face after deployment. If testing data is too different from real use, a high test score may not mean the model works in practice. Matching test data to real conditions gives a trustworthy estimate.

The model has already seen those emails, so it can recognize them from memory rather than understanding what makes an email spam. The accuracy reflects memorization, not true skill on new emails. A fair test needs emails the model has never seen.

Labels are the correct answers attached to each training example, such as 'cat' or 'dog' for an image. During training the model compares its guesses to the labels and adjusts to reduce mistakes. Without labels, supervised learning cannot know what is correct.

If the training data over-represents some groups and under-represents others, the model learns those imbalances and may treat groups unfairly. For example, a face recognizer trained mostly on one skin tone may work poorly on others. Balanced, representative data helps reduce this bias.

If you use information from the testing data to make decisions, the test is no longer a fair, unseen check. Your model may look better than it truly is, leading to overconfidence. Keeping the test set untouched until the end protects an honest evaluation.

Poor performance on both sets usually means underfitting, where the model is too simple or has not learned enough. The model has failed to capture the pattern even in the data it studied. Solutions include using a more capable model or better features.

Unseen data refers to examples the model did not encounter during training. Testing on unseen data shows how the model will behave on new inputs in the real world. The whole point of testing is to measure performance on data the model has never seen.

With very few test examples, a couple of lucky or unlucky predictions can swing the score a lot. A small sample may not represent the variety of real-world cases. Larger test sets give more stable and trustworthy estimates.

The main goal is for the model to learn patterns from training data so it can make accurate predictions on new, unseen data. Training is not about memorizing examples but about understanding the general rule behind them. Success is measured by performance on testing data.

When you study, you practice with known problems (training) and then take an exam with new questions (testing). A model learns from training examples and is then checked on testing examples it has not seen. Both reveal whether real learning, not memorizing, took place.

If the real world differs from the testing data, for example new trends or conditions the test did not include, the model may struggle. This mismatch is sometimes called data drift. Models often need monitoring and retraining as the world changes.

Stratified splitting keeps the proportion of each class roughly the same in both sets. For example, if 30% of emails are spam, both training and testing sets keep about 30% spam. This is helpful when some classes are rare so they appear fairly in both sets.

train_test_split(X, y, test_size=0.2,
    stratify=y, random_state=42)

The model never saw winter patterns during training, so it cannot generalize to conditions it has not learned. Its training data does not represent the full range of real situations. Including data from all seasons would make it more reliable year-round.

Errors, duplicates, or mislabeled examples can confuse the model and produce misleading test results. Cleaning first ensures both sets contain accurate, useful information. Good results depend on good data, often summed up as 'garbage in, garbage out'.

Training accuracy shows how well the model fits the examples it learned from, while testing accuracy shows how well it handles new, unseen examples. Testing accuracy is the more honest measure of real performance. A big gap between them often signals overfitting.