Data cleaning and preparation

Data cleaning is the process of finding and fixing problems in a dataset, such as wrong values, missing entries, or duplicates, so the data is accurate and ready to use. Clean data helps a computer learn correct patterns instead of being confused by mistakes. It is usually one of the first steps before training a model.

It means that if you feed a model bad or messy data, it will produce bad or unreliable results. A model can only learn from the examples it is given, so poor-quality input leads to poor-quality output. This is why cleaning data first is so important.

A missing value is a spot in the data where information should exist but is empty or unknown. For example, a survey row might have no age recorded for one person. Missing values must be handled, because many models cannot work with empty cells.

One way is to remove the rows or columns that have missing data, which is simple but can lose useful information. Another way is to fill in (impute) the missing value with something sensible, like the average, the most common value, or a placeholder. The best choice depends on how much data is missing and why.

A duplicate record is the same data row that appears more than once in a dataset. Duplicates can be a problem because they make some examples count too much, which can bias what the model learns. Removing duplicates helps each real example carry the right amount of importance.

An outlier is a value that is very different from most of the other values, like a person listed as 200 years old. Outliers can be real rare events or they can be mistakes. During cleaning, you check whether an outlier is a true value or an error to fix.

People type data in different ways, using different spellings, capitalization, or abbreviations. To a computer these look like different values even though they mean the same place. Standardizing them to one form is part of data cleaning so the model treats them as the same.

Standardizing means converting values into one consistent format, such as making all text lowercase or trimming extra spaces. This way "Apple", "apple ", and "APPLE" all become the same value. It prevents the computer from treating identical things as different.

text = "  Apple "
clean = text.strip().lower()  # 'apple'

Each row usually represents one example or record, like one student or one customer. Each column represents one feature or attribute, like name, age, or score. Understanding this layout helps you know what to clean and how.

A data type tells what kind of value is in a column, such as a number, text, or date. It matters because math only works on numbers, and a number stored as text (like "42") can cause errors. Cleaning often includes converting values to the correct data type.

If a number is stored as text, the computer cannot do math like adding or averaging with it. Converting it to a real number lets the model and calculations use it correctly. This kind of type conversion is a common cleaning step.

value = "42"
number = int(value)  # now you can do math
total = number + 8   # 50

Normalization changes numbers so they fall into a common range, such as 0 to 1, even if they started with very different sizes. This helps because a feature with huge values (like income) won't unfairly dominate a feature with small values (like age). It makes learning fairer across features.

x = 80
low, high = 0, 100
scaled = (x - low) / (high - low)  # 0.8

You train the model on the training set and then check it on the testing set, which it has not seen before. This shows whether the model truly learned patterns or just memorized examples. Keeping the sets separate gives an honest measure of performance.

A label is the correct answer attached to an example, such as marking an email as "spam" or "not spam". The model learns by comparing its guesses to these labels. Accurate labels are essential, so checking and fixing wrong labels is part of data preparation.

If examples have wrong labels, the model learns incorrect patterns and will make more mistakes. For instance, calling a cat photo a "dog" teaches the model the wrong thing. That is why reviewing and correcting labels is an important cleaning step.

Trimming whitespace means removing extra spaces, tabs, or line breaks from the start, end, or inside a value. Hidden spaces can make "cat" and "cat " look like two different values to a computer. Trimming keeps values consistent.

name = "  Maya  "
print(name.strip())  # 'Maya'

Deleting rows is easy, but if many rows are removed you can lose a lot of useful information and shrink the dataset too much. Sometimes the missing data even has a pattern that matters. So you should think about why values are missing before deciding to drop them.

Imputation means filling in missing values with estimated ones instead of leaving them empty. Common choices are the average (mean), the middle value (median), or the most frequent value. It lets you keep rows that would otherwise be unusable.

The mean (average) can be pulled far up or down by a few extreme values, so it may not represent typical data. The median is the middle value and is less affected by outliers. So when outliers exist, the median often gives a more realistic fill-in.

Consistent units means all values in a column use the same measurement, like all heights in centimeters instead of some in inches. Mixing units makes the numbers meaningless to compare and confuses the model. Converting everything to one unit is an important cleaning step.

A feature is one measurable piece of information about each example, like a person's height or a house's number of rooms. Features are the inputs the model uses to make predictions. Preparing good, clean features helps the model learn better.

Encoding turns categories (like "red", "green", "blue") into numbers, because most models work only with numbers. For example, you might assign red=0, green=1, blue=2, or use a one-hot column for each color. Without encoding, the model cannot use the text categories.

colors = {"red": 0, "green": 1, "blue": 2}
print(colors["green"])  # 1

Charts let you see your data visually, making strange values, gaps, or outliers easy to spot. For example, a scatter plot might reveal one point sitting far away from the rest. Visualizing data before modeling helps you catch problems you might miss in a table of numbers.

Biased data over-represents some groups or situations and under-represents others, so the model learns unfair patterns. For example, a face dataset with mostly one skin tone may work poorly for others. Checking that data fairly represents everyone is both a cleaning task and an ethical responsibility.

Data may contain private details like names, addresses, or health information that could harm people if leaked or misused. Ethical preparation includes removing or hiding (anonymizing) such details and only keeping what is needed. Protecting privacy keeps people safe and builds trust.

Anonymizing means removing or disguising information that could identify a specific person, such as names or ID numbers. This lets you use the data for learning while protecting people's privacy. It is an important ethical step when datasets involve real individuals.

A data dictionary is a document that explains each column, what it means, its units, and what values are allowed. It helps you understand the data correctly so you clean it the right way. Without it, you might misread a column and make mistakes.

The minimum and maximum quickly reveal impossible or suspicious values, such as a negative age or a test score of 500 out of 100. These extremes often point to errors that need fixing. It is a fast way to catch obvious data problems.

Validation means checking that values follow expected rules, like an age being a reasonable number. Values that break the rule are flagged as likely errors to investigate or fix. Validation rules help catch mistakes automatically during cleaning.

age = 150
if age < 0 or age > 120:
    print("Invalid age!")

Feature engineering is creating new, more useful features from existing data. For example, from a "date of birth" column you could compute an "age" column, which a model may use more easily. Good feature engineering can help a model learn better patterns.

Different datasets may use different column names, units, or value formats for the same thing, like "DOB" versus "birthdate". Before merging them you must align these so matching records line up correctly. Otherwise the combined data can be inconsistent or wrong.

Raw data is the original data as collected, which may contain errors, missing values, and inconsistencies. Clean or processed data has been checked and fixed so it is accurate and ready for analysis or training. Most projects move from raw to clean before any modeling.

Cleaning involves changing data, and sometimes a change is a mistake or you need to redo it differently. Keeping the original (raw) copy lets you go back and start over safely. It also lets others check what changes you made.