Feature engineering

Feature engineering is the process of using domain knowledge to create, transform, or select input variables (features) that make machine learning models work better. It matters because the quality of features often influences model performance more than the choice of algorithm itself. Good features expose the underlying patterns in data so the model can learn them more easily.

Many algorithms compute distances or weighted sums, so features with larger numeric ranges can dominate those with smaller ranges, biasing the model. Scaling brings all features onto a comparable scale so each contributes fairly. For distance-based methods like k-NN, unscaled features can completely distort which points are considered 'nearest'; for gradient-based models like logistic regression, scaling also speeds up and stabilizes convergence.

Standardization rescales a feature to have mean 0 and standard deviation 1 using (x - mean) / std, and it is not bounded to a fixed range. Min-max normalization rescales values to a fixed range, usually [0, 1], using (x - min) / (max - min). Min-max is more sensitive to extreme minimum and maximum values, since a single outlier can compress all other values toward one end of the range.

z = (x - x.mean()) / x.std()
mm = (x - x.min()) / (x.max() - x.min())

One-hot encoding converts a categorical variable into several binary (0/1) columns, one per category, where exactly one column is 1 for each row. It is used for nominal categories that have no inherent order, because it avoids implying a false ranking between categories. For example, 'Red', 'Green', 'Blue' become three separate indicator columns.

import pandas as pd
pd.get_dummies(df['color'], dtype=int)

Label encoding assigns ordered integers, which implies a numeric ordering and relative magnitude between categories that does not actually exist for nominal data. A model may wrongly interpret category 2 as 'greater than' category 1, distorting learned relationships. For unordered categories, one-hot encoding is usually safer.

Ordinal encoding is appropriate when a categorical variable has a meaningful, natural order, such as 'Low', 'Medium', 'High' or education levels. The integers then correctly preserve the ranking, which can help the model. It would be wrong for unordered categories like city names where no inherent order exists.

Feature selection chooses a subset of the original features and discards the rest, keeping the remaining features unchanged. Feature extraction creates new features by combining or transforming the originals, such as with PCA. Selection keeps interpretability of original variables, while extraction may produce more compact but less interpretable features.

PCA is a dimensionality reduction technique that transforms correlated features into a smaller set of uncorrelated components ranked by how much variance they capture. By keeping the top components, you reduce dimensionality while retaining most of the information, which can reduce noise and overfitting. The new components are linear combinations of the original features. PCA assumes features are scaled first, since it is sensitive to differing feature ranges.

from sklearn.decomposition import PCA
PCA(n_components=2).fit_transform(X)

A common strategy is imputation, such as filling missing values with the column's mean or median. The risk is that it can reduce the feature's variance and distort relationships, and the median is often safer than the mean when outliers are present. Adding a 'was missing' indicator column can preserve the information that a value was originally absent.

df['age'] = df['age'].fillna(df['age'].median())

Data leakage happens when information from outside the training data, often from the test set or the target itself, is used to create features, giving the model an unfair preview of the answer. It produces overly optimistic validation scores that collapse in real-world use. A common cause is scaling or imputing using statistics computed over the whole dataset before splitting.

If you compute scaling statistics using the whole dataset, information from the test set leaks into training, causing data leakage and overly optimistic results. The correct approach is to fit the scaler on the training data and then apply the same fitted transformation to the test data. This simulates how the model would handle genuinely unseen data.

scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

Binning converts a continuous numeric feature into discrete intervals or categories, such as turning ages into 'child', 'teen', 'adult', 'senior'. It can capture non-linear relationships, reduce the effect of small fluctuations or outliers, and make patterns more robust. The trade-off is some loss of fine-grained information.

pd.cut(df['age'], bins=[0,12,19,60,120], labels=['child','teen','adult','senior'])

A log transformation compresses large values more than small ones, reducing right-skew and making a distribution closer to normal. This can stabilize variance and lessen the influence of extreme outliers, helping models that assume roughly symmetric or linear relationships. It is commonly applied to features like income or population, and log1p is used so that zero values are handled safely.

import numpy as np
df['log_income'] = np.log1p(df['income'])

An interaction feature combines two or more existing features, often by multiplying them, to capture effects that depend on their joint values. It helps when the influence of one feature on the target changes depending on another feature. For example, the effect of 'room area' on house price may depend on 'location', so their product can be informative.

df['area_x_rooms'] = df['area'] * df['rooms']

A datetime value can be decomposed into components like year, month, day of week, hour, and flags such as 'is weekend' or 'is holiday'. These derived features expose cyclical and seasonal patterns that a raw timestamp hides. For cyclic features like hour or month, sine and cosine transforms can preserve their circular nature.

df['dow'] = df['date'].dt.dayofweek
df['is_weekend'] = df['dow'] >= 5

Cyclical features wrap around, so hour 23 and hour 0 are actually adjacent, but a plain numeric encoding treats them as far apart. Encoding the value as sin(2*pi*x/period) and cos(2*pi*x/period) places it on a circle, so the model sees that the end of the cycle is close to the start. This preserves the true cyclical distances.

df['hr_sin'] = np.sin(2*np.pi*df['hour']/24)
df['hr_cos'] = np.cos(2*np.pi*df['hour']/24)

The curse of dimensionality refers to problems that arise when the number of features grows very large, causing data to become sparse and distances between points to become less meaningful. This can hurt model performance and require exponentially more data. Feature engineering addresses it through selection and dimensionality reduction to keep only the most informative features.

One-hot encoding a high-cardinality feature creates a huge number of sparse columns, increasing memory use and the risk of overfitting. Alternatives include grouping rare categories into an 'Other' bucket, target/mean encoding, or learned embeddings. These keep dimensionality manageable while preserving useful information.

Target encoding replaces each category with a statistic of the target for that category, typically the mean target value. It is compact and can be powerful for high-cardinality features, but it risks data leakage and overfitting because it uses the target. To mitigate this, it should be computed using cross-validation folds or with smoothing.

Outliers can distort statistics like mean and standard deviation and dominate distance-based and linear models, skewing the learned relationships. A robust approach is to use the median and interquartile range, apply transformations like log, or clip/winsorize extreme values to a chosen percentile. Robust scaling uses the median and IQR instead of mean and std.

lo, hi = df['x'].quantile([0.01, 0.99])
df['x'] = df['x'].clip(lo, hi)

Polynomial features are new features created by raising existing features to powers or multiplying them, such as x^2, x^3, or x1*x2. They let linear models capture non-linear and interaction relationships in the data. However, high-degree polynomials greatly increase the number of features and can cause overfitting.

from sklearn.preprocessing import PolynomialFeatures
PolynomialFeatures(degree=2).fit_transform(X)

Highly correlated features carry overlapping information, which adds little new signal while increasing dimensionality. In linear models this causes multicollinearity, making coefficient estimates unstable and hard to interpret. Removing or combining redundant features can improve stability and interpretability.

Filter methods rank features using statistical measures (like correlation or chi-square) independent of any model. Wrapper methods evaluate subsets by actually training a model and measuring performance, which is accurate but computationally expensive. Embedded methods perform selection as part of model training, such as Lasso regularization shrinking some coefficients to zero.

L1 regularization adds a penalty proportional to the absolute value of the coefficients, which can shrink some coefficients exactly to zero. Features with zero coefficients are effectively removed from the model, so Lasso performs automatic feature selection during training. This is an example of an embedded selection method.

from sklearn.linear_model import Lasso
Lasso(alpha=0.1).fit(X, y)

Bag-of-words represents a piece of text as a vector of word counts (or presence indicators) across a fixed vocabulary, ignoring word order. Each column corresponds to a vocabulary word and each value is how many times it appears in the document. It is simple and effective but loses grammar, word order, and context.

from sklearn.feature_extraction.text import CountVectorizer
CountVectorizer().fit_transform(corpus)

TF-IDF weights each word by its term frequency in a document multiplied by its inverse document frequency across the corpus. This down-weights very common words (like 'the') that appear everywhere and up-weights words that are distinctive to specific documents. It therefore highlights more informative terms better than raw counts.

from sklearn.feature_extraction.text import TfidfVectorizer
TfidfVectorizer().fit_transform(corpus)

A feature is an input variable used to make predictions, such as a person's age or height. The target (label) is the output variable the model is trying to predict, such as whether a customer churns. Feature engineering focuses on improving the features so the model can better predict the target.

Domain knowledge helps you create features that capture meaningful real-world relationships that raw data may not express directly. For example, knowing that BMI = weight / height^2 lets you build a single informative feature instead of leaving weight and height separate. Such features encode expert insight that improves model performance and interpretability.

df['bmi'] = df['weight_kg'] / (df['height_m'] ** 2)

Fitting computes and stores the parameters needed for the transformation, such as the mean and standard deviation from the training data. Transforming applies those stored parameters to convert the data. You fit on the training set, then transform both training and test sets using the same learned parameters to avoid leakage.

scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)

Frequency (count) encoding replaces each category with how often it appears in the data, either as a raw count or a proportion. It is compact, useful for high-cardinality features, and can be informative when a category's commonness relates to the target. A limitation is that two different categories with the same frequency become indistinguishable.

freq = df['city'].value_counts()
df['city_freq'] = df['city'].map(freq)

A model expects inputs in exactly the same format and scale it was trained on, so the same transformations must be applied consistently to new data. If, for example, the training data was standardized but production data is not, predictions become unreliable. Using a saved pipeline ensures the identical steps and parameters are reused at prediction time.

from sklearn.pipeline import Pipeline

Feature hashing maps categories or words to a fixed number of columns using a hash function, instead of building an explicit vocabulary. Its advantage is a fixed, memory-efficient output size that handles very high-cardinality or streaming data without storing a growing dictionary. The downside is possible hash collisions where different inputs map to the same column.

A ratio or aggregate feature combines raw values into a more meaningful quantity, such as a rate, average, or proportion. For example, 'spending per transaction' (total spend / number of transactions) often captures behavior better than either raw number alone. Aggregates like a customer's average monthly purchases summarize patterns across many records into a single useful feature.

df['spend_per_txn'] = df['total_spend'] / df['num_transactions']

The test set is meant to estimate performance on truly unseen data, so using it to choose or build features leaks information and gives a falsely optimistic evaluation. All feature decisions and fitted parameters should come only from the training data (using cross-validation if needed). This keeps the final test score an honest measure of generalization.