Computer vision basics

A digital image is a 2D grid of small samples called pixels (picture elements), where each pixel stores a numeric value representing brightness or color at that location. In a grayscale image each pixel is a single intensity number (commonly 0-255), while in a color image each pixel holds several channel values. The image's resolution is given by its width and height in pixels.

Each pixel can take an integer value from 0 to 255, because 8 bits give 2^8 = 256 possible levels. A value of 0 represents pure black (no brightness) and 255 represents pure white (maximum brightness), with intermediate values being shades of gray. This range is a common convention for storing intensity efficiently in one byte.

An RGB image has three channels: Red, Green, and Blue, each typically an 8-bit intensity value. A pixel's color is formed by additive mixing of these three intensities, so equal high values of all three produce white and all zeros produce black. For example (255, 0, 0) is pure red and (255, 255, 0) is yellow.

Grayscale reduces each pixel from three color values to a single intensity, cutting the data and computation by about a factor of three. Many tasks such as edge detection, thresholding, or shape analysis depend mainly on brightness/structure rather than color, so grayscale preserves the needed information while simplifying processing. This makes algorithms faster and often more robust to color variations like lighting.

A color image is usually stored as a 3D array of shape (height, width, channels), for example (480, 640, 3). The first dimension is the number of rows (height), the second is the number of columns (width), and the third is the color channels. Note that OpenCV stores channels in BGR order rather than RGB by default.

import cv2
img = cv2.imread('photo.jpg')
print(img.shape)  # e.g. (480, 640, 3)

An image histogram is a plot showing how many pixels fall into each intensity level (for example, counts for values 0 to 255). It reveals the distribution of brightness in the image, indicating whether it is dark, bright, low-contrast, or high-contrast. A histogram bunched on the left means a dark image, while values spread across the range indicate good contrast.

Histogram equalization redistributes pixel intensities so they spread more uniformly across the full range, which increases the global contrast of the image. It is especially useful for images that are too dark or washed out, helping bring out detail in underexposed or overexposed regions. The transformation is based on the cumulative distribution function of the original intensities.

Convolution slides a small matrix called a kernel (or filter) over the image, and at each position computes a weighted sum of the overlapping pixel values to produce one output pixel. The kernel's weights determine the effect, such as blurring, sharpening, or edge detection. This local operation is the core building block of classical filtering and convolutional neural networks.

# A 3x3 averaging (blur) kernel
import numpy as np
kernel = np.ones((3,3)) / 9.0

A Gaussian blur smooths an image by replacing each pixel with a weighted average of its neighbors, where nearer pixels get higher weights from a Gaussian (bell-curve) function. This reduces high-frequency noise and small details. It is applied before edge detection because edge detectors are sensitive to noise, and smoothing prevents spurious edges caused by random pixel fluctuations.

An edge is a location where image intensity changes sharply, typically marking the boundary between objects or regions. It is identified by a large gradient (rate of change) of intensity, meaning neighboring pixels differ strongly. Edge detectors compute these intensity gradients to highlight such boundaries.

The Sobel operator is a pair of small convolution kernels that estimate the intensity gradient of an image in the horizontal (x) and vertical (y) directions. By combining these gradients, it highlights edges and can give both edge strength (magnitude) and direction (orientation). It is a classic, simple method for edge detection.

The Canny edge detector first smooths the image with a Gaussian filter to reduce noise, then computes intensity gradients (magnitude and direction). It applies non-maximum suppression to thin edges to single-pixel width, and finally uses double thresholding with hysteresis to keep strong edges and connected weak edges while discarding the rest. The result is clean, thin, well-connected edges.

Thresholding converts a grayscale image into a simpler form by comparing each pixel's intensity to a threshold value. In binary thresholding, pixels above the threshold are set to one value (e.g., white) and those below to another (e.g., black), producing a black-and-white image. It is commonly used to separate foreground objects from the background.

Otsu's method automatically chooses the optimal threshold value for binarizing a grayscale image, rather than requiring a human to pick it manually. It analyzes the image histogram and selects the threshold that minimizes the variance within the two pixel groups (or equivalently maximizes between-class variance). This works best when the histogram is bimodal, with clearly separated foreground and background.

Image segmentation is the task of partitioning an image into meaningful regions or segments, typically grouping pixels that belong to the same object or surface. It produces a label for regions or individual pixels, so that, for example, all road pixels share one label and all car pixels share another. It is more detailed than classification because it localizes objects at the pixel or region level.

Image classification assigns a single label to an entire image (e.g., 'cat'). Object detection both classifies and localizes multiple objects, usually drawing bounding boxes around each. Segmentation goes further by labeling individual pixels, giving the precise shape of each region or object rather than just a box.

A bounding box is a rectangle that tightly encloses a detected object, marking its location in the image. It is commonly represented by coordinates such as the top-left corner plus width and height (x, y, w, h), or by two corners (x_min, y_min, x_max, y_max). Detection models output one box per detected object along with a class label and confidence score.

IoU measures how well a predicted bounding box overlaps with the ground-truth box. It is computed as the area of their intersection divided by the area of their union, giving a value from 0 (no overlap) to 1 (perfect overlap). A prediction is usually counted as correct if its IoU with the true box exceeds a chosen threshold such as 0.5.

def iou(a, b):
    # boxes as (x1,y1,x2,y2)
    xi1, yi1 = max(a[0],b[0]), max(a[1],b[1])
    xi2, yi2 = min(a[2],b[2]), min(a[3],b[3])
    inter = max(0, xi2-xi1) * max(0, yi2-yi1)
    area_a = (a[2]-a[0])*(a[3]-a[1])
    area_b = (b[2]-b[0])*(b[3]-b[1])
    return inter / (area_a + area_b - inter)

CNNs use small filters that share weights across the whole image, dramatically reducing the number of parameters compared to fully connected layers that would need a separate weight for every pixel connection. This weight sharing also gives translation invariance, so a feature can be detected wherever it appears. Convolutions exploit the spatial structure of images, learning local patterns like edges and textures efficiently.

A pooling layer downsamples feature maps, reducing their spatial size to lower computation and memory while retaining the most important information. Max pooling, the most common type, takes the maximum value in each small region (e.g., 2x2), which keeps the strongest activations. This also provides some robustness to small shifts and distortions in the input.

Early convolutional layers tend to learn simple, low-level features such as edges, corners, and color blobs. Deeper layers combine these into more complex, higher-level features like textures, object parts, and eventually whole objects. This hierarchy of increasingly abstract features is a key reason CNNs are powerful for vision.

A feature map is the output produced when a filter is convolved across the input, with each value indicating how strongly that filter's pattern is present at a location. A convolutional layer with many filters produces many feature maps, each highlighting a different learned feature. Together they form the layer's representation of the image.

Padding adds extra border pixels (often zeros) around an image before convolution. Without padding, the output feature map shrinks after each convolution and border pixels get less attention. 'Same' padding adds just enough border so the output has the same spatial size as the input, preserving resolution through layers.

Stride is the step size by which the filter moves across the image between computations. A stride of 1 moves one pixel at a time, while a larger stride (e.g., 2) skips positions, producing a smaller output feature map. Larger strides downsample the data and reduce computation but capture less detail.

The output size for one dimension is (input - filter)/stride + 1 = (32 - 5)/1 + 1 = 28. Since both height and width are 32 and the filter is square, the output feature map is 28x28. With no padding the map shrinks by (filter size - 1) along each dimension.

out = (32 - 5) // 1 + 1  # = 28

Data augmentation artificially expands a training set by applying label-preserving transformations to images, such as flipping, rotating, cropping, scaling, or changing brightness. This exposes the model to more variation, helping it generalize better and reducing overfitting. It is especially valuable when labeled data is limited.

Transfer learning reuses a model already trained on a large dataset (such as ImageNet) and adapts it to a new, often smaller, task by fine-tuning or replacing its final layers. Because the pretrained model has already learned general visual features like edges and textures, it needs far less data and training time to reach good accuracy. This makes it practical when you lack a huge labeled dataset.

RGB describes color as amounts of red, green, and blue light, while HSV describes it as Hue (the color type), Saturation (its vividness), and Value (its brightness). HSV is often useful for color-based segmentation because it separates color information (hue) from lighting (value), making it easier to detect a specific color under varying illumination. For example, isolating a red object is simpler by selecting a hue range than by juggling three RGB values.

Resizing changes an image's pixel dimensions, using interpolation (e.g., nearest, bilinear, bicubic) to estimate new pixel values from existing ones. Downsampling to a smaller size discards pixels, so fine details and high-frequency content can be lost permanently and cannot be fully recovered by upscaling later. Upscaling, conversely, must invent plausible detail it does not truly have.

Salt-and-pepper noise appears as random isolated black and white pixels, while Gaussian noise adds small random variations to all pixels following a normal distribution. A median filter is best for salt-and-pepper noise because it replaces each pixel with the median of its neighborhood, removing extreme outliers without heavily blurring edges. Gaussian noise is usually better reduced with a Gaussian/averaging filter.

This weighted formula converts RGB to grayscale by accounting for the human eye's differing sensitivity to each color, being most sensitive to green and least to blue. A plain average would misrepresent perceived brightness, so green is weighted highest and blue lowest. The result, called luminance, better matches how bright a color appears to people.

gray = 0.299*R + 0.587*G + 0.114*B

Softmax converts the network's raw output scores (logits) into a probability distribution over the classes, where each value is between 0 and 1 and all values sum to 1. This lets the model express confidence per class, and the class with the highest probability is chosen as the prediction. It is the standard output activation for multi-class classification.

Overfitting occurs when a model learns the training images too specifically, including their noise, so it performs well on training data but poorly on new, unseen images. A common sign is a large gap where training accuracy is high but validation accuracy is much lower. Remedies include data augmentation, dropout, regularization, or gathering more training data.

OpenCV reads and stores images in BGR channel order by default, whereas most other libraries (like Matplotlib or PIL) and conventions use RGB. If you display or feed an OpenCV image into an RGB-expecting function without converting, the red and blue channels are swapped, producing wrong colors or degraded model accuracy. You convert with cv2.cvtColor using COLOR_BGR2RGB to fix this.

import cv2
rgb = cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB)