Python and data libraries

A NumPy ndarray stores elements of a single fixed type in contiguous memory, enabling fast vectorized operations in compiled C code, whereas a Python list stores heterogeneous objects via pointers and requires slow element-by-element Python loops. This makes ndarrays far more memory-efficient and faster for large numerical datasets.

Broadcasting lets NumPy perform arithmetic between arrays of different shapes by virtually stretching the smaller array across the larger one without copying data, as long as their dimensions are compatible. For example, adding a scalar or a 1D row vector to a 2D matrix applies it across all rows.

import numpy as np
a = np.array([[1,2,3],[4,5,6]])
b = np.array([10,20,30])
print(a + b)  # b broadcast across both rows

A Series is a one-dimensional labeled array (like a single column with an index), while a DataFrame is a two-dimensional labeled table made up of multiple Series sharing a common row index. Each column of a DataFrame is itself a Series.

'.loc[]' selects rows and columns by their labels (index names or column names), while '.iloc[]' selects by integer position (0-based). For example, df.loc['Mon'] uses the label 'Mon', whereas df.iloc[0] uses positional index 0.

df.loc['row_label', 'col_name']  # label-based
df.iloc[0, 2]                     # position-based

Vectorization expresses operations on whole arrays at once, delegating the looping to optimized, compiled C code rather than the slow Python interpreter. This greatly improves speed and produces cleaner, more readable code.

# Vectorized (fast):
result = a * b
# vs loop (slow):
# result = [a[i]*b[i] for i in range(len(a))]

It splits the DataFrame into groups based on the unique values in 'col', then computes the mean of the numeric columns within each group (non-numeric columns are excluded), and combines the results into a new DataFrame indexed by the group keys. This follows the split-apply-combine pattern.

df.groupby('city')['temperature'].mean()

plt.show() renders and displays the figure(s) you have built with plotting commands, opening a window (or rendering inline in notebooks). Until it is called, the plot is constructed in memory but not displayed.

import matplotlib.pyplot as plt
plt.plot([1,2,3],[4,5,6])
plt.show()

The .shape attribute returns a tuple giving the size of the array along each dimension (axis). For example, a 2D array with 3 rows and 4 columns has shape (3, 4).

import numpy as np
a = np.zeros((3,4))
print(a.shape)  # (3, 4)

NaN (Not a Number) represents missing data, arising from gaps in source data, failed conversions, or misaligned joins. df.dropna() removes rows (or columns) that contain any NaN values, returning a cleaned DataFrame.

df_clean = df.dropna()  # drop rows with any NaN

df.fillna(0) replaces missing (NaN) values with a specified value (here 0), keeping all rows, while df.dropna() removes the rows or columns that contain NaN. fillna preserves data size but may introduce bias; dropna loses information but keeps only complete records.

df.fillna(0)     # replace NaN with 0
df.dropna()      # remove rows with NaN

It produces a 1D NumPy array of evenly spaced values starting at 0, up to (but not including) 10, in steps of 2: array([0, 2, 4, 6, 8]). The stop value is exclusive, like Python's range.

import numpy as np
print(np.arange(0,10,2))  # [0 2 4 6 8]

Both create a 2x3 array, but np.zeros fills every element with 0.0 and np.ones fills every element with 1.0. They are commonly used to initialize arrays before filling them with computed values.

np.zeros((2,3))  # all 0.0
np.ones((2,3))   # all 1.0

Boolean indexing lets you select only the elements or rows that satisfy a condition by passing an array of True/False values. This enables concise, fast filtering, such as selecting all values greater than a threshold, without writing explicit loops.

a = np.array([1,5,2,8])
print(a[a > 3])  # [5 8]
df[df['age'] > 18]

df.head() returns the first 5 rows of the DataFrame by default (you can pass a number, e.g. head(10)). It is useful for quickly inspecting the structure and sample contents of a dataset without printing all of it.

df.head()    # first 5 rows
df.head(10)  # first 10 rows

df.describe() returns summary statistics for the numeric columns, including count, mean, standard deviation, minimum, maximum, and the 25th, 50th (median), and 75th percentiles. It gives a quick statistical overview of the data distribution.

df.describe()

reshape returns a new array with the same data arranged in the given shape, and when possible it returns a view that shares memory with the original (changes affect both). If the data cannot be laid out contiguously for the new shape, NumPy returns a copy instead.

a = np.arange(12)
b = a.reshape(3,4)  # often a view sharing memory

axis=0 applies the operation down the rows (collapsing rows to produce a result per column), while axis=1 applies it across the columns (collapsing columns to produce a result per row). So df.sum(axis=0) sums each column, and df.sum(axis=1) sums each row.

df.sum(axis=0)  # sum each column
df.sum(axis=1)  # sum each row

It returns a Series counting how many times each unique value appears in the column, sorted in descending order of frequency by default. It is commonly used to inspect the distribution of a categorical variable.

df['city'].value_counts()

It performs matrix multiplication between A and B following linear-algebra rules: the number of columns of A must equal the number of rows of B, producing a matrix of shape (rows of A, columns of B). This is distinct from element-wise multiplication (A * B).

import numpy as np
A = np.array([[1,2],[3,4]])
B = np.array([[5,6],[7,8]])
print(A @ B)  # matrix product

A * B multiplies corresponding elements at the same positions (requiring compatible or broadcastable shapes), whereas A @ B performs true matrix multiplication using rows-times-columns dot products. They generally give different results and have different shape requirements.

A * B   # element-wise
A @ B   # matrix product

The Index is the set of labels identifying the rows (and similarly columns) of a Series or DataFrame. It enables fast lookups, automatic data alignment during operations between objects, and meaningful label-based selection with .loc.

df.set_index('date')

Pandas automatically aligns the two Series on their index labels before computing, matching equal labels together. Labels present in only one Series produce NaN in the result because there is no corresponding value to combine.

s1 = pd.Series([1,2], index=['a','b'])
s2 = pd.Series([10,20], index=['b','c'])
print(s1 + s2)  # a:NaN, b:12, c:NaN

It sets the seed of NumPy's pseudo-random number generator so that the sequence of 'random' numbers produced afterward is reproducible. Running the same code with the same seed yields identical results, which is essential for reproducible experiments.

np.random.seed(42)
print(np.random.rand(3))  # same output every run

A histogram shows the distribution of a single numeric variable by grouping values into bins and showing the frequency in each bin, while a scatter plot shows the relationship between two numeric variables by plotting points at (x, y) positions. Histograms reveal distribution shape; scatter plots reveal correlation or patterns between variables.

plt.hist(data)              # distribution of one variable
plt.scatter(x, y)           # relationship of two variables

It joins two DataFrames by matching rows where the 'id' column values are equal, keeping only rows whose key appears in both DataFrames (an inner join). The result combines columns from both, similar to a SQL inner join.

df.merge(other, on='id', how='inner')

An inner merge keeps only keys present in both DataFrames, whereas a left merge keeps all rows from the left DataFrame and fills NaN for keys not found in the right one. Left joins preserve the full left dataset even when matches are missing.

df.merge(other, on='id', how='left')

apply() runs a given function on each element of a Series (or each row/column of a DataFrame) and returns the transformed results. It enables custom, flexible transformations beyond built-in vectorized operations, though it is usually slower than vectorized methods.

df['name_len'] = df['name'].apply(len)

Many ML algorithms (like gradient descent, k-NN, and SVMs) are sensitive to the scale of features, so a feature with a large range can dominate one with a small range. Scaling brings all features to a comparable range (e.g., 0 to 1), improving convergence and fairness across features.

x_scaled = (x - x.min()) / (x.max() - x.min())

It computes the mean along axis 0, meaning it averages down the rows for each column, producing one mean value per column. The result is a 1D array with length equal to the number of columns.

a = np.array([[1,2],[3,4]])
print(np.mean(a, axis=0))  # [2. 3.]

pd.concat stacks or glues DataFrames together along an axis (rows or columns) based on their position/index alignment, while pd.merge combines them by matching values in key column(s) like a database join. Use concat to append data and merge to relate data on keys.

pd.concat([df1, df2])           # stack rows
df1.merge(df2, on='id')         # join on key

They add descriptive text to a plot: plt.xlabel() and plt.ylabel() label the horizontal and vertical axes, and plt.title() adds a heading above the chart. These labels make the visualization interpretable and communicate what the data represents.

plt.xlabel('Year')
plt.ylabel('Sales')
plt.title('Annual Sales')

df.isnull() produces a boolean DataFrame marking each cell as True if it is missing (NaN). Calling .sum() on it counts the True values per column, giving the number of missing values in each column, which is useful for assessing data quality.

df.isnull().sum()

int64 stores whole numbers and float64 stores decimal numbers, both as efficient fixed-size numeric types, while object usually stores Python objects such as strings (text) or mixed types. Knowing column dtypes matters because numeric operations and memory usage depend on them.

df.dtypes

It returns the elements at indices 1, 2, and 3 (the stop index 4 is excluded). Importantly, basic slicing in NumPy returns a view that shares memory with the original array, so modifying the slice also modifies the original.

a = np.array([0,10,20,30,40])
s = a[1:4]   # [10 20 30], a view
s[0] = 99    # also changes a[1]

It returns a new DataFrame with the rows reordered by the 'score' column from highest to lowest (descending). The original DataFrame is unchanged unless inplace=True is used.

df.sort_values(by='score', ascending=False)

Seaborn is built on top of matplotlib and provides high-level functions for attractive statistical visualizations (like box plots, heatmaps, and regression plots) with concise code and sensible default styling. It also integrates directly with pandas DataFrames, simplifying common exploratory plots.

import seaborn as sns
sns.heatmap(df.corr(), annot=True)

df.corr() computes the pairwise Pearson correlation coefficients between the numeric columns, measuring the strength and direction of their linear relationships. Values range from -1 (perfect negative) through 0 (no linear relationship) to +1 (perfect positive).

df.corr()

a ** 2 squares every element in a single fast operation executed in compiled C, while a Python loop processes elements one at a time through the interpreter. For large arrays the vectorized version is typically many times faster and more concise.

a = np.arange(1_000_000)
squares = a ** 2  # fast, vectorized

df['col'] selects a single column and returns it as a Series (1D), while df[['col']] passes a list of column names and returns a DataFrame (2D) containing those column(s). The double brackets are needed to keep a DataFrame structure or select multiple columns.

s  = df['col']     # Series
sub = df[['col']]  # DataFrame

It groups the rows by the 'class' column and, for the 'score' column within each group, computes both the mean and the maximum. The result is a DataFrame indexed by class with two columns, 'mean' and 'max'.

df.groupby('class')['score'].agg(['mean','max'])