Cheat sheet · Data & MLOps

Advanced Pandas cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

Pandas is the de-facto Python library for tabular, labeled data, built on NumPy (with optional Apache Arrow backends) and a columnar memory model. This cheat sheet covers the data model, vectorized manipulation, joins/reshaping, groupby internals, time series, and performance — the areas interviewers actually probe — and reflects pandas 2.x/3.0 (2026) behavior.

Core Data Model & Selection

Two primary objects: Series (1-D, labeled) and DataFrame (2-D, dict-of-Series sharing one Index). The Index is a first-class, possibly non-unique, hashable axis — not just row numbers.
Selection: use .loc for label-based and .iloc for positional access. Avoid chained indexing (df[mask][col] = ...) — it may write to a temporary copy; under Copy-on-Write (default in pandas 3.0) such writes never propagate, so always assign via a single .loc[mask, col] = ...
Boolean masking is the idiomatic filter: df[df['x'] > 0]; combine conditions with & | ~ and parenthesize each one. Use .query('x > 0 and y < 5') for readability on large expressions.
dtypes matter: prefer nullable dtypes (Int64, Float64, boolean, string) and category for low-cardinality columns to cut memory and enable proper NA handling.
object dtype usually means boxed Python objects (slow); diagnose with df.info(memory_usage='deep') and df.dtypes. Pandas 2.x+ adds first-class Arrow-backed dtypes via dtype_backend='pyarrow' (read_* functions or .convert_dtypes()).

Missing Data, Cleaning & Vectorization

Missing markers: legacy float NaN, NaT for datetimes, and pd.NA for nullable/Arrow dtypes. NaN != NaN — use .isna()/.notna(), never == np.nan.
Handle NA with dropna(subset=...), fillna(value=...), or interpolate(); use the ffill()/bfill() methods directly — fillna(method='pad'/'backfill') is removed in pandas 3.0.
Vectorize: operate on whole columns/NumPy arrays instead of iterrows/itertuples or apply. A vectorized op is often 10-100x faster than a Python-level loop.
Use .map (Series, elementwise via callable/dict), .apply (along an axis), and DataFrame.map (elementwise; .applymap is deprecated since 2.1). Prefer np.where / np.select / .clip for conditional logic.
String ops via the .str accessor (df['s'].str.lower().str.contains(...)) and datetimes via .dt; both are vectorized. duplicated()/drop_duplicates(subset=, keep=) for dedup.

GroupBy: Split-Apply-Combine

Model: split rows by key(s) -> apply a function per group -> combine results. df.groupby('k')['v'].agg(...) is the workhorse.
agg accepts named aggregations for clean output columns: .agg(avg=('v', 'mean'), n=('v', 'size')). It also takes a list of funcs or a per-column dict.
transform returns a result aligned to the original index (same shape) — ideal for group-wise normalization or filling: df['z'] = g.transform(lambda s: (s - s.mean()) / s.std()).
filter keeps/drops whole groups by a predicate; apply is the most flexible but slowest — reach for built-in agg/transform first.
observed=True is important when grouping on categoricals (avoids the Cartesian explosion of unused category combinations); in pandas 3.0 observed defaults to True. Use as_index=False or .reset_index() to flatten keys back to columns.

Merge, Join & Reshape

merge is SQL-style: how in {inner, left, right, outer, cross}; specify on/left_on/right_on. Use validate='m:1' (etc.) to assert key cardinality and catch silent fan-out duplicates.
Row-count surprises usually mean duplicate keys on one side causing a many-to-many blowup. Use indicator=True to audit which side rows came from.
join aligns on index (convenient for index-keyed frames); concat stacks along axis=0 (rows) or axis=1 (columns) and aligns on the other axis.
Long<->wide: pivot_table(index, columns, values, aggfunc) aggregates (handles duplicate keys); pivot does not (it raises on duplicates). melt goes wide->long; stack/unstack move between index and column levels.
MultiIndex enables hierarchical reshaping; select with .loc[(a, b), :], pd.IndexSlice, or .xs(). Flatten afterwards if downstream code expects single-level columns.

Time Series

Use a DatetimeIndex (pd.to_datetime, set as index) to unlock slicing by partial strings (df.loc['2026-01']) and time-aware ops.
resample is groupby-over-time: df.resample('D').sum() / .ohlc() / .agg(...). Distinguish downsample (aggregate) from upsample (introduces NA -> ffill/interpolate). Note pandas 2.2+ aliases: use 'ME'/'YE'/'h'/'min' ('M'/'Y'/'H'/'T' are deprecated).
Rolling/expanding windows: df.rolling(window=7, min_periods=1).mean(), .ewm(span=...) for exponential weighting; window can be an offset like '7D' for time-based windows (requires a sorted DatetimeIndex).
shift/diff/pct_change for lags and period-over-period change; combine with groupby for per-entity lags (g['v'].shift(1)).
Timezones: tz_localize to attach, tz_convert to translate; store/compute in UTC and convert for display to avoid DST bugs.

Performance & Scale

Memory: downcast numerics, use category for repeated strings, and Arrow-backed string dtype; df.memory_usage(deep=True) to measure.
Avoid growing a DataFrame in a loop (repeated concat is O(n^2)); collect rows in a list and build once, or use vectorized construction.
I/O: read only needed columns (usecols), set dtype on read, and prefer columnar formats (Parquet/Feather) over CSV for speed and type fidelity. read_csv(chunksize=) for out-of-core streaming.
Copy-on-Write (CoW) is the default in pandas 3.0 — it removes most SettingWithCopy ambiguity and avoids defensive copies via lazy copy semantics. On pandas 2.x enable it early with pd.set_option('mode.copy_on_write', True) to match 3.0 behavior.
When data exceeds memory or a single core: scale out with Polars, DuckDB (SQL over DataFrames/Parquet), or Dask rather than forcing pandas. eval()/query() can speed large arithmetic via numexpr.

Common pitfalls

Chained indexing (df[df.a > 0]['b'] = 1): the write may target a copy and silently no-op (always so under Copy-on-Write); assign with a single df.loc[df.a > 0, 'b'] = 1.
Comparing to NaN with == always returns False; use .isna()/.notna(). pd.NA propagates in comparisons and raises if forced into a Python bool.
Forgetting parentheses around boolean conditions: df[df.a > 0 & df.b < 1] misparses because & binds tighter than comparisons — write df[(df.a > 0) & (df.b < 1)].
merge silently exploding rows due to duplicate keys (many-to-many); always sanity-check shape and use validate=.
Using iterrows/apply for everything — often orders of magnitude slower than vectorized/NumPy operations.
Mutating a slice/view and expecting the original to change (or vice versa); under Copy-on-Write writes never propagate to a parent — be explicit and use .copy() when you need an independent frame.
inplace=True is rarely a performance win (it often still copies), breaks method chaining, and is being de-emphasized.
Leaving high-cardinality text as object dtype balloons memory; grouping categoricals without observed=True (pre-3.0) creates huge sparse outputs.
Mixing tz-naive and tz-aware datetimes, or assuming local time; leads to off-by-hours and DST errors.
Relying on automatic index alignment unawares: arithmetic between misaligned Series produces NaN where labels don't match.

Interview tips

.loc vs .iloc and the dangers of chained indexing / SettingWithCopyWarning (and how Copy-on-Write changes it) — a near-universal screening question.
Explain split-apply-combine and the difference between agg, transform, filter, and apply (shapes returned and when to use each).
Walk through merge types and how to detect/prevent duplicate-key row explosions (validate, indicator, shape checks).
Reshaping: melt vs pivot vs pivot_table (why pivot_table when keys repeat), and stack/unstack with MultiIndex.
Vectorization mindset: rewrite a loop/apply into vectorized or np.where/np.select form and reason about complexity.
Handling missing data correctly: NaN vs NaT vs pd.NA, nullable dtypes, and imputation strategies (and their leakage risks in ML — fit imputers on train only).
Time series mechanics: resample vs rolling, downsample vs upsample, lags via shift/diff/pct_change, timezone handling.
Performance/scale: how to reduce memory (category/Arrow), why CSV is slow vs Parquet, and when to graduate to Polars/DuckDB/Dask.
Awareness of pandas 2.x/3.0 changes: Arrow backend and Copy-on-Write-by-default semantics signal you are current (2026).

Read the full lesson Test yourself — 13 question quiz