Python, Pandas and NumPy

Chained indexing creates an intermediate object (often a copy), so the assignment may silently fail to modify the original and triggers a `SettingWithCopyWarning`. `df.loc[df.a > 0, 'b'] = 1` performs a single, atomic indexing operation on the original frame, guaranteeing the write lands on the source DataFrame. Under Copy-on-Write (Pandas 3.0 default), chained assignment never propagates to the original and raises a `ChainedAssignmentError`.

df.loc[df.a > 0, 'b'] = 1  # safe
df[df.a > 0]['b'] = 1     # SettingWithCopyWarning

Broadcasting aligns dimensions from the trailing axis; each size-1 dimension is virtually stretched to match the other operand. The result has shape (3,4) without materializing the expanded arrays in memory.

import numpy as np
(np.zeros((3,1)) + np.zeros((1,4))).shape  # (3,4)

`apply(axis=1)` invokes the Python function once per row, incurring per-row Python overhead and losing the benefit of NumPy's C-level loops. The fix is to express the logic with vectorized column operations, `np.where`, or `np.select`, which operate on whole arrays at once.

# slow
df.apply(lambda r: r.a + r.b, axis=1)
# fast
df['a'] + df['b']

`float32` uses 4 bytes with ~7 significant decimal digits, halving memory and often improving cache/SIMD throughput, but accumulating more rounding error. `float64` (8 bytes, ~15-16 digits) is the NumPy default and preferred when numerical precision matters, e.g. in reductions over large arrays.

`merge` defaults to joining on common column names (or explicit `on`/`left_on`/`right_on`) and `how='inner'`, whereas `DataFrame.join` aligns on the index by default and uses `how='left'`. Both can be made equivalent by specifying keys explicitly.

With `observed=True`, only category values actually present in the data form groups; with `observed=False` (the historical default), every defined category—even unused ones—produces a group, often yielding empty/NaN rows. As of Pandas 2.x the default for categorical group-bys was deprecated and flipped toward `observed=True`, so it matters for both correctness and performance.

`np.dot` dispatches to optimized BLAS routines (e.g., OpenBLAS/MKL) that exploit multithreading, SIMD, and cache blocking, giving orders-of-magnitude speedups over interpreted Python loops. It also avoids Python object overhead per element.

Copy-on-Write makes any DataFrame derived from another behave as if it were an independent copy, deferring the actual copy until a write occurs. This eliminates ambiguous view-vs-copy behavior, removes most `SettingWithCopyWarning` cases, and can reduce unnecessary copies for read-only chains.

`np.where` returns a new array selecting elementwise from `a` or `b` based on `cond`, leaving inputs unchanged. Boolean masking with assignment mutates the original array in place only where the mask is True, and does not provide an else-branch.

With `deep=True`, Pandas accounts for the actual memory held by Python objects in `object` dtype columns (e.g., the strings themselves), not just the 8-byte pointers. This is essential for diagnosing memory bloat from string columns that should be converted to `category` or the `string[pyarrow]` dtype.

IEEE 754 binary floating point cannot represent 0.1, 0.2, or 0.3 exactly, so rounding errors make the sum slightly off. Use `math.isclose` or `np.isclose(0.1+0.2, 0.3)` with appropriate tolerances instead of `==`.

`@` performs matrix multiplication (`np.matmul`). For two 1-D arrays it computes the inner product (a scalar); for higher dimensions it broadcasts over leading batch axes and contracts the last two dimensions, unlike `np.dot` which has different higher-dim semantics.

`usecols` avoids parsing/loading unneeded columns, and explicit `dtype` skips type inference and prevents accidental upcasting to `object` or `float64`. Together they cut memory and parse time substantially, and avoid the two-pass inference that can mis-detect integer columns containing NaN.

`Series.map` substitutes values using a dict/Series/function elementwise; `Series.apply` applies a function elementwise (with more flexibility for return types); `DataFrame.map` (which replaced the deprecated `applymap` in Pandas 2.1) applies a function to every scalar cell of a DataFrame. `DataFrame.apply` instead operates on whole rows or columns.

`np.nan` is unequal to everything including itself, and `np.mean` propagates NaN to return NaN if any element is NaN. `np.nanmean` ignores NaNs and averages the remaining values, which is what's usually wanted for dirty data.

`iterrows` yields each row as a Series, which forces a common dtype across mixed-type columns (often upcasting ints to float/object) and is slow. Prefer vectorization; if iteration is unavoidable use `itertuples()` (faster, namedtuples, dtype-preserving) or convert to NumPy arrays.

The -1 tells NumPy to infer that dimension so total size is preserved (here, 3 columns and inferred rows). Reshape returns a view when the data is contiguous and compatible with the new shape; if a view is impossible (e.g., due to non-contiguous strides), it returns a copy.

`how='any'` drops a row/column if it contains at least one NaN, while `how='all'` drops it only if every value is NaN. The axis and an optional `thresh` (minimum non-null count) further control which entries are removed.

`validate` checks the merge key relationship before joining and raises a `MergeError` if it is violated—e.g., `one_to_many` asserts keys are unique in the left frame but may repeat in the right. This catches unexpected duplicate keys that would otherwise silently cause a fan-out (row explosion).

It indicates whether the array's memory is laid out in C (row-major) contiguous order. Contiguity matters because many operations (vectorized loops, BLAS, reshaping to a view, `.ravel()` without copy) are faster or copy-free on contiguous data, and slicing can produce non-contiguous strided views.

`agg` reduces each group to a scalar (or several) producing one row per group; `transform` returns a result aligned to the original index with the same shape (broadcasting group results back to rows); `filter` keeps or drops entire groups based on a boolean group-level predicate.

df['z'] = df.groupby('k')['v'].transform('mean')  # group mean per row

Categorical dtype stores an integer code array plus a categories index; it saves memory only when the number of distinct values is small relative to the row count. For high-cardinality (nearly unique) columns it adds the overhead of the categories index without code reuse, and can even increase memory and slow some operations.

It computes standard matrix multiplication (sum over the shared index j). `einsum` advantages include expressing complex tensor contractions/reductions concisely in one call, often avoiding intermediate arrays and giving fine control over which axes are summed.

Providing an explicit format skips per-element format guessing, making parsing far faster and deterministic, and avoids ambiguous interpretations (e.g., day-vs-month order). In recent Pandas, mixed or inconsistent formats raise rather than silently mis-parse, so specifying the format prevents subtle date bugs.

Basic slicing returns a view sharing the same underlying buffer, so modifying `b` also changes the corresponding elements of `a`. To get an independent array you must call `a[2:5].copy()`; fancy/boolean indexing, by contrast, always returns a copy.

A sorted index lets Pandas use binary search (O(log n)) for label lookups and enables efficient range slicing on MultiIndex/datetime indexes, whereas an unsorted index falls back to slower scans and can raise `PerformanceWarning` or errors on partial slicing. It's a standard optimization for repeated label-based access.

The legacy global `seed`/functions share one mutable global state, which is hard to make reproducible across libraries and not thread-safe. `default_rng()` returns an independent `Generator` with a better algorithm (PCG64), explicit seeding, and is the recommended API for reproducible, isolated random streams.

rng = np.random.default_rng(42)
rng.normal(size=3)

`pivot_table` aggregates duplicate index/column combinations using `aggfunc` (and supports margins/fill_value), whereas `pivot` requires the index/column pairs to be unique and raises on duplicates. Use `pivot` for pure reshaping and `pivot_table` when aggregation is needed.

`axis=0` stacks rows (vertically) aligning on columns by label; `axis=1` stacks columns (horizontally) aligning on the row index. Non-matching labels produce NaNs unless `join='inner'` is used to keep only the intersection.

NumPy fixed-width integers wrap around (modular arithmetic) on array overflow rather than promoting to a bigger type, so uint8 255 + 1 becomes 0. This silent wraparound is a common source of bugs versus Python's arbitrary-precision ints (note: under NEP 50, overflowing pure-Python scalar operands can instead raise).

`Int64` is a nullable integer extension dtype that can hold `pd.NA` without upcasting the whole column to `float64` (which is what plain `int64` requires to store missing values). This preserves integer semantics in the presence of missing data.

`np.vectorize` is essentially a convenience wrapper that loops in Python under the hood, so it provides broadcasting/elementwise semantics but no C-level speedup. Real performance comes from native ufuncs/vectorized operations or compiled approaches (Numba, Cython).

With `method='min'`, tied values all receive the smallest rank in their group and the next distinct value skips ranks; with `method='dense'`, ties also share a rank but the next distinct value increments by exactly 1 (no gaps). Dense ranking yields contiguous rank labels.

Default arguments are evaluated once at function definition time, so a mutable default like a list is shared across all calls and retains mutations between invocations. The idiom is to default to `None` and create a fresh list inside the function body.

def f(x, acc=None):
    if acc is None: acc = []
    acc.append(x); return acc

`eval`/`query` parse a string expression and can use the `numexpr` engine to evaluate it without creating large intermediate temporaries, reducing memory traffic on big frames. They also allow concise, readable filtering, though for small frames the parsing overhead may outweigh the benefit.

`argsort` returns the indices that would sort the array, so `arr[np.argsort(arr)]` reproduces the sorted array. The index array can also reorder a parallel array consistently (e.g., sort one column and apply the same permutation to others).

`normalize=True` returns relative frequencies (proportions summing to ~1) instead of raw counts, while `dropna=False` includes NaN as its own category in the tally. They are independent options that can be combined.

Floating-point keys can differ by tiny rounding errors so seemingly equal values fail the exact equality merge requires, dropping rows unexpectedly. Safer practice is to merge on integer or string keys, or round/quantize the float keys to a fixed precision before merging.

`np.clip` limits each element to the range [lo, hi], replacing values below lo with lo and above hi with hi. It's commonly used to bound probabilities away from 0/1 before taking logs (numerical stability), cap outliers, or constrain gradients/activations.

p = np.clip(p, 1e-7, 1 - 1e-7)  # before log

Named aggregation gives explicit, flat output column names and avoids the deprecated/ambiguous dict-of-lists syntax that produced MultiIndex columns. Each output is defined as `new_name=(column, func)`, making multi-metric aggregations readable and deterministic.

df.groupby('k').agg(total=('v','sum'), avg=('v','mean'))