Cheat sheet · Natural Language Processing

NLP Pipeline cheat sheet

The key ideas on one page — plus common pitfalls and interview tips. Skim it before an interview, or print it for revision.

A practical, interview-grade walkthrough of the modern NLP pipeline -- from text ingestion and tokenization through representation, modeling, evaluation, and production serving -- reflecting the LLM/transformer era (2026) while retaining the classical foundations interviewers still probe.

1. Pipeline Overview & Stages

Canonical flow: ingest/clean -> normalize -> tokenize -> represent (embeddings) -> model (classify/generate/extract) -> evaluate -> serve/monitor, with a feedback loop back to data.
Classical NLP front-loaded heavy preprocessing (stopword removal, stemming, lemmatization); transformer/LLM pipelines do far less -- subword tokenization on near-raw text is the norm.
Two dominant paradigms in 2026: fine-tuned encoder models (BERT/RoBERTa/DeBERTa-v3 family) for discriminative tasks, and decoder LLMs (GPT, Claude, Llama, Qwen, Gemini) for generative and few-shot/zero-shot tasks.
RAG (Retrieval-Augmented Generation) is a first-class pipeline shape: chunk -> embed -> index in a vector DB -> retrieve top-k (often + rerank) -> stuff into prompt -> generate.
Decide build-vs-prompt early: a frozen LLM behind a good prompt often beats a custom-trained model for low-volume, high-variety tasks; train/fine-tune when you have data volume, tight latency/cost budgets, or need a small specialist.
Design the eval + feedback loop as part of the pipeline, not an afterthought -- data drift and prompt/model regressions are the main production failure modes.

2. Text Preprocessing & Normalization

Unicode normalization (NFC/NFKC) first -- collapses visually identical glyphs and prevents tokenizer mismatches; critical for multilingual and user-generated text.
Task-dependent cleaning: HTML/markup removal, whitespace collapse, optional lowercasing/accent stripping -- but cased models (NER, multilingual) usually need case and diacritics preserved.
Stemming (Porter/Snowball, crude suffix chopping) vs lemmatization (dictionary + POS, returns a valid lemma): lemmatization is more accurate but slower; both are mostly obsolete for subword transformer models.
Stopword removal can help sparse bag-of-words/TF-IDF models but hurts transformers (they rely on function words for syntax) -- do not strip stopwords from transformer/LLM inputs.
Sentence segmentation and language detection matter for chunking and routing in multilingual or long-document pipelines (spaCy, pysbd, fastText langid).
Keep preprocessing deterministic and version-controlled -- train/serve skew from mismatched cleaning is a silent accuracy killer.

3. Tokenization & Vocabulary

Subword tokenization is the modern default: BPE (GPT), WordPiece (BERT, '##' prefix marks word continuations), Unigram/SentencePiece (T5, DeBERTa-v3, multilingual). Rare words split into known pieces, so there is no true OOV.
BPE iteratively merges the most frequent adjacent symbol pair until a target vocab size (greedy, frequency-driven). WordPiece instead picks the merge that maximizes training-corpus likelihood. Unigram starts large and prunes tokens by likelihood loss.
Special tokens carry structure: [CLS]/[SEP] for BERT pooling and pair tasks, <s>/</s>, <pad>, <unk>, plus chat templates (system/user/assistant roles) for instruct LLMs -- getting the chat template wrong silently degrades quality.
1 token is roughly ~4 chars / ~0.75 words in English; non-Latin scripts and code are far less efficient (more tokens per char), driving cost and context-window budgeting.
Token count drives context limits, API cost, latency, and truncation. Always measure with the model's own tokenizer; never approximate across model families.
Mismatched tokenizer and model checkpoint = garbage output; always load them as a paired unit from the same checkpoint.

4. Text Representation & Embeddings

Sparse classical: Bag-of-Words and TF-IDF -- still strong, fast, interpretable baselines for short-text classification with linear models or gradient boosting.
TF-IDF weights term t in doc d by its term frequency times an inverse document frequency that downweights terms common across the corpus (N docs, df = document frequency). Use sklearn's smoothed form to keep IDF non-negative.
Static embeddings (Word2Vec/GloVe/fastText) give one vector per word with no context; fastText adds char n-grams for OOV/morphology. Largely superseded but useful when compute is tight.
Contextual embeddings (BERT, sentence-transformers, hosted embedders like OpenAI/Cohere/Voyage) produce context-aware vectors; sentence-embedding models (SBERT, BGE, E5) are the workhorse for semantic search/RAG.
Similarity is almost always cosine; L2-normalize vectors so dot product equals cosine, then use ANN indexes (HNSW, IVF-PQ in FAISS / Qdrant / pgvector) for scale.
Pooling matters for encoders: mean-pooling token states usually beats the raw [CLS] vector for sentence similarity unless the model was trained with a CLS/contrastive objective.

5. Modeling: Classical, Transformers & LLMs

Core task types: classification (sentiment/intent/topic), sequence labeling (NER/POS via BIO tagging), seq2seq (translation/summarization), QA/extraction, and open-ended generation.
Baseline ladder: TF-IDF + Logistic Regression / linear SVM -> fine-tuned encoder (DeBERTa-v3, RoBERTa) -> LLM (zero/few-shot or fine-tuned). Always establish the cheap baseline first.
Transformer core is self-attention; each token attends to all others. Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V -- the 1/sqrt(d_k) scaling keeps softmax logits and gradients stable as dimension grows.
Fine-tuning in 2026 is usually parameter-efficient: LoRA/QLoRA adapt a small set of low-rank weights, enabling tuning large models on a single GPU; full fine-tuning is reserved for big budgets/specialists.
LLM prompting toolkit: zero-shot, few-shot exemplars, chain-of-thought for reasoning, structured/JSON output with schema constraints, and tool/function calling for grounded actions.
RAG vs fine-tuning: RAG for fresh/factual/private knowledge that changes often; fine-tuning for style, format, and task behavior. They compose -- RAG context plus a lightly tuned model is common.

6. Evaluation Metrics

Classification: precision, recall, F1 (use macro-F1 for class imbalance, weighted-F1 to reflect support); accuracy alone is misleading on skewed data.
F1 = 2PR/(P+R), the harmonic mean of precision and recall -- it punishes lopsided precision/recall.
Sequence labeling (NER): report entity-level (span-exact) F1 via seqeval, NOT token-level -- partial spans must be scored correctly.
Generation: BLEU/ROUGE/METEOR measure n-gram overlap (translation/summarization) but correlate weakly with quality; BERTScore adds semantic similarity. Perplexity measures LM fit, not task usefulness.
LLM-as-judge is standard for open-ended outputs (faithfulness, relevance, coherence) -- mitigate position/verbosity/self-preference bias with explicit rubrics, pairwise comparison, and randomized ordering.
RAG-specific: retrieval recall@k / MRR / nDCG for the retriever, plus faithfulness/groundedness and answer-relevance (RAGAS-style) for the end-to-end system.

7. Productionization, Serving & Monitoring

Serving choices: batch vs real-time; quantization (8-bit/4-bit, GPTQ/AWQ) and distillation to shrink models; vLLM/TGI/TensorRT-LLM for high-throughput LLM inference with paged KV-cache and continuous batching.
Latency/cost levers for LLMs: streaming tokens, prompt/KV caching, speculative decoding, shorter prompts, and routing easy queries to smaller models (cascades).
Guardrails: input validation, PII redaction, prompt-injection defense (especially in RAG over untrusted docs), output schema validation, and refusal/safety filters.
Reproducibility: pin tokenizer + model + preprocessing versions together; eliminate train/serve skew; log prompts, params, and seeds.
Monitoring: track data/embedding drift, latency/throughput/cost, hallucination and groundedness rates, and user feedback; auto-flag low-confidence or out-of-distribution inputs for review.
Continuous improvement loop: capture production traces -> build eval sets -> regression-test prompts/models in CI before shipping changes.

Common pitfalls

Over-preprocessing for transformers: stripping stopwords, lowercasing cased models, or aggressive stemming destroys signal that LLMs/transformers rely on.
Train/serve skew: applying different normalization or a different tokenizer at inference than at training -- a top silent accuracy killer.
Mixing a tokenizer with a mismatched model checkpoint, producing garbage embeddings/outputs.
Reporting raw accuracy on imbalanced data instead of macro-F1 / precision-recall, or doing token-level NER scoring instead of entity-span F1.
Trusting BLEU/ROUGE as quality measures for generation -- they reward surface overlap, not correctness or faithfulness.
Data leakage: fitting TF-IDF vectorizers or scalers on the full dataset before the train/test split, or letting duplicate documents straddle splits.
Ignoring tokenization cost/limits -- silent truncation drops the end of long inputs, and non-English text inflates token counts and cost.
Defaulting to fine-tuning when RAG (fresh knowledge) or simple prompting would solve it cheaper and faster.
Naive fixed-size RAG chunking that splits sentences/tables mid-thought, with no reranking -- kills retrieval quality.
Treating an LLM-as-judge score as ground truth without controlling for verbosity, position, and self-preference bias.

Interview tips

Walk the full pipeline end to end and justify which classical preprocessing steps you would SKIP for a transformer/LLM and why.
Explain subword tokenization (BPE vs WordPiece vs Unigram/SentencePiece) and why subwords eliminate the OOV problem.
Derive scaled dot-product attention and explain the role of the 1/sqrt(d_k) scaling factor.
Compare TF-IDF, static (Word2Vec/GloVe/fastText), and contextual (BERT/SBERT) embeddings -- and when each is the right tool.
Know when to choose prompting vs RAG vs fine-tuning (and PEFT/LoRA), with cost, latency, freshness, and data-volume tradeoffs.
Select metrics by task: macro-F1 for imbalance, entity-level F1 for NER, retrieval@k + faithfulness for RAG, and the limits of BLEU/ROUGE.
Show production awareness: quantization, vLLM/continuous batching, drift monitoring, prompt-injection guardrails, and avoiding train/serve skew.
Expect a system-design prompt (e.g. 'build a support-ticket classifier' or 'build a RAG chatbot over docs') -- structure the answer as data -> model -> eval -> serve -> monitor.

Read the full lesson