NLP basics: teaching computers language

NLP is a branch of AI that helps computers understand, read, and generate human language — the kind people use every day. It bridges the gap between how people communicate naturally and how machines process information. Examples include virtual assistants like Siri or Google Assistant that can answer spoken questions.

Tokenization is the process of splitting a sentence or paragraph into smaller pieces called tokens, which are usually words or punctuation marks. For example, a proper tokenizer turns 'I love NLP!' into ['I', 'love', 'NLP', '!']. This is almost always the very first step when a computer starts processing any text, because it needs individual units to work with.

sentence = "I love NLP!"
tokens = sentence.split()
print(tokens)  # ['I', 'love', 'NLP!']
# Note: a real tokenizer also splits punctuation:
# → ['I', 'love', 'NLP', '!']

Stop words are extremely common words like 'the', 'is', 'a', and 'in' that carry very little meaningful information on their own. NLP systems often remove them before analyzing text so the model can focus on words that actually reveal the topic or feeling of the text. For example, removing stop words from 'The cat sat on the mat' leaves 'cat sat mat', which still captures the key idea.

Sentiment analysis is an NLP technique that automatically figures out whether a piece of text expresses a positive, negative, or neutral feeling. Companies use it to scan thousands of customer reviews quickly without humans having to read each one individually. For example, the model would classify 'This movie was absolutely amazing!' as positive and 'Total waste of time' as negative.

A corpus (plural: corpora) is a large collection of text used to train or test NLP models. It could be millions of news articles, books, or web pages gathered together. The bigger and more varied the corpus, the more language patterns the model can learn, making it better at understanding new text it has never seen.

A spam filter reads the words and phrases in an email and decides whether it is junk or legitimate. It learns that phrases like 'You've won a prize!' or 'Click here now to claim' appear very often in spam. By training on thousands of labeled examples, it gets better and better at spotting unwanted messages without humans having to write every rule by hand.

Autocorrect uses NLP to detect misspelled words and suggest or automatically replace them with the most likely intended word. It compares what you typed against a dictionary and uses context clues from surrounding words to guess your meaning. For example, 'teh' gets corrected to 'the' because 'the' is far more common and fits naturally in most sentences.

Machine translation is the automatic conversion of text from one human language to another using a computer, like Google Translate does. Early systems worked by looking words up in a dictionary one by one, but modern systems use AI trained on millions of paired sentences in two languages. The model learns grammar, idioms, and phrasing patterns on its own from these examples.

A chatbot is a computer program that uses NLP to hold a conversation with a human through text or voice. Simple chatbots follow pre-written rules and scripts, while more advanced ones use language models trained on vast amounts of text to respond more naturally. They are widely used in customer support, education apps, and entertainment.

Raw text scraped from the internet often contains HTML tags, random symbols, inconsistent capitalization, and typos that would confuse a learning model. Cleaning removes or standardizes these issues so the model sees consistent, meaningful language. Without it, the model might treat 'Dog', 'dog', and 'DOG' as three entirely different and unrelated words.

Stemming reduces a word to its root form by chopping off common word endings. For example, 'running', 'runs', and 'runner' all become 'run'. This helps a model recognize that these different word forms share the same base meaning, reducing the vocabulary size it needs to handle. Stemming is fast but sometimes produces root forms that are not real dictionary words.

# stemming examples
# "playing" → "play"
# "played"  → "play"
# "happily" → "happili"  (not a real word!)

Both techniques reduce words to a base form, but lemmatization uses vocabulary knowledge and grammar rules to return the actual dictionary word called the lemma. Stemming simply chops off word endings and can produce non-words — for example, 'happiness' might become 'happi'. Lemmatization correctly reduces 'better' to 'good', while stemming would just give 'better'. Lemmatization is more accurate but slower to compute.

A Bag of Words represents a sentence by counting how many times each word appears, completely ignoring word order. So 'cat sat on mat' and 'mat on sat cat' produce the exact same BoW representation. While simple and fast, this model loses meaning that comes from word order — for example, it cannot distinguish 'dog bites man' from 'man bites dog'.

TF-IDF stands for Term Frequency–Inverse Document Frequency. TF measures how often a word appears in one specific document; IDF measures how rare the word is across all documents. These two scores are multiplied together, so a word gets a high score only when it appears frequently in one document but is rare across all others. Common filler words like 'the' score very low because they appear in every document.

NER is an NLP task where the model scans text and labels 'named entities' such as people's names, organizations, locations, and dates. For example, in 'Elon Musk founded Tesla in California', NER tags 'Elon Musk' as PERSON, 'Tesla' as ORGANIZATION, and 'California' as LOCATION. This lets computers automatically extract structured facts from unstructured text like news articles.

An N-gram is a sequence of N consecutive words from a piece of text. A bigram (2-gram) from 'I love pizza' produces the pairs ['I love', 'love pizza']. N-grams capture some word order and context that a simple word-count model misses. They are used in autocomplete systems, spell checkers, and simple language models to predict what word is likely to come next.

Text classification is the task of automatically assigning a predefined category or label to a piece of text. Common examples include spam detection (spam vs. not-spam), news topic labeling (sports, politics, technology), and sentiment analysis (positive or negative). The model learns by studying many labeled examples during training and then applies those patterns to categorize new, unseen text.

One-hot encoding represents each word as a long list of zeros with a single '1' placed at the position matching that word in the vocabulary list. If the vocabulary contains 10,000 words, every word becomes a list of 10,000 numbers — all zeros except for one. While easy to understand, it creates very large and memory-hungry vectors and does not capture any meaning or similarity between words.

# vocabulary: ["cat", "dog", "fish"]
# cat  = [1, 0, 0]
# dog  = [0, 1, 0]
# fish = [0, 0, 1]

A word embedding represents each word as a compact list of decimal numbers (a vector) where words with similar meanings end up with numerically similar vectors. Unlike one-hot encoding, 'king' and 'queen' will have vectors close together in this space because they appear in similar contexts during training. This lets computers recognize that words like 'car' and 'automobile' are related, something one-hot encoding completely misses.

A language model is an AI trained to predict the next word (or token) in a sequence of text. Given the phrase 'The sky is', it predicts that 'blue' is far more likely than 'purple' or 'heavy'. By training on enormous amounts of text, language models learn grammar rules, common facts, and writing styles. Modern large language models (LLMs) like GPT power today's chatbots and writing assistants.

POS tagging labels each word in a sentence with its grammatical role: noun, verb, adjective, adverb, preposition, and so on. In 'The quick fox jumps', tagging gives: The(DETERMINER) quick(ADJECTIVE) fox(NOUN) jumps(VERB). Understanding a word's grammatical role helps computers parse sentence structure correctly, which is important for tasks like question answering and machine translation.

Human language is extremely varied — the same idea can be expressed in countless different ways, with different accents, dialects, vocabulary choices, and sentence structures. A model trained on too little data will only recognize the exact patterns it has seen and fail on new sentences. More diverse training data means the model discovers broader patterns and can generalize to the huge range of real-world text it will encounter.

Lowercasing converts all text to lowercase so that 'Apple', 'apple', and 'APPLE' are treated as the same word. Without this step, a model would think they are three separate, unrelated words, wasting the learning it gets from seeing all three forms. It is one of the simplest yet most impactful preprocessing steps in any NLP pipeline.

text = "Hello World"
text = text.lower()
print(text)  # → "hello world"

Automatic text summarization is an NLP task where a model creates a shorter version of a long document that captures the key points. Extractive summarization picks the most important existing sentences directly from the original text; abstractive summarization generates brand-new shorter sentences that paraphrase the content. Search engines and news apps use summarization to show quick article previews.

Word2Vec is a neural network trained to create word embeddings by predicting words from their neighbors in a large text corpus. A famous and surprising result is that these embeddings support vector arithmetic that captures real-world meaning: 'king' − 'man' + 'woman' ≈ 'queen'. This shows the model learned genuine relational meaning — not just which words appear together — purely from reading text.

The attention mechanism lets a model decide which words in a sentence are most relevant when interpreting any particular word. For example, in 'The animal didn't cross the street because it was too tired', attention helps the model resolve that 'it' refers to 'animal' and not 'street'. Without attention, older models had to compress all the context of a long sentence into a single fixed-size vector, which caused information loss.

The Transformer is a neural network architecture introduced in 2017 that relies entirely on attention mechanisms rather than processing words one by one in sequence. Because it processes all words in a sentence simultaneously, it trains much faster and handles long-range dependencies better than older models. Models like BERT and GPT are both built on the Transformer architecture and power nearly all modern NLP applications.

Transfer learning means taking a model already trained on a huge general-purpose dataset and fine-tuning it on a smaller, specific task. For instance, a model pre-trained on billions of web pages already understands grammar, vocabulary, and facts; you can then teach it to classify medical reports using only a few thousand labeled examples instead of millions. This saves enormous amounts of time, computing power, and data collection effort.

BERT (Bidirectional Encoder Representations from Transformers) reads every sentence from both left to right AND right to left simultaneously, giving it rich two-way context for understanding meaning. GPT reads only from left to right and is optimized for generating new text word by word. Because of this, BERT excels at classification and question-answering tasks, while GPT excels at text generation and completion.

An NLP model is biased when it produces unfair or skewed results because its training data reflected human social stereotypes — for example, associating 'nurse' more with women and 'engineer' more with men. When such models are used in real products like hiring tools, loan applications, or content moderation, they can make decisions that unfairly disadvantage certain groups of people. Detecting, measuring, and reducing bias is a critical area of AI ethics research today.

Zero-shot classification means a model can correctly sort text into categories it was never directly trained on, without needing any labeled examples for those categories. It works by comparing the meaning of the text to the meaning of the category labels themselves, using language understanding built up during pre-training. For example, a model can classify a restaurant review as 'positive' or 'negative' even if it was never specifically trained on sentiment data.

Perplexity measures how 'surprised' a language model is when shown a new piece of text — a lower perplexity score means the model predicted the text well and understood the language. It is calculated from the average probability the model assigns to each word in a test set: a model that assigns high probabilities to the actual words will have low perplexity. Researchers use perplexity to compare different language models and track training progress.

Syntax refers to the grammatical structure and rules of a language — whether a sentence is put together correctly according to grammar. Semantics refers to the actual meaning of words and sentences. The famous example 'Colorless green ideas sleep furiously' is syntactically correct (it follows English grammar) but semantically nonsensical (the meaning is impossible). NLP systems need to understand both to accurately process human language.

Coreference resolution is the NLP task of figuring out when different words or phrases in a text refer to the same real-world entity. In the sentence 'Maria entered the room. She sat down immediately', the model must determine that 'She' refers to 'Maria' and not someone else. This is essential for comprehension-heavy tasks like summarization, question answering, and dialogue systems, where keeping track of who did what is critical.