Natural language processing basics

NLP stands for Natural Language Processing. Its core goal is to enable computers to understand, interpret, generate, and respond to human language in text or speech form in a useful way.

Tokenization is the process of breaking text into smaller units called tokens, such as words, subwords, or characters. It is usually the first step in a text pipeline because models operate on discrete units rather than raw strings.

text = "I love NLP"
tokens = text.split()
print(tokens)  # ['I', 'love', 'NLP']

Stop words are very common words such as 'the', 'is', 'and', and 'of' that often carry little distinguishing meaning. They are sometimes removed to reduce noise and dimensionality so models focus on more informative words, though removal is not always beneficial.

Stemming chops word endings using crude rules to get a root form (e.g., 'studies' to 'studi'), which may not be a real word. Lemmatization uses vocabulary and grammar rules to return the proper dictionary base form or lemma (e.g., 'studies' to 'study'), so it is more accurate but slower.

Bag of Words represents a text as a count or presence of each word from a vocabulary, ignoring grammar and word order. Each document becomes a vector of word frequencies, which is simple but loses information about sequence and context.

TF-IDF (Term Frequency-Inverse Document Frequency) weights a word by how often it appears in a document (TF) times how rare it is across all documents (IDF). It is useful because it boosts informative, distinctive words while down-weighting common words that appear everywhere.

A word embedding is a dense vector of real numbers that represents a word in a continuous space, where words with similar meanings have similar vectors. Unlike one-hot encoding, embeddings capture semantic relationships, so 'king' and 'queen' lie close together.

One-hot encoding creates a vector as long as the vocabulary with a single 1 and the rest 0s, so it becomes very high-dimensional and sparse. It also treats every word as equally different, capturing no similarity or meaning between words.

POS tagging is the task of labeling each word in a sentence with its grammatical category, such as noun, verb, adjective, or adverb. It helps downstream tasks like parsing, information extraction, and disambiguation of word meaning.

Named Entity Recognition is the task of identifying and classifying named entities in text, such as people, organizations, locations, dates, and amounts. For example, in 'Apple opened a store in Paris in 2020', NER tags 'Apple' as an organization, 'Paris' as a location, and '2020' as a date.

Sentiment analysis is the NLP task of determining the emotional tone or opinion expressed in text, typically classifying it as positive, negative, or neutral. It is widely used to analyze product reviews, social media posts, and customer feedback.

An n-gram is a contiguous sequence of n items (usually words or characters) from a text. For example, in 'I love NLP', the bigrams (2-grams) are 'I love' and 'love NLP'; n-grams help capture some local word-order context.

words = "I love NLP".split()
bigrams = list(zip(words, words[1:]))
print(bigrams)  # [('I','love'), ('love','NLP')]

A corpus is a large, structured collection of text (or speech) used to train and evaluate NLP models. Examples include collections of news articles, books, or tweets, and the quality and size of a corpus strongly affect model performance.

The vocabulary is the set of all unique tokens (usually words or subwords) that a model recognizes, often built from the training corpus. Tokens not in the vocabulary are out-of-vocabulary (OOV) and may be replaced by a special unknown token.

Cosine similarity measures the cosine of the angle between two vectors, capturing how similar their directions are regardless of magnitude. The value ranges from -1 to 1 in general, but for non-negative vectors like word counts it ranges from 0 to 1. In NLP it is commonly used to compare word or document embeddings to judge how semantically similar they are.

import numpy as np
a, b = np.array([1,0,1]), np.array([1,1,0])
cos = a.dot(b)/(np.linalg.norm(a)*np.linalg.norm(b))
print(round(cos,3))  # 0.5

Syntax refers to the rules of sentence structure and word arrangement (grammar), while semantics refers to the meaning conveyed by words and sentences. A sentence can be syntactically correct but semantically nonsensical, like 'Colorless green ideas sleep furiously'.

Ambiguity occurs when a word, phrase, or sentence has more than one possible meaning, such as 'bank' (river edge vs. financial institution). It challenges NLP because the system must use context to choose the correct interpretation, which is hard to do reliably.

Text normalization transforms text into a consistent standard form so variations are treated the same. Examples include lowercasing all letters and expanding contractions or removing punctuation, which reduce unnecessary variation before analysis.

s = "Hello, WORLD!"
print(s.lower())  # 'hello, world!'

Machine translation is the automatic conversion of text from one language to another. The Transformer architecture, introduced in 2017 with its attention mechanism, dramatically improved translation quality and underpins modern systems like neural translators.

Attention lets a model weigh the importance of different words in the input when producing each part of the output, so it can focus on the most relevant context. This helps capture long-range dependencies that older sequential models struggled with.

A language model assigns probabilities to sequences of words, essentially learning how likely a given sentence or next word is. This lets it predict or generate text, powering applications like autocomplete, chatbots, and translation.

An out-of-vocabulary word is one that does not appear in the model's known vocabulary, often because it is rare, new, or misspelled. Such words are typically mapped to a special unknown token or handled using subword tokenization.

Subword tokenization splits words into smaller frequent pieces, like 'unhappiness' into 'un', 'happi', 'ness'. It solves the out-of-vocabulary problem by letting the model represent rare or new words as combinations of known subword units while keeping the vocabulary manageable.

Word order carries meaning, since 'dog bites man' differs from 'man bites dog'. The Bag of Words model ignores word order entirely, treating text as an unordered collection of word counts, which is one of its key limitations.

Text classification is the task of assigning a predefined category or label to a piece of text. A common real-world example is spam detection, where emails are labeled as 'spam' or 'not spam' based on their content.

A feature vector is a numeric representation of text that a machine learning algorithm can process, since models cannot use raw strings directly. Methods like Bag of Words, TF-IDF, or embeddings convert text into these numeric vectors before classification or other tasks.

The distributional hypothesis states that words appearing in similar contexts tend to have similar meanings, often summarized as 'a word is known by the company it keeps'. Word embedding methods exploit this by learning vectors from the surrounding words in large corpora.

A chatbot typically understands user input through intent recognition and entity extraction (natural language understanding), decides on a response using dialog logic, and then produces a reply via natural language generation. This lets it interpret what the user wants and respond appropriately.

NLU focuses on a machine comprehending and extracting meaning from human language input, such as identifying intent or entities. NLG focuses on producing fluent, human-readable language as output, such as writing a summary or a chatbot reply.

Lowercasing collapses capitalization differences, which can erase signals like proper nouns or sentence-initial emphasis. For example, 'Apple' the company and 'apple' the fruit become identical, and acronyms like 'US' may be confused with the word 'us'.

Sarcasm uses positive-sounding words to express a negative meaning, like 'Great, my phone died again', so the literal words mislead the model. Because surface words conflict with the true sentiment, models that rely mainly on word polarity often misclassify sarcastic text.

A document-term matrix is a table where each row is a document, each column is a term from the vocabulary, and each cell holds a count or weight of that term in that document. It is the structured numeric form produced by methods like Bag of Words or TF-IDF.

Neural networks usually require inputs of equal length within a batch, but sentences vary in length. A padding token fills shorter sequences up to a fixed length so they can be processed together, and a mask is used so the model ignores the padded positions.

Models learn patterns from their training data, so a small or biased corpus leads to poor generalization and can reinforce stereotypes present in the text. Large, diverse corpora help the model handle varied language, dialects, and topics more accurately and reduce some biases.