Learning by rewards (RL made simple)

Reinforcement learning is a type of machine learning where an agent learns by interacting with an environment and receiving rewards or penalties for its actions. Instead of being given labeled correct answers, the agent discovers the best behavior through trial and error. Over many interactions, it learns to take actions that maximize its total reward.

An agent is the learner or decision-maker in a reinforcement learning system. It observes the current situation (state), chooses an action, and receives a reward based on that action. For example, an AI playing a video game is the agent — it decides every move.

The environment is everything the agent interacts with — it provides the agent with a current state, accepts the agent's actions, and sends back a reward and the next state. For a chess-playing AI, the chessboard and pieces form the environment. The environment sets the rules of the world the agent lives in.

A reward is a numerical signal the agent receives after taking an action, telling it how good or bad that action was. Positive rewards encourage the agent to repeat the action, while negative rewards (penalties) discourage it. The agent's overall goal is to maximize its total accumulated reward over time.

A state is a snapshot of the current situation that the agent observes before making a decision. For example, in a maze game, the state might include the agent's position, nearby walls, and remaining health. The agent uses the state to decide which action to take next.

A policy is the agent's strategy — a rule or function that tells the agent which action to take in each state. A good policy leads to actions that earn the most rewards over time. Policies can range from simple lookup tables to complex neural networks.

An episode is one complete run of the agent's experience, from a starting state all the way to a final (terminal) state — like one full game of chess or one attempt to navigate a maze. After each episode, the agent updates what it has learned and begins again. Thousands of episodes are often needed before the agent becomes skilled.

Exploration means the agent tries new, unknown actions to discover if they lead to better rewards, while exploitation means sticking with actions already known to give good rewards. Too much exploitation can cause the agent to miss better strategies it has never tried; too much exploration wastes time on actions that may be bad. A good RL agent must carefully balance both.

Q-learning is an algorithm where the agent learns a number called a Q-value for each (state, action) pair, estimating how much total future reward that action will produce from that state. By always choosing the action with the highest Q-value, the agent gradually learns the best strategy. Q-values are updated after every step using the actual reward received.

Because the agent starts with no instructions — it tries different actions, observes the reward it receives, and gradually keeps the strategies that work well. Over many trials, it learns which actions lead to the best outcomes, just like a person learning to ride a bicycle by trying, falling, and adjusting. The key is that feedback comes from experience, not from a teacher telling it the right answer.

Delayed rewards occur when the benefit of an action only appears much later, not immediately after the action. For example, in chess, sacrificing a piece early might lead to winning the game 20 moves later. The agent must learn to connect early actions to future outcomes, which is much harder than learning from instant feedback.

A robot is given a positive reward for moving forward and a negative reward (penalty) for falling over. Through thousands of simulation trials, it learns which leg movements keep it balanced and moving efficiently. This process mirrors how a baby learns to walk — through practice and natural feedback from falling.

DeepMind's AlphaGo learned in two stages: it first studied millions of human expert Go games using supervised learning to build a starting strategy, then used reinforcement learning through self-play — competing against copies of itself — to continuously improve beyond the level of any human. It received a positive reward for winning each game. By combining deep neural networks, supervised pre-training, and self-play RL, it defeated the world champion in 2016, which was considered impossible just a few years earlier.

A positive reward signals that an action was good and encourages the agent to repeat it, while a negative reward (penalty) signals that an action was bad and discourages it. For example, in a driving game, collecting fuel might give +10 points and crashing might give −100 points. The agent learns over time to seek positive rewards and avoid negative ones.

Reward hacking happens when an agent finds an unexpected loophole to earn high rewards without actually achieving the intended goal. For example, an RL agent in a racing game might learn to drive in circles to collect speed bonuses instead of completing the race. This shows why designing reward functions very carefully is critical — a poorly defined reward can produce clever but useless behavior.

The discount factor (written as γ, or gamma) controls how much the agent values future rewards relative to immediate ones. A discount factor close to 1 makes the agent plan far into the future, treating distant rewards nearly as valuable as immediate ones. A value close to 0 makes it focus mainly on the very next reward. This lets the agent handle uncertainty about events far in the future.

A value function estimates how good it is for the agent to be in a particular state by predicting the total future reward expected from that point onward. States with higher values are more desirable because they are expected to lead to more reward. The agent uses the value function to prefer better states when choosing actions.

Deep reinforcement learning (deep RL) combines reinforcement learning with deep neural networks. Instead of storing Q-values in a huge table (impossible for complex games), a neural network learns to predict Q-values directly from raw inputs like pixel images. This allowed AI to master Atari video games by looking at the screen just as a human would.

In epsilon-greedy, the agent picks a random action with probability ε (epsilon) to explore new possibilities, and picks the best-known action with probability 1−ε to exploit what it already knows. A high epsilon means more exploration; as training progresses, epsilon is often gradually reduced so the agent exploits its knowledge more. This balances trying new things with using what already works.

In MARL, multiple agents interact with the same environment and may cooperate or compete with each other — for example, two teams of AI playing soccer. This adds complexity because each agent's best strategy depends on what all other agents do, and those strategies keep changing as everyone learns. Examples include simulating traffic systems or training cooperative AI teams.

In RLHF, human raters compare different AI-generated responses and indicate which is better, creating a reward signal from human preferences. A reward model is trained on these preferences and then used to fine-tune the chatbot using RL. This helps align the AI's behavior with what humans actually want — making it more helpful, honest, and safe.

Sparse rewards occur when the agent only receives a reward signal very rarely — for example, only when it wins or loses, with no feedback in between. This makes learning very slow because the agent has almost no signal to guide its actions toward the goal. Techniques like reward shaping (adding intermediate rewards) or curriculum learning help address this challenge.

RL agents optimize for a reward function, but in combat scenarios it is extremely difficult to define a reward that captures all human values, laws of war, and ethical constraints. An agent might discover strategies that maximize the reward in unexpected and harmful ways — for example, harming civilians — if those outcomes are not explicitly penalized. Most ethics experts argue that life-or-death decisions should always require human oversight, not just an AI's reward signal.

Robots are often trained inside computer simulations because real-world training is slow, expensive, and risky. However, the simulated world is never perfectly accurate — differences in friction, lighting, and physical forces cause the robot to fail when moved to the real world. Researchers close this 'reality gap' by making simulations more realistic or by fine-tuning the agent on small amounts of real-world data.

A Markov Decision Process (MDP) is a mathematical framework that describes how an agent interacts with an environment: it defines states, actions, transition probabilities (what happens after each action), and rewards. The key 'Markov property' says the next state depends only on the current state and action — not on the full history of everything that happened before. Almost all RL algorithms are designed to find the best policy within an MDP.

Curriculum learning means starting the agent on easy tasks and gradually increasing the difficulty as it improves — just like how students learn basic arithmetic before calculus. This prevents the agent from being overwhelmed by hard problems where rewards are too rare to learn from at the start. It has helped train agents for complex games and robotic manipulation tasks.

Reward shaping means adding extra intermediate reward signals on top of the main reward to guide the agent more quickly toward the goal. For example, instead of only rewarding a robot for reaching a destination, you also give it small rewards for each step it moves closer. This speeds up learning significantly when the natural reward is sparse or very delayed.

In simulations, the agent can fail thousands of times without any real harm. But in the real world — such as with surgical robots or self-driving cars — a bad action can cause injury or property damage. Safe exploration methods restrict the agent to actions known to be safe while it is still learning, preventing dangerous mistakes during training.

Video games provide a clear numerical reward (the score), well-defined rules, fast simulation speed, and a safe environment where failures have no real-world consequences. Games like Atari, chess, and StarCraft have become standard benchmarks for comparing RL algorithms. Breakthroughs in games often inspire new techniques applied to real-world problems.

Self-play is when an RL agent trains by competing against copies of itself rather than against a fixed opponent or human player. As the agent improves, it always faces a challenging opponent — an equally skilled version of itself — which keeps driving improvement. AlphaZero used self-play to reach superhuman performance in chess and Go within days.

Yes — an RL agent can overfit by memorizing the specific details of its training environment instead of learning a general strategy. For example, it might memorize the exact layout of one training maze rather than learning general maze-solving skills. This means it performs well in training but fails when placed in a slightly different environment it has never seen.

RL agents for driving learn to steer, accelerate, and brake by receiving rewards for safe, smooth, and efficient driving and penalties for collisions or traffic violations. Training is usually done in realistic driving simulators before the agent is tested on real roads. The key challenge is making sure the agent handles all rare but dangerous edge cases, like a child suddenly running into the road.

In supervised learning, the model trains on a labeled dataset where every correct answer is already provided. In reinforcement learning, there are no labeled correct answers — the agent only receives a reward signal after taking an action and must figure out the best strategy entirely through experience. RL is especially suited for sequential decision-making tasks where collecting labeled data is impossible.

This code captures the core intuition of reinforcement learning: if an action produces a positive reward, keep doing it; if it does not, try something different. It is a simplified illustration of how real RL agents update their strategy based on feedback from the environment. Real algorithms like Q-learning are far more sophisticated, but this captures the essential idea of learning from rewards.

if reward > 0:
    keep_action()
else:
    try_different_action()