Temperature and Top-p Explained: Why the Same Prompt Gives Different Answers

📎 This article includes 1 downloadable practice file ↓

An LLM does not choose the next word directly. It gives every possible token a score, turns the scores into probabilities, then samples. Temperature and top-p control that last step.

A worked example

Suppose three candidate tokens have scores (logits) 2.0, 1.0 and 0.1. Divide by the temperature, then apply softmax:

Low temperature sharpens the distribution toward the top choice; high temperature flattens it so unlikely tokens appear more often. Temperature 0 means “always take the top token” (nearly deterministic).

import numpy as np
def softmax_t(logits, t):
    z = np.array(logits) / t
    e = np.exp(z - z.max())
    return (e / e.sum()).round(2)
print(softmax_t([2.0, 1.0, 0.1], 0.5), softmax_t([2.0, 1.0, 0.1], 2.0))

Top-p (nucleus sampling)

Top-p keeps only the smallest set of tokens whose probabilities add up to p, then samples among them. With top_p = 0.9 in the default row above, A + B = 90%, so C is never chosen. It cuts off the long tail of weird tokens while still allowing variety.

Which settings to use

We implement sampling ourselves in Lesson 8 of the free Build Your Own LLM course.