You make your first call to a model API, and there it is: a wall of parameters — temperature, top_p, max_tokens, frequency_penalty, stop, seed… Most people touch temperature, pray, and leave the rest to chance. A shame: each knob does one precise thing, and knowing which to turn changes everything.
Today’s image: a mixing console. You’re the sound engineer, each fader and dial has a role, and the skill is knowing which to push for the effect you want — not pushing them all to the max. Let’s walk the console, group by group. You’ll see: it’s not rocket science.
The “randomness” group: temperature and top_p
The two best-known faders — and the most misused. They dose the randomness in the choice of each word (the real mechanism — softmax, nucleus and friends — is a specialists’ affair; today we stay practical):
temperature: the width of the draw. Low (→ 0), the model almost always takes the most probable word — sober, repetitive, reliable. High (→ 1 or 2 depending on the provider), it dares improbable choices — creative, diverse, adventurous. It’s the knob the hallucinations article will come back to in a few days.top_p(nucleus): instead of dosing randomness, it cuts the tail of unlikely words — “keep only the candidates weighing p%”. At 0.1, ultra-restrictive; at 1, anything goes.
The golden rule nobody states: don’t push both at once. They act on the same distribution, and combining them makes behavior unpredictable. Pick one: temperature for most cases, top_p if you want to firmly bound the drift. Leave the other at its default.
(On open models, two cousins appear: top_k — “keep the k best candidates” — and min_p — “nothing below X% of the favorite”. Same family, finer tuning.)
Enough theory — turn the knobs yourself. The widget below runs the real sampling formula (softmax + nucleus top-p) on four toy distributions. Pick a scenario, adjust the sliders — and above all, the most telling trick: keep the same settings while switching scenarios. You’ll watch the same temperature flatten an open story while leaving an established fact perfectly intact.
The “length” group: max_tokens
A single fader, but a classic trap. max_tokens (sometimes max_completion_tokens) caps the length of the response — output tokens, the most expensive. Two things you absolutely must know:
- It truncates, it doesn’t summarize. Hitting the limit cuts the response mid-sentence — it’s not “be shorter”, it’s “stop dead”. For shorter, ask in the prompt; the parameter is a seatbelt, not a style instruction.
- Reasoning models count their thinking in it. On those models, the invisible “thinking” tokens consume the budget before the visible answer even starts — cap too low, and you get… nothing. Budget generously.
The “repetition” group: frequency and presence
Two dials to fight the model that rambles — useful in long generation:
| Knob | What it penalizes | The effect |
|---|---|---|
frequency_penalty |
words already used often | breaks literal repetitions (“very very very”) |
presence_penalty |
words that already appeared (even once) | pushes toward new topics, more variety |
Typical values from -2 to 2. A slight positive (0.3–0.6) usually suffices; too high, the model forbids itself necessary words and gets weird. Leave at zero by default, touch only if you see the rambling.
The “control” group: stop, seed, n
The knobs that frame the output without touching the style:
stop(orstop_sequences): strings that cut generation as soon as they appear. Essential when you generate homemade structure (“stop at###”) or a single line in a dialogue.seed: a seed to attempt to reproduce an output. The important word is attempt — it’s never guaranteed across a fleet of GPUs. Useful in development to replay a case, not a promise of determinism.n: request several answers in one call — handy for picking the best or exploring variants (beware, it multiplies the output bill).logit_bias: force or ban specific words. Powerful, surgical, rarely needed — the expert’s knob.
The “structure & thinking” group
The most modern settings on the console:
response_format/ structured outputs: impose a JSON schema. It’s not a style setting but a contract that constrains generation — valid JSON by construction. The right reflex for anything feeding a screen or an import.- Reasoning effort (
reasoning_effortor a thinking budget depending on the provider): on reasoning models, dosing how much the model “thinks” before answering. Low = fast and cheap; high = slower but better on hard problems. It’s the same logic as choosing the right model, but for compute intensity.
The recipes: which setting for which use
The table to keep handy — a starting point, not a law:
| Use | temperature | The rest |
|---|---|---|
| Factual, extraction, classification | 0 – 0.3 | response_format if structured output |
| Code | 0 – 0.2 | large max_tokens (reasoning included) |
| Writing, rewriting | 0.6 – 0.8 | slight frequency_penalty if repetitive |
| Brainstorm, ideation | 0.9 – 1.2 | n > 1 to vary the angles |
| Chat / assistant | 0.5 – 0.7 | stop on the speaking turn |
The word of honesty
- Names and ranges vary by provider. OpenAI, Anthropic, open models don’t expose exactly the same knobs or bounds (temperature up to 2 for one, 1 for another;
stopvsstop_sequences…). Always check your provider’s docs — this guide gives the map, not the exact territory. - The defaults are good. In the vast majority of cases, touch only the temperature. Cargo-culting settings (“I copied top_p 0.92 from a tutorial”) does more harm than good.
- Tune with evals, not by feel. A parameter that’s “better” over three tries might be luck. Measure before carving.
In summary
- An API call is a mixing console: each knob does one thing, the art is touching few.
- Randomness:
temperatureortop_p, never both. Length:max_tokenstruncates (doesn’t summarize) and includes reasoning. - Repetition:
frequency/presence_penaltyagainst rambling, dose lightly. Control:stop,seed(best-effort),n,logit_bias. - Structure:
response_formatfor guaranteed JSON; reasoning effort to dose the thinking. - And above all: defaults are fine, names vary by provider, and you tune with evals.
The console has many knobs, but a good mix pushes three, not thirty. Start with temperature, add a fader when a precise need calls for it, and measure. And that, honestly… is not rocket science.
