Designing system prompts

5 min read · Updated 2026-08-10

The system prompt is the constitution of your prompt: it sets the rules that apply to every call, before the model ever sees a user message. Getting it right is the highest-leverage thing you can do, because a good system prompt makes every downstream request more consistent.

What belongs in the system prompt

Put the things that should never change per request:

• Role — who the model is acting as (“a support triage assistant,” “a senior SQL reviewer”).
• Hard rules — the constraints it must always obey, including refusals (“never output SQL that drops a table”).
• Tone and style — terse vs. friendly, reading level, language.
• Output contract — the exact format every response must take (often a JSON schema).

Everything that changes per call — the ticket, the document, the user’s question — goes in the user message, not the system prompt.

Structure it, don’t ramble

A system prompt is easier for the model to follow when it’s organized. A reliable pattern:

You are a support triage assistant for a SaaS product.

Rules:
- Classify priority as exactly one of: low, medium, high.
- If the ticket doesn't indicate a priority, use "unknown". Never guess.
- Be terse and factual. No greetings, no apologies.

Output:
Return only JSON matching:
{ "priority": "low|medium|high|unknown", "summary": string }

Short, ordered, and explicit beats a long paragraph every time.

State the negative space

Models behave well when you tell them what not to do and what to do when they can’t comply. “If the input is not a support ticket, return { "error": "not_a_ticket" }” prevents the model from improvising. Defining these edges in the system prompt is what makes behavior predictable across thousands of calls.

Keep it stable for caching and versioning

Two practical reasons to keep the system prompt clean and constant:

• Prompt caching. Providers can cache a stable prefix across calls, cutting latency and cost. A system prompt that changes every request can’t be cached.
• Versioning. When the system prompt is a fixed, named artifact, you can review changes, compare versions, and roll back. Treat it like code, not a string you tweak in place.

Don’t overload it

A system prompt that tries to encode a five-step workflow is a sign the task should be a flow — several governed prompts chained together — rather than one mega-prompt. Each step is then easier to write, test, and evaluate.

Checklist

• Role, rules, tone, and output contract are all present and separated.
• There’s defined behavior for invalid or out-of-scope input.
• No per-request data is baked into the system prompt.
• It’s stable enough to cache and version.
• It’s been tested on each target model (system-prompt adherence varies by model).