Message Roles: The Chat Protocol
Modern LLMs use a structured conversation format with three distinct roles. Each role shapes how the model interprets and responds to text.
System Message
Sets the model's persona, constraints, and behavior rules. Processed first and treated as high-priority instructions. Not shown to the end user in most UIs.
Controls: tone, format, boundaries, persona, output constraints
User Message
The human's input — questions, instructions, data to process. Each turn adds a new user message. This is the primary "prompt" in most interactions.
Contains: questions, instructions, context, data, examples
Assistant Message
The model's responses. In multi-turn conversations, prior assistant messages become part of the context. Can be "pre-filled" to steer output format.
Used for: conversation history, output steering, few-shot examples
Key insight: The model sees ALL messages as one long token sequence. The "roles" are just special tokens that help the model understand the structure. The system message is not a hard security boundary — at the token level it's just text. But modern models are explicitly trained to prioritize system/developer instructions over user input (OpenAI's "instruction hierarchy"). So it's privileged by training, not magic — and not bulletproof: injections can still override it.
What Makes a Good Prompt
Prompt engineering is about reducing ambiguity. The clearer and more structured your instructions, the more reliable the output.
Weak Prompt
Summarize this article about climate change.
Problems: no length guidance, no audience, no format, no focus area. The model has to guess everything.
Strong Prompt
Summarize this article in 3 bullet points for a non-technical executive audience. Focus on: economic impact, timeline, and policy recommendations. Use plain language, no jargon.
Clear on: length, audience, focus, format, and style. The model knows exactly what to produce.
The Six Principles of Effective Prompts
"Extract all dates and dollar amounts" beats "find the important stuff." Concrete instructions produce concrete results.
Tell the model who the audience is, what the output is for, and what domain you are working in. Context eliminates ambiguity.
"Return a JSON object with keys: name, date, amount" is far better than "give me the data." Show the model the shape you want.
A single input-output example is worth a paragraph of instructions. Show, don't just tell. This is the power of few-shot prompting.
"Do not include personal opinions." "Respond only with the answer, no explanations." Constraints narrow the output space.
"You are an experienced tax accountant" activates domain-relevant patterns. Role assignment primes the model's knowledge and tone.
Zero-Shot vs Few-Shot vs Chain-of-Thought
Three prompting strategies that trade complexity for reliability. Click each tab to compare them on the same task.
| Strategy | Examples Given | Reasoning Shown | Best When | Token Cost |
|---|---|---|---|---|
| Zero-shot | None | No | Simple, well-defined tasks | Low |
| Few-shot | 2-5 examples | No | Format matters, task is learnable from examples | Medium |
| Chain-of-thought | 0-3 + reasoning | Yes | Multi-step reasoning, math, logic | High |
Interactive: Prompt Playground
Edit the system prompt and user message to see how structure affects the model's output. This simulates structured output generation.
How this works in production: The system prompt defines the schema. The model is constrained (via JSON mode or function calling) to output valid JSON matching that schema. No free-text — just structured data your code can parse reliably.
Structured Output: JSON Mode & Function Calling
Going beyond free-text responses. Modern APIs let you constrain the model to output valid JSON or call predefined functions.
JSON Mode
Forces the model to output syntactically valid JSON — but not necessarily matching your schema (keys/types can still be wrong). To guarantee the schema, use Structured Outputs (json_schema with strict: true), which constrains decoding to your schema.
{
"model": "gpt-4o",
"response_format": { "type": "json_object" },
"messages": [
{
"role": "system",
"content": "Return JSON with: name, age, city"
},
{
"role": "user",
"content": "John is 30, lives in Seattle"
}
]
}
// Output (guaranteed valid JSON):
{
"name": "John",
"age": 30,
"city": "Seattle"
}
Function/Tool Definitions
Provide a JSON Schema that describes available functions. The model decides when to call them and fills in the arguments.
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City name"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"]
}
},
"required": ["location"]
}
}
}
Why Structured Output Matters
Free text output requires regex or NLP to parse. Brittle, breaks on format changes, impossible to validate programmatically.
Parse with json.loads(). JSON mode guarantees valid syntax; Structured Outputs additionally guarantees the schema (required fields, types). Integrates directly into data pipelines.
The model becomes an intent classifier + argument extractor. It decides which function to call and with what parameters. Your code executes the actual function.
Tool & Function Calling Architecture
The LLM does not execute tools — it generates structured requests that your code executes. The results flow back as context for the next generation.
The Tool-Use Loop
Click each step to follow a complete tool-calling interaction.
Full Architecture Flow
"What's the weather
in Tokyo?"
Generates tool call:
get_weather("Tokyo")
Executes the API call
weather_api.get("Tokyo")
{"temp": 22, "sky":
"partly_cloudy"}
"It's 22C and partly
cloudy in Tokyo"
What the LLM does
- Decides whether a tool is needed
- Selects which tool to call
- Fills in the arguments (from user input)
- Synthesizes the tool result into a natural answer
What your code does
- Defines tool schemas (name, description, parameters)
- Receives the LLM's tool call request
- Executes the actual function / API call
- Returns the result back to the LLM
Critical insight: The LLM never directly accesses databases, APIs, or the filesystem. It generates a structured JSON request describing what it wants. Your application code acts as the execution layer — with full control over validation, rate limiting, and security. This is why tool calling is fundamentally different from giving the model direct access to external systems.
Prompt Injection Attacks
When untrusted user input can override system instructions. The most important security concern in LLM applications.
Direct Injection
The user directly includes instructions that override the system prompt.
The model may comply because it sees the injection as a high-priority instruction, overriding the system prompt.
Indirect Injection
Malicious instructions are hidden in data the model processes (documents, web pages, emails).
More dangerous because the user may not even know the document contains injected instructions.
Defense Strategies
If the model can call tools, limit which tools it has access to. Don't give a customer service bot access to database deletion functions. This is the strongest structural defense.
Require explicit human approval before the model executes high-stakes or irreversible actions (sending email, purchases, deletes, code execution).
Check the model's output before acting on it. If it tries to call unexpected functions or produce forbidden content, block it at the application layer.
Input filtering ("ignore previous instructions"...) and delimiter separation raise the bar but are easily bypassed (paraphrasing, encoding, indirect injection, spoofed delimiters). Defense-in-depth only.
No complete fix: prompt injection cannot be fully eliminated at the LLM level — it's an architectural limitation. Rely on structural controls (least privilege, human approval), not on filtering the prompt.
Prompt Versioning & A/B Testing
Prompts are code. They should be versioned, tested, and evaluated systematically — not edited ad hoc in production.
The Prompt Lifecycle
Best Practices
When Prompts Are Not Enough
Prompt engineering is powerful but not unlimited. It is one control surface among several. The decision tree below helps determine when to escalate beyond prompting.
Interactive Decision Tree
Click a starting question to walk through the decision process.
The Intervention Ladder
| Level | Intervention | When to Use | Cost | Iteration Speed |
|---|---|---|---|---|
| 1. Prompt | Rewrite the prompt | Format, tone, simple behavior changes | Free | Minutes |
| 2. Few-shot | Add examples | Pattern matching, output format consistency | Free | Minutes |
| 3. RAG | Add retrieval | Knowledge gaps, freshness, grounding | Medium | Days |
| 4. Tools | Add function calling | Real-time data, calculations, external actions | Medium | Days |
| 5. Fine-tune | Train on your data | Persistent style, domain adaptation, latency | High | Weeks |
| 6. Custom model | Train from scratch | Unique modality, extreme specialization | Very high | Months |