How Can a Generative LLM Classify?
LLMs are trained to generate text, but they can also be powerful classifiers. Instead of a dedicated classification head, they use instruction-following to map inputs to labels.
A generative LLM performs classification by being prompted to map an input into one label from a defined set. It uses its language understanding to produce the target class, often along with a justification or structured output.
Traditional Classifiers
Require labeled training data, a fixed label set, and retraining when labels change. Fast inference, predictable output format. Think logistic regression, BERT fine-tuned with a classification head.
LLM-Based Classification
Labels described in natural language. Works well when classes are nuanced, inputs are messy, or examples are scarce. Trade-off: slower and less stable unless you constrain the output carefully.
Zero-Shot vs Few-Shot Classification
Two prompting strategies that determine how much guidance the model gets before classifying.
Zero-Shot
Only label definitions or instructions are given. The model relies entirely on its pretraining knowledge. Works best when labels are intuitive and standard.
Best for Clear, well-known categories
Few-Shot
Provides a handful of examples showing how inputs map to classes. The examples act as a tiny on-the-fly training signal (in-context learning). Particularly helpful when labels are subtle, overlapping, or organization-specific.
Best for Ambiguous or custom taxonomies
Key insight: Few-shot examples help the model infer boundaries, edge cases, and formatting expectations more reliably. GPT-3 popularized this style of in-context learning, where performance can improve substantially with well-chosen examples (Brown et al., 2020). The quality of examples matters more than quantity.
When to Prompt vs Fine-Tune
The real classification decision is not only accuracy; it is how much label drift, ambiguity, scale, and governance the product can tolerate.
| Approach | Best When | Operational Note | Iteration Speed |
|---|---|---|---|
| Prompted LLM | Labels are evolving or nuanced | Easy to iterate, but cost and consistency need control | Minutes |
| Few-Shot LLM | Edge cases matter, examples improve framing | Helpful for pilot phases and policy-heavy tasks | Minutes |
| Fine-Tuned Classifier | Labels are stable and volume is high | Better throughput and cost once taxonomy settles | Days |
| Hybrid Approach | Need automation plus human escalation | Useful when uncertain cases must be triaged safely | Varies |
Decision Flow
Designing a Label Taxonomy
Many classification failures come from unclear class definitions rather than weak models. If humans cannot classify consistently, the LLM will not fix the ontology for you.
Bad Taxonomy
Semantically entangled labels cause the model to mirror that ambiguity.
Good Taxonomy
Mutually understandable, operationally useful, non-overlapping labels with clear inclusion/exclusion rules.
Taxonomy Design Checklist
Interactive: Classify Text with Confidence
Type any customer message and see how an LLM-style classifier would assign labels with confidence scores. This simulates keyword-and-pattern-based classification logic.
Try these examples:
Multi-Label vs Single-Label Classification
Multi-label is not just a small extension of single-label classification; it changes the decision structure.
Single-Label
"My order hasn't arrived"
Exactly one class must be chosen. Uses argmax or softmax over the label set. Clean, simple decision boundary.
Multi-Label
"My order hasn't arrived and I was charged twice"
Multiple labels may apply simultaneously. Each label has an independent inclusion threshold (sigmoid, not softmax). Requires stronger thresholding, validation, and audit logic.
| Aspect | Single-Label | Multi-Label |
|---|---|---|
| Decision | Pick the one best label | Decide which labels cross the threshold |
| Output function | Softmax (sums to 1) | Sigmoid per label (independent) |
| Failure mode | Forces a choice even when uncertain | Can under-tag or over-tag |
| Calibration | Simpler | Harder: each label needs its own threshold |
Metrics That Matter
Accuracy is a starting point, but precision, recall, F1, and confusion matrices are usually more informative. The best metric is the one aligned to the cost of being wrong.
Precision
"Of everything I labeled X, how many actually were X?"
Optimize when false positives are costly (e.g., spam filter blocking real email)
Recall
"Of everything that actually was X, how many did I catch?"
Optimize when false negatives are costly (e.g., fraud detection, medical triage)
F1 Score
"Harmonic mean of precision and recall"
Use macro-F1 in imbalanced settings for fairer view
Accuracy
"What fraction did I get right overall?"
Misleading when classes are imbalanced (99% accuracy by always predicting majority)
Interactive Confusion Matrix
Click cells to adjust counts. Watch how precision, recall, and F1 change in real-time. The matrix shows predicted vs actual labels for a 3-class classifier.
Click a cell to increment (+1). Right-click to decrement (-1).
Estimating Confidence
Raw verbal confidence statements from the model are not reliable enough to use as the only confidence signal. Measure confidence externally whenever possible.
Unreliable: Self-Reported
The model's stated confidence is often uncalibrated. It may say 95% for things it gets wrong 30% of the time. This is not a real probability.
Better: External Signals
Production pattern: Combine model scores, retrieval evidence, schema validity, and historical error patterns to decide when to auto-route versus escalate to human review. A confidence score below the threshold triggers escalation, not rejection.
Human-in-the-Loop Pipeline
Human review is a precision tool, not a sign of system weakness. A mature design routes easy cases automatically and reserves reviewer attention for the cases where it creates the most risk reduction.
When to include humans
Feedback loop: Human corrections on escalated cases become the most valuable training and evaluation data. Use them to retrain, update prompts, or refine the taxonomy.
Handling Class Imbalance
Imbalance is both a data problem and a decision-policy problem. The right evaluation metric should reflect the priority of catching minority classes.
Interactive: See the imbalance problem
Mitigation Strategies
Common Production Failure Modes
Classification quality depends on prompts, data definitions, evaluation sets, routing policies, and review loops. If you monitor only a single accuracy number, you will miss the real reasons the system is failing.
Label Drift
Taxonomy changes are made upstream but the classifier prompt or training data is not updated. The model keeps predicting against the old schema. Especially dangerous in silent failures where the system returns a valid-looking but wrong label.
Prompt Brittleness
Small changes to prompt wording cause large shifts in classification behavior. "Categorize this" vs "Label this" should be equivalent but may not be. Version-control your prompts and run regression tests.
Hidden Format Errors
The model returns "billing_issue" instead of "Billing Issue", or wraps the label in quotes, or adds explanation text. Schema enforcement (JSON mode, structured outputs) is essential for downstream parsing.
False Confidence on Ambiguity
The model produces a crisp answer with high apparent confidence even when the input is genuinely ambiguous. It has no built-in mechanism to say "I'm not sure" unless the prompt explicitly enables abstention.
Minority Class Neglect
Prompt-only systems may overpredict broad majority classes unless the prompt explicitly describes minority cases. Rare but important categories (fraud, safety violations) get systematically under-detected.
Silent Upstream Changes
Preprocessing, retrieval, or data format changes alter the input the classifier sees without anyone updating the classifier itself. The system degrades gradually with no obvious trigger event.