📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=ysPbXH0LpIE
Mastering Prompt Engineering: Lessons from Anthropic’s Applied AI Team
Discover how to transform vague AI responses into precise, reliable insights using real-world insurance claim analysis. Hannah and Christian from Anthropic reveal the iterative science behind effective prompting, moving from simple queries to sophisticated, structured systems.
Core Question: How can structural prompts and contextual grounding eliminate hallucinations and improve accuracy in complex visual and linguistic tasks?
Highlights
- The “Iterative Science” of prompt engineering: moving from vague guesses to factual certainty.
- Utilizing XML tags and Markdown to provide clear data boundaries for the model.
- The “Form-First” reasoning strategy: guiding Claude to analyze structured data before interpreting subjective sketches.
- Advanced output control techniques like prefilling responses and leveraging extended thinking for complex logic.
⏱️ Reading time: approx. 4 minutes · Saves you about 21 minutes vs. watching.
Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇
The Iterative Journey: From Skiing to Insurance
Why Basic Prompts Fail
Prompt engineering is an empirical science. It is the practice of communicating with a language model by providing clear instructions, context, and organizational structure to achieve a specific result.
When Christian and Hannah first fed a Swedish accident report into Claude with a minimal prompt, the model hallucinated. Because the street name sounded familiar in a recreational context, Claude assumed it was analyzing a skiing accident rather than a car crash. This “innocent error” happened because the model lacked a defined persona and situational awareness.
It’s not magic; it’s empirical science.
To fix this, the team moved to “Version 2,” where they defined the assistant’s role. By explicitly stating, “You are an assistant for a car insurance claims adjuster,” they provided the necessary guardrails. They also established a tone of factual confidence: if the model isn’t sure, it should say so rather than guessing.
💡 Digging Deeper
Q: Why did Claude think the car accident was a skiing trip?
A: Without a system role, the model relied on its internal training data intuition for Swedish street names, which overlapped with popular ski areas, rather than the specific context of the images provided.
Q: What is the most important setting for these experiments?
A: The team recommends setting temperature to zero to ensure consistency and using a large token budget to avoid cutting off complex reasoning.
Structuring the Message
Using XML Tags for Data Clarity
Claude thrives on structure. While humans prefer conversational flow, models perform better when information is partitioned into clear, labeled segments.
The Anthropic team heavily recommends using XML tags—like <form_context> or <user_preferences>—to wrap different parts of the prompt. This allows the model to “lookup” specific information more efficiently.
In the insurance scenario, the Swedish form has 17 specific rows. Instead of letting Claude guess what each checkbox means, the team provided a detailed breakdown of the form’s architecture within the system prompt. This included explaining that “Vehicle A” is on the left and “Vehicle B” is on the right, and that humans might mark the form with circles or scribbles rather than perfect “X” marks.
Providing this structural context ahead of time drastically reduces the model’s cognitive load.
💡 Digging Deeper
Q: Why use XML tags instead of just bullet points?
A: XML tags provide unambiguous start and end points for data, making it easier for the model to distinguish between instructions, background documents, and the actual user query.
Q: How does storage or “Prompt Caching” play into this?
A: Since the structure of a standard insurance form never changes, this detailed background info is a perfect candidate for prompt caching, saving latency and cost on repeated queries.
Guiding the Reasoning Path
The Order of Analysis Matters
A critical breakthrough in the demonstration was controlling the order in which Claude processed information. Hannah pointed out that if a human looks at a messy accident sketch first, they might be confused. However, if they read the factual checkboxes on the form first, the sketch suddenly makes sense.
They instructed Claude to follow this exact sequence:
- Carefully examine the checkboxes in the form.
- List every marked item to establish a factual baseline.
- Compare the sketch against the established facts from the form.
- Reach a final verdict on fault.
By forcing the model to “show its work” inside tags like <analysis>, the team ensured that the final verdict—identifying “Vehicle B” as the party at fault—was based on logic rather than a lucky guess.
Advanced Controls and Output Formatting
Prefilling and Extended Thinking
For real-world applications, you rarely want a chatty AI; you want data you can pipe into a database. Christian demonstrated how to use “prefilled responses” to force Claude into a specific format. By starting the model’s response with a specific XML tag like <verdict>, you bypass the conversational preamble (“Here is my analysis…”) and get straight to the data.
Furthermore, with newer models like Claude 3.7, “extended thinking” allows the model to use a private scratchpad to deliberate over complex, gray-area claims. This is particularly useful for insurance cases that aren’t black and white.
It’s about putting words in the model’s mouth to get the JSON or XML you need.
Key Takeaways
The journey of prompt engineering is one of constant refinement. You start with the “what” (the task), move to the “who” (the persona), and eventually master the “how” (the structure). By treating the prompt as a piece of software architecture—using delimiters, specific examples, and logical sequencing—you move from unreliable AI interactions to robust, production-ready tools.
The most effective prompts aren’t just instructions; they are environments. By providing Claude with the “Swedish Form Dictionary” and a step-by-step reasoning protocol, the team turned a confused model into a reliable insurance assistant. Whether you are analyzing car crashes or legal documents, the principles remain the same: ground the model in context, structure the data, and guide the reasoning.
Q&A
Q1: What is the benefit of “few-shot” prompting in this context?
A: Providing examples (even encoding images in base64) of “difficult” cases allows Claude to understand the nuances of human error, such as messy handwriting or ambiguous sketches.
Q2: Should I always include the task description at the beginning?
A: Yes. Anthropic recommends defining the role and task first, followed by background data, and then specific instructions to ensure the model has the right mindset before processing data.
Q3: How do XML tags prevent hallucinations?
A: They help the model reference specific data points. If you tell the model to “refer to <document>,” it is less likely to pull information from its general training data.
Q4: What if the human sketch is completely unreadable?
A: The prompt should include an instruction allowing the model to be “uncertain.” Telling Claude “only answer if you are confident” prevents it from making up details about an illegible drawing.
Q5: Can I use Markdown instead of XML?
A: Markdown is helpful for readability, but XML tags are generally superior for programmatic parsing and giving the model very clear boundaries between different data types.
Q6: How does “prefilling” help with JSON output?
A: By starting the response with a curly bracket {, you ensure the model doesn’t include conversational text, making the output immediately serializable for your code.
