All notes

· Per Lilja

Write your evals before your prompts

Why an evaluation set is the most valuable artefact in any LLM project, and how to start one in an afternoon.

The fastest way to improve an LLM feature is not a better prompt. It is a way to know whether the prompt you just wrote is better.

Start small

Fifty real examples with a clear definition of a good answer will tell you more than a week of vibes-based testing. Grow the set every time a user reports a bad answer.

Make it run on every change

An eval that runs only when someone remembers is an eval that doesn’t run. Put it in CI next to your unit tests.

Placeholder article. Replace with your own writing.

Have a hard problem? Let’s talk.

A 30-minute intro call, no pitch deck. You describe the problem, I tell you honestly whether I’m the right person for it.