Write down the answers you expect before you test, then check the model against them. Cover three things: facts it should get right, requests it should refuse, and rules it must never break. Run that set before launch, and again on a schedule, because passing once is not passing next month.
This is the step people skip, and skipping it is what makes AI testing feel impossible. If you read an answer without knowing what you wanted, it will almost always sound reasonable; these systems are good at sounding reasonable. Deciding the right answer in advance turns a matter of taste into a matter of fact.
Twenty questions is a genuinely useful starting set for a small business. Forty is better. You do not need hundreds to learn something important in the first week.
Pass, fail, or borderline is enough to begin with. Count them, write the date, keep the file. The score matters far less than the trend; three fails this week against zero last week is a signal worth acting on, and no amount of precision in the scoring changes that.
One warning about grading with another AI model, which is a popular shortcut. It works reasonably well for clear-cut cases and poorly for the subtle ones, which are the cases you most want to catch. If you use it, spot-check its grading by hand.
Every complaint is a new test case. Every time you edit your documents or your instructions, the set should be run again. That is the whole discipline: a fixed set of expectations, run repeatedly, with the results kept.
It is unglamorous, and it is the difference between believing your assistant works and being able to show that it does.
Twenty is a real start, forty is comfortable, and the number matters less than running them repeatedly. Add one every time something goes wrong.
No. A spreadsheet and an hour a week will get you most of the value. Software helps when you want it to run on its own, keep the history, and tell you when the results move.
Test them explicitly and write down the refusal you expect. A model that is helpful about the wrong things is a more common problem than a model that is unhelpful.
Point it at your own model, run the tests on demand or on a schedule, and get a score and a report you can hand to someone else. Runs on demand or on a schedule you choose, from $30 a month.