Yes, and for a first pass they probably should. A developer can write twenty test cases in an afternoon. The expensive part is what comes after: running them on a schedule, storing the history, noticing the day the results move, and keeping the set current as the product changes.
An afternoon of a developer's time gets you a test file. What it does not get you is a scheduler, somewhere to keep months of results, a way to compare this week against last, alerts when something moves, and a report you can hand to a customer. Those are the parts that take weeks rather than hours, and they are the parts that quietly stop working when the person who built them moves on to something else.
So the question is not whether your developer can write tests. It is whether you want to own a small internal product forever.
Model providers run extensive evaluations before they ship. Those evaluations are about the model in general; they are not about your documents, your instructions, your rules or your customers. Nobody outside your business knows that your assistant must never quote a discount, and nobody else is checking whether it still refuses to.
That gap is the entire subject. The provider guarantees a component. What you deploy is a system, and the system is yours to check.
Build it yourself if you have developer time you are not otherwise using, if you enjoy owning the tooling, or if your testing needs are unusual enough that no general tool fits.
Buy it if your developer time is worth more on the product, if you want the history and the reports without maintaining them, or if you will need to show someone outside the business that you check. For reference, Aggi ARIA Shape B is $30 a month for up to four models and twenty runs a day, with a seven-day money-back guarantee. An hour of developer time usually costs more than a month of that, which is the comparison worth making honestly.
They test their model. They do not test your instructions, your documents, your rules or your customers' questions, because they do not know them. That part is yours.
A day or two to build something workable, then a recurring cost in attention: running it, keeping the history, and updating the set. The build is the part people estimate; the maintenance is the part that surprises them.
When you need the results kept and comparable over time, when you want to hand the report to someone else, or when your own hours are better spent on the product.
Point it at your own model, run the tests on demand or on a schedule, and get a score and a report you can hand to someone else. $30 a month for up to four models and twenty runs a day, with a seven-day money-back guarantee.