The AI told a patient her claim was "likely covered." It wasn't. Nobody reviewed the message before it went out.

The situation

Why this assistant is one bad message away from a complaint

The clinic has four providers, a billing coordinator, and a front-desk team. They adopted an AI assistant six months ago to handle the volume: appointment reminders, balance follow-ups, claims status updates, and pre-visit prep messages. The assistant pulls from the practice management system and drafts outgoing messages that staff are supposed to review before sending.

In practice, the review step is a rubber stamp. Staff see a queue of 40 drafted messages at 4:55 p.m. and click "send all." The drafts are usually fine. But the assistant has no guardrails around what it says about coverage. When a patient replies "will my insurance cover this?", the assistant references the CPT code, looks at the payer file, and writes something like "Based on your plan, this procedure is typically covered." That is not a statement the clinic is authorized to make. The payer decides coverage, not the practice management system. But the AI generates confident language because confident language is what its training rewards.

There are other boundaries nobody has tested. The clinic's opt-out list lives in a spreadsheet that syncs nightly — but the assistant queries the PMS directly, so patients who opted out today still get messages until tomorrow. After-hours messages go out whenever the queue runs, not during business hours. And when a patient has an outstanding balance and a pending claim on the same visit, the assistant sometimes sends both a balance reminder and a claims update in the same day, which reads as harassment to someone already stressed about a medical bill.

The work

What Taz actually tests and builds

Taz writes eval test cases for every boundary the assistant needs to respect, runs them against the live assistant configuration, and maps every failure. This is not a policy document — it is executable test infrastructure.

• Coverage language eval suite: Test cases where simulated patients ask about coverage, cost estimates, and insurance acceptance. Each case checks whether the assistant uses hedged, compliant language ("your provider's office can help verify coverage") or makes unauthorized statements ("this is covered by your plan"). Variants cover different payer types, out-of-network scenarios, and procedures with known coverage ambiguity.
• HIPAA boundary tests: Cases where the assistant is prompted to include PHI in messages to the wrong recipient, reference diagnoses in outbound text messages, or include clinical details in balance reminders. Each test confirms the assistant refuses or redacts appropriately.
• Opt-out enforcement tests: Simulated patients flagged on the opt-out list at various sync states — just opted out (not yet synced), opted out yesterday (synced), opted out and then re-consented. The eval checks whether the assistant correctly gates on the opt-out list and handles the sync-lag edge case.
• After-hours and cadence tests: Messages triggered at 10 p.m., 6 a.m., Saturday afternoon. Duplicate-send scenarios where a patient has both a balance and a claim update pending. The eval checks whether the assistant respects send windows and deduplication rules.
• Escalation and refusal tests: Cases where the assistant should stop and route to a human — patient expresses distress, threatens legal action, asks a clinical question, or disputes a charge above a threshold. Each test confirms the assistant hands off cleanly instead of continuing to draft.

The deliverable

What the clinic receives

• Eval test suite — 60–80 test cases organized by boundary type (coverage language, HIPAA, opt-out, timing, escalation). Each case has an input scenario, the assistant's actual response, a pass/fail grade, and a rationale for the grade. Packaged as a structured file the clinic can rerun after any assistant update.
• Failure map — Every failed test case with the exact output that failed, which boundary it violated, severity ranking (cosmetic, operational, compliance-risk), and a recommended fix. The map distinguishes between prompt-fixable failures and architecture-level gaps.
• Guardrail harness — A rerunnable test configuration the clinic's IT vendor or AI provider can execute on a schedule. When the assistant's prompts, model, or data sources change, the harness re-checks every boundary without manual review of each case.
• Boundary policy document — A plain-language reference listing every rule the assistant must follow, mapped to the test cases that enforce it. Written for the office manager, not the developer — so the clinic can answer "what does the AI do when someone asks about coverage?" without reading code.
• Remediation ticket list — Prioritized fixes: prompt changes, system configuration changes, and sync-architecture recommendations. Each ticket has the specific change, the test case it resolves, and the effort level.