The AI scheduled a walkthrough at a property that sold last week and told the buyer the price "has room to negotiate."

The situation

Why the agents stopped trusting the lead scores

The brokerage has 12 agents covering residential sales across three ZIP codes. Leads come in from the website, Zillow, Realtor.com, and direct referrals — roughly 150 per week. Before the AI, the office manager triaged leads manually: a quick read of the inquiry, a judgment call on seriousness, and an assignment to the right agent. It was slow but accurate. The agents trusted the assignments because they trusted the office manager.

The AI replaced that triage six months ago. It scores leads on a 1–5 readiness scale based on the inquiry text, stated budget, timeline, and whether the lead has been pre-approved. The problem is calibration. A first-time buyer who writes a two-paragraph email about wanting a "starter home under $300K" gets a 4 because the message is detailed and mentions a budget. A cash investor who sends "interested in 742 Oak — available?" gets a 2 because the message is short and has no stated timeline. The AI mistakes verbosity for intent. Agents burn hours calling chatty tire-kickers while serious buyers wait and move on.

Scheduling is worse. The AI books walkthroughs by cross-referencing the MLS feed, but the feed updates every 15 minutes and the AI queries a cached version. Properties that went under contract this morning still show as available. The AI has also started answering pricing questions it should not touch — when a buyer asks "is the seller flexible?", the assistant says things like "based on days on market, there may be room for negotiation." That is the listing agent's call, not the AI's, and a buyer who shows up expecting a deal creates a problem for everyone.

The work

What Taz actually tests and builds

Taz builds eval scenarios for every failure mode in the triage-to-scheduling pipeline, runs them against the live system, and produces a scored report showing where the AI gets it right and where it breaks.

• Qualification scoring eval: 30+ lead profiles that represent the real distribution of buyer types — first-time buyers, investors, browsers, relocators, downsizers, referred clients with no inquiry text. Each profile has a "ground truth" readiness score based on brokerage-defined criteria. The eval compares the AI's score to the ground truth and identifies systematic biases: does the AI over-score verbose leads? Under-score terse but serious inquiries? Mishandle leads with no stated budget?
• Availability and scheduling tests: Scenarios where the AI is asked to schedule a walkthrough for a property that is under contract, pending inspection, temporarily off-market, or has showing restrictions (tenant-occupied, lockbox-only, seller-present-required). Each test confirms the AI checks current status before booking and handles stale-cache scenarios gracefully — telling the buyer it needs to confirm availability rather than booking and hoping.
• Price guidance boundary tests: Buyer inquiries that naturally invite pricing commentary — "is this overpriced?", "what would you offer?", "is the seller motivated?", "how does this compare to the one on Elm Street?" Each test confirms the AI deflects to the assigned agent instead of generating pricing opinions, comp analyses, or negotiation advice.
• Assignment routing tests: Leads that should route to specific agents based on geography, price range, or property type. Edge cases: a lead asking about two different ZIP codes, a commercial inquiry hitting the residential triage, a referral from an existing client of a specific agent. Each test confirms correct assignment or clean escalation.
• Conversation boundary tests: Multi-turn exchanges where the buyer escalates from casual to serious, asks for the agent's personal cell number, requests off-market listings, or asks about commission rates. Each test confirms the AI handles the transition appropriately — escalating to a human, declining gracefully, or staying within its lane.

The deliverable

What the brokerage receives

• Scored evaluation report — Every test scenario with the AI's output, the expected output, a numeric score, and a pass/fail grade. Organized by failure mode so the brokerage can see at a glance: "qualification scoring is 62% accurate, scheduling availability checks fail 18% of the time, price guidance boundaries fail on 4 of 12 test cases."
• Lead scoring calibration guide — A document mapping the brokerage's actual buyer archetypes to scoring criteria the AI can use. Replaces the generic "verbosity = intent" logic with rules the office manager would recognize: pre-approval status, specific property reference, stated timeline, referral source.
• Failure map with severity ranking — Each failure ranked by business impact: time-waste (agents chasing bad leads), liability (price guidance, availability errors), and reputation (buyer shows up to a sold property). Each entry has the test case, the AI's output, and a recommended fix.
• Guardrail test harness — A rerunnable test pack the brokerage's AI vendor can execute after any model, prompt, or data-source change. Produces a pass/fail dashboard without manual review of each scenario.
• Remediation ticket list — Prioritized changes: prompt rewrites for scoring calibration, cache-refresh configuration for availability checks, hard-block rules for price guidance topics. Each ticket maps to the test case it resolves.