PatientAI Voice Tester
Help me test a clinic's AI phone assistant. We'll call as pretend patients and listen for mistakes.
The story
Some clinics let an AI answer the phone. Can you trust it with a worried patient?
PatientAI Voice Tester makes real phone calls to the clinic's AI and plays pretend patients: one almost out of pills, one confused, one with signs of an emergency. It saves every word and the recording, and a person reviews each call. It is not online, because every call costs money.
My role
I designed and directed the tool from idea to working MVP: the call flow, the eight patient scenarios, the choice of tools and the test calls. Solo project, built with AI tools: it began in Google AI Studio, continued in Manus, which shaped most of the dashboard's look, and I finished it with Claude.
Overview
PatientAI Voice Tester is a dashboard that stress-tests an AI phone assistant for a medical office. It places real calls through Twilio, plays a patient persona with an LLM, and stores each call's transcript and recording for human review. It started as a technical challenge, and I kept going on my own to an MVP with my own scenarios.
My role. I designed and directed it: the call flow, the eight scenarios, the tools and the test calls. Built with AI tools.
Architecture
- Dashboard: React and Tailwind, served by an Express server. It starts calls and shows the transcript, recording and bug notes.
- Call flow: the server saves the call in SQLite and asks Twilio to dial with recording on. The bot stays silent so the assistant speaks first.
- Each turn: Twilio transcribes the assistant's speech and posts it to the server, which sends the scenario and the full transcript to OpenAI (
gpt-4o). The reply is spoken with an Amazon Polly voice. - Review: "AI Analyze" drafts a list of problems that the tester can edit, and a script exports a DOCX report.
Key decisions
- Taking turns, not live audio. The live voice version was slower and less reliable. Each turn is now one HTTP request I can log and replay, at the cost of some overlap (Stop 4).
- The assistant speaks first. When my bot greeted too, the voices collided, and a real clinic line answers first anyway.
- State in the database. The transcript lives in SQLite and goes back to the model each turn.
- AI drafts, a human decides. The analyst only produces a first list.
Testing and evaluation
There are no automated tests. I ran five or six full calls, since each one costs money. I deleted the early ones for privacy, and none are saved in the repo, so I do not cite results. Known limits: the assistant's words come from speech recognition, so some "mistakes" may be recognition errors, and I have not measured how accurate the analyst is.
What I'd do next
- Record one sample call and publish it with its analysis.
- Give the analyst each scenario's pass criteria and compare it with calls I reviewed by hand.
- Add tests for the scenarios and call handlers, with Twilio and OpenAI mocked.
- TypeScript
- Node.js
- Express
- React
- Tailwind CSS
- Motion
- Vite
- Twilio Voice (TwiML)
- OpenAI API
- Amazon Polly voice
- SQLite (better-sqlite3)
- docx